You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The nlohmann/json test suite includes 8 CMake integration tests (comprising 10 individual test executions) that are labeled as `not_reproducible`. These tests verify different methods of consuming the library in CMake-based projects but create build artifacts with timestamps and cached state that prevent bit-for-bit reproducible builds.
6
+
7
+
The non-reproducibility of all these tests is reasonable and justified.
8
+
9
+
These can be identified and found by searching for "LABELS not_reproducible" in the project.
10
+
11
+
## The 8 Non-Reproducible CMake Integration Tests
12
+
13
+
### 1. cmake_import (2 tests)
14
+
15
+
**Location:**`tests/cmake_import/`
16
+
17
+
**Test Names:**
18
+
-`cmake_import_configure`
19
+
-`cmake_import_build`
20
+
21
+
**What it tests:**
22
+
Verifies that the library can be found and used via CMake's `find_package(nlohmann_json)` mechanism after installation. This simulates how end users would consume an installed version of the library.
23
+
24
+
**Why non-reproducible:**
25
+
- Requires `JSON_Install=ON` to generate CMake config files (`nlohmann_jsonConfig.cmake`)
26
+
- Uses `find_package()` which searches for and reads installed CMake configuration files
27
+
- Generates CMakeCache.txt with absolute paths and timestamps in the test build directory
28
+
- Creates compiled executables (`with_namespace_target`, `without_namespace_target`) with timestamps
29
+
- Multiple runs produce different timestamps on all generated artifacts
30
+
31
+
---
32
+
33
+
### 2. cmake_import_minver (2 tests)
34
+
35
+
**Location:**`tests/cmake_import_minver/`
36
+
37
+
**Test Names:**
38
+
-`cmake_import_minver_configure`
39
+
-`cmake_import_minver_build`
40
+
41
+
**What it tests:**
42
+
Similar to `cmake_import`, but specifically tests that CMake's version constraint mechanism works correctly with `find_package(nlohmann_json 3.2.0 REQUIRED)`. Ensures the library properly declares its version in the CMake config files.
43
+
44
+
**Why non-reproducible:**
45
+
- Same reasons as `cmake_import`
46
+
- Additionally verifies version metadata in generated config files
47
+
- Creates timestamped build artifacts and CMake cache files
48
+
49
+
---
50
+
51
+
### 3. cmake_add_subdirectory (2 tests)
52
+
53
+
**Location:**`tests/cmake_add_subdirectory/`
54
+
55
+
**Test Names:**
56
+
-`cmake_add_subdirectory_configure`
57
+
-`cmake_add_subdirectory_build`
58
+
59
+
**What it tests:**
60
+
Verifies that the library can be embedded directly into a CMake project using `add_subdirectory()`. This is the most common integration method for projects that vendor their dependencies.
61
+
62
+
**Why non-reproducible:**
63
+
- Runs a nested CMake configuration in an isolated test directory
64
+
- Generates CMakeCache.txt with absolute source/binary paths and timestamps
Verifies that the library headers can be used via direct `target_include_directories()` calls with both `PRIVATE` and `SYSTEM` variants. Also tests a specific regression from [discussion #2281](https://github.com/nlohmann/json/discussions/2281) regarding transitive include directories.
82
+
83
+
**Why non-reproducible:**
84
+
- Creates a full CMake build with libraries (Foo, Bar) and executables
85
+
- Generates CMakeCache.txt with absolute paths to source includes
86
+
- All compiled artifacts have timestamps
87
+
- Static libraries and executables accumulate with different modification times on each run
88
+
89
+
---
90
+
91
+
## Assertions and Reproducibility
92
+
93
+
Additionally, the nlohmann/json README notes that **assertions must be disabled** for fully reproducible builds (see [discussion #4494](https://github.com/nlohmann/json/discussions/4494)). This is because:
94
+
95
+
- Assertions often use `__FILE__` which embeds absolute source paths
96
+
- Different build machines have different paths
97
+
- This embeds environment-specific information into binaries
98
+
99
+
To ensure reproducible builds, compile with:
100
+
```bash
101
+
cmake -DCMAKE_BUILD_TYPE=Release # Typically defines NDEBUG
102
+
# or explicitly:
103
+
cmake -DCMAKE_CXX_FLAGS="-DNDEBUG"
104
+
```
105
+
106
+
## Running Reproducible Tests Only
107
+
108
+
To run only reproducible tests:
109
+
```bash
110
+
# Exclude non-reproducible tests
111
+
ctest -LE not_reproducible
112
+
113
+
# Exclude both non-reproducible and git-required tests
114
+
ctest -LE "not_reproducible|git_required"
115
+
```
116
+
117
+
The project has a dedicated CI target that automatically excludes these tests:
Copy file name to clipboardExpand all lines: TSF/trustable/assertions/TA-ITERATIONS_CONTEXT.md
+12-12Lines changed: 12 additions & 12 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -35,21 +35,21 @@ For releases, additional documentation should summarise all changes across the i
35
35
36
36
- list of components with source
37
37
- source code
38
-
-**Answer**:
38
+
-**Answer**: Provided by JLS-10.
39
39
- build instructions
40
-
-**Answer**:
40
+
-**Answer**: Provided by JLS-10.
41
41
- test code
42
-
-**Answer**:
42
+
-**Answer**: Provided by JLS-10.
43
43
- test results summary
44
-
-**Answer**:
44
+
-**Answer**: Provided by JLS-10.
45
45
- attestations
46
-
-**Answer**:
46
+
-**Answer**: Provided by JLS-52.
47
47
48
48
- list of components where source code is not available
49
49
- risk analysis
50
-
-**Answer**:
50
+
-**Answer**: There are no components without source code within this project.
51
51
- attestations
52
-
-**Answer**:
52
+
-**Answer**: There are no components without source code within this project.
53
53
54
54
**Confidence scoring**
55
55
@@ -63,16 +63,16 @@ Confidence scoring for TA-ITERATIONS based on
63
63
64
64
- How much of the software is provided as binary only, expressed as a
65
65
fraction of the BoM list?
66
-
-**Answer**:
66
+
-**Answer**: 0% is provided as binary only. The entire library consists of source code in the form of a single C++ header file.
67
67
- How much is binary, expressed as a fraction of the total storage footprint?
68
-
-**Answer**:
68
+
-**Answer**: 0% is provided as binary. The entire library consists of source code in the form of a single C++ header file.
69
69
- For binaries, what claims are being made and how confident are we in the
70
70
people/organisations making the claims?
71
-
-**Answer**:
71
+
-**Answer**: 0% is provided as binary. The entire library consists of source code in the form of a single C++ header file.
72
72
- For third-party source code, what claims are we making, and how confident
73
73
are we about these claims?
74
-
-**Answer**:
74
+
-**Answer**: The nlohmann/json library has no external dependencies/accesses to third party source code, as stated in JLS-34.
75
75
- For software developed by us, what claims are we making, and how confident
76
76
are we about these claims?
77
-
-**Answer**:
77
+
-**Answer**: There is no software developed by us, as we just use the original nlohmann/json library within eclipse_score/inc_nlohmann_json. For the code developed in nlohmann/json, all claims and confidence are provided in the statements. See e.g., all the no-json-faults (NJF) items.
Copy file name to clipboardExpand all lines: TSF/trustable/assertions/TA-RELEASES_CONTEXT.md
+13-11Lines changed: 13 additions & 11 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -39,16 +39,16 @@ to TT-RESULTS analysis.
39
39
**Evidence**
40
40
41
41
- list of reproducible SHAs
42
-
-**Answer**:
42
+
-**Answer**: JLS-14 ensures that the SHA value of the nlohmann/json library used within eclipse-score/inc_nlohmann_json coincides with the SHA value provided by Niels Lohmann (for the same version).
43
43
- list of non-reproducible elements with:
44
44
- explanation and justification
45
-
-**Answer**:
45
+
-**Answer**: The only elements which are not reproducible are 8 CMake integration tests. The list of these, together with explanation and justification, is provided in TSF/docs/non_reproducible_tests.md.
46
46
- details of what is not reproducible
47
-
-**Answer**:
47
+
-**Answer**: The only elements which are not reproducible are 8 CMake integration tests. The list of these, together with explanation and justification, is provided in TSF/docs/non_reproducible_tests.md.
48
48
- evidence of configuration management for build instructions and infrastructure
49
-
-**Answer**:
49
+
-**Answer**: Provided by JLS-10 and JLS-19.
50
50
- evidence of repeatable builds
51
-
-**Answer**:
51
+
-**Answer**: Provided by JLS-53.
52
52
53
53
**Confidence scoring**
54
54
@@ -67,15 +67,17 @@ R / (R + N + B + M / (M + X))
67
67
68
68
- How confident are we that all components are taken from within our
69
69
controlled environment?
70
-
-**Answer**:
70
+
-**Answer**: We are very confident that all components are taken from within our controlled environment, as there are currently no external components used within the nlohmann/json library (as documented in JLS-34).
71
71
- How confident are we that all of the tools we are using are also under our
72
72
control?
73
-
-**Answer**:
73
+
-**Answer**: All tools used by nlohmann/json are mirrored within our controlled environment eclipse-score/inc_nlohmann_json. Therefore, these tools are under full control of the Eclipse S-Core organisation.
74
74
- Are our builds repeatable on a different server, or in a different context?
75
-
-**Answer**:
75
+
-**Answer**: Since there is no "build" of the header-only library, yes.
76
76
- How sure are we that our builds don't access the internet?
77
-
-**Answer**:
77
+
-**Answer**: We are confident that our builds don't access the internet, since there is no "build" of the header-only library.
78
78
- How many of our components are non-reproducible?
79
-
-**Answer**:
79
+
-**Answer**: All of our components are reproducible, since we only use a single component, the nlohmann/json library.
80
80
- How confident are we that our reproducibility check is correct?
81
-
-**Answer**:
81
+
-**Answer**: We are very confident that our reproducibility check is correct.
Copy file name to clipboardExpand all lines: TSF/trustable/assertions/TA-TESTS_CONTEXT.md
+10-9Lines changed: 10 additions & 9 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -26,13 +26,13 @@ TA-ITERATIONS.
26
26
**Evidence**
27
27
28
28
- Test build environment reproducibility
29
-
-**Answer**:
29
+
-**Answer**: Provided by JLS-62.
30
30
- Test build configuration
31
-
-**Answer**:
31
+
-**Answer**: Provided by JLS-16.
32
32
- Test build reproducibility
33
-
-**Answer**:
33
+
-**Answer**: Provided by JLS-62.
34
34
- Test environment configuration
35
-
-**Answer**:
35
+
-**Answer**: Provided by JLS-16.
36
36
37
37
**Confidence scoring**
38
38
@@ -42,13 +42,14 @@ tooling and their build environments are repeatable and reproducible.
42
42
**CHECKLIST**
43
43
44
44
- How confident are we that our test tooling and environment setups used for tests, fault inductions, and analyses are reproducible?
45
+
-**Answer**: The test can be reproduced any time on any machine running the versions of the operating systems and compilers as provided, as described in AOU-14.
45
46
- Are any exceptions identified, documented and justified?
46
-
-**Answer**:
47
+
-**Answer**: To the best of our knowledge, there are no exceptions identified.
47
48
- How confident are we that all test components are taken from within our controlled environment?
48
-
-**Answer**:
49
+
-**Answer**: All tests are either self-contained or download test data from [within Eclipse S-CORE](https://github.com/eclipse-score/inc_nlohmann_json/tree/json_test_data_version_3_1_0_mirror).
49
50
- How confident are we that all of the test environments we are using are also under our control?
50
-
-**Answer**:
51
+
-**Answer**: Very confident, as the environments are standard docker images of ubuntu and standard versions of compilers that are executed in our CI pipeline.
51
52
- Do we record all test environment components, including hardware and infrastructure used for exercising tests and processing input/output data?
52
-
-**Answer**:
53
+
-**Answer**: No, since the tests are independent from hardware, there is no record of hardware or infrastructure.
53
54
- How confident are we that all tests scenarios are repeatable?
54
-
-**Answer**:
55
+
-**Answer**: All test scenarios are repeated daily in the CI pipeline.
Copy file name to clipboardExpand all lines: TSF/trustable/statements/JLS-02.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -25,4 +25,4 @@ score:
25
25
Erikhu1: 1.0
26
26
---
27
27
28
-
Fuzz testing is used in the original nlohmann/json repository (https://github.com/nlohmann/json) to uncover edge cases and failure modes throughout development. (https://github.com/nlohmann/json/blob/develop/tests/fuzzing.md)
28
+
Fuzz testing is used in the nlohmann/json repository to uncover edge cases and failure modes throughout development.
0 commit comments