halnasri-Revisit_TT_INDICATORS (#28)

* revisit TA-INDICATORS

* fix dotstop

* added tow statement about AWIs for the testing

* added new coverage_gate to parent workflow

* small fix

* edited the coverage_gate workflow

* rewrite statement JLS-55

* fix workflow

* add pr_count_gat to parent-workflow

* fix pr_count_gate

* fix pr_count_gate

* edited statements and context file of TA-INDICATORS

* edited context file

* Update TSF/trustable/statements/JLS-55.md

Co-authored-by: LucaFue <luca.fueger@d-fine.de>
Signed-off-by: halnasri <hatem.alnasri@d-fine.com>

* Update TSF/trustable/statements/JLS-55.md

Co-authored-by: LucaFue <luca.fueger@d-fine.de>
Signed-off-by: halnasri <hatem.alnasri@d-fine.com>

* Update TSF/trustable/statements/JLS-54.md

Co-authored-by: LucaFue <luca.fueger@d-fine.de>
Signed-off-by: halnasri <hatem.alnasri@d-fine.com>

* Update TSF/trustable/statements/JLS-54.md

Co-authored-by: LucaFue <luca.fueger@d-fine.de>
Signed-off-by: halnasri <hatem.alnasri@d-fine.com>

* Edited JLS-55

Signed-off-by: halnasri <hatem.alnasri@d-fine.com>

* fixed some answers of the context file

Signed-off-by: halnasri <hatem.alnasri@d-fine.com>

* Update coverage threshold to 99.19%

Signed-off-by: halnasri <hatem.alnasri@d-fine.com>

* Update answers in TA-INDICATORS_CONTEXT.md

Clarified answers regarding risk analyses and monitoring mechanisms in the context of AWIs.

Signed-off-by: halnasri <hatem.alnasri@d-fine.com>

---------

Signed-off-by: halnasri <hatem.alnasri@d-fine.com>
Co-authored-by: LucaFue <luca.fueger@d-fine.de>

Author: 2 people

.dotstop.dot

@@ -82,13 +82,8 @@ digraph G {
 "JLS-51" [sha="190e17d59795c9ed3b25a0a8bf57497de1e0d06ab90b3f6ba47b543c95edea43"];
 "JLS-52" [sha="8539f924c31974a2722615d2410a25336a5d6a9f399f16dc485be83f7f87a5ff"];
 "JLS-53" [sha="d9f7e732e34b0ec79305dde4c5b3d60906559ef1d90bc3ce2906e28a90293844"];
-"JLS-54" [sha="e1d713fcbd323526a04ea0c7f963a4425bab1567cc7d701546d29cb05f8ac92f"];
+"JLS-54" [sha="f1ff5d3aff697980bcf072e27e00f9916052aceb4796fe1417486ddc80d5e241"];
 "JLS-55" [sha="321c775e2acf0267ac1753eb772fedf1bcca18029714071c6e55d999f6dd008a"];
-"JLS-56" [sha="6fdd3ab1172c1df5658f140d104d51559080aa32389a5a842b93cfbd394593b3"];
-"JLS-57" [sha="6261b6cf44be2e742af1e1d687f1233161ab7cdaf6f1c0a6e31e671a7451adc5"];
-"JLS-58" [sha="dbdb83427fd82fd3be5e90ab761945a0346b33740b9ea80fd37122dfa6baaa60"];
-"JLS-59" [sha="2ac670fb73ace2f582d7351baf2ea3bbd16a00dbe01cc56997ff86b564d69840"];
-"JLS-60" [sha="035303bc28dfc2e2a80d5b4b9de3a45406508a0c5c27d511e676159db227d9b9"];
 "JLS-61" [sha="151f1cda2384ae4935d29d300c3424bca710378fa3689bbcff69b06dc86bb692"];
 "JLS-62" [sha="60848232c2989d0282b64792d7da7a57c04ff368d2ac9deae09c3743251dfc79"];
 "JLS-63" [sha="2b50e79c3b43c6815b5dc15c7909ce5fb513e98fadb28ddfa40938f20f5d0427"];

.github/workflows/coverage_gate.yml

@@ -0,0 +1,54 @@
+name: Coverage Gate
+
+on:
+  workflow_call:
+    inputs:
+      artifact_id:
+        description: "Unused, kept for consistency with parent"
+        required: true
+        type: string
+
+jobs:
+  coverage_gate:
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Download coverage HTML artifact
+        uses: actions/download-artifact@v4
+        with:
+          name: code-coverage-report
+          path: coverage_html
+
+      - name: Debug list coverage files
+        run: |
+          echo "=== coverage_html contents ==="
+          ls -R coverage_html
+
+      - name: Enforce coverage threshold
+        run: |
+          THRESHOLD=99.19
+
+          echo "=== Extracting line coverage for 'Lines:' from index.html ==="
+
+          HEADER_BLOCK=$(grep -A1 'Lines:' coverage_html/index.html || true)
+          echo "$HEADER_BLOCK"
+
+          LINE_COV=$(echo "$HEADER_BLOCK" | grep -oE "[0-9]+(\.[0-9]+)?" | head -n1 || true)
+          echo "Extracted Line coverage: '${LINE_COV}'"
+
+          if [ -z "$LINE_COV" ]; then
+            echo "Could not extract line coverage for 'Lines:' from index.html"
+            exit 1
+          fi
+
+          COMPARE=$(awk -v cov="$LINE_COV" -v thr="$THRESHOLD" 'BEGIN { if (cov < thr) print "lt"; else print "ge"; }')
+
+          if [ "$COMPARE" = "lt" ]; then
+            echo "Coverage below threshold, failing job."
+            exit 1
+          fi
+
+          echo "Coverage is above threshold."

.github/workflows/parent-workflow.yml

@@ -10,6 +10,7 @@ on:
 
 permissions:
   contents: read
+  pull-requests: read
 
 jobs:
   labeler:
@@ -63,6 +64,17 @@ jobs:
     with:
       artifact_id: "ubuntu-${{ github.sha }}"
 
+  coverage_gate:
+    name: Run Coverage Gate Workflow
+    needs: [ubuntu]
+    uses: ./.github/workflows/coverage_gate.yml
+    with:
+      artifact_id: "coverage_gate-${{ github.sha }}"
+
+  pr_count_gate:
+    name: Run PR Count Gate Workflow
+    uses: ./.github/workflows/pr_count_gate.yml
+
   dependency_review:
     name: Run dependency_review Workflow
     if: ${{ github.event_name == 'pull_request' }}  # only run dependency_review for PRs
@@ -73,7 +85,7 @@ jobs:
   collect_artifacts_pr:
     name: "Collect Results & Deploy (PR)"
     if: github.event_name == 'pull_request'
-    needs: [labeler, SME_review_checker, check_amalgamation, test_trudag_extensions, dependency_review, codeql, ubuntu]
+    needs: [labeler, SME_review_checker, check_amalgamation, test_trudag_extensions, dependency_review, codeql, ubuntu, coverage_gate, pr_count_gate]
     runs-on: ubuntu-latest
     strategy:
       matrix:
@@ -106,7 +118,7 @@ jobs:
   collect_artifacts_non_pr:
     name: "Collect Results & Deploy (Non-PR)"
     if: github.event_name != 'pull_request'
-    needs: [labeler, test_trudag_extensions, codeql, ubuntu]  # no check_amalgamation, dependency_review or SME_review_checker if non PR
+    needs: [labeler, test_trudag_extensions, codeql, ubuntu, coverage_gate, pr_count_gate]  # no check_amalgamation, dependency_review or SME_review_checker if non PR
     runs-on: ubuntu-latest
     strategy:
       matrix:

.github/workflows/pr_count_gate.yml

@@ -0,0 +1,37 @@
+name: PR Count Gate
+
+on:
+  workflow_call:
+
+jobs:
+  pr_count_gate:
+    runs-on: ubuntu-latest
+
+    steps:
+            - name: Count open pull requests
+              id: pr-count
+              uses: actions/github-script@v7
+              with:
+                github-token: ${{ secrets.GITHUB_TOKEN }}
+                script: |
+                  const { owner, repo } = context.repo;
+                  const per_page = 100;
+                  let page = 1;
+                  let total = 0;
+
+                  while (true) {
+                    const { data } = await github.rest.pulls.list({
+                      owner,
+                      repo,
+                      state: 'open',
+                      per_page,
+                      page,
+                    });
+                    if (data.length === 0) break;
+                    total += data.length;
+                    if (data.length < per_page) break;
+                    page++;
+                  }
+
+                  core.info(`Open pull requests: ${total}`);
+                  core.setOutput('open_prs', total.toString());

TSF/trustable/assertions/TA-INDICATORS_CONTEXT.md

@@ -57,19 +57,19 @@ set of AWIs is incorrect, or the tolerance is too lax.
 **Evidence**
 
 - Risk analyses
-  - **Answer**: 
+  - **Answer**: There is no dedicated TSF risk analysis focusing specifically on runtime behaviour of `eclipse-score/inc_nlohmann_json`.
 - List of advance warning indicators
-  - **Answer**: 
+  - **Answer**: The only two introduced AWIs are provided in JLS-54 and JLS-55.
 - List of Expectations for monitoring mechanisms
-  - **Answer**: 
+  - **Answer**: There are no dedicated monitoring mechanisms defined. Any expectations for monitoring apply at system or integration level and are expected to be specified and implemented by the integrator.
 - List of implemented monitoring mechanisms
-  - **Answer**: 
+  - **Answer**: There are no dedicated monitoring mechanisms for AWIs in the sense of continuous or in-field monitoring. The two AWIs (coverage and PR count) are evaluated only when CI workflows are executed and are used as quality gates at CI time, rather than as a separate, continuous monitoring system (see JLS-54 and JLS-55).
 - List of identified misbehaviours without advance warning indicators
-  - **Answer**: 
+  - **Answer**: Provided by JLS-69.
 - List of advance warning indicators without implemented monitoring mechanisms
-  - **Answer**: 
+  - **Answer**: All currently defined AWIs (JLS-54 and JLS-55) are evaluated via CI runs, but there is no additional, dedicated monitoring mechanism beyond these CI executions.
 - Advance warning signal data as time series (see TA-DATA)
-  - **Answer**: 
+  - **Answer**: The only AWIs in JLS-54 and JLS-55 are implemented as part of the CI and therefore saved as time series (see JLS-18 and JLS-45).
 
 **Confidence scoring**
 
@@ -80,27 +80,27 @@ monitoring mechanisms have been implemented to collect the required data.
 **Checklist**
 
 - How appropriate/thorough are the analyses that led to the indicators?
-  - **Answer**: Since no misbehaviours for the use of the library for parsing and verification of JSON data according to RFC8259 have been identified, no warning indicators are implemented.
+  - **Answer**: For eclipse-score/inc_nlohmann_json, the library itself is a statically integrated, header-only component without stream processing loops. No runtime misbehaviours specific to this repository have been identified, and therefore no runtime AWIs are implemented for the library itself. The two AWIs that do exist (coverage threshold and PR-count limit) are based on the assumption that CI test results and review load correlate with potential misbehaviours in the library and its evolution, and are therefore focused on test and process quality rather than runtime behaviour (see JLS-54 and JLS-55).
 - How confident can we be that the list of indicators is comprehensive?
-  - **Answer**:  
+  - **Answer**:  For the scope of `eclipse-score/inc_nlohmann_json` as a static library, we are reasonably confident that CI-based indicators on test coverage and the count of open PRs are sufficient.
 - Could there be whole categories of warning indicators still missing?
-  - **Answer**:  Yes, there could. Within S-CORE, however, any warning indicator that is not natively implemented within the original nlohmann/json should be implemented in the wrapper defining the interface between the library and the project using it.
+  - **Answer**: Yes, there could. In particular, runtime performance or stability indicators in systems that use the library are not covered here. Any missing warning indicators are expected to be implemented by the integrator (see AOU-09).
 - How has the list of advance warning indicators varied over time?
-  - **Answer**: 
+  - **Answer**: The current AWIs (coverage threshold and pr count threshold on protected branches) were introduced as CI-based quality gates. No additional AWIs have been added or removed so far (see JLS-54 and JLS-55).
 - How confident are we that the indicators are leading/predictive?
-  - **Answer**: 
+  - **Answer**: The indicators are leading in the sense that they prevent changes which reduce test coverage, or are made in an overloaded PR situation, from entering protected branches and being used as a basis for integration and release.
 - Are there misbehaviours that have no advance warning indicators?
-  - **Answer**: 
+  - **Answer**: Potential runtime misbehaviours in consuming systems are not covered by AWIs in this repository.
 - Can we collect data for all indicators?
-  - **Answer**: 
+  - **Answer**: Both indicators (JLS-54 and JLS-55) are derived from CI runs, and the required data (coverage and pr count) is collected automatically for each CI execution.
 - Are the monitoring mechanisms used included in our Trustable scope?
-  - **Answer**: 
+  - **Answer**: There are no continuously running runtime monitoring mechanisms. The only related mechanisms are the CI workflows implementing JLS-54 and JLS-55, they run only when CI is executed.
 - Are there gaps or trends in the data?
-  - **Answer**: 
+  - **Answer**: There is no trend analysis preformed on AWIs. However, there is trend analysis done as a proof of concept to the failure rate of CI test (see JLS-17). Potential gaps could arrise during the integration of the library (see AOU-09).
 - If there are gaps or trends, are they analysed and addressed?
-  - **Answer**: 
+  - **Answer**: There are no trends identified (see the question above). Any gaps should be closed by the integrator.
 - Is the data actually predictive/useful?
-  - **Answer**: 
+  - **Answer**: Yes, the CI data from the AWIs is useful to prevent regressions in the tested behaviour of the library and possible issues introduced due to a large number of open PRs from entering protected branches. 
 - Are indicators from code, component, tool, or data inspections taken into
 consideration? 
-  - **Answer**: 
+  - **Answer**: Yes, all types of indicator are taken into consideration.

TSF/trustable/statements/JLS-54.md

@@ -1,6 +1,13 @@
 ---
 level: 1.1
 normative: true
+references: 
+    - type: verbose_file
+      path: "./.github/workflows/coverage_gate.yml"
+      description: "GitHub Actions workflow enforcing a minimum coverage threshold."
+    - type: verbose_file
+      path: "./.github/workflows/parent-workflow.yml"
+      description: "Parent CI workflow that calls the coverage_gate workflow."
 ---
 
-In eclipse-score/inc_nlohmann_json, a GitHub workflow continuously monitors the fraction of failing unit and integration tests on protected branches (e.g., main, release branches). If the failure rate exceeds a defined threshold over a configurable number of consecutive runs, the workflow blocks further merges to the affected branch, and restores the last known-good commit (last fully passing pipeline) as the default basis for integration and release.
+In the eclipse-score/inc_nlohmann_json repository, code coverage for unit and integration tests is measured in every CI run, and a minimum coverage threshold is defined for each protected branch. If coverage for a change would fall below this threshold, the CI workflow blocks the merge until coverage is restored or the change is rejected.

TSF/trustable/statements/JLS-55.md

@@ -1,6 +1,13 @@
 ---
 level: 1.1
 normative: true
+references: 
+    - type: verbose_file
+      path: "./.github/workflows/pr_count_gate.yml"
+      description: "GitHub Actions workflow enforcing a limit on open PRs."
+    - type: verbose_file
+      path: "./.github/workflows/parent-workflow.yml"
+      description: "Parent CI workflow that calls the pr_count_gate workflow."
 ---
 
-In eclipse-score/inc_nlohmann_json, a GitHub workflow tracks CI pipeline duration (build + tests) over time. If the median runtime increases beyond a defined relative threshold compared to a rolling baseline, the workflow flags the regression, blocks releases from the affected commit(s), and opens or updates an issue to investigate performance-related misbehaviours.
+In eclipse-score/inc_nlohmann_json, a GitHub Actions workflow checks the number of open pull requests in the main branch. If the number exceeds a defined threshold, the workflow fails and blocks further merges until the number of open pull requests is reduced below that threshold.