馃殌 Feature Request
Add a SARIF 2.1.0 exporter alongside the existing JUnit and Hub exporters, so a suite result can be uploaded to GitHub code scanning.
result = await suite.run(agent)
result.to_sarif("giskard.sarif")
- uses: github/codeql-action/upload-sarif@v3
with:
sarif_file: giskard.sarif
馃攬 Motivation
to_junit_xml gets scan results into a CI test report, which answers "did the build pass". SARIF answers a different question: it puts each finding in the GitHub Security tab and as a PR annotation, with a rule, a severity and a fingerprint that keeps an alert stable across runs.
For a scanner that produces security findings rather than test failures, that seems like the more natural surface. It also means a team can gate merges on new findings using the same machinery they already use for CodeQL and dependency alerts.
The tag vocabulary already on scenarios (threat-type:*) maps cleanly onto SARIF rule tags, so findings arrive already categorised.
馃洶 Alternatives
- Keep JUnit only. Works for pass/fail gating but findings don't reach the Security tab.
- Leave it to users. The mapping from
SuiteResult to SARIF rules and results is non-obvious enough that everyone would rewrite it slightly differently.
- Put it in the Hub rather than OSS, if surfacing findings is meant to be a Hub concern. Fair, and worth saying so if that's the plan.
馃搸 Additional context
I've written this against main to check it's practical: ~150 lines in checks/export/sarif.py mirroring junit.py, a SuiteResult.to_sarif() method next to to_junit_xml(), and tests. Output validates against the published SARIF 2.1.0 schema. No new dependencies.
Design points I'd want your view on before proposing it:
- Only failures and errors become results. Passing checks are omitted, since an alert per passing check would bury the real ones.
- Rules keyed by
check_kind, so one rule covers every instance rather than one rule per scenario.
- Scenario tags become rule tags. This is the part I'd most like a second opinion on, because it's where an OWASP or MITRE mapping would slot in later if you ever wanted one.
Happy to open a PR if this is wanted, or to drop it if exporting findings belongs on the Hub side.
馃殌 Feature Request
Add a SARIF 2.1.0 exporter alongside the existing JUnit and Hub exporters, so a suite result can be uploaded to GitHub code scanning.
馃攬 Motivation
to_junit_xmlgets scan results into a CI test report, which answers "did the build pass". SARIF answers a different question: it puts each finding in the GitHub Security tab and as a PR annotation, with a rule, a severity and a fingerprint that keeps an alert stable across runs.For a scanner that produces security findings rather than test failures, that seems like the more natural surface. It also means a team can gate merges on new findings using the same machinery they already use for CodeQL and dependency alerts.
The tag vocabulary already on scenarios (
threat-type:*) maps cleanly onto SARIF rule tags, so findings arrive already categorised.馃洶 Alternatives
SuiteResultto SARIF rules and results is non-obvious enough that everyone would rewrite it slightly differently.馃搸 Additional context
I've written this against
mainto check it's practical: ~150 lines inchecks/export/sarif.pymirroringjunit.py, aSuiteResult.to_sarif()method next toto_junit_xml(), and tests. Output validates against the published SARIF 2.1.0 schema. No new dependencies.Design points I'd want your view on before proposing it:
check_kind, so one rule covers every instance rather than one rule per scenario.Happy to open a PR if this is wanted, or to drop it if exporting findings belongs on the Hub side.