Skip to content

feat(cli): add structured exit codes for proper error signaling#1670

Closed
frankentini wants to merge 1 commit intoNVIDIA:mainfrom
frankentini:fix/cli-exit-codes
Closed

feat(cli): add structured exit codes for proper error signaling#1670
frankentini wants to merge 1 commit intoNVIDIA:mainfrom
frankentini:fix/cli-exit-codes

Conversation

@frankentini
Copy link
Copy Markdown

Summary

Closes #1221

Adds an ExitCode enum to garak/exception.py with well-defined integer codes following common Unix conventions, and updates cli.main() to return them instead of using bare exit(1) calls or returning None.

Exit Codes

Code Name When
0 SUCCESS Normal completion (list, version, scan, etc.)
1 RUNTIME_ERROR General / uncaught errors, GarakException
2 USAGE_ERROR Invalid CLI arguments or parameter values
3 CONFIG_ERROR Missing/invalid config files, fixer migrations
4 PLUGIN_ERROR Missing API keys, bad generator, plugin config
5 INTERRUPTED User-initiated Ctrl-C / SIGINT

Changes

  • garak/exception.py: Added ExitCode(IntEnum) with the six codes above
  • garak/cli.py: main() now returns an ExitCode int instead of None; replaced bare exit(1) calls with typed returns; catches specific exception subclasses (APIKeyMissingError, PluginConfigurationError, ConfigFailure) before the generic GarakException handler for more precise exit codes
  • garak/__main__.py: Propagates cli.main() return value to sys.exit()
  • tests/cli/test_cli_exit_codes.py: Tests for enum values, return types, and exit codes for success paths and various failure scenarios

Motivation

As noted in #1221, proper exit codes help wrapping entities (CI pipelines, orchestration tools, garak-as-a-service) distinguish between different failure modes without parsing stderr output. The codes follow the common Unix convention where 0 = success, 1 = general error, and 2 = usage error.

Closes NVIDIA#1221

Adds an ExitCode enum with well-defined integer codes following Unix
conventions:

  0 = SUCCESS        - normal completion
  1 = RUNTIME_ERROR  - general / uncaught errors
  2 = USAGE_ERROR    - invalid CLI arguments or parameter values
  3 = CONFIG_ERROR   - missing/invalid config files, fixer migrations
  4 = PLUGIN_ERROR   - missing API keys, bad generator, plugin config
  5 = INTERRUPTED    - user-initiated Ctrl-C / SIGINT

Changes:
- garak/exception.py: add ExitCode(IntEnum) with the six codes above
- garak/cli.py: main() now returns an ExitCode int instead of None;
  replaces bare exit(1) calls with typed returns; catches specific
  exception subclasses (APIKeyMissingError, PluginConfigurationError,
  ConfigFailure) before the generic GarakException handler
- garak/__main__.py: propagates cli.main() return to sys.exit()
- tests/cli/test_cli_exit_codes.py: validates enum values, return
  types, and exit codes for success/failure scenarios
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 5, 2026

DCO Assistant Lite bot:
Thank you for your submission, we really appreciate it. Like many open-source projects, we ask that you sign our Developer Certificate of Origin before we can accept your contribution. You can sign the DCO by just posting a Pull Request Comment same as the below format.


I have read the DCO Document and I hereby sign the DCO


You can retrigger this bot by commenting recheck in this Pull Request

@jmartin-tech
Copy link
Copy Markdown
Collaborator

Closing as duplicate of #1651, this can be reopened if that PR does not land.

@github-actions github-actions bot locked and limited conversation to collaborators Apr 6, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

arch: give exit codes on improper exit

2 participants