Skip to content

feat(scan): generation progress and quiet mode on scan helpers #2670

Description

@davidberenstein1957

Feature Request

Show progress (or structured logs) while generate_suite runs, label scan phases (generating vs running), and add a verbose switch on quality_scan / vulnerability_scan so CI can suppress Rich UI and print_report while still returning SuiteResult.

Motivation

Suite.run already has Rich progress. Generation (adversarial LLM + HF downloads) can take minutes with no feedback. Scan helpers always print reports. Opt-in check-level logging (#2607) does not cover generation or a first-class quiet API.

Implementation plan

  1. generate_suite: Rich progress (or logging INFO) over generators — i/n + generator class/name; update on completion/failure.
  2. Scan helpers: set phase before generate vs run; pass verbose into Suite.run (existing flag) and gate result.print_report().
  3. Non-TTY: fall back to log lines (no broken Rich escape codes in CI).
  4. Complement feat(checks): opt-in progress logging for scenario/check execution (#… #2607; do not replace logging.
  5. Tests with spies: verbose=False never calls print_report; fake multi-generator suite emits ≥1 progress/log event during generate.

API usage

from giskard.scan import generate_suite, vulnerability_scan

# Interactive (default): progress + printed report
result = await vulnerability_scan(
    target=my_agent,
    description="...",
    languages=["en"],
    verbose=True,  # default
)

# CI / captured logs: no Rich suite bar, no print_report
result = await vulnerability_scan(
    target=my_agent,
    description="...",
    languages=["en"],
    verbose=False,
)
assert result.passed is False or True  # still a SuiteResult
print(result.pass_rate)  # inspect programmatically

# Generation only
suite = await generate_suite(
    description="...",
    languages=["en"],
    max_scenarios=20,
    verbose=True,
)

Example terminal output (verbose generate)

Generating suite  [2/7] AdversarialScenarioGenerator
Generating suite  [3/7] GOATAttackScenarioGenerator
Generated 36 scenarios.
Running suite …
████████████░░░░  18/36  pass=14 fail=3 error=1

Example log lines (non-TTY / verbose generate)

INFO giskard.scan.catalog: generate_suite: start generators=7 max_scenarios=40
INFO giskard.scan.catalog: generate_suite: done generator=AdversarialScenarioGenerator scenarios=12
INFO giskard.scan.catalog: generate_suite: finished total_scenarios=36
INFO giskard.scan: vulnerability_scan: phase=running scenarios=36

Quiet mode (verbose=False)

No Rich bars, no automatic print_report(). Return value unchanged:

SuiteResult(scenarios=[...], ...)  # same type as today

Acceptance criteria

  • generate_suite alone is not a silent multi-minute hang
  • Scan helpers accept verbose and can suppress print_report
  • Tests cover quiet mode + at least one generation progress/log event

Out of scope

  • Token / cost aggregation (separate)
  • Changing default scenario budgets

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions