Skip to content

ci: make integration tests more frugal #2695

Description

@davidberenstein1957

Problem

Each authorized PR runs ~21 integration jobs that hit live LLM APIs. Recent runs showed:

  • Redundant provider coverage (google + gemini, openai + bare, multiple Azure aliases)
  • Triple Python version matrix (3.12–3.14) despite Main CI already running unit tests on all three
  • A both agents backend job that re-runs the union of giskard-llm and litellm suites
  • A checks job that runs make test-functional but collects zero tests (giskard-checks uses @pytest.mark.integration, not functional)

Failures are often infra-related (e.g. OpenAI credits exhausted) rather than regressions, while API cost scales with every PR.

Proposal (phase 1 — workflow only)

Shrink the integration matrix without changing test code:

  1. Python 3.12 only for integration jobs (unit tests keep 3.12–3.14 on Main CI)
  2. Drop alias LLM providers — keep openai, google, anthropic, azure; drop bare, gemini, azure_ai, azure_foundry_v1
  3. Drop both agents backendgiskard-llm and litellm jobs already cover both install layouts
  4. Remove dead checks integration job until checks uses functional marks or a dedicated smoke tier

Expected impact: ~21 jobs → 6 jobs, ~70% fewer live API calls per PR.

Follow-ups (phase 2+)

  • Introduce @pytest.mark.functional_smoke for PR CI vs full suite on nightly/main
  • Path-filter integration workflow to LLM/agents changes only
  • Wire giskard-checks integration tests (--run-integration) into a smoke job when ready

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions