Problem
Each authorized PR runs ~21 integration jobs that hit live LLM APIs. Recent runs showed:
- Redundant provider coverage (
google + gemini, openai + bare, multiple Azure aliases)
- Triple Python version matrix (3.12–3.14) despite Main CI already running unit tests on all three
- A
both agents backend job that re-runs the union of giskard-llm and litellm suites
- A
checks job that runs make test-functional but collects zero tests (giskard-checks uses @pytest.mark.integration, not functional)
Failures are often infra-related (e.g. OpenAI credits exhausted) rather than regressions, while API cost scales with every PR.
Proposal (phase 1 — workflow only)
Shrink the integration matrix without changing test code:
- Python 3.12 only for integration jobs (unit tests keep 3.12–3.14 on Main CI)
- Drop alias LLM providers — keep
openai, google, anthropic, azure; drop bare, gemini, azure_ai, azure_foundry_v1
- Drop
both agents backend — giskard-llm and litellm jobs already cover both install layouts
- Remove dead
checks integration job until checks uses functional marks or a dedicated smoke tier
Expected impact: ~21 jobs → 6 jobs, ~70% fewer live API calls per PR.
Follow-ups (phase 2+)
Problem
Each authorized PR runs ~21 integration jobs that hit live LLM APIs. Recent runs showed:
google+gemini,openai+bare, multiple Azure aliases)bothagents backend job that re-runs the union of giskard-llm and litellm suiteschecksjob that runsmake test-functionalbut collects zero tests (giskard-checksuses@pytest.mark.integration, notfunctional)Failures are often infra-related (e.g. OpenAI credits exhausted) rather than regressions, while API cost scales with every PR.
Proposal (phase 1 — workflow only)
Shrink the integration matrix without changing test code:
openai,google,anthropic,azure; dropbare,gemini,azure_ai,azure_foundry_v1bothagents backend —giskard-llmandlitellmjobs already cover both install layoutschecksintegration job until checks usesfunctionalmarks or a dedicated smoke tierExpected impact: ~21 jobs → 6 jobs, ~70% fewer live API calls per PR.
Follow-ups (phase 2+)
@pytest.mark.functional_smokefor PR CI vs full suite on nightly/maingiskard-checksintegration tests (--run-integration) into a smoke job when ready