Motivation
Right now, moving an eval dataset (or its already-computed results) between Giskard and another eval tool means hand-translating it — there's no shared format. This gets in the way when a team wants to evaluate the same test cases across multiple frameworks (e.g. compare Giskard's LLM-as-judge checks against another tool's graders on identical inputs), or wants to reuse a Suite built for Giskard somewhere else, or vice versa.
EvalPort is an open interchange format (Apache 2.0) for exactly this: portable LLM evaluation datasets — test cases, graders, eval suites, and result sets as plain JSON, framework-agnostic. It's already integrated with UK AISI's Inspect AI (PR approved) and has standalone adapter packages for AutoGen, CrewAI, Ragas, LangSmith, Braintrust, and MLflow, so a Giskard adapter would put it in company with the rest of the eval ecosystem rather than isolated.
Feature description
A to_openeval() / from_openeval() pair targeting giskard.checks.Suite, since that's the natural unit ("many scenarios run together") mapping onto EvalPort's EvalSuite:
from giskard_openeval_adapter import to_openeval, from_openeval
# Export a Giskard Suite (+ its scenarios/checks) to an EvalPort suite
suite = to_openeval(giskard_suite)
from openeval.validate import validate_suite
assert validate_suite(suite).valid
# ...and the other direction: build Giskard Scenarios from an EvalPort suite
scenarios = from_openeval(suite)
Each Scenario's interact() input/output would map to an EvalPort TestCase, and each attached Check (Groundedness, Conformity, LLMJudge, string/regex/similarity checks) would map to its own EvalPort grader — llm_judge for the LLM-as-judge checks, exact_match/regex/semantic_similarity for the deterministic ones where the mapping is direct.
I'd be happy to build this myself, either as a standalone giskard-openeval-adapter package (matching the shape of the adapters linked above) if that's a better fit outside this repo, or scoped to fit giskard-checks directly if maintainers would rather it live in-tree. Given the note in AUTONOMOUS.md, I'm posting this as a feature-request issue first rather than opening a PR — happy to talk through the exact mapping (particularly how best to round-trip a Check's judge configuration) before writing any code.
Motivation
Right now, moving an eval dataset (or its already-computed results) between Giskard and another eval tool means hand-translating it — there's no shared format. This gets in the way when a team wants to evaluate the same test cases across multiple frameworks (e.g. compare Giskard's LLM-as-judge checks against another tool's graders on identical inputs), or wants to reuse a
Suitebuilt for Giskard somewhere else, or vice versa.EvalPort is an open interchange format (Apache 2.0) for exactly this: portable LLM evaluation datasets — test cases, graders, eval suites, and result sets as plain JSON, framework-agnostic. It's already integrated with UK AISI's Inspect AI (PR approved) and has standalone adapter packages for AutoGen, CrewAI, Ragas, LangSmith, Braintrust, and MLflow, so a Giskard adapter would put it in company with the rest of the eval ecosystem rather than isolated.
Feature description
A
to_openeval()/from_openeval()pair targetinggiskard.checks.Suite, since that's the natural unit ("many scenarios run together") mapping onto EvalPort'sEvalSuite:Each
Scenario'sinteract()input/output would map to an EvalPortTestCase, and each attachedCheck(Groundedness,Conformity,LLMJudge, string/regex/similarity checks) would map to its own EvalPort grader —llm_judgefor the LLM-as-judge checks,exact_match/regex/semantic_similarityfor the deterministic ones where the mapping is direct.I'd be happy to build this myself, either as a standalone
giskard-openeval-adapterpackage (matching the shape of the adapters linked above) if that's a better fit outside this repo, or scoped to fitgiskard-checksdirectly if maintainers would rather it live in-tree. Given the note in AUTONOMOUS.md, I'm posting this as a feature-request issue first rather than opening a PR — happy to talk through the exact mapping (particularly how best to round-trip aCheck's judge configuration) before writing any code.