StringMatching and RegexMatching pass unconditionally when the keyword or pattern is empty

[WatchTree-19]

Summary

StringMatching with an empty or whitespace-only keyword, and RegexMatching with an empty pattern, return PASS against any text. The check reports success without testing anything.

I'm raising this as an issue rather than a PR because two existing tests assert the current behaviour, so this is a design question for you rather than an obvious defect.

Reproduction

from giskard.checks import StringMatching, RegexMatching, Trace

await StringMatching(text="The capital of France is Paris.", keyword="").run(Trace())
# CheckStatus.PASS  -- "The answer contains the keyword ''."

await StringMatching(text="The capital of France is Paris.", keyword="   ").run(Trace())
# CheckStatus.PASS  -- whitespace normalisation reduces "   " to ""

await StringMatching(text="The capital of France is Paris.", keyword="\n").run(Trace())
# CheckStatus.PASS

await RegexMatching(text="anything at all", pattern="").run(Trace())
# CheckStatus.PASS  -- an empty regex matches everything

"" is a substring of every string and an empty regex matches everything, so these are Python's semantics working as designed. The question is whether a check framework should surface that as a green result.

Why it matters in practice

Nobody writes keyword="" on purpose. But keyword_key and pattern_key resolve the matcher from a trace field:

StringMatching(target_key="trace.last.outputs.answer",
               keyword_key="trace.last.inputs.expected_keyword")

If expected_keyword is blank or whitespace in a row of test data, that check passes unconditionally and silently. In a suite of hundreds, a vacuously green check is indistinguishable from a real one, and it's the failure mode you least want in a testing tool: it reports assurance it did not earn.

The two checks also disagree on whitespace. StringMatching(keyword=" ") passes because it normalises whitespace before matching; RegexMatching(pattern=" ") fails because it does not.

Suggested behaviour

Return CheckStatus.ERROR for an empty matcher, consistent with how _extract_and_validate already treats a non-string matcher or an unresolvable key — both configuration problems rather than assertion outcomes.

I'd leave RegexMatching(pattern=" ") as a genuine FAIL, since matching literal spaces is a legitimate intent.

Existing tests that assert the current behaviour

# tests/builtin/test_string_matching.py
async def test_empty_keyword() -> None:
    check = StringMatching(text="Hello", keyword="")
    result = await check.run(Trace())
    # Empty string should be found in any text
    assert result.status == CheckStatus.PASS

# tests/builtin/test_regex_matching.py
async def test_empty_pattern_regex_mode() -> None:
    ...
    # Empty regex matches any string
    assert result.status == CheckStatus.PASS

These read to me like characterisation tests pinning observed behaviour rather than a deliberate product decision, but you'd know. If it was deliberate, feel free to close this.

Branch ready if you want it

Implemented, with the two tests above updated and a parametrised case added for whitespace-only keywords:

https://github.com/WatchTree-19/giskard-oss/tree/proposal-empty-matcher-vacuous-pass

All 64 tests in test_string_matching.py and test_regex_matching.py pass on it, and the wider tests/builtin suite is unchanged apart from those two assertions. Two pre-existing failures in test_rego_policy.py are unrelated — they fail on a clean checkout too.

Happy to open it as a PR if you'd like it, or to drop it.

--

Contributing as WatchTree-19; you merged my Gemini finish_reason enum-value fix a while back.

[linear-code]

OSS-141