Skip to content

fix: preserve raw reranker scores for calibrated [0,1] providers - #1846

Merged
nicoloboschi merged 1 commit into
mainfrom
fix/reranker-passthrough-normalization
May 29, 2026
Merged

nicoloboschi merged 1 commit into
mainfrom
fix/reranker-passthrough-normalization

Conversation

@nicoloboschi

Copy link
Copy Markdown
Collaborator

Summary

  • Replace rank-based normalization with passthrough for reranker scores already in [0, 1]. Calibrated rerankers (Cohere, Jina, llama.cpp/Qwen) return meaningful absolute confidence — rank normalization was inflating weak candidates (e.g. 0.007 → 1.0) simply for being top-ranked among bad results.
  • Sigmoid normalization for logit-based local models is unchanged.
  • Downstream multiplicative boosts (recency, temporal, proof count) are scale-invariant and require no adjustment.

Closes #1823

Test plan

  • Updated test_reranker_score_normalization.py — passthrough tests replace rank normalization tests
  • New test_low_confidence_scores_preserved validates the core issue scenario (all-weak candidates stay low)
  • test_combined_scoring.py passes unchanged (boosts are scale-invariant)
  • Verify with a calibrated external reranker (Cohere/llama.cpp) that recall scores reflect actual confidence

Replace rank-based normalization with passthrough for reranker scores
already in [0, 1]. Calibrated rerankers (Cohere, Jina, llama.cpp/Qwen)
return meaningful absolute confidence — rank normalization was inflating
weak candidates (e.g. 0.007) to 1.0 simply for being top-ranked.

Closes #1823
@nicoloboschi
nicoloboschi merged commit 18b9c59 into main May 29, 2026
73 of 74 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Reranker raw relevance scores are lost after rank-normalization, causing misleading high recall scores

1 participant