Skip to content

Feature/compute cometkiwi metric#57

Open
tanhaow wants to merge 3 commits intodevelopfrom
feature/compute-cometkiwi-metric
Open

Feature/compute cometkiwi metric#57
tanhaow wants to merge 3 commits intodevelopfrom
feature/compute-cometkiwi-metric

Conversation

@tanhaow
Copy link

@tanhaow tanhaow commented Mar 19, 2026

Associated Issue(s): resolves #52

Changes in this PR

Include all key changes in this pull request

  • Added compute_cometkiwi() function to metrics.py using the Unbabel/wmt22-cometkiwi-da model
  • Integrated CometKiwi metric into evaluate_corpus.py script; CSV output now includes cometkiwi column alongside chrf and comet
  • Added HuggingFace authentication error handling with helpful user guidance for the gated HuggingFace model (--> or should we move this to DeveloperNotes instead?)

Notes

  • First run will download ~2GB model and cache it for subsequent use (Added model caching via LOADED_METRICS dictionary to reuse loaded models across evaluations)

Reviewer Checklist

Include discrete checks that should be done by the reviewer beyond looking through
code and/or file changes. Note that this check list will correspond to tasks within
the PR overview page.

  • Verify compute_cometkiwi() function signature matches pattern (takes tr_text and src_text, returns float)
  • Check that evaluate_corpus.py correctly computes and writes CometKiwi scores to CSV

@tanhaow tanhaow self-assigned this Mar 19, 2026
@tanhaow tanhaow requested a review from laurejt March 19, 2026 18:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant