Skip to content

Sort qrel and results so the keys lexical order fits the dict order.#183

Open
vd-omermazig wants to merge 1 commit intobeir-cellar:mainfrom
vd-omermazig:fix_evaluator_results_order_considiration
Open

Sort qrel and results so the keys lexical order fits the dict order.#183
vd-omermazig wants to merge 1 commit intobeir-cellar:mainfrom
vd-omermazig:fix_evaluator_results_order_considiration

Conversation

@vd-omermazig
Copy link

This is necessary because when the results are more than 1, we need to keep the order that they were retrieved at for some metrics to work. Example:

qrel = {
    'q1': {
        'd1': 0,
        'd2': 1,
        'd3': 1,
    },
}

run = {
    'q1': {
        'd1': 1,
        'd2': 1,
        'd3': 1,
    }
}
from beir.retrieval.evaluation import EvaluateRetrieval

evaluator = EvaluateRetrieval()
ndcg, _map, recall, precision = evaluator.evaluate(qrel, run, [1,2,3])
print(recall)

Old Version:
{'Recall@1': 0.5, 'Recall@2': 1.0, 'Recall@3': 1.0}
New version:
{'Recall@1': 0.0, 'Recall@2': 0.5, 'Recall@3': 1.0}

The new version is correct because for Recall@1 the d1 result should have been considered and not d3.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant