Look into integrating `mteb` with huggingfaces community evals

[KennethEnevoldsen]

Currently it seems to be mostly relevant for generative models (blog), but talked with @tomaarsen and it sounds like we could do something similar but where mteb replaces inspect AI as the runner.

---

I would say that we need the following:

• 1. Include the required metadata in the dataset repo (eq. to the eval.yaml). I suppose we need the metadata as well as any additional settings such as input_column_name, label_column_name etc. - we could of course also allow a script itself as the config. We could even allow a "mteb version" tag for debugging.
• 2. We need to make this format loadable into MTEB
• 3. We need to a way for the community to push results

(the leaderboard in this case will only be a task-specific leaderboard, not for a full benchmark)

fixing 1 and 2 would also allow us to convert a lot of our code into metadata files that can simply be loaded in

[KennethEnevoldsen]

@tomaarsen, @isaac-chung and @Samoed let me know if there is something I am missing here

[Samoed]

1. We would need to create a script to export our format to eval.yaml. I think including script would be easier and more reproducible than just column names
2. Not sure about this, because we store more information in our resulta. I think we can update scores based on results from results repo to keep reproducibility. For metadata we can keep main in repo and just export it to eval.yaml
3. I think we can keep our current pipeline

[KennethEnevoldsen]

1. I don't nec. think we need to use the eval.yaml format. Let us just start with the format that would work best for us - then we can work on compatibility (we can introduce it as a beta feature to allow us to make changes) - I suspect something likemteb_task_config.toml
   1.1) re column names, it is more that we need more than just the metadata to run the task. I have added an example of how the data could look below.
2. Hmm I think there is a misunderstanding here - it is more about making the mteb_task_config.toml (or whatever we decide to use) loadable
3. def for the leaderboard - I was more thinking for what is displayed on the dataset page (see image above) for now though let us focus on 1 and 2

---

Example implementation

Writing the config (1):

task = mteb.get_task("LccSentimentClassification")
task.to_config(name="mteb_task_config.toml") # default

Example of the config:

[task]
abstask="AbsTaskClassification"
samples_per_label = 16

[task.metadata]
name = "LccSentimentClassification"
description = "The leipzig corpora collection, annotated for sentiment"
reference = "https://github.com/fnielsen/lcc-sentiment"
type = "Classification"
category = "t2c"
modalities = ["text"]
eval_splits = ["test"]
eval_langs = ["dan-Latn"]
main_score = "accuracy"
date = ["2006-01-01", "2006-12-31"]
domains = ["News", "Web", "Written"]
task_subtypes = ["Sentiment/Hate speech"]
license = "cc-by-4.0"
annotations_creators = "expert-annotated"
dialect = []
sample_creation = "found"
prompt = "Classify texts based on sentiment"

dataset = { path = "DDSC/lcc", revision = "de7ba3406ee55ea2cc52a0a41408fa6aede6d3c6" }

bibtex_citation = """
@inproceedings{quasthoff-etal-2006-corpus,
  address = {Genoa, Italy},
  author = {Quasthoff, Uwe  and
Richter, Matthias  and
Biemann, Christian},
  booktitle = {Proceedings of the Fifth International Conference on Language Resources and Evaluation ({LREC}{'}06)},
  editor = {Calzolari, Nicoletta  and
Choukri, Khalid  and
Gangemi, Aldo  and
Maegaard, Bente  and
Mariani, Joseph  and
Odijk, Jan  and
Tapias, Daniel},
  month = may,
  publisher = {European Language Resources Association (ELRA)},
  title = {Corpus Portal for Search in Monolingual Corpora},
  url = {http://www.lrec-conf.org/proceedings/lrec2006/pdf/641_pdf.pdf},
  year = {2006},
}
"""

I could also imagine something like:

[mteb]
version=2.7.28
...

to help us with reproducibility when debugging (but I am not sure exactly what it should contain).

Loading the config (2):

# from file
mteb.get_task("mteb_task_config.toml") # will give a warning is metadata is incomplete
mteb.get_tasks(task_names=["mteb_task_config.toml"])

# from huggingface
mteb.get_task("DDSC/lcc") # checks for mteb_task_config.toml if it does not exist it will fail

We could of course also make all of this work with an mteb_task_config.py, which allows for custom filtering, but I am not sure I would want to encourage that (will cause more backward compatibility issues and conflicts)

[Samoed]

1&2. Yes, I agree that we can add support for benchmark tasks, but keep the main source of tasks in our repo

3. I think we can just export results from our results repo to dataset's leaderboard, otherwise would be hard to track everything

[NielsRogge]

Hi folks, we updated the spec a bit, for each HF dataset, one needs to add the following:

name: MTEB
description: MTEB is (...)
evaluation_framework: mteb
 
tasks:
- id: <dataset-id>
config: default
split: test

See Math Arena as a recent example: https://huggingface.co/datasets/MathArena/aime_2026

[Samoed]

Great! I'll try to integrate soon

[NielsRogge]

Btw, first you need to register your evaluation_framework to the hub here: https://github.com/huggingface/huggingface.js/blob/main/packages/tasks/src/eval.ts. Then you can refer the evaluation_framework in your eval.yaml.

[Samoed]

Seems it's already registered https://github.com/huggingface/huggingface.js/blob/be08e0e051e3b5703971671196dcfc2043b60c11/packages/tasks/src/eval.ts#L15-L19

[tomaarsen]

cc @burtenshaw re. the evaluation registration. It seems like you're indeed already in there.

• Tom Aarsen

[burtenshaw]

re. the evaluation registration. It seems like you're indeed already in there.

Correct @Samoed . We just need this eval.yaml to start showing reported scores in the dataset repo.

name: MTEB
description: MTEB is (...)
evaluation_framework: mteb
 
tasks:
- id: <dataset-id>
config: default
split: test

[KennethEnevoldsen]

@Samoed, it seems like we can just add this to our push_to_hub and re-push/update datasets.

[Samoed]

Yes, wanted to add to it. Only question how we utilize different scores. I think we can do:

1. Task name with scores for all splits/subsets
2. Task-config-split for specific splits and subsets

[KennethEnevoldsen]

@burtenshaw is there a spec for the eval.yaml?