@@ -11,13 +11,18 @@ geekdocBreadcrumb: false
1111# Contribute to LEPISZCZE
1212
1313We invite the community to contribute to ` LEPISZCZE ` by submitting model results. You can either manually fill in your submissions or use the ` embeddings ` library for automatic generation.
14+ <br />
15+ <br />
1416
1517## Table of Contents
16- * [ 1A. Manually Filled Submissions] ( #1a-manually-filled-submissions )
17- * [ 1B. Automatically Generated Submissions] ( #1b-generation-submission-using-embeddings-library )
18+ * [ 1.A Manually Filled Submissions] ( #1a-manually-filled-submissions )
19+ * [ 1.B Example Submissions] ( #1b-example-submissions )
20+ * [ 1.C Automatically Generated Submissions] ( #1c-generation-submission-using-embeddings-library )
1821 * [ 2. Submitting submission as PR] ( #2-submit-via-pull-request )
1922
20- ## 1A. Manually Filled Submissions
23+ <br />
24+
25+ ## 1.A Manually Filled Submissions
2126
2227Submissions ** must include** the following information:
2328
@@ -40,11 +45,14 @@ There are also optional submission keys, but we strongly recommend including all
4045| ** hparams** | Mapping of hyperparameters with their values. |
4146| ** packages** | Mapping of packages used for model training and evaluation, along with their versions. |
4247
43- Submissions should be in ` .json ` format.
44-
45-
46- ### Examples
48+ Submissions should be in ` .json ` format.
49+ <br />
4750
51+ ``` python
52+ print (" Hello, Hugo!" )
53+ ```
54+ ## 1.B Example Submissions
55+ \
4856{{< collapse title="Information Retrieval sample submission file without optional fields." >}}
4957{
5058 "submission_name": "msmarco_bm_25",
@@ -71,8 +79,7 @@ Submissions should be in `.json` format.
7179 "averaged_over": 1
7280}
7381{{< /collapse >}}
74-
75-
82+ \
7683{{< collapse title="Question Answering sample submission file with packages provided." >}}
7784
7885{
@@ -129,61 +136,59 @@ Submissions should be in `.json` format.
129136 },
130137 "averaged_over": 1
131138}
132-
133139{{< /collapse >}}
134-
135-
136-
137- ## 1B. Generation Submission using Embeddings library
140+ \
141+ ## 1.C Generation Submission using Embeddings library
138142
139143
140144- Install ` embeddings ` package
141145
142- ``` bash
143- pip install clarinpl-embeddings
144- ```
146+ ``` bash
147+ pip install clarinpl-embeddings
148+ ```
145149
146150- Put your data in accordance with comments
147151
148- ` ` ` python
149- import datasets
150- import numpy as np
151-
152- from embeddings.evaluator.evaluation_results import Predictions
153- from embeddings.evaluator.leaderboard import get_dataset_task
154- from embeddings.evaluator.submission import AveragedSubmission
155- from embeddings.utils.utils import get_installed_packages
156-
157- DATASET_NAME = " clarin-pl/polemo2-official"
158- TARGET_COLUMN_NAME = " target"
159-
160- hparams = {" hparam_name_1" : 0.2, " hparam_name_2" : 0.1} # put your hyperparameters here!
161-
162- dataset = datasets.load_dataset(DATASET_NAME)
163- y_true = np.array(dataset["test"][TARGET_COLUMN_NAME])
164- # put your predictions from multiple runs below!
165- predictions = [
166- Predictions(
167- y_true=y_true, y_pred=np.random.randint(low=0, high=4, size=len(y_true))
168- )
169- for _ in range(5)
170- ]
171-
172- # make sure you are running on a training env or put exported packages below!
173- packages = get_installed_packages()
174- submission = AveragedSubmission.from_predictions(
175- submission_name="your_submission_name", # put your submission here!
176- dataset_name=DATASET_NAME,
177- dataset_version=dataset["train"].info.version.version_str,
178- embedding_name="your_embedding_model", # put your embedding name here!
179- predictions=predictions,
180- hparams=hparams,
181- packages=packages,
182- task=get_dataset_task(DATASET_NAME),
152+ ``` python
153+ import datasets
154+ import numpy as np
155+
156+ from embeddings.evaluator.evaluation_results import Predictions
157+ from embeddings.evaluator.leaderboard import get_dataset_task
158+ from embeddings.evaluator.submission import AveragedSubmission
159+ from embeddings.utils.utils import get_installed_packages
160+
161+ DATASET_NAME = " clarin-pl/polemo2-official"
162+ TARGET_COLUMN_NAME = " target"
163+
164+ hparams = {" hparam_name_1" : 0.2 , " hparam_name_2" : 0.1 } # put your hyperparameters here!
165+
166+ dataset = datasets.load_dataset(DATASET_NAME )
167+ y_true = np.array(dataset[" test" ][TARGET_COLUMN_NAME ])
168+ # put your predictions from multiple runs below!
169+ predictions = [
170+ Predictions(
171+ y_true = y_true, y_pred = np.random.randint(low = 0 , high = 4 , size = len (y_true))
183172 )
184-
185- submission.save_json()
186- ` ` `
173+ for _ in range (5 )
174+ ]
175+
176+ # make sure you are running on a training env or put exported packages below!
177+ packages = get_installed_packages()
178+ submission = AveragedSubmission.from_predictions(
179+ submission_name = " your_submission_name" , # put your submission here!
180+ dataset_name = DATASET_NAME ,
181+ dataset_version = dataset[" train" ].info.version.version_str,
182+ embedding_name = " your_embedding_model" , # put your embedding name here!
183+ predictions = predictions,
184+ hparams = hparams,
185+ packages = packages,
186+ task = get_dataset_task(DATASET_NAME ),
187+ )
188+
189+ submission.save_json()
190+ ```
191+ <br />
187192
188193## 2. Submit via pull request
189194
0 commit comments