Spaces:
Sleeping
Sleeping
Create Live_Challenge_Day_and_Dry_Test_Instructions.md
Browse files
Operational_Instructions/Live_Challenge_Day_and_Dry_Test_Instructions.md
ADDED
|
@@ -0,0 +1,26 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
**Live Challenge Day (May 12)**
|
| 2 |
+
|
| 3 |
+
1. We will conduct two live sessions on May 12 as part of the Live Challenge Day (calendar invites will be sent shortly):
|
| 4 |
+
* Session 1: 7:00 – 9:00 UTC
|
| 5 |
+
* Session 2: 15:00 – 17:00 UTC
|
| 6 |
+
|
| 7 |
+
2. Just before your session start time, your team leader will receive an email containing the Question file (500 questions) — see Question file [json schema](Question_File.json.schema) and [example](Question_File_Example.json)
|
| 8 |
+
|
| 9 |
+
3. You must generate and submit your Answer file — see Answer file [json schema](Answe_File.json.schema) and [example](Answer_File_Example.json) — within 2 hours from your session start time\
|
| 10 |
+
3.1 **Remark:** Details regarding the exact submission process will be provided soon
|
| 11 |
+
|
| 12 |
+
4. Please refer to the LiveRAG Challenge [Evaluation Guidelines](LiveRAG_Evaluation_Guidelines.md) for important information about the evaluation process
|
| 13 |
+
|
| 14 |
+
5. You must share your RAG system Git repository with us by May 13 AoE via email at sigir2025-liverag-gen@tii.ae to enable result reproduction
|
| 15 |
+
|
| 16 |
+
6. Additional information:\
|
| 17 |
+
6.1 The automatic evaluation results leaderboard will be published on the HuggingFace Challenge page once the evaluation process is completed
|
| 18 |
+
6.2 The top-performing teams will undergo manual evaluation to determine the final winners, who will be announced on July 17 during the SIGIR 2025 LiveRAG Workshop
|
| 19 |
+
|
| 20 |
+
**Dry Test (May 5)**
|
| 21 |
+
|
| 22 |
+
1. To ensure a smooth Live Challenge Day, we will conduct a Dry Test on May 5 (calendar invites will be sent shortly)
|
| 23 |
+
|
| 24 |
+
2. Each Dry Test Question file will contain 50 questions
|
| 25 |
+
|
| 26 |
+
3. Your answers will not be evaluated, and no leaderboard will be published for the Dry Test. Otherwise, instructions 1–4 from the Live Challenge Day also apply
|