---
license: apache-2.0
language:
- ar
library_name: sentence-transformers
pipeline_tag: sentence-similarity
base_model: NAMAA-Space/AraModernBert-Base-STS
base_model_relation: finetune
tags:
- sentence-transformers
- sentence-similarity
- feature-extraction
- dense
- bi-encoder
- modernbert
- arabic
- quran
- hadith
- retrieval
- islamic-nlp
- loss:ContrastiveLoss
widget:
- source_sentence: ما هو أعظم الذنوب؟
  sentences:
  - إن الله لا يغفر أن يشرك به ويغفر ما دون ذلك لمن يشاء ومن يشرك بالله فقد ضل ضلالا بعيدا.
  - ألهاكم التكاثر. حتى زرتم المقابر. كلا سوف تعلمون. ثم كلا سوف تعلمون.
  - إنا أعطيناك الكوثر. فصل لربك وانحر. إن شانئك هو الأبتر.
- source_sentence: أين تكتب الملائكة أعمال بني آدم ؟
  sentences:
  - كلا بل تكذبون بالدين. وإن عليكم لحافظين. كراما كاتبين. يعلمون ما تفعلون.
  - والعصر. إن الإنسان لفي خسر. إلا الذين آمنوا وعملوا الصالحات وتواصوا بالحق وتواصوا بالصبر.
  - كلا إن الإنسان ليطغى. أن رآه استغنى. إن إلى ربك الرجعى.
---

# islamqa-retriever-quran

An Arabic **bi-encoder for Qur'anic passage retrieval**, fine-tuned from
[`NAMAA-Space/AraModernBert-Base-STS`](https://huggingface.co/NAMAA-Space/AraModernBert-Base-STS)
through a 5-stage curriculum. This repository holds **stage 3 (`quqa`)** of that curriculum — the stage
selected as the project's production Qur'an retriever.

Built for the **IslamicEval 2025 shared task, Subtask 2** (ArabicNLP @ EMNLP 2025): given a free-text
question in Modern Standard Arabic, rank passages from the Thematic Qur'anic Passage Collection (1,266
passages) and Sahih Al-Bukhari (2,254 hadiths).

- **Type:** sentence-transformers bi-encoder (`ModernBertModel` + mean pooling)
- **Parameters:** ~149M · **Embedding dim:** 768 · **Similarity:** cosine
- **Max sequence length:** 512 tokens (tokenizer `model_max_length`; the backbone's
  `max_position_embeddings` is 8192, but 512 is the effective limit — index and query must use the same)
- **Pooling:** **mean** · **Prompts/prefixes:** none (empty for both `query` and `document`)
- **Language:** Arabic (MSA)

---

## ⚠️ Read this before using it: the Qur'an/Hadith seesaw

This checkpoint is **Qur'an-specialised, and it paid for that in hadith recall.** The curriculum's
per-stage evaluation shows a genuine seesaw between the two corpora — training on Qur'anic data
(Tafseer, QuQA) lifts Qur'an recall and depresses hadith recall, and the hadith stage reverses it
catastrophically.

**Dense-only Recall (no BM25, no reranker), measured on the project's two dev splits:**

| Model | Qur'an dev R@10 | R@30 | R@70 | Hadith dev R@10 | R@30 | R@70 |
|---|---|---|---|---|---|---|
| Base `AraModernBert-Base-STS` (zero-shot) | 0.322 | **0.466** | 0.584 | 0.688 | **0.750** | 0.875 |
| stage_1 `tydi` | 0.329 | 0.428 | 0.512 | 0.469 | 0.625 | 0.750 |
| stage_2 `tafseer` | 0.347 | 0.565 | 0.676 | 0.188 | 0.344 | 0.500 |
| **stage_3 `quqa` — this model** | **0.433** | **0.562** | **0.683** | 0.313 | 0.531 | 0.656 |
| stage_4 `haqa` | 0.105 | 0.134 | 0.229 | 0.688 | **0.781** | 0.875 |
| stage_5 `task` | 0.372 | 0.469 | 0.606 | 0.219 | 0.375 | 0.625 |

Qur'an dev = 34 answerable AyaTEC v1.3 dev questions over the 1,266-passage QPC corpus.
Hadith dev = 32 questions derived from HaQA over the 2,254-hadith Sahih-Bukhari corpus (a thin,
directional split — see caveats below).

**Headline:** Qur'an dev **R@30 0.466 → 0.562** (+20.6% relative) and **R@10 0.322 → 0.433** (+34.5%)
against the zero-shot base, at the cost of hadith dev R@30 0.750 → 0.531.

**So: do not use this model alone on hadith traffic.** stage_3 was selected because it dominates every
*other* fine-tuned stage on **both** splits simultaneously, not because it beats the base model
everywhere — it does not.

### The recommended production setup is *dual*: this model **+ the base model**

The parent project runs two encoders, one per corpus, and merges with per-encoder affine calibration
constants fitted on shared reference passages:

| Configuration | Pooled mixed-traffic MAP@10 (72 questions) |
|---|---|
| single: base model | lower |
| single: this model (stage_3) | 0.3646 |
| **calibrated dual: this model (Qur'an) + base model (hadith)** | **0.3696** |

The calibrated dual beats every single-model configuration on the mixed reading that most closely
proxies the hidden test set, and it also passes the hadith dev split. Raw concatenation of the two
encoders' scores does **not** work — the base model's cosine scale sits ~0.27 below this model's, so
un-calibrated merging silently suppresses the entire hadith corpus.

### End-to-end contribution (full pipeline, Qur'an dev)

Within the complete hybrid pipeline (dense + BM25 → Reciprocal Rank Fusion → `NAMAA-Space/GATE-Reranker-V1`
cross-encoder → answerability threshold gate):

| Pipeline | MAP@10 |
|---|---|
| BM25 only | 0.086 |
| dense (zero-shot base) only | 0.169 |
| hybrid + GATE, zero-shot base | 0.3135 |
| **hybrid + GATE, this model** | **0.3303** |

Over the full 3,520-passage two-corpus store, R@30 rises 0.363 → 0.489. Note that much of the retrieval
gain does not reach MAP unless the answerability threshold is **re-calibrated** for this model's score
distribution — a better retriever shifts the distribution the gate sits in.

---

## Usage

```python
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("IdealisticSolutions/islamqa-retriever-quran")

questions = ["ما هو أعظم الذنوب؟"]
passages  = [
    "إن الله لا يغفر أن يشرك به ويغفر ما دون ذلك لمن يشاء ومن يشرك بالله فقد ضل ضلالا بعيدا.",
    "ألهاكم التكاثر. حتى زرتم المقابر. كلا سوف تعلمون. ثم كلا سوف تعلمون.",
]

q = model.encode(questions, normalize_embeddings=True)
p = model.encode(passages,  normalize_embeddings=True)
print(model.similarity(q, p))   # cosine
```

No prompt prefix or instruction template is needed — unlike e5 (`query:` / `passage:`) or Qwen3, this
model was trained without prefixes and adding them will hurt.

**Text normalisation must match training.** Qur'anic passages (Tanzil simple-clean) carry **no**
diacritics; hadith text **does**. Strip diacritics (`[ً-ْٰ]`) from hadith before encoding,
and use the identical normalisation at index time and at query time. A retriever that normalises
differently on the two sides is a silent quality bug, not an error.

### Dual-encoder sketch (Qur'an + hadith)

```python
quran_enc  = SentenceTransformer("IdealisticSolutions/islamqa-retriever-quran")
hadith_enc = SentenceTransformer("NAMAA-Space/AraModernBert-Base-STS")
# Retrieve from each corpus with its own encoder, then merge with per-encoder
# affine calibration (NOT raw score concatenation — see the seesaw section).
```

---

## Training

**Curriculum:** TyDiQA-ar → Jalalayn Tafseer → **QuQA (this checkpoint)** → HaQA → task data.
Each stage initialises from the previous stage's weights and is evaluated on both dev splits;
stage 3 is where the run was cut.

| | |
|---|---|
| Base model | `NAMAA-Space/AraModernBert-Base-STS` |
| Init weights for this stage | stage_2 (`tafseer`), itself from stage_1 (`tydi`), itself from the base |
| Stage dataset | QuQA — 2,176 questions, 1,269 passages |
| Training pairs | **23,436** rows (3,365 positive / 20,071 negative) |
| Hard negatives | mined from top-K retrieval, K = 70; 2,000 questions probed, 9,972 negatives added |
| Loss | `ContrastiveLoss`, margin 0.5 |
| Epochs / steps | 1.0 / 733 |
| Batch size | 32 (grad accumulation 1) |
| LR / schedule | 2e-5, linear, warmup ratio 0.1, weight decay 0.01 |
| Seed | 42 |
| Final train loss | 0.0133 |
| Hardware / time | 1× NVIDIA RTX 5090 Laptop, 145 s for this stage |
| Frameworks | sentence-transformers 6.0.0 · transformers 5.15.1 · torch 2.11.0+cu128 |
| Trained | 2026-08-19/20 |

Positive-only training loses roughly 20% recall on this task — the hard negatives are not optional.

### Training data sources

- **QuQA** — Qur'anic question–answer pairs (this stage)
- **TyDiQA-ar** (stage 1), **Jalalayn Tafseer** (stage 2), **HaQA** (stage 4), AyaTEC task data (stage 5)
- Corpora: Thematic Qur'anic Passage Collection v1.1; Sahih Al-Bukhari v1.0

All of these are third-party datasets distributed with the IslamicEval 2025 shared task; their own terms
apply to the data, independent of this model's weights.

---

## Evaluation caveats (please read)

- **The official dev split is Qur'an-only.** Every team in the 2025 shared task saw its score collapse
  from dev to the mixed hidden test set. Never select a checkpoint or a threshold on the Qur'an dev split
  alone — that trap is exactly why stage_5 (which looks fine on dev) is not the shipped checkpoint.
- **The hadith dev split is thin (32 questions) and directional, not precise.** It is derived by grounding
  held-out HaQA questions onto Sahih-Bukhari at a precision-first match threshold; only ~17% ground.
- **The hadith dev split contains no unanswerable questions**, so it can validate ranking but can never
  calibrate a zero-answer gate.
- Dense-only numbers above are recall over a single corpus; the end-to-end MAP numbers are over the full
  two-corpus 3,520-passage store. They are not comparable to each other.
- Sample sizes are small (34 and 32 questions). Treat differences under ~0.02 as noise.

## Intended use, and what this model must not be used for

**Intended:** ranking and retrieving *existing, verbatim* passages of the Qur'an and of hadith
collections in response to Arabic natural-language questions; semantic similarity over Arabic religious
text; a retrieval component inside a larger RAG or QA system.

**Not intended, and explicitly out of scope:**

- **This model must never be used to generate, paraphrase, complete, correct, or "modernise" Qur'anic or
  hadith text.** It is an embedding model with no generation head, and the surrounding system should
  return stored passage IDs and verbatim stored text only. Hallucinated or misattributed religious
  content is the primary failure mode this entire project exists to prevent.
- It is **not** a source of religious rulings, and retrieval rank is not a statement of authenticity,
  authority, or theological correctness.
- It is Arabic-only. Non-Arabic questions must be translated to MSA before encoding; measured
  end-to-end loss for translated English questions is roughly −13% MAP@10, and about 70% of that loss is
  translation-induced ranking scatter that no answerability threshold recovers.
- If any translation is displayed to a user, the Arabic source must be shown verbatim and the translation
  clearly labelled as approximate — never presented as the source text.

**Bias and limitations:** trained on one Qur'anic QA dataset and one tafseer (Jalalayn), so it inherits
their coverage, phrasing, and interpretive perspective. Hadith coverage is limited to Sahih Al-Bukhari.
Recall@10 on Qur'an dev is 0.433 — this model is a *candidate generator*, and the pipeline that produced
the MAP numbers above depends on a cross-encoder reranker downstream.

## License and attribution

Licensed **Apache-2.0**, inherited from the base model
[`NAMAA-Space/AraModernBert-Base-STS`](https://huggingface.co/NAMAA-Space/AraModernBert-Base-STS)
(Apache-2.0), which is itself a fine-tune of
[`NAMAA-Space/AraModernBert-Base-V1.0`](https://huggingface.co/NAMAA-Space/AraModernBert-Base-V1.0).
These weights are a derivative work of that checkpoint; credit for the Arabic ModernBERT backbone and its
STS fine-tune belongs to NAMAA-Space. The training datasets are third-party and carry their own terms.

## Citation

The shared task and the reference pipeline this work reproduces and extends:

```bibtex
@inproceedings{islamiceval2025,
  title     = {IslamicEval 2025 Shared Task, Subtask 2: Qur'an and Hadith Passage Retrieval},
  booktitle = {Proceedings of ArabicNLP @ EMNLP 2025},
  year      = {2025}
}
```

Backbone: Warner et al., *Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder*
(ModernBERT), 2024. Training framework: Reimers & Gurevych, *Sentence-BERT*
([arXiv:1908.10084](https://arxiv.org/abs/1908.10084)).