NAMAA-Space/alexandriax-nllb-1.3b-lora

AlexandriaX-2026 · Subtask 1 — NLLB-200-1.3B, QLoRA adapter

English to dialectal Arabic dialogue translation over 13 Arabic varieties, from the NAMAA Community submission to AlexandriaX-2026 (ArabicNLP 2026 / EMNLP). A LoRA adapter over facebook/nllb-200-1.3B, trained in 4-bit (QLoRA), with the target variety selected by NLLB's own language codes rather than by a text prefix.

The interesting negative result here: NLLB-200 shares its SentencePiece vocabulary with the metric itself (spBLEU tokenises with FLORES-200's SentencePiece model) and exposes native codes for several Arabic varieties, yet it finished last of the five small fine-tunes — 2.4 spBLEU behind a 368M encoder-decoder with no language codes at all. Vocabulary alignment with the metric does not buy score.

Task	AlexandriaX-2026 Subtask 1 (context-aware EN→DA dialogue translation)
Base model	facebook/nllb-200-1.3B (encoder-decoder, 1.3B parameters)
Adapter	LoRA, r=16, α=32, dropout 0.05 — 94.5 MB, approx. 23.6M trainable parameters
Dialect control	per-dialect forced_bos_token_id (NLLB target code)
Context	none — each turn is translated in isolation
Dev (12,250 turns, 11 countries)	21.83 spBLEU · 38.13 chrF++
Blind test	not run (this system was not part of a submitted bundle)
Track	constrained (provided data only, ≤5B parameters)
License	CC-BY-NC-4.0, inherited from NLLB-200 — non-commercial

Per-dialect target codes

NLLB has native codes for seven of the thirteen varieties. The remaining four with training data plus the two test-only varieties are mapped to their nearest neighbour, which acts as a decoding bias that fine-tuning then adapts:

Country	NLLB code	Native or proxy
EG Egyptian	arz_Arab	native
MA Moroccan	ary_Arab	native
TN Tunisian	aeb_Arab	native
SA Saudi	ars_Arab	native
SY Syrian	apc_Arab	native (North Levantine)
LB Lebanese	apc_Arab	native (North Levantine)
JO Jordanian	ajp_Arab	native (South Levantine)
PS Palestinian	ajp_Arab	native (South Levantine)
YE Yemeni	acq_Arab	native (Ta'izzi-Adeni)
OM Omani	ars_Arab	proxy — no native code
MR Mauritanian	ary_Arab	proxy — no native code
LY Libyan	arz_Arab	proxy — no native code, test-only
SD Sudanese	arz_Arab	proxy — no native code, test-only

Source language is always eng_Latn; the fallback code is arb_Arab (MSA).

Usage

import torch
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
from peft import PeftModel

BASE = "facebook/nllb-200-1.3B"
ADAPTER = "NAMAA-Space/alexandriax-nllb-1.3b-lora"

tok = AutoTokenizer.from_pretrained(BASE)
base = AutoModelForSeq2SeqLM.from_pretrained(BASE, torch_dtype=torch.bfloat16,
                                             device_map="auto")
model = PeftModel.from_pretrained(base, ADAPTER).eval()

NLLB_CODE = {"EG": "arz_Arab", "MA": "ary_Arab", "TN": "aeb_Arab", "SA": "ars_Arab",
             "SY": "apc_Arab", "LB": "apc_Arab", "JO": "ajp_Arab", "PS": "ajp_Arab",
             "YE": "acq_Arab", "OM": "ars_Arab", "MR": "ary_Arab",
             "LY": "arz_Arab", "SD": "arz_Arab"}

def translate(sentences, country, num_beams=5, max_new_tokens=128):
    tok.src_lang = "eng_Latn"
    enc = tok(sentences, return_tensors="pt", padding=True, truncation=True, max_length=200)
    enc = {k: v.to(model.device) for k, v in enc.items()}
    bos = tok.convert_tokens_to_ids(NLLB_CODE[country])
    with torch.no_grad():
        out = model.generate(**enc, forced_bos_token_id=bos,
                             num_beams=num_beams, max_new_tokens=max_new_tokens)
    return tok.batch_decode(out, skip_special_tokens=True)

print(translate(["Good morning. How much for the whole quantity?"], "EG"))

Reference decoding for every number in this card: beam search with 5 beams, max_new_tokens=128, source truncated at 200 tokens. Loading the base in 4-bit (BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.bfloat16)) reproduces the training-time numerics and fits the whole thing in well under 8 GB.

Intended use

Research on dialect-code conditioning in massively multilingual MT: what NLLB's Arabic codes do and do not buy, and how far a proxy code can be pushed for a variety the model has never had a code for. Non-commercial only, per the base model's licence.

---

The shared task

AlexandriaX-2026 (ArabicNLP 2026 / EMNLP) — Context-Aware Dialectal Arabic MT and MT Evaluation. This model was built for Subtask 1: Context-Aware English-to-Dialectal Arabic Dialogue Translation.

Given one English dialogue turn together with its conversation history and metadata — target country/dialect, domain, participant roles, speaker, and speaker→addressee gender direction — the system must produce the turn in the requested country's spoken Arabic, preserving meaning while adapting lexical, morphological, pragmatic and sociolinguistic choices to that variety.

Two tracks: constrained (provided data only, ≤5B parameters) and unconstrained (any external data or model). Ranking is by spBLEU (primary) and chrF++ (secondary), each macro-averaged over countries.

Official data (UBC-NLP/alexandria)

Split sizes in turns, as published by the organisers:

Split	EG	JO	LB	LY	MA	MR	OM	PS	SA	SD	SY	TN	YE	Total
train	3,108	5,501	8,906	0	2,573	5,515	6,280	14,933	8,470	0	6,071	2,034	3,089	66,480
dev	1,113	1,113	1,118	0	1,110	1,114	1,109	1,110	1,110	0	1,119	1,116	1,118	12,250
public test	1,118	1,107	1,106	1,109	1,115	1,112	1,118	1,109	1,113	1,106	1,114	1,109	1,106	14,442
private (blind) test	1,113	1,109	1,110	1,309	1,111	1,119	1,107	1,111	1,114	915	1,114	1,114	1,113	14,459

Libyan (LY) and Sudanese (SD) appear only at test time — they are zero-shot for every system trained on this data.

Conversation-level counts: 21,146 train / 3,963 dev / 4,706 public-test conversations; mean 3.13 turns per conversation (range 1–5). Mean length 102 characters of English source, 74 characters of dialectal target.

Dialects (13 countries). Egyptian, Jordanian, Lebanese, Libyan, Moroccan, Mauritanian, Omani, Palestinian, Saudi, Sudanese, Syrian, Tunisian, Yemeni. Labels are country + sub-dialect, and several countries carry more than one: Palestinian 10 (Nabulsi and Albira urban, plus Falahi varieties of Surif, Kobar, Noba, Ni'lin, Shuqba, Aboud, Silwad, Ramallah), Omani 5 (Suri, Rustaqi, Al-Wafi, Ibri, Seebi), Saudi 3 (Southern, Hijazi, Khaleeji), Yemeni 3 (Taiz, San'ani, Central), Syrian 2 (Levantine Standard, Homsi). The remaining countries carry one label each (e.g. Egyptian Arabic (Cairene), Moroccan Standard Darija, Mauritanian Hassaniya, Libyan Arabic (Misrati/Central)).

Domains (11, near-uniform). Agriculture and farming, Commerce and transactions, Construction and real estate, Education and academia, Energy and resources, Everyday and social, Healthcare and medical, Legal and financial, Logistics and transportation, Professional and workplace, Tourism and hospitality.

Speaker direction (turns, train+dev+public test): female→male 30,636 · male→female 30,203 · male→male 20,465 · female→female 11,868. The corpus carries 76 distinct translator IDs and 44 reviewer IDs.

Code-switching in the gold is strongly dialect-specific — the share of gold turns containing Latin characters runs from TN 39.1% / MA 33.8% / LB 18.1% down to SY 1.2% / YE 0.8%. Systems that normalise every borrowing into Arabic script are penalised hardest on Maghrebi references (see Known limitations).

Evaluation protocol

• spBLEU — sacrebleu.BLEU(tokenize="flores200"), corpus-level per country, then averaged over countries.
• chrF++ — sacrebleu.CHRF(word_order=2), same averaging.
• Decoding is turn-by-turn: at turn n the conversation history contains the system's own previous outputs, never the gold ones. (An early evaluation harness in this project leaked gold previous-turn Arabic into the prompt and inflated scores by ≈2.4 spBLEU; every number reported here comes from the corrected, self-conditioned harness.)

---

Training data

The official Subtask-1 training conversations only — 63,130 English→dialect turn pairs over the 11 countries with a train split, each tokenised under its own target code (the dataset is built per code and concatenated). No context, no auxiliary corpus, no back-translation.

PS	LB	SA	OM	SY	MR	JO	YE	EG	MA	TN
14,183	8,464	8,035	5,965	5,760	5,234	5,224	2,946	2,943	2,443	1,933

LY and SD contribute no training data; at test time they borrow arz_Arab and rely on whatever transfers from Egyptian.

Training procedure — every hyperparameter

Extracted verbatim from AlexandriaX_NB7_NLLB_1.3B_Full.ipynb. Runnable single-file version: train_nllb13b_qlora.py.

Model, adapter and data

Setting	Value	Note
Base model	facebook/nllb-200-1.3B	encoder-decoder, 24+24 layers, d_model 1024, FFN 8192
Regime	QLoRA	4-bit frozen base + LoRA adapters
LoRA rank r	16
LoRA alpha	32	scaling 32/16 = 2
LoRA dropout	0.05
LoRA bias	"none"	biases not adapted
task_type	SEQ_2_SEQ_LM
Target modules	q_proj, k_proj, v_proj, out_proj, fc1, fc2	attention and feed-forward, encoder and decoder
Trainable parameters	≈23.6M of 1.37B (1.7%)	94.5 MB adapter
max_length source / target	200 / 200 tokens	shorter than the AraT5 runs
Label padding	-100
Training examples	63,130 turn pairs	tokenised per country under its own target code, then concatenated
Dialect conditioning	tokenizer.tgt_lang per country at tokenisation time	see the code table above
Shuffle seed	42

Quantisation

Setting	Value
load_in_4bit	True
bnb_4bit_quant_type	nf4
bnb_4bit_use_double_quant	True
bnb_4bit_compute_dtype	bfloat16 (fp16 fallback if bf16 unsupported)
device_map	"auto"

Optimisation

Setting	Value	Note
Optimiser	paged_adamw_8bit	paged, so optimiser state can spill to host memory
Learning rate	2e-4	the usual QLoRA range
LR scheduler	cosine
Warmup	100 steps
Epochs	3	the notebook notes 1 epoch is often as good and ~3× faster
per_device_train_batch_size	16
gradient_accumulation_steps	2
Effective batch	32
Optimiser steps	≈5,920	63,130 / 32 × 3
Label smoothing	none	the repo's first auto-generated card claimed 0.1; it was never applied
Weight decay / clipping	0.0 / 1.0	framework defaults

Memory and the two NLLB-specific fixes

Setting	Value	Why
prepare_model_for_kbit_training(..., use_gradient_checkpointing=False)		so checkpointing is not double-enabled
gradient_checkpointing_enable(use_reentrant=False)	called exactly once, after get_peft_model	otherwise M2M100/NLLB raises "cannot specify both decoder_input_ids and decoder_inputs_embeds"
TrainingArguments(gradient_checkpointing=False)		already enabled above; setting it here too re-triggers the crash
enable_input_require_grads()	called	gradients must reach the frozen 4-bit base under checkpointing
decoder_start_token_id	left untouched	overriding NLLB's native value breaks generation
config.pad_token_id	set from the tokenizer
use_cache	False during training

Precision, bookkeeping, hardware

Setting	Value
Hardware	1 × A100-80GB
bf16 / fp16	bf16 True (fp16 as fallback)
TF32	enabled (matmul and cudnn)
logging_steps / save_steps / save_total_limit	25 / 200 / 2
eval_strategy	"no"
Resume	automatic from the highest checkpoint-*
Libraries	transformers, peft 0.19.1, bitsandbytes

Inference (used for every score in this card)

Setting	Value
Decoding	beam search, num_beams=5
forced_bos_token_id	the target country's NLLB code
tokenizer.src_lang	eng_Latn
max_new_tokens	128
Generation batch	48
Context	none

Results

Per-country, official dev set (12,250 turns, 11 countries)

	EG	JO	LB	MA	MR	OM	PS	SA	SY	TN	YE	macro
spBLEU	23.23	27.09	23.18	13.19	9.03	21.07	26.02	25.29	31.33	20.77	19.97	21.83
chrF++	38.95	43.73	39.19	28.98	25.78	38.06	41.70	42.14	47.38	36.48	37.00	38.13

Against the other constrained systems (same dev set)

System	Params	dev spBLEU	dev chrF++
AraT5v2, full fine-tune	368M	25.12	40.66
Qwen2.5-1.5B LoRA	1.5B	23.71	40.51
NileChat-3B QLoRA, context-free	3B	23.54	39.68
Gemma-3-1B LoRA	1B	22.71	38.86
NLLB-200-1.3B QLoRA (this model)	1.3B	21.83	38.13

What the ranking says

Three facts sit awkwardly together and are the reason this checkpoint is released even though it lost:

1. NLLB's SentencePiece model is the spBLEU tokeniser, so this system and the metric segment text identically — and it still finished last.
2. It is the only system in the bank with native dialect codes, and they did not rescue the varieties that have them (MA 13.19 with a native ary_Arab, against 19.75 for AraT5v2 with a plain English prefix).
3. Its worst two countries are exactly the two proxy-coded ones with data (MR 9.03 with ary_Arab, OM 21.07 with ars_Arab), which is the expected direction, but the native-code varieties did not compensate.

The reading consistent with the whole bank: this metric rewards fidelity to the annotators' conventions, which is acquired from the target side during fine-tuning, more than it rewards either fluency or vocabulary alignment. A 368M dialect-pretrained encoder-decoder learns those conventions better than a 1.3B general-purpose translator with 1.7% of its weights adapted.

Known limitations

• Non-commercial licence (CC-BY-NC-4.0) inherited from NLLB-200. Research use only.
• Proxy codes are approximations. LY, SD, OM and MR have no native NLLB code; the mapping to arz/ars/ary biases decoding toward a neighbouring variety and shows up as the two lowest scores in the table.
• Mauritanian Hassaniya is effectively unsolved here (9.03 spBLEU), the weakest score of any system in the bank on any country.
• No conversational context, so no cross-turn anaphora or politeness tracking.
• Sub-dialects are not addressed — one code per country, while the corpus labels up to 10 Palestinian and 5 Omani sub-dialects.
• LY and SD are zero-shot and untested at development time.
• Not evaluated on the blind test, so no 13-country number exists for this checkpoint.
• Metric-only evaluation: no human judgement, no neural metric.

---

Where this model sits in the NAMAA system

All Subtask-1 systems built by the team, scored on the official 12,250-turn dev set (11 countries) and, where they were run, on the 14,459-turn private blind test (13 countries). Country-macro spBLEU / chrF++.

System	Params / arch.	dev spBLEU	dev chrF++	blind spBLEU	blind chrF++	Released
Gemma, beam search (submitted, constrained)	~3.1B, dec-only	—	—	27.413	42.58	no
Routed ensemble (submitted, unconstrained)	—	—	—	27.412	43.05	n/a
Gemini 2.5 Flash, 5-shot	API	—	—	26.68	42.49	n/a
Claude Sonnet 4.5, 5-shot	API	—	—	26.36	42.26	n/a
AraT5v2 full fine-tune	368M, enc–dec	25.12	40.66	23.26	39.03	alexandriax-arat5v2-base
Qwen2.5-1.5B LoRA	1.5B, dec-only	23.71	40.51	21.24	38.06	no
NileChat-3B QLoRA, context-free	3B, dec-only	23.54	39.68	—	—	alexandriax-nilechat-lora
NileChat-3B QLoRA, +context	3B, dec-only	22.87	39.11	—	—	no
NileChat-3B QLoRA, +context +back-translation	3B, dec-only	22.77	38.71	—	—	alexandriax-nilechat-ctx-aux
Gemma-3-1B LoRA	1B, dec-only	22.71	38.86	20.09	36.20	no
NLLB-200-1.3B QLoRA	1.3B, enc–dec	21.83	38.13	—	—	alexandriax-nllb-1.3b-lora
AraT5v2, dialect-rebalanced	368M, enc–dec	void run¹		—	—	alexandriax-arat5v2-balanced
mT5-large, dialect-rebalanced	1.23B, enc–dec	not evaluated²		—	—	alexandriax-mt5-large-balanced
MBR over 3 NileChat variants	—	23.57	39.86	—	—	n/a
MBR over 5 samples, one model	—	20.09	37.50	—	—	n/a
Linear adapter merge	—	19.90	35.33	—	—	n/a

¹ That run was trained against destroyed targets — a tokenizer fallback substituted t5-base (32,100 English tokens) for AraT5v2's 110,208-token vocabulary, so every Arabic character became <unk>. It scored 0.00 spBLEU and cannot be recovered without retraining; the post-mortem and a fixed training script are in its card.

² That run stopped at step 2,500 of a planned 31,568 (epoch 0.63 of 8) and was never decoded on the development set, so no score exists for it. Its card carries the full recovered configuration.

Two findings from this bank of models are worth carrying elsewhere.

1. Parameter count does not predict rank below the cap. The 368M encoder–decoder AraT5v2 beats every larger decoder-only fine-tune on identical data, and among the decoder-only models spBLEU falls Qwen2.5-1.5B > NileChat-3B > Gemma-3-1B — the reverse of their size order. A reading consistent with this: the metric rewards fidelity to the annotators' conventions over generative fluency. A translator fine-tuned on the provided targets acquires those conventions; a decoder-only model several times its size contributes fluency n-gram overlap does not credit.
2. Combination is not free. Fitted and evaluated on disjoint halves of the dev conversations: routing per country +0.07, per country + sub-dialect +0.28, per country + domain −0.32, MBR consensus over 5 systems −0.57, MBR over the top-2 per dialect −0.81 — against a best single system of 24.98. The per-turn oracle reaches 32.25 (+7.27), so the right output is usually in the pool and the failure is in selection: three NileChat variants agree with one another and outvote the single strongest system, so consensus weights model-family size rather than quality. The submitted system therefore routes per dialect under a ±0.40 spBLEU margin guard instead of voting.

---

The collection

All released artefacts live in NAMAA at AlexandriaX-2026:

Repo	What it is
alexandriax-arat5v2-base	AraT5v2-base full fine-tune — best small fine-tune, 25.12 dev / 23.26 blind spBLEU
alexandriax-arat5v2-balanced	the same recipe on a temperature-rebalanced dialect mixture — void run, released for the post-mortem and the fixed script
alexandriax-nilechat-lora	NileChat-3B QLoRA, context-free — best of the three NileChat variants, 23.54 dev spBLEU
alexandriax-nilechat-ctx-aux	NileChat-3B QLoRA, context + back-translation — the augmentation ablation, 22.77 dev spBLEU
alexandriax-nllb-1.3b-lora	NLLB-200-1.3B QLoRA with per-dialect language codes, 21.83 dev spBLEU
alexandriax-mt5-large-balanced	mT5-large on the rebalanced mixture — partial run (2,500/31,568 steps), never evaluated
alexandria-backtranslated-pairs	348,787 synthetic EN→dialect pairs over 14 varieties

Every model repo above carries a single-file train_*.py reproduction script with the exact hyperparameters that produced its checkpoint; the dataset repo carries build_backtranslated_pairs.py.

Official task data: UBC-NLP/alexandria. Base models: UBC-NLP/AraT5v2-base-1024, UBC-NLP/NileChat-3B-Base, facebook/nllb-200-1.3B, google/mt5-large.

---

Team

NAMAA Community — Fatimah Emad Eldin (Cairo University) · Omer Nacar (Tuwaiq Academy) · Khloud Al Jallad (Arab International University) · Mona Abdelazim (Ain Shams University).

Citation

Coming soon. The NAMAA system-description paper for AlexandriaX-2026 is under review for the ArabicNLP 2026 (EMNLP) proceedings; this card will be updated with the final ACL Anthology reference and DOI when the proceedings are published. Until then, please cite as:

@inproceedings{namaa-alexandriax-2026,
  title     = {{NAMAA} Community at {AlexandriaX-2026}: Prompting, Fine-Tuning and Agreement
               Voting for Dialectal Arabic Translation and Evaluation},
  author    = {Emad Eldin, Fatimah and Nacar, Omer and Al Jallad, Khloud and Abdelazim, Mona},
  booktitle = {Proceedings of the Fourth Arabic Natural Language Processing Conference
               (ArabicNLP 2026)},
  year      = {2026},
  note      = {To appear. Citation coming soon.}
}

Please also cite the shared task and the base model:

@inproceedings{alexandriax2026,
  title     = {{AlexandriaX-2026} Shared Task: Context-Aware Dialectal Arabic Machine
               Translation and MT Evaluation},
  author    = {El Mekki, Abdellah and Elmadany, AbdelRahim A. and Magdy, Samar M. and
               Ezzini, Saad and El-Haj, Mo and Jarrar, Mustafa and El-Beltagy, Samhaa and
               Abbas, Mourad and Zaraket, Fadi and Al Mandhari, Salim and Alyafeai, Zaid and
               Ghanem, Bernard and Abdul-Mageed, Muhammad},
  booktitle = {Proceedings of the Fourth Arabic Natural Language Processing Conference
               (ArabicNLP 2026)},
  year      = {2026},
  note      = {Overview paper. Citation coming soon.}
}

Acknowledgements

Thanks to the AlexandriaX-2026 organisers for the data, the evaluation infrastructure and their responsiveness during the evaluation phases.