medelharchaoui/t5-small-xa-only-squad

T5-small — Cross-Attention-Only fine-tune on SQuAD

Encoder–decoder QA model fine-tuned from google-t5/t5-small (60.5M) by training only the decoder cross-attention (EncDecAttention) blocks plus the decoder final layer norm — ~6.3M trainable params (10.4%). The encoder, decoder self-attention, and feed-forward weights are frozen.

TL;DR: a 60M-param encoder–decoder, tuning just 6.3M of those weights, reaches F1 0.7075 / EM 0.5293 on SQuAD — outperforming a fully fine-tuned GPT-2-large (774M, F1 0.5041) that is 13× larger. The strongest evidence in this study that architecture beats scale for context-grounded tasks.

Results (SQuAD validation)

Model	Architecture	Trainable / Total	EM	Token F1
GPT-2-large	decoder-only	774M / 774M	0.3516	0.5041
T5-small XA-only (this model)	enc-dec	6.3M / 60.5M	0.5293	0.7075
T5-large XA-only	enc-dec	100.7M / 737M	0.6406	0.8128
T5-large LoRA r=8	enc-dec	2.4M / 740M	0.6445	0.8152

Validation loss 0.4208, perplexity 1.52. Trained on only 3,000 SQuAD examples.

Eval note: 256-example SQuAD-validation generation subset, 4-beam search.

How to use

Trained with the input prefix answer question: prepended to a question: ... context: ... source string — match it exactly:

from transformers import T5ForConditionalGeneration, AutoTokenizer

repo = "medelharchaoui/t5-small-xa-only-squad"
tok = AutoTokenizer.from_pretrained(repo)
model = T5ForConditionalGeneration.from_pretrained(repo)

question = "What culture do 'bairn' and 'hyem' originate from?"
context = ("'bairn' and 'hyem' are geordie words with origins in scandinavia; barn and hjem "
           "are the corresponding modern norwegian and danish words.")
text = f"answer question: question: {question} context: {context}"
ids = tok(text, return_tensors="pt", truncation=True, max_length=384).input_ids
print(tok.decode(model.generate(ids, num_beams=4, max_new_tokens=16)[0], skip_special_tokens=True))

Training

Setting	Value
Base model	google-t5/t5-small (60.5M)
Trainable params	*.EncDecAttention.* + decoder.final_layer_norm (~6.3M)
Dataset	rajpurkar/squad, 3,000 train examples
Precision	bf16
Optimizer steps	1,500 (batch 4 × grad-accum 8 = eff. batch 32)
LR / warmup	2e-4, 150 warmup, weight decay 0.01
Source / target max len	384 / 32
Seed	37
Hardware	1× NVIDIA RTX 3060 (12 GB), local

Limitations

• English SQuAD-style extractive QA only; short answer spans grounded in the supplied context.
• Small model trained on only 3,000 examples — strong for its size on SQuAD, but not a general-purpose QA system.
• Evaluated on a held-out validation subset, not the official SQuAD test server.

Citation

Part of an encoder–decoder vs decoder-only paradigm study (OptimiAI, 2026).