Arabic NLI Binary Classifier — s27-sawb-arabert-large-nli

Binary Arabic text classifier (0 = faithful, 1 = unfaithful). Fine-tuned for Arabic NLI-based binary text classification.

Base model

HassanB4/sawb-arabert-large (AraBERT-Large, 340M params, previously fine-tuned for Arabic factual inconsistency detection in the ICAIRE work, F1=0.9788 there)

Input format

nli — [CLS] gold_answer [SEP] model_answer [SEP] (NLI framing: gold_answer as premise, model_answer as hypothesis)

Dev results

• AUC-ROC (official, full dev n=1300): 0.9547
• AUC-ROC (clean dev, n=800, excludes ~100 questions also seen in train): 0.9295
• Macro F1 (official, threshold=0.55 via grid search): 0.9113

Note: the official-dev number is inflated by 500 dev rows whose questions also appear in the training set (near-memorization). The clean-dev AUC-ROC (0.9295) is the honest generalization estimate and the number to use for ranking against other runs. It sits between s02-camelbert-nli/s03-marbert-nli (0.927) and s05-mbert-nli (0.9310), below s04-arbert-nli (0.9408, current best single). Interestingly, this run's epoch-1 AUC (0.8885) was well below every other run's epoch-1 (all ≥0.93) — the ICAIRE checkpoint's factual inconsistency-aware encoder, previously fine-tuned on a different (qa-style) input format, needed extra steps to adapt to the NLI framing before catching up. It then climbed monotonically for all 5 epochs without plateauing (0.8885 → 0.9293 → 0.9427 → 0.9519 → 0.9547), unlike the other runs which peaked around epoch 2-3. As a 340M AraBERT-Large architecture (vs the 110-180M BERT-base models used elsewhere in this series), it adds architectural diversity to the ensemble. All far exceed the published CAMeLBERT baseline (0.7093 dev AUC-ROC).

Training data

Arabic training set — 4,705 Arabic (question, gold_answer, model_answer) triples, 5 source LLMs, 13 knowledge domains.

Usage

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

tokenizer = AutoTokenizer.from_pretrained("HassanB4/s27-sawb-arabert-large-nli")
model = AutoModelForSequenceClassification.from_pretrained("HassanB4/s27-sawb-arabert-large-nli")

inputs = tokenizer(gold_answer, model_answer, return_tensors="pt", truncation=True, max_length=512)
with torch.no_grad():
    logits = model(**inputs).logits
score = torch.softmax(logits, dim=-1)[0][1].item()  # unfaithfulness score
predicted_label = int(score > 0.55)  # best threshold from grid search