TigreGotico/arabic-MARBERT-dialect-identification-city-onnx

Arabic MARBERT dialect identification (city-level) - ONNX

ONNX export of Ammar-alhaj-ali/arabic-MARBERT-dialect-identification-city, a MARBERT (UBC-NLP backbone) fine-tune for city-level Arabic dialect identification - finer-grained than the country/region-level dialect identifiers already common on the Hub.

Licence: not declared by the source repository at the time of this conversion. Recorded honestly, not assumed - treat as all-rights-reserved until the author publishes a licence; converted per this org's standing policy that licence ambiguity does not block mirroring.

Provenance note: this is a single-author community fine-tune (264 downloads, 12 likes on the source repo at the time of conversion), not an institutional release. It was spot-checked before conversion effort was invested: it loads correctly and produces plausible, non-degenerate dialect predictions on Gulf/MSA/Egyptian/Levantine/Maghrebi test sentences (see parity section) - it is not a broken or abandoned checkpoint. Treat its accuracy claims with the same caution due any single-author model without institutional validation.

Files

File	Size	Purpose
model.onnx	622 MB	fp32 graph
model.int8.onnx	157 MB	dynamic int8 graph

Usage

import numpy as np
import onnxruntime as ort
from transformers import AutoTokenizer, AutoConfig
from huggingface_hub import snapshot_download

d = snapshot_download("TigreGotico/arabic-MARBERT-dialect-identification-city-onnx")
tok = AutoTokenizer.from_pretrained(d)
config = AutoConfig.from_pretrained(d)
sess = ort.InferenceSession(f"{d}/model.onnx", providers=["CPUExecutionProvider"])

def detect(text):
    enc = tok(text, return_tensors="np", truncation=True)
    inputs = {k: v for k, v in enc.items() if k in [i.name for i in sess.get_inputs()]}
    logits = sess.run(None, inputs)[0]
    idx = int(np.argmax(logits, axis=-1)[0])
    return config.id2label[idx]

print(detect("شلونك اليوم؟ شخبارك؟"))
# -> Doha

For the int8 build, load model.int8.onnx instead.

Parity with the original PyTorch model

5 sentences spanning Gulf, MSA, Egyptian, Levantine and Maghrebi Arabic, compared against the original model's own transformers.pipeline("text-classification", model="...").

parity:
  sample_size: 5
  metric: top1_exact_match
  fp32: 1.00   # 5/5
  int8: 0.80   # 4/5

Model	Top-1 agreement with reference
model.onnx (fp32)	100.00% (5/5)
model.int8.onnx	80.00% (4/5)

Differential check: all 5 distinct dialect inputs produced 5 distinct predicted city labels (Doha, MSA, Aswan, Beirut, Tunis) in the fp32 build - no collapse-to-one-label bug.

The one int8 disagreement is a near-miss, not a wrong-dialect-family error: for an Egyptian-dialect greeting the reference predicts Aswan and int8 predicts Cairo - both Egyptian cities, a fine-grained within-country confusion rather than cross-dialect noise.

Caveat on sample size: n=5 is a small smoke-test establishing conversion fidelity, not a systematic accuracy evaluation.