Model card update

README.md

@@ -4,57 +4,233 @@ base_model: Jacaranda/AfroLlama_V1
 tags:
 - base_model:adapter:Jacaranda/AfroLlama_V1
 - lora
+- qlora
+- int4
 - transformers
+- africa
+- low-resource
 pipeline_tag: text-generation
 model-index:
 - name: Caracal_AfroLlama_int4
-  results: []
+  results:
+  - task:
+      type: text-generation
+      name: AfriXNLI (Natural Language Inference)
+    dataset:
+      name: IrokoBench AfriXNLI
+      type: irokobench/afrixnli
+    metrics:
+    - type: accuracy
+      value: 34.01
+      name: Accuracy
+license: apache-2.0
+language:
+- amh
+- tir
+- ada
+- kln
+- dav
+- orm
+- fra
+- lin
+- kin
+- kam
+- luy
+- gpe
+- pcm
+- som
+- ibo
+- luo
+- kik
+- lug
+- en
+- eng
+- yor
+- hau
+- swa
+- xho
 ---
 
-<!-- This model card has been generated automatically according to the information the Trainer had access to. You
-should probably proofread and complete it, then remove this comment. -->
+<!-- This model card has been generated according to the AfriLLMQuant pilot project telemetry report.
+You should probably proofread and complete it, then remove this comment. -->
 
 # Caracal_AfroLlama_int4
 
-This model is a fine-tuned version of [Jacaranda/AfroLlama_V1](https://huggingface.co/Jacaranda/AfroLlama_V1) on an unknown dataset.
-It achieves the following results on the evaluation set:
-- Loss: 1.8681
+## Model Description
 
-## Model description
+**Caracal_AfroLlama_int4** is an INT4-quantized, instruction-following variant of [Jacaranda/AfroLlama_V1](https://huggingface.co/Jacaranda/AfroLlama_V1), produced as part of the **AfriLLMQuant** pilot project (OpenToken / Carnegie Mellon University, Africa). It was adapted via **continued pre-training (CPT)** on African-language text, then **instruction-tuned** on the MURI dataset, then compressed to 4-bit precision using **parameter-efficient fine-tuning (PEFT) via QLoRA** — training low-rank adapters rather than the full model.
 
-More information needed
+The goal of the project was to make instruction-following African-language models deployable on resource-constrained hardware, rather than requiring non-consumer-grade GPUs.
 
-## Intended uses & limitations
+| | |
+|---|---|
+| **Base model** | Jacaranda/AfroLlama_V1 |
+| **Adapter method** | LoRA / QLoRA (PEFT) |
+| **Quantization** | INT4 |
+| **Memory footprint** | ~16 GB (FP) → **~5 GB** (INT4) |
+| **Quantization training time** | ~4 days on 1x NVIDIA A100 80GB |
+| **License** | Apache 2.0 |
 
-More information needed
+---
+
+## Supported Languages
+
+The model covers languages drawn from the CPT and instruction-tuning corpora, with a primary focus on **East Africa** and broader Sub-Saharan African coverage. Perplexity (PPL) figures below are from multi-domain CPT evaluation; lower is better, and a PPL under 100 is generally considered functional for a low-resource language.
+
+### Strong performance (PPL < 20)
+
+| Language | ISO | Avg PPL | Samples |
+|---|---|---:|---:|
+| Amharic | `amh` | 1.98 | 1,000 |
+| Tigrinya | `tir` | 1.99 | 1,000 |
+| Adangme | `ada` | 2.75 | 1,000 |
+| Kalenjin | `kln` | 2.84 | 1,000 |
+| Dawida | `dav` | 3.02 | 1,000 |
+| Code-switched | — | 4.47 | 288 |
+| English (Kenyan Agri. domain) | `en` | 5.44 | 300 |
+| Wakhal transcriptions | — | 5.96 | 1,000 |
+
+### Moderate performance (PPL 20–250)
+
+| Language | ISO | Avg PPL | Samples |
+|---|---|---:|---:|
+| Oromo | `orm` | 9.82 | 945 |
+| French | `fra` | 10.97 | 1,000 |
+| Lingala | `lin` | 10.97 | 545 |
+| KenSQuAD domain | — | 11.23 | 1,000 |
+| Kinyarwanda | `kin` | 12.14 | 1,000 |
+| Kamba | `kam` | 12.54 | 132 |
+| Luhya | `luy` | 13.34 | 153 |
+| Nigerian Pidgin (gpe) | `gpe` | 17.79 | 1,000 |
+| Nigerian Pidgin (pcm) | `pcm` | 22.19 | 568 |
+| Somali | `som` | 24.46 | 1,000 |
+| Igbo | `ibo` | 39.63 | 1,000 |
+| Luo (Dholuo) | `luo` | 47.08 | 1,000 |
+| Kikuyu (Gĩkũyũ) | `kik` | 59.26 | 1,000 |
+| Luganda | `lug` | 97.74 | 1,000 |
+| English (general) | `en`/`eng` | 118.29 / 136.52 | 1,000 / 1,000 |
+| Yoruba | `yor` | 178.19 | 1,000 |
+| Hausa | `hau` | 205.84 | 1,000 |
+
+### Weaker performance (PPL > 250)
+
+| Language | ISO | Avg PPL | Samples |
+|---|---|---:|---:|
+| Swahili | `swa` | 272.91 | 1,000 |
+| Xhosa | `xho` | 457.27 | 1,000 |
+| Acholi | `ach` | 1,619.32 | 1,000 |
+| Shona | `sna` | 2,079.58 | 1,000 |
+| Zulu | `zul` | 2,564.68 | 1,000 |
+
+> **Note:** The project report attributes the high-perplexity languages (Acholi, Shona, Zulu) to limited training-data coverage rather than a fundamental modeling limitation, and recommends targeted corpus expansion for these languages in future work. Treat this as a stated hypothesis rather than a verified result — it has not been independently confirmed by additional experiments.
+
+---
+
+## Training Data
+
+- **Continued Pre-training (CPT) corpus:** ~0.5B tokens after filtering (from a larger ~1.2B token raw collection), capped at 5M tokens per language. Sources included open-license web data, Wikipedia, and Bible translations for Somali, Dholuo/Luo, Luhya, Gikuyu, Acholi, Luganda, Kamba, Gusii, Kinyarwanda, and Kalenjin, alongside other African and cross-lingual corpora.
+- **Instruction-tuning dataset:** ~60,000 examples from the [MURI dataset](https://arxiv.org/abs/2409.12958) (Apache 2.0 licensed), focused on low-resource language instruction-following with emphasis on Sub-Saharan African languages.
+- **PII and toxicity:** Source data was reviewed to remove personally identifiable information and screened for toxic content prior to training.
+
+### Quantization / Adapter Training
+
+Caracal_AfroLlama_int4 used **PEFT via QLoRA** to fine-tune and quantize AfroLlama_V1 to INT4 over a 4-day training run — distinct from the full Quantization-Aware Training (QAT) used for the sibling Caracal-instruct model.
+
+---
+
+## Evaluation
+
+| Benchmark | Metric | Score |
+|---|---|---:|
+| AfriXNLI (IrokoBench) | Accuracy | **34.01%** |
+
+AfriXNLI is a 3-way natural language inference benchmark for African languages. The reported score is close to the ~33% random-baseline rate for 3-way classification, so this result should be treated as a preliminary signal rather than strong evidence of NLI capability, pending further evaluation.
+
+---
+
+## Usage
+
+```python
+from peft import PeftModel
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+base_model_id = "Jacaranda/AfroLlama_V1"
+adapter_id = "amidblue/Caracal_AfroLlama_int4"  # update with actual repo path
+
+tokenizer = AutoTokenizer.from_pretrained(base_model_id)
+base_model = AutoModelForCausalLM.from_pretrained(base_model_id, load_in_4bit=True, device_map="auto")
+model = PeftModel.from_pretrained(base_model, adapter_id)
 
-## Training and evaluation data
+prompt = "<s><Input>\nWho is the president of Kenya?\n</Input>\n<Answer>\n"
+inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+output = model.generate(**inputs, max_new_tokens=64)
+print(tokenizer.decode(output[0], skip_special_tokens=True))
+```
 
-More information needed
+### Expected input/output format
 
-## Training procedure
+The model expects the `<Input>...</Input>` / `<Answer>...</Answer>` instruction format used during SFT, e.g.:
+
+```
+<s> <Input>
+Asha ana kalamu 7 mwanzoni. Kisha ananunua pakiti 3 za kalamu, ambapo kila pakiti ina kalamu 4. Baadaye, anampatia rafiki yake kalamu 5. Asha anabaki na kalamu ngapi?
+</Input>
+<Answer>
+$$\boxed{14}$$ </Answer>
+```
+
+---
+
+## Known Limitations
+
+Manual spot-checks during the pilot surfaced several recurring failure modes that should be kept in mind before relying on this model:
+
+- **Factual unreliability on current events and named entities.** The model can produce confident, fluent, but incorrect answers about real-world facts such as current political office-holders — including conflating different countries, leaders, and historical periods in the same answer. It performed correctly on some queries (e.g., correctly identifying Kenya's president) but not others, so accuracy is inconsistent rather than uniformly wrong.
+- **Fabricated entities.** In some cases the model generates plausible-sounding but nonexistent names or institutions, rather than indicating uncertainty.
+- **Topic drift in free-form generation.** On open-ended Swahili prompts, the model sometimes continues into unrelated tangents and run-on text not requested in the prompt.
+- **Unreliable on health/medical content.** Responses to health-related prompts (e.g., explaining medical conditions) can include inaccurate or fabricated claims stated with unwarranted confidence. **This model should not be used as a source of medical information.**
+- **Inconsistent performance on structured/multiple-choice reasoning.** On simple causal-reasoning (COPA-style) tasks the model can answer correctly, but on basic multiple-choice tasks (e.g., selecting which listed number is even) it has been observed to select an incorrect option while still producing a well-formatted answer — indicating the failure is in reasoning, not output formatting.
+- **High perplexity for several Bantu and Nilotic languages** (notably Acholi, Shona, and Zulu), reflecting limited representation in the training corpus relative to other supported languages.
+
+Given these patterns, outputs — especially factual claims, named entities, and anything health- or safety-related — should be independently verified before use in any downstream application.
+
+---
+
+## Intended Uses & Limitations
+
+**Intended for:** research into low-resource African-language modeling, lightweight on-device deployment experiments, and as a base for further fine-tuning on downstream African-language tasks.
+
+**Not intended for:** medical, legal, financial, or other high-stakes advice; uses requiring reliable factual/current-events accuracy; production deployment without further evaluation and safety review.
+
+---
+
+## Training Procedure
 
 ### Training hyperparameters
 
-The following hyperparameters were used during training:
+The following hyperparameters were used during adapter training:
+
 - learning_rate: 2e-05
 - train_batch_size: 16
 - eval_batch_size: 16
 - seed: 42
 - gradient_accumulation_steps: 2
 - total_train_batch_size: 32
-- optimizer: Use paged_adamw_8bit with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
+- optimizer: paged_adamw_8bit (betas=(0.9, 0.999), epsilon=1e-08)
 - lr_scheduler_type: cosine
 - lr_scheduler_warmup_steps: 0.03
 - num_epochs: 2
 
 ### Training results
 
-| Training Loss | Epoch | Step  | Validation Loss |
-|:-------------:|:-----:|:-----:|:---------------:|
-| 1.8109        | 1.0   | 17746 | 1.9012          |
-| 1.7053        | 2.0   | 35492 | 1.8681          |
+| Training Loss | Epoch | Step | Validation Loss |
+|:---:|:---:|:---:|:---:|
+| 1.8109 | 1.0 | 17,746 | 1.9012 |
+| 1.7053 | 2.0 | 35,492 | 1.8681 |
+
+### Compute
 
+Trained on a single dedicated **NVIDIA A100 80GB SXM** instance (LeafCloud, Netherlands), provided under an attribution/SAAC Tier-1 free allocation as part of a 3-week OpenToken grant period (May 2026). Estimated equivalent commercial cost: ~$1,260 (504 hrs × $2.50/hr at hyperscaler rates).
 
 ### Framework versions
 
@@ -62,4 +238,28 @@ The following hyperparameters were used during training:
 - Transformers 5.5.1
 - Pytorch 2.12.0+cu126
 - Datasets 4.8.5
-- Tokenizers 0.22.2
+- Tokenizers 0.22.2
+
+---
+
+## Citation
+
+```bibtex
+@misc{caracal-afrollama-int4,
+  author    = {Theophilus Lincoln Owiti},
+  title     = {Caracal\_AfroLlama\_int4: INT4-Quantized Instruction-Following AfroLlama for African Languages},
+  year      = {2026},
+  publisher = {Hugging Face},
+  note      = {Carnegie Mellon University, Africa / AfriLLMQuant},
+  url       = {https://huggingface.co/amidblue/Caracal_AfroLlama_int4}
+}
+```
+
+**Authors:**
+- **Theophilus Lincoln Owiti** — Carnegie Mellon University, Africa
+
+---
+
+## Model Card Authors
+
+Theophilus Lincoln Owiti