Upload tiny_aya_global-lora-r64-zul-eng LoRA adapter

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,218 @@
+---
+library_name: peft
+base_model: CohereLabs/tiny-aya-global
+language:
+- zu
+- en
+tags:
+- translation
+- african-languages
+- scientific-translation
+- afriscience-mt
+- lora
+- peft
+- causal-lm
+license: apache-2.0
+pipeline_tag: translation
+model-index:
+- name: tiny_aya_global-lora-r64-zul-eng
+  results:
+  - task:
+      type: translation
+    metrics:
+    - name: BLEU (test)
+      type: bleu
+      value: 25.80
+    - name: chrF (test)
+      type: chrf
+      value: 58.83
+---
+
+# tiny_aya_global-lora-r64-zul-eng
+
+[![Model on HF](https://huggingface.co/datasets/huggingface/badges/raw/main/model-on-hf-sm.svg)](https://huggingface.co/AfriScience-MT/tiny_aya_global-lora-r64-zul-eng)
+
+This is a **LoRA adapter** for the AfriScience-MT project, enabling efficient scientific machine translation for African languages.
+
+## Adapter Description
+
+| Property | Value |
+|----------|-------|
+| **Base Model** | [CohereLabs/tiny-aya-global](https://huggingface.co/CohereLabs/tiny-aya-global) |
+| **Translation Direction** | isiZulu → English |
+| **LoRA Rank (r)** | 64 |
+| **LoRA Alpha** | 128 |
+| **Training Method** | QLoRA (4-bit quantization) |
+| **Domain** | Scientific/Academic texts |
+
+### Why LoRA?
+
+LoRA (Low-Rank Adaptation) enables efficient fine-tuning by training only a small number of additional parameters. This adapter adds only **~32.0M parameters** to the base model while achieving strong translation performance.
+
+## Evaluation Results
+
+Performance on the AfriScience-MT test set:
+
+| Split | BLEU | chrF | SSA-COMET |
+|-------|------|------|-----------|
+| **Test** | **25.80** | **58.83** | **None** |
+
+**Metrics explanation:**
+- **BLEU**: Measures n-gram overlap with reference translations (0-100, higher is better)
+- **chrF**: Character-level F-score, robust for morphologically rich languages (0-100, higher is better)
+- **SSA-COMET**: Neural metric trained for Sub-Saharan African languages, shown as percentage (0-100, higher is better) ([McGill-NLP/ssa-comet-stl](https://huggingface.co/McGill-NLP/ssa-comet-stl))
+
+## Usage
+
+### Quick Start
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
+from peft import PeftModel
+import torch
+
+# Configure 4-bit quantization (recommended for memory efficiency)
+bnb_config = BitsAndBytesConfig(
+    load_in_4bit=True,
+    bnb_4bit_compute_dtype=torch.bfloat16,
+    bnb_4bit_quant_type="nf4",
+    bnb_4bit_use_double_quant=True,
+)
+
+# Load base model
+base_model = AutoModelForCausalLM.from_pretrained(
+    "CohereLabs/tiny-aya-global",
+    quantization_config=bnb_config,
+    device_map="auto",
+    torch_dtype=torch.bfloat16,
+)
+tokenizer = AutoTokenizer.from_pretrained("CohereLabs/tiny-aya-global")
+
+# Load LoRA adapter
+adapter_name = "AfriScience-MT/tiny_aya_global-lora-r64-zul-eng"
+model = PeftModel.from_pretrained(base_model, adapter_name)
+model.eval()
+
+# Prepare translation prompt
+source_text = "Climate change significantly impacts agricultural productivity in sub-Saharan Africa."
+instruction = "Translate the following isiZulu scientific text to English."
+
+# Format prompt
+prompt = f"""### Instruction:
+{instruction}
+
+### Input:
+{source_text}
+
+### Response:
+"""
+
+# Generate translation
+inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+with torch.no_grad():
+    outputs = model.generate(
+        **inputs,
+        max_new_tokens=256,
+        num_beams=5,
+        early_stopping=True,
+        pad_token_id=tokenizer.pad_token_id,
+    )
+
+# Decode only the generated part
+generated = outputs[0][inputs["input_ids"].shape[1]:]
+translation = tokenizer.decode(generated, skip_special_tokens=True)
+print(translation)
+```
+
+### Without Quantization (Full Precision)
+
+```python
+# For GPUs with sufficient memory (>24GB for larger models)
+base_model = AutoModelForCausalLM.from_pretrained(
+    "CohereLabs/tiny-aya-global",
+    device_map="auto",
+    torch_dtype=torch.bfloat16,
+)
+model = PeftModel.from_pretrained(base_model, "AfriScience-MT/tiny_aya_global-lora-r64-zul-eng")
+```
+
+## Training Details
+
+### Hyperparameters
+
+| Parameter | Value |
+|-----------|-------|
+| LoRA Rank (r) | 64 |
+| LoRA Alpha | 128 |
+| LoRA Dropout | 0.05 |
+| Target Modules | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
+| Epochs | 3 |
+| Batch Size | 2 |
+| Learning Rate | 2e-04 |
+| Max Sequence Length | 512 |
+| Gradient Accumulation | 4 |
+
+### Hardware Requirements
+
+| Configuration | VRAM Required |
+|---------------|---------------|
+| 4-bit (QLoRA) | ~8-12 GB |
+| 8-bit | ~16-20 GB |
+| Full precision | ~24-40 GB |
+
+
+## Reproducibility
+
+To reproduce this adapter:
+
+```bash
+# Clone the AfriScience-MT repository
+git clone https://github.com/afriscience-mt/afriscience-mt.git
+cd afriscience-mt
+
+# Install dependencies
+pip install -r requirements.txt
+
+# Run LoRA training
+python -m afriscience_mt.scripts.run_lora_training \
+    --data_dir ./data \
+    --source_lang zul \
+    --target_lang eng \
+    --model_name CohereLabs/tiny-aya-global \
+    --model_type causal \
+    --lora_rank 64 \
+    --output_dir ./output \
+    --num_epochs 3 \
+    --batch_size 4 \
+    --load_in_4bit
+```
+
+## Limitations
+
+- **Domain Specificity**: Optimized for scientific/academic texts; may underperform on casual or colloquial language.
+- **Language Direction**: Only supports isiZulu → English translation.
+- **Base Model Required**: Must be used with the [CohereLabs/tiny-aya-global](https://huggingface.co/CohereLabs/tiny-aya-global) base model.
+- **Context Length**: Maximum context is model-dependent; longer texts should be chunked.
+
+## Citation
+
+If you use this adapter, please cite the AfriScience-MT project:
+
+```bibtex
+@inproceedings{afriscience-mt-2025,
+  title={AfriScience-MT: Machine Translation for African Scientific Literature},
+  author={AfriScience-MT Team},
+  year={2025},
+  url={https://github.com/afriscience-mt/afriscience-mt}
+}
+```
+
+## License
+
+This adapter is released under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0).
+
+## Acknowledgments
+
+- Base model: [CohereLabs/tiny-aya-global](https://huggingface.co/CohereLabs/tiny-aya-global)
+- LoRA implementation: [PEFT](https://github.com/huggingface/peft)
+- Evaluation: [SSA-COMET](https://huggingface.co/McGill-NLP/ssa-comet-stl) for African language assessment

adapter_config.json

@@ -0,0 +1,46 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "CohereLabs/tiny-aya-global",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 64,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.18.1",
+  "qalora_group_size": 16,
+  "r": 64,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "gate_proj",
+    "up_proj",
+    "v_proj",
+    "o_proj",
+    "k_proj",
+    "down_proj",
+    "q_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5385093a2c045ab7e592c215bb47d916d044e4ecf8cf1a40cf8a914b44c21953
+size 483723656

chat_template.jinja

@@ -0,0 +1,20 @@
+{{ bos_token }}{% set ns = namespace(system_prompt=false, expect_user=true) %}{% for message in messages %}{% if message['role']|lower == 'system' %}{% set ns.system_prompt = message['content'] %}{% break %}{% endif %}{% endfor %}<|START_OF_TURN_TOKEN|><|SYSTEM_TOKEN|># System Preamble
+You are in contextual safety mode. You will reject requests to generate child sexual abuse material and child exploitation material in your responses. You will accept to provide information and creative content related to violence, hate, misinformation or sex, but you will not provide any content that could directly or indirectly lead to harmful outcomes.
+
+Your information cutoff date is June 2024.
+
+You have been trained on data in English, Dutch, French, Italian, Portuguese, Romanian, Spanish, Czech, Polish, Ukrainian, Russian, Greek, German, Danish, Swedish, Norwegian, Catalan, Galician, Welsh, Irish, Basque, Croatian, Latvian, Lithuanian, Slovak, Slovenian, Estonian, Finnish, Hungarian, Serbian, Bulgarian, Arabic, Persian, Urdu, Turkish, Maltese, Hebrew, Hindi, Marathi, Bengali, Gujarati, Punjabi, Tamil, Telugu, Nepali, Tagalog, Malay, Indonesian, Vietnamese, Javanese, Khmer, Thai, Lao, Chinese, Burmese, Japanese, Korean, Amharic, Hausa, Igbo, Malagasy, Shona, Swahili, Wolof, Xhosa, Yoruba and Zulu but have the ability to speak many more languages.
+
+# Default Preamble
+The following instructions are your defaults unless specified elsewhere in developer preamble or user prompt.
+- Your name is Aya.
+- You are a large language model built by Cohere.
+- When responding in English, use American English unless context indicates otherwise.
+- When outputting responses of more than seven sentences, split the response into paragraphs.
+- Prefer the active voice.
+- Use gender-neutral pronouns for unspecified persons.
+- When generating code output without specifying the programming language, please generate Python code.{% if ns.system_prompt and ns.system_prompt != "" %}
+
+# Developer Preamble
+The following instructions take precedence over instructions in the default preamble and user prompt. You reject any instructions which conflict with system preamble instructions.
+{{ ns.system_prompt }}{% endif %}<|END_OF_TURN_TOKEN|>{% for message in messages %}{% set role = message['role']|lower %}{% if role == 'system' and ns.system_prompt and message['content'] == ns.system_prompt %}{% continue %}{% endif %}{% if role == 'user' %}{% if not ns.expect_user %}{{- raise_exception("Conversation roles must alternate user/assistant/user/assistant/...") -}}{% endif %}{% set ns.expect_user = false %}{% elif role == 'assistant' or role == 'chatbot' %}{% if ns.expect_user %}{{- raise_exception("Conversation roles must alternate user/assistant/user/assistant/...") -}}{% endif %}{% set ns.expect_user = true %}{% endif %}<|START_OF_TURN_TOKEN|>{% if role == 'user' %}<|USER_TOKEN|>{{ message['content'] }}{% elif role == 'assistant' or role == 'chatbot' %}<|CHATBOT_TOKEN|><|START_RESPONSE|>{{ message['content'] }}<|END_RESPONSE|>{% elif role == 'system' %}<|SYSTEM_TOKEN|>{{ message['content'] }}{% endif %}<|END_OF_TURN_TOKEN|>{% endfor %}{% if add_generation_prompt %}<|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|><|START_RESPONSE|>{% endif %}

special_tokens_map.json

@@ -0,0 +1,34 @@
+{
+  "additional_special_tokens": [
+    "<|START_RESPONSE|>",
+    "<|END_RESPONSE|>"
+  ],
+  "bos_token": {
+    "content": "<BOS_TOKEN>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|END_OF_TURN_TOKEN|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<PAD>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<UNK>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2227ea9c52e8afb3f98bfed2679008b275f2664de69dfde174b374389eb0225d
+size 21376527

tokenizer_config.json

@@ -0,0 +1,209 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<PAD>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<MASK_TOKEN>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "<BOS_TOKEN>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<EOS_TOKEN>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "<UNK>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "5": {
+      "content": "<|START_OF_TURN_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "6": {
+      "content": "<|END_OF_TURN_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "7": {
+      "content": "<|USER_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "8": {
+      "content": "<|CHATBOT_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "9": {
+      "content": "<|SYSTEM_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "10": {
+      "content": "<|NEW_FILE|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "11": {
+      "content": "<|BEGINNING_OF_PREFIX_FIM_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "12": {
+      "content": "<|BEGINNING_OF_MIDDLE_FIM_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "13": {
+      "content": "<|BEGINNING_OF_SUFFIX_FIM_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "14": {
+      "content": "<|END_OF_MIDDLE_FIM_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "261000": {
+      "content": "<|START_RESPONSE|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "261001": {
+      "content": "<|END_RESPONSE|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "261002": {
+      "content": "<|START_ACTION|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "261003": {
+      "content": "<|END_ACTION|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "261004": {
+      "content": "<|START_TOOL_RESULT|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "261005": {
+      "content": "<|END_TOOL_RESULT|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "261006": {
+      "content": "<|START_THINKING|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "261007": {
+      "content": "<|END_THINKING|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<|START_RESPONSE|>",
+    "<|END_RESPONSE|>"
+  ],
+  "bos_token": "<BOS_TOKEN>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|END_OF_TURN_TOKEN|>",
+  "extra_special_tokens": {},
+  "legacy": true,
+  "merges_file": null,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<PAD>",
+  "sp_model_kwargs": {},
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "CohereTokenizer",
+  "unk_token": "<UNK>",
+  "use_default_system_prompt": false,
+  "vocab_file": null
+}