Instructions to use theophilusowiti/Caracal_AfroLlama_int4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use theophilusowiti/Caracal_AfroLlama_int4 with PEFT:

from peft import PeftModel
from transformers import AutoModelForCausalLM

base_model = AutoModelForCausalLM.from_pretrained("Jacaranda/AfroLlama_V1")
model = PeftModel.from_pretrained(base_model, "theophilusowiti/Caracal_AfroLlama_int4")

Transformers

How to use theophilusowiti/Caracal_AfroLlama_int4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="theophilusowiti/Caracal_AfroLlama_int4")

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("theophilusowiti/Caracal_AfroLlama_int4", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use theophilusowiti/Caracal_AfroLlama_int4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "theophilusowiti/Caracal_AfroLlama_int4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "theophilusowiti/Caracal_AfroLlama_int4",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/theophilusowiti/Caracal_AfroLlama_int4

SGLang

How to use theophilusowiti/Caracal_AfroLlama_int4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "theophilusowiti/Caracal_AfroLlama_int4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "theophilusowiti/Caracal_AfroLlama_int4",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "theophilusowiti/Caracal_AfroLlama_int4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "theophilusowiti/Caracal_AfroLlama_int4",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use theophilusowiti/Caracal_AfroLlama_int4 with Docker Model Runner:
```
docker model run hf.co/theophilusowiti/Caracal_AfroLlama_int4
```

model description

by Papizo - opened Jun 28

base: refs/heads/main

←

from: refs/pr/3

Discussion Files changed

+61

-7

Papizo

Jun 28

proposed Changes
Heading

Update Caracal_AfroLlama_int4 Model Card with Regional Language Performance Metrics

Overview

This Pull Request modifies the documentation repository for the Caracal_AfroLlama_int4 model asset. The primary motivation is to resolve placeholder omissions (More information needed) and standardize performance reporting in alignment with our broader Caracal ecosystem updates (such as Caracal_instruct)

Changes made and from where

Metadata Expansion : Appended essential discoverability tags including africa, low-resource, int4, and quantization to the YAML header block to align index capabilities across the Hugging Face Hub landscape.
Document Baseline Refactoring : Replaced boilerplate placeholder sections with informative contextual overviews detailing explicit Intended Uses, Architectural Limitations, and Dataset Profiles.
Linguistic Evaluation Integration: Inserted a structurally isolated Top Performing Languages segment profiling optimal cross-lingual validation nodes matching core optimization matrices.
These adjustments are synthesized directly from our standardized AfriLLMQuant project telemetry profiles and localized evaluation evaluation pipelines, bridging foundational benchmark data over to the fine-tuned PEFT adapter branch.
Added usage and output section.

model descriptionfd2cfdd2

theophilusowiti

Owner Jun 28

Changes Accepted with subtle review.

theophilusowiti changed pull request status to merged Jun 28

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment