Instructions to use mesolitica/mallam-1.1B-4096 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use mesolitica/mallam-1.1B-4096 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="mesolitica/mallam-1.1B-4096")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("mesolitica/mallam-1.1B-4096")
model = AutoModelForCausalLM.from_pretrained("mesolitica/mallam-1.1B-4096", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use mesolitica/mallam-1.1B-4096 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "mesolitica/mallam-1.1B-4096"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mesolitica/mallam-1.1B-4096",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/mesolitica/mallam-1.1B-4096

SGLang

How to use mesolitica/mallam-1.1B-4096 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "mesolitica/mallam-1.1B-4096" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mesolitica/mallam-1.1B-4096",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "mesolitica/mallam-1.1B-4096" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mesolitica/mallam-1.1B-4096",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use mesolitica/mallam-1.1B-4096 with Docker Model Runner:
```
docker model run hf.co/mesolitica/mallam-1.1B-4096
```

MaLLaM 🌙 1.1B (Malaysia Large Language Model), Pretrain 1.1B 4096 context length on Malaysian text

Pretrain from scratch 1.1B parameters using Mistral architecture on 90B Malaysian text tokens.

README at https://github.com/mesolitica/malaya/tree/5.1/pretrained-model/mistral

Trained on 90B tokens, gathered at https://github.com/malaysia-ai/dedup-text-dataset/tree/main/pretrain-llm
We use Ray cluster to train on 5 nodes of 4x A100 80GB, https://github.com/malaysia-ai/jupyter-gpu/tree/main/ray

WandB, https://wandb.ai/mesolitica/pretrain-mistral-1.1b?workspace=user-husein-mesolitica

WandB report, https://wandb.ai/mesolitica/pretrain-mistral-3b/reports/Pretrain-Larger-Malaysian-Mistral--Vmlldzo2MDkyOTgz

Technical report, https://github.com/mesolitica/malaya/wiki/MaLLaM-%F0%9F%8C%99-Malaysia-Large-Language-Model

how-to

from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
import torch

TORCH_DTYPE = 'bfloat16'
nf4_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type='nf4',
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=getattr(torch, TORCH_DTYPE)
)

tokenizer = AutoTokenizer.from_pretrained('mesolitica/mallam-1.1B-4096')
model = AutoModelForCausalLM.from_pretrained(
    'mesolitica/mallam-1.1B-4096',
    use_flash_attention_2 = True,
    quantization_config = nf4_config
)
prompt = '<s>nama saya'
inputs = tokenizer([prompt], return_tensors='pt', add_special_tokens=False).to('cuda')

generate_kwargs = dict(
    inputs,
    max_new_tokens=512,
    top_p=0.95,
    top_k=50,
    temperature=0.9,
    do_sample=True,
    num_beams=1,
    repetition_penalty=1.05,
)
r = model.generate(**generate_kwargs)

Downloads last month: 114

Safetensors

Model size

1B params

Tensor type

BF16

Model tree for mesolitica/mallam-1.1B-4096

Adapters

1 model

Spaces using mesolitica/mallam-1.1B-4096 2

Collection including mesolitica/mallam-1.1B-4096

MaLLaM 🌙

Collection

Pretrain from scratch 4096 context length on 90B tokens Malaysian text, https://huggingface.co/papers/2401.14680 • 8 items • Updated Mar 2 • 16