Instructions to use seacorn/llama3.1-8b-reasoning-summarizer with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use seacorn/llama3.1-8b-reasoning-summarizer with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="seacorn/llama3.1-8b-reasoning-summarizer")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForMultimodalLM

tokenizer = AutoTokenizer.from_pretrained("seacorn/llama3.1-8b-reasoning-summarizer")
model = AutoModelForMultimodalLM.from_pretrained("seacorn/llama3.1-8b-reasoning-summarizer")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use seacorn/llama3.1-8b-reasoning-summarizer with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "seacorn/llama3.1-8b-reasoning-summarizer"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "seacorn/llama3.1-8b-reasoning-summarizer",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/seacorn/llama3.1-8b-reasoning-summarizer

SGLang

How to use seacorn/llama3.1-8b-reasoning-summarizer with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "seacorn/llama3.1-8b-reasoning-summarizer" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "seacorn/llama3.1-8b-reasoning-summarizer",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "seacorn/llama3.1-8b-reasoning-summarizer" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "seacorn/llama3.1-8b-reasoning-summarizer",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use seacorn/llama3.1-8b-reasoning-summarizer with Docker Model Runner:
```
docker model run hf.co/seacorn/llama3.1-8b-reasoning-summarizer
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

See axolotl config

axolotl version: 0.8.0.dev0

base_model: meta-llama/Llama-3.1-8B-Instruct
# optionally might have model_type or tokenizer_type
model_type: LlamaForCausalLM
tokenizer_type: AutoTokenizer
# Automatically upload checkpoint and final model to HF
hub_model_id: seacorn/llama3.1-8b-reasoning-summarizer

load_in_8bit: true
load_in_4bit: false
strict: false

seed: 42

datasets:
  - path: output.jsonl
    type: chat_template
dataset_prepared_path:
val_set_size: 0.05
output_dir: ./lora-out

sequence_len: 8192
sample_packing: true
eval_sample_packing: false
pad_to_sequence_len: true

adapter: lora
lora_model_dir:
lora_r: 16
lora_alpha: 32
lora_dropout: 0.05
lora_target_linear: true
lora_fan_in_fan_out:
lora_modules_to_save:
  - embed_tokens
  - lm_head

wandb_project: huggingface
wandb_entity:
wandb_watch:
wandb_name: llama3.1-8b-reasoning-summarizer
wandb_log_model:

gradient_accumulation_steps: 4
micro_batch_size: 2
num_epochs: 2
optimizer: adamw_bnb_8bit
lr_scheduler: cosine
learning_rate: 0.0002

train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: false

gradient_checkpointing: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true
s2_attention:

warmup_ratio: 0.05
evals_per_epoch: 4
eval_table_size:
eval_max_new_tokens: 128
saves_per_epoch: 5
debug:
deepspeed:
weight_decay: 0.0
fsdp:
fsdp_config:
special_tokens:
   pad_token: <|end_of_text|>

llama3.1-8b-reasoning-summarizer

This model is a fine-tuned version of meta-llama/Llama-3.1-8B-Instruct on the seacorn/news-summarizer-reasoner dataset. It achieves the following results on the evaluation set:

Loss: 1.1173

Intended uses & limitations

The model performs best in summarization tasks, specifically in English and maybe Chinese. The model provides reasoning ON/OFF via system prompt trigger, all instructions should be contained within the user prompt.

Reasoning off example:

messages = [
    {"role": "system", "content": "reasoning off"},
    {"role": "user", "content": "Summarize the following into 5 bullet points, each with 20 words max.\n\nMarch 28 (Reuters) -..."}
]

# output
- Elon Musk's xAI acquires X ...

Reasoning on example:

messages = [
    {"role": "system", "content": "reasoning on"},
    {"role": "user", "content": "Summarize the following into 5 bullet points, each with 20 words max.\n\nMarch 28 (Reuters) -..."}
]

# output
<think>
Okay, I need to summarize this article into 5 bullet points, each with a maximum of 20 words. ...
</think>

- Musk's xAI acquires X ...

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 0.0002
train_batch_size: 2
eval_batch_size: 2
seed: 42
gradient_accumulation_steps: 4
total_train_batch_size: 8
optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: cosine
lr_scheduler_warmup_steps: 56
num_epochs: 2.0

Training results

Training Loss	Epoch	Step	Validation Loss
2.0396	0.0018	1	1.7982
1.3908	0.2506	141	1.2241
1.8534	0.5011	282	1.1842
1.5745	0.7517	423	1.1560
0.9261	1.0018	564	1.1288
1.2359	1.2523	705	1.1344
1.1835	1.5029	846	1.1223
0.9898	1.7534	987	1.1173

Framework versions

PEFT 0.15.0
Transformers 4.50.0
Pytorch 2.5.1+cu124
Datasets 3.4.1
Tokenizers 0.21.1

Downloads last month: 3

Safetensors

Model size

8B params

Tensor type

BF16

Model tree for seacorn/llama3.1-8b-reasoning-summarizer

Base model

meta-llama/Llama-3.1-8B

Finetuned

meta-llama/Llama-3.1-8B-Instruct

Finetuned

(2789)

this model

Quantizations

1 model

seacorn
/

llama3.1-8b-reasoning-summarizer