README.md

---
library_name: transformers
license: apache-2.0
base_model: h4bbo/FuseLLM-112M-Completion
tags:
  - habbo
  - code
  - chatml
  - sft
language:
  - en
pipeline_tag: text-generation
---

# FuseLLM-112M (Chat)

This is the **ChatML chat** variant of FuseLLM-112M — a 112M-parameter, from-scratch
Qwen3-architecture decoder-only model supervised-fine-tuned (SFT) on on-domain Habbo instruction
pairs derived **from the Habbo source corpus itself** (no external/teacher data).

The base model, [h4bbo/FuseLLM-112M-Completion](https://huggingface.co/h4bbo/FuseLLM-112M-Completion),
was trained only on raw Habbo code in completion mode. Its tokenizer already shipped a Qwen3
ChatML template, but the weights had never seen a chat turn, so chat-formatted prompts produced
poor, repetitive output. This checkpoint teaches the weights to follow ChatML turns and to **emit
`<|im_end|>`** at the end of an assistant answer (the turn terminator — conveniently the same token,
151645, the base model was trained to use as a document boundary), which is what stops the runaway
repetition.

## Use

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "h4bbo/FuseLLM-112M"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, torch_dtype=torch.bfloat16, device_map="auto", attn_implementation="sdpa")

messages = [
    {"role": "system", "content": "You are a Habbo Hotel emulator code assistant. Reply with concise, correct code or a brief explanation."},
    {"role": "user", "content": "Implement this Java method:\n```java\npublic static void sendRoomPacket(Session s, int header) { }\n```"},
]
text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
ids = tok(text, return_tensors="pt").to(model.device)
out = model.generate(**ids, max_new_tokens=256, eos_token_id=151645, pad_token_id=151643,
                     do_sample=True, temperature=0.7, top_p=0.9, repetition_penalty=1.05)
print(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True))
```

It is designed for Habbo-coding instructions (method/class completion, code continuation,
doc→code). It is **not** a general assistant — outside its narrow domain it will produce poor or
repetitive output.

## Architecture

`Qwen3ForCausalLM`, hidden 512, 8 layers (full attention), 8 heads / 8 KV heads, intermediate
1408, vocab 151,936, max context 2048, `tie_word_embeddings=true`. Trained in bf16.

## Training data

10,000 ChatML conversations derived deterministically from the Habbo corpus (the same corpus the
base was trained on — 84,925 unique files, ~210M est. tokens; top languages Java 26,335 /
C# 19,048 / PHP 10,969 / ActionScript 3,801). No external data, no teacher model. Templates:

- **Method completion** — given a method signature with empty body, return the real body.
- **Code continuation** — given a file prefix, return the real suffix.
- **Doc → method** — given a Javadoc/PHPDoc/`///` summary, return the real method.

Distribution in this build: Java ~4,936 · C# ~3,752 · PHP ~989 · ActionScript ~323. Decompiler
noise (JD-Core `/* N:M */` line markers and `/* Location: … */` footers) is stripped. Secrets are
scrubbed to `[REDACTED]` (currently `xX!elgps`) before extraction; the training file is verified to
contain 0 secret occurrences.

## Training config

- Full fine-tune (no LoRA — 112M is small enough to train every weight on 24 GB).
- TRL `SFTTrainer` + `SFTConfig`, `messages` format auto-detected, ChatML applied by the base
  tokenizer's chat template. `packing=False`, `max_length=2048` (model's native max position; only
  11/10,000 conversations exceed it). Full-sequence causal-LM loss (the Qwen3 chat template has no
  `{% generation %}` markers, so `assistant_only_loss` is unset — at inference we prompt through
  `<|im_start|>assistant\n` and stop at `<|im_end|>`).
- bf16, sdpa attention, `adamw_torch`, cosine schedule, 3 epochs, LR 2e-5, warmup 0.03,
  effective batch 16 (BS 4 × GA 4), `max_grad_norm=1.0`, seed 42. 1,875 steps.
- `generation_config.json` is written with `eos_token_id=151645` (`<|im_end|>`),
  `pad_token_id=151643`, `temperature=0.7`, `top_p=0.9`, `repetition_penalty=1.05`.

## Training result

Loss dropped from ~1.83 (step 10) to ~0.45 by the end of epoch 1 and held in the ~0.4–0.5 range
through epochs 2–3. Final: `train_loss 0.521`, `mean_token_accuracy 0.9144`, 1,875 steps, ~16.5 min
on a single RX 7900 XTX (ROCm). Verified behavior: method-completion and code-continuation prompts
produce coherent on-domain Habbo code, close the fenced block, and **stop at `<|im_end|>`**;
doc→method and free-form explanation prompts tend to ramble (see Limitations).

## Redaction

Secrets (currently `xX!elgps`) are scrubbed to `[REDACTED]` in all training content before
tokenisation. The output training file is checked to contain 0 occurrences. No known credentials
enter the weights.

## License

Released under **Apache-2.0**. See the base model
[h4bbo/FuseLLM-112M-Completion](https://huggingface.co/h4bbo/FuseLLM-112M-Completion) for its
license terms.

## GGUF

A non-quantized **bf16** GGUF — `FuseLLM-112M.bf16.gguf` (~220 MB, a bit-exact copy of the bf16
safetensors weights, so truly lossless) — is included in this repo for use with
[llama.cpp](https://github.com/ggml-org/llama.cpp) / [Ollama](https://ollama.com). The ChatML chat
template and the EOS token (`<|im_end|>`, 151645) are embedded as GGUF metadata, so the model loads
in chat mode automatically. No quantized (Q4/Q5/Q8) variant is shipped here.

Example with llama.cpp:

```bash
llama-cli -m FuseLLM-112M.bf16.gguf -cnv \
  --temp 0.7 --top-p 0.9 --repeat-penalty 1.05 -n 256 \
  -p "Implement this Java method:\n```java\npublic static void sendRoomPacket(Session s, int h) { }\n```"
```

## Intended use

Domain-specialist code assistant for the Habbo Hotel emulator ecosystem (server/client tooling).
Not affiliated with or endorsed by Sulake/Habbo.

## Limitations

- 112M parameters — narrow capacity; expect errors and repetition on long or off-domain prompts.
- Trained only on on-domain code-instruction pairs; not a general chat / instruction model.
- Doc→method and free-form explanation prompts often ramble past `<|im_end|>` despite
  `repetition_penalty`; keep `max_new_tokens` modest and prefer the SFT templates
  (method completion / code continuation).