Initial upload: FuseLLM-112M base causal LM + tokenizer/chat_template + fp16/Q4_K_M GGUF

.gitattributes

@@ -33,3 +33,6 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+gguf/FuseLLM-112M.Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+gguf/FuseLLM-112M.fp16.gguf filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,96 @@
+---
+library_name: transformers
+license: other  # TODO: set the license you want to release this model under
+language:
+  - code
+tags:
+  - qwen3
+  - causal-lm
+  - code-completion
+  - habbo
+  - from-scratch
+---
+
+# FuseLLM-112M
+
+A small **112M-parameter decoder-only language model trained from scratch** (no base
+checkpoint, no LoRA) on a corpus of Habbo emulator / game-server source code. The
+goal is a tiny, fast model for **code completion** in that Java codebase, not a
+general-purpose or instruction-following model.
+
+## Model details
+
+| | |
+|---|---|
+| Architecture | Qwen3 (decoder-only causal LM) |
+| Parameters | ~112M (tied input/output embeddings) |
+| Hidden size | 512 |
+| Layers | 8 (all full attention) |
+| Attention heads | 8 (8 KV heads) |
+| Vocab size | 151,936 |
+| Max context | 2048 |
+| Precision | float32 (safetensors) |
+| Training | From scratch, 4 epochs, 16,188 steps |
+| Final train loss | ~0.58 |
+
+`tie_word_embeddings: true` — the output `lm_head` shares the input embedding
+matrix, so checkpoints store only one copy. This is expected, not a missing weight.
+
+## Intended use
+
+- **Code completion** for Habbo-style Java server code (raw prompt → continuation).
+- Local experimentation / distillation base.
+
+## What it is NOT
+
+- **Not instruction-tuned / not a chat model.** It was trained only on raw source
+  code, never on chat/instruction data.
+- The Qwen3 ChatML chat template is included (it ships with the tokenizer) for
+  tokenizer/tool compatibility, but the model has **not** learned to follow chat
+  turns. Passing chat-formatted prompts will produce poor, often repetitive output.
+  Use it in **completion mode**, not conversation mode.
+
+## Usage
+
+### transformers (recommended for completion)
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+m   = AutoModelForCausalLM.from_pretrained("h4bbo/FuseLLM-112M")
+tok = AutoTokenizer.from_pretrained("h4bbo/FuseLLM-112M")
+
+prompt = "public class Room {\n  public void onEnter(Player p) {\n    "
+ids = tok(prompt, return_tensors="pt").input_ids
+out = m.generate(ids, max_new_tokens=64, do_sample=False,
+                 repetition_penalty=1.1, pad_token_id=tok.eos_token_id)
+print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))
+```
+
+### llama.cpp (completion mode)
+
+The repo includes GGUF files (`FuseLLM-112M.fp16.gguf`, `FuseLLM-112M.Q4_K_M.gguf`)
+verified to load and generate in `llama.cpp`.
+
+```bash
+# Completion mode — pass the raw code seed, do NOT use chat/conversation mode.
+llama-cli -m FuseLLM-112M.Q4_K_M.gguf -cnv -st --no-jinja \
+  -f seed.txt -n 64 --temp 0.0 --repeat-penalty 1.1 --no-display-prompt < /dev/null
+```
+
+`--no-jinja` keeps the prompt raw (the embedded chat template exists but the model
+isn't chat-tuned, so conversation mode is not meaningful for this model).
+
+## Files
+
+- `model.safetensors`, `config.json`, `generation_config.json` — HF model
+- `tokenizer.json`, `tokenizer_config.json`, `chat_template.jinja` — tokenizer + ChatML template
+- `FuseLLM-112M.fp16.gguf` — lossless fp16 GGUF (~220 MB)
+- `FuseLLM-112M.Q4_K_M.gguf` — 4-bit quantized GGUF (~88 MB), the practical llama.cpp file
+
+## Notes
+
+- Small model + limited-domain corpus: expect repetition on long generations; use
+  a repetition penalty and keep continuations short.
+- Trained from scratch, so this is fully independent of any upstream Qwen weights.
+  The Qwen3 architecture/tokenizer are reused for compatibility.

chat_template.jinja

@@ -0,0 +1,61 @@
+{%- if tools %}
+    {{- '<|im_start|>system\n' }}
+    {%- if messages[0].role == 'system' %}
+        {{- messages[0].content + '\n\n' }}
+    {%- endif %}
+    {{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
+{%- else %}
+    {%- if messages[0].role == 'system' %}
+        {{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- for message in messages %}
+    {%- if message.content is string %}
+        {%- set content = message.content %}
+    {%- else %}
+        {%- set content = '' %}
+    {%- endif %}
+    {%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
+        {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {{- '<|im_start|>' + message.role + '\n' + content }}
+        {%- if message.tool_calls %}
+            {%- for tool_call in message.tool_calls %}
+                {%- if (loop.first and content) or (not loop.first) %}
+                    {{- '\n' }}
+                {%- endif %}
+                {%- if tool_call.function %}
+                    {%- set tool_call = tool_call.function %}
+                {%- endif %}
+                {{- '<tool_call>\n{"name": "' }}
+                {{- tool_call.name }}
+                {{- '", "arguments": ' }}
+                {%- if tool_call.arguments is string %}
+                    {{- tool_call.arguments }}
+                {%- else %}
+                    {{- tool_call.arguments | tojson }}
+                {%- endif %}
+                {{- '}\n</tool_call>' }}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- content }}
+        {{- '\n</tool_response>' }}
+        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+{%- endif %}

config.json

@@ -0,0 +1,44 @@
+{
+  "architectures": [
+    "Qwen3ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": null,
+  "dtype": "float32",
+  "eos_token_id": 151645,
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 512,
+  "initializer_range": 0.02,
+  "intermediate_size": 1408,
+  "layer_types": [
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention"
+  ],
+  "max_position_embeddings": 2048,
+  "max_window_layers": 28,
+  "model_type": "qwen3",
+  "num_attention_heads": 8,
+  "num_hidden_layers": 8,
+  "num_key_value_heads": 8,
+  "pad_token_id": 151643,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "rope_theta": 10000.0,
+    "rope_type": "default"
+  },
+  "sliding_window": null,
+  "tie_word_embeddings": true,
+  "transformers_version": "5.13.0",
+  "use_cache": false,
+  "use_sliding_window": false,
+  "vocab_size": 151936,
+  "_name_or_path": "FuseLLM-112M"
+}

generation_config.json

@@ -0,0 +1,11 @@
+{
+  "_from_model_config": true,
+  "eos_token_id": [
+    151645
+  ],
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "pad_token_id": 151643,
+  "transformers_version": "5.13.0",
+  "use_cache": false
+}

gguf/FuseLLM-112M.Q4_K_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a97a2cbfb373b092e12dc7de9d0c9a9f555962fe9f4a22fa124bffae930420db
+size 91304832

gguf/FuseLLM-112M.fp16.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:94a71078a8e5379be43466f2889dbf6214ce2e2a110469deb4c78d4488ce71bf
+size 229719424

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d3ebdb08b1e9f44ccf2bf768c7425bbb3971ecfd4f08fc59b5083b718a0d63fb
+size 447532792

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:be75606093db2094d7cd20f3c2f385c212750648bd6ea4fb2bf507a6a4c55506
+size 11422650

tokenizer_config.json

@@ -0,0 +1,30 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "extra_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "is_local": false,
+  "local_files_only": false,
+  "model_max_length": 1010000,
+  "pad_token": "<|endoftext|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}