---
title: ThingAI
type: org
tags:
- slm
- llm
- pytorch
- bash
- shell
- code
- tokenizer
- rust
---

# ThingsAI

Building efficient, specialist Small Language Models that run on consumer hardware. Zero telemetry. Open weights. Everything from tokenizer to training script is public.

## Models

* **Dwarf-15M** 
  A 15.54M parameter shell/bash specialist. 12 layers, d_model=320, GQA 5Q/1KV, SwiGLU, RMSNorm, RoPE. Custom 8202-token vocabulary via DwarfGoToken.  1390:1 token-to-parameter ratio across 11 datasets spanning raw shell, Python, C, instruction pairs, and English web text. Target use case: CLI tool that translates natural language into bash commands with user review before execution.

* **Quark-270M**
  Our largest model. 252M effective parameters, 32 layers, d_model=768, GQA 12Q/4KV, 65K bilingual vocabulary (Italian + English). Trained on curated multilingual data. Available as Base and Instruct variants.

* **Quark-135M**
  Bilingual (Italian + English) general-purpose model. 135M parameters, 30 layers, 9 attention heads (3 KV, GQA), SwiGLU, RMSNorm, RoPE θ=10k. Trained on 15B+ tokens. Published benchmarks: HellaSwag 31.37%, ARC-Easy 41.46%, PIQA 61.26%.

* **Quark-72M** *(archived — research artifact)*
  A 71.7M parameter model that taught us an expensive lesson. With vocab_size=65536 and d_model=512, the embedding matrix consumed ~33.5M of the 71.7M total parameters — nearly half the budget in pure lookup table. Effective transformer capacity was ~35M parameters, explaining why it underperformed the nominally smaller Quark-135M on every benchmark (PIQA 54.57% vs 58.32%, ARC-Easy 32.10% vs 47.73%). Additionally, zero-shot chain-of-thought prompting actively degraded performance, dropping ARC-Easy from 33% to 25.5% (random guess level). This model remains published with its limitations honestly documented. Every architectural decision in Dwarf-15M — the compact 8K vocabulary, the syntax-aware tokenizer, the instruction data mixed into pretraining — was a direct response to what went wrong here.

* **Quark-Mod**
  Multi-label content moderation model. 9 categories: toxic, severe_toxic, obscene, threat, insult, identity_hate, cyberbullying, hate_speech, offensive.

## Tokenizers

* **DwarfGoToken**
  An 8202-token BPE tokenizer built for shell/bash. Uses ByteLevel pre-tokenization with syntax-aware protected tokens for shell operators (2>&1, &&, >>, ||) that would otherwise be split by standard BPE. Built on a 51MB corpus of shell, Python, C, and English text. Two critical bugs were found and fixed during Dwarf-15M training: space loss from incorrect pre-tokenizer configuration, and short shell keywords (fi, do, if) matching as substrings inside English words.

* **GoToken**
  A BPE tokenizer written in Rust with Python bindings via PyO3. Published on crates.io and PyPI. Provides syntax-aware pre-tokenization for shell/bash patterns. Used as the foundation for DwarfGoToken.

## What We Focus On

* **Specialist over generalist:** A 15M model can't do everything, but it can excel at one thing. Dwarf targets shell/bash; future models will target math/physics.
* **Honest failure documentation:** When something doesn't work (72M vocabulary problem, zero-shot CoT degradation, tokenizer bugs), we publish the failure and what we learned.
* **Extreme overtraining for small models:** Following the Phi/SmolLM philosophy — small models need more tokens per parameter, not fewer. Dwarf trains at 2500:1, 125x beyond Chinchilla optimal.
* **Custom tooling from scratch:** Tokenizers (gotoken, DwarfGoToken), training scripts with multi-source streaming, and inference tools — all built in-house, all open.
* **Consumer hardware:** Everything runs on an RTX 3070 (8GB) or equivalent. No datacenter required.

## Links

* Models and tokenizers: [HuggingFace](https://huggingface.co/ThingAI)
* Script & Tool: [GitHub](https://github.com/overcastlab)
* Website: [things-ai.org](https://things-ai.org)
* GoToken: [crates.io](https://crates.io/crates/gotoken) · [PyPI](https://pypi.org/project/gotoken/)