docs: regenerate card from current README — engine, browser demo, fast-path dispatcher
Browse files
README.md
CHANGED
|
@@ -69,6 +69,9 @@ model-index:
|
|
| 69 |
value: 23.5
|
| 70 |
name: BFCL v4 single-turn
|
| 71 |
---
|
|
|
|
|
|
|
|
|
|
| 72 |
# 🧵 Thimble
|
| 73 |
|
| 74 |
### A tool-calling layer, not a language model.
|
|
@@ -76,11 +79,17 @@ model-index:
|
|
| 76 |
**Your schemas in, validated calls out, at 48M parameters.**
|
| 77 |
|
| 78 |
[](https://github.com/nikshepsvn/thimble/blob/master/LICENSE)
|
| 79 |
-
[](#numbers)
|
| 81 |
[](#the-contract)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 82 |
|
| 83 |
-
|
| 84 |
|
| 85 |
It does not converse, reason, or write prose — it was never trained to. It reads
|
| 86 |
a catalog of typed functions and a request, and returns the calls to make or an
|
|
@@ -98,42 +107,90 @@ Three guarantees hold on **any** catalog, with no training and no configuration,
|
|
| 98 |
because they come from a grammar compiled out of your schemas rather than from
|
| 99 |
the weights:
|
| 100 |
|
| 101 |
-
- **Output is always well-formed JSON.**
|
| 102 |
-
|
|
|
|
|
|
|
| 103 |
- **Calls to tools you did not declare cannot be emitted.**
|
| 104 |
|
| 105 |
The model is consulted at exactly five choice points: refuse-or-call, which tool,
|
| 106 |
include this optional, what value, stop or continue. Everything else — braces,
|
| 107 |
-
quotes, commas, every argument key — is determined before it runs.
|
| 108 |
-
|
| 109 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 110 |
|
| 111 |
## Does this fit your problem?
|
| 112 |
|
| 113 |
-
**
|
| 114 |
-
`annotate variant rs4988235 against build GRCh38`. Identifiers, codes,
|
| 115 |
-
numbers, enum picks — copied, not inferred. Chains are fine: two-plus-call
|
| 116 |
-
score **73.5%** on a catalog it knows.
|
| 117 |
|
| 118 |
The gate is how *extractive* the request is, not what domain it belongs to. In a
|
| 119 |
pair of small probes, an unseen biomedical catalog in `dot.notation` scored 0.75
|
| 120 |
while a familiar-looking app catalog with conversational phrasing scored 0.57.
|
| 121 |
-
|
|
|
|
| 122 |
|
| 123 |
**Adapt it when** you need conversational phrasing, disciplined handling of
|
| 124 |
-
optional arguments, or calibrated refusal
|
|
|
|
| 125 |
|
| 126 |
**Use something else when** you have an open-world catalog, need Java or
|
| 127 |
-
JavaScript schema dialects, or need parallel instantiations of one schema.
|
| 128 |
-
can afford 600M parameters, fine-tune Qwen instead — it will probably
|
| 129 |
-
higher. This is for when you cannot
|
|
|
|
| 130 |
|
| 131 |
## Numbers
|
| 132 |
|
| 133 |
-
Ordered strict exact match —
|
| 134 |
-
|
| 135 |
-
Compute, 45M params, 153B training tokens), their
|
| 136 |
-
metric, so the left column
|
|
|
|
| 137 |
|
| 138 |
**Catalog represented in training** (eval rows firewalled out):
|
| 139 |
|
|
@@ -152,54 +209,117 @@ metric, so the left column has a scale.
|
|
| 152 |
| Seal-Tools out-of-domain (654) | 28.1 | 28.7 |
|
| 153 |
| BFCL v4 single-turn (3,641) | 23.5 | 42.6 |
|
| 154 |
|
| 155 |
-
The spread is visible *inside a single suite* — the cleanest
|
| 156 |
-
project, because only one variable moves:
|
| 157 |
|
| 158 |
-
|
|
|
|
|
|
|
|
|
|
| 159 |
|
| 160 |
Name-sequence accuracy tracks row accuracy exactly. The model is not failing to
|
| 161 |
-
extract arguments on unfamiliar catalogs — it is failing to pick the right
|
|
|
|
| 162 |
|
| 163 |
**Disclosures.** Mobile Actions' public train split (8,693 rows, disjoint from
|
| 164 |
-
eval) is in the training mix — that is what the first table's heading means.
|
| 165 |
-
|
| 166 |
-
|
| 167 |
-
|
| 168 |
-
|
|
|
|
|
|
|
| 169 |
|
| 170 |
## Adapting it to your catalog
|
| 171 |
|
| 172 |
-
```
|
| 173 |
-
|
| 174 |
-
|
|
|
|
|
|
|
| 175 |
|
| 176 |
-
|
| 177 |
-
hf download flashvenom/thimble thimble-v6.pt --local-dir checkpoints/
|
| 178 |
-
hf download flashvenom/thimble tokenizer.json --local-dir data/
|
| 179 |
|
| 180 |
-
|
| 181 |
-
|
| 182 |
-
|
|
|
|
|
|
|
| 183 |
|
| 184 |
-
|
| 185 |
-
python scripts/adapt.py --catalog my_tools.json --name mydomain
|
| 186 |
|
| 187 |
-
|
| 188 |
-
|
| 189 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 190 |
```
|
| 191 |
|
| 192 |
-
|
| 193 |
-
|
| 194 |
-
scratch and 33.1 annealed into the LR-decay phase. **Keep the guard data** —
|
| 195 |
-
annealing purely on your catalog trades away the competence you are building on.
|
| 196 |
|
| 197 |
-
|
| 198 |
-
|
|
|
|
|
|
|
| 199 |
|
| 200 |
-
|
| 201 |
-
|
| 202 |
-
|
|
|
|
|
|
|
|
|
|
| 203 |
|
| 204 |
## How it works
|
| 205 |
|
|
@@ -222,10 +342,13 @@ so the tokenizer's own share of the gain was never isolated.
|
|
| 222 |
**The loss is weighted by those same five decisions** — structure 1x, keys 1.5x,
|
| 223 |
names 2x, values 4x, stop-decision 6x — matched to the measured error
|
| 224 |
distribution. The model is optimized for the choices it will be asked to make,
|
| 225 |
-
not for tokens it will never emit.
|
|
|
|
|
|
|
|
|
|
| 226 |
|
| 227 |
**The decoder consults the model only at those points.** Everything else is
|
| 228 |
-
determined before it runs, which is where the contract
|
| 229 |
from. One call, start to finish — `MODEL` marks the only places the network is
|
| 230 |
asked anything:
|
| 231 |
|
|
@@ -284,7 +407,9 @@ what the decoder will supply — cost **12 points** in a controlled twin run. Th
|
|
| 284 |
tokens carry the call-sequencing signal: the model learns *when a call ends*
|
| 285 |
through structure it never has to emit. Remove them and it breaks.
|
| 286 |
|
| 287 |
-
|
|
|
|
|
|
|
| 288 |
|
| 289 |
- **Retriever** — `retrieve(query, tools, emitted=...)`, a DTDR-style
|
| 290 |
(arXiv 2512.17052) refresh conditioned on the *partial plan*, so the candidate
|
|
@@ -302,7 +427,11 @@ through structure it never has to emit. Remove them and it breaks.
|
|
| 302 |
(arXiv 2607.18363) finds architecture choices at this scale worth hundredths of
|
| 303 |
a nat at matched parameters. The co-design above is the part that matters.
|
| 304 |
|
| 305 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 306 |
|
| 307 |
Row accuracy factors as `P(name sequence) x p^n`, where `p` is per-call argument
|
| 308 |
accuracy. Each version measured which factor was binding and attacked only that:
|
|
@@ -319,108 +448,69 @@ canonical date forms, ~29 were unwinnable noise in the gold. Mid-run causal
|
|
| 319 |
check: **+3.3 points at constant LR** from the corrective corpus alone. That loop
|
| 320 |
is what `adapt.py` automates for your catalog.
|
| 321 |
|
| 322 |
-
|
| 323 |
|
| 324 |
-
|
| 325 |
|
| 326 |
-
|
| 327 |
-
|
| 328 |
-
|
| 329 |
-
|
| 330 |
-
|
| 331 |
-
|
| 332 |
-
|
| 333 |
-
| Beam / RL / best-of-N | oracle-capped below target |
|
| 334 |
-
| RLOO fine-tune on the annealed checkpoint | diverges at every LR — sharp minima and policy gradients don't mix |
|
| 335 |
-
| Matching Seal's gold numeric typing | not learnable — 74% of params are mixed-convention noise |
|
| 336 |
|
| 337 |
-
|
|
|
|
| 338 |
|
| 339 |
-
##
|
| 340 |
|
| 341 |
- **Unfamiliar catalogs.** Out-of-domain name-sequence accuracy is 79% against
|
| 342 |
-
88% in-domain. Every out-of-domain deficit traces
|
| 343 |
-
|
| 344 |
-
|
| 345 |
-
|
| 346 |
-
|
| 347 |
-
|
| 348 |
-
|
| 349 |
-
Seal-Tools in-domain. The call count is not the
|
| 350 |
-
|
| 351 |
-
|
| 352 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 353 |
- **768-token context.** 151 of 3,641 BFCL rows (4.1%) do not fit and score as misses.
|
| 354 |
- **Microcontrollers.** ~11.5MB at 2-bit is a property of the parameter count,
|
| 355 |
not a shippable artifact: 2-bit would need quantization-aware retraining this
|
| 356 |
model never had. The smallest thing that actually runs is the 48MB int8
|
| 357 |
-
|
| 358 |
-
|
| 359 |
-
Scale
|
| 360 |
-
|
| 361 |
-
|
| 362 |
-
##
|
| 363 |
-
|
| 364 |
-
**
|
| 365 |
-
|
| 366 |
-
|
| 367 |
-
|
| 368 |
-
|
| 369 |
-
|
| 370 |
-
|
| 371 |
-
|
| 372 |
-
|
| 373 |
-
|
| 374 |
-
|
| 375 |
-
|
| 376 |
-
|
| 377 |
-
|
| 378 |
-
|
| 379 |
-
|
| 380 |
-
|
| 381 |
-
|
| 382 |
-
|
| 383 |
-
|
| 384 |
-
## Files & usage
|
| 385 |
-
|
| 386 |
-
- `thimble-v6.pt` — checkpoint (`torch.load(..., weights_only=False)` → `{"model": state_dict, "cfg": dict}`)
|
| 387 |
-
- `tokenizer.json` — BPE vocab + merges
|
| 388 |
-
|
| 389 |
-
The guarantees live in the decoding harness, so inference goes through the repo:
|
| 390 |
-
|
| 391 |
-
```bash
|
| 392 |
-
git clone https://github.com/nikshepsvn/thimble
|
| 393 |
-
cd thimble && uv venv && uv pip install -e ".[hub]"
|
| 394 |
-
|
| 395 |
-
# both files come from this repo; neither is in git
|
| 396 |
-
hf download flashvenom/thimble thimble-v6.pt --local-dir checkpoints/
|
| 397 |
-
hf download flashvenom/thimble tokenizer.json --local-dir data/
|
| 398 |
-
|
| 399 |
-
python demo.py "make a reservation at Nobu for 2 people at 7pm and text Sam saying dinner is on"
|
| 400 |
-
# [{"name": "createReservation",
|
| 401 |
-
# "arguments": {"partySize": 2, "restaurant": "Nobu", "time": "7pm"}},
|
| 402 |
-
# {"name": "sendMessage",
|
| 403 |
-
# "arguments": {"body": "dinner is on", "contact": "Sam"}}]
|
| 404 |
-
|
| 405 |
-
python demo.py "sing me a happy birthday song"
|
| 406 |
-
# [] (refused: no tool applies)
|
| 407 |
-
|
| 408 |
-
python scripts/final_eval.py --ckpt thimble-v6 --suite seal-tools-in # reproduce the table
|
| 409 |
-
```
|
| 410 |
-
|
| 411 |
-
Real output, not a mock — typed integers, two-call composition, and refusal,
|
| 412 |
-
with structure guaranteed by the grammar.
|
| 413 |
-
|
| 414 |
-
## Integrity
|
| 415 |
-
|
| 416 |
-
Public corpora (xlam, ToolACE, Dolci, Glaive, official benchmark train splits)
|
| 417 |
-
plus stepwise-validated, evidence-filtered synthetic data. Every training row
|
| 418 |
-
passed an **8-gram contamination firewall against every evaluation query of
|
| 419 |
-
every reported suite** (BFCL included). Champion selection by held-out dev loss
|
| 420 |
-
only; nothing was ever tuned on an eval set; every negative result is published.
|
| 421 |
-
|
| 422 |
-
*Built by one person and an AI assistant in about a week of evenings, for about
|
| 423 |
-
the price of a game console. The failures are the useful part.*
|
| 424 |
|
| 425 |
## Citation
|
| 426 |
|
|
|
|
| 69 |
value: 23.5
|
| 70 |
name: BFCL v4 single-turn
|
| 71 |
---
|
| 72 |
+
|
| 73 |
+
<div align="center">
|
| 74 |
+
|
| 75 |
# 🧵 Thimble
|
| 76 |
|
| 77 |
### A tool-calling layer, not a language model.
|
|
|
|
| 79 |
**Your schemas in, validated calls out, at 48M parameters.**
|
| 80 |
|
| 81 |
[](https://github.com/nikshepsvn/thimble/blob/master/LICENSE)
|
| 82 |
+
[](https://huggingface.co/flashvenom/thimble)
|
| 83 |
[](#numbers)
|
| 84 |
[](#the-contract)
|
| 85 |
+
[](https://github.com/nikshepsvn/thimble/blob/master/REPRODUCING.md)
|
| 86 |
+
|
| 87 |
+
<picture>
|
| 88 |
+
<source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/nikshepsvn/thimble/master/assets/results-dark.png">
|
| 89 |
+
<img alt="Accuracy by suite, split by whether the tool catalog appeared in training" src="https://raw.githubusercontent.com/nikshepsvn/thimble/master/assets/results-light.png" width="100%">
|
| 90 |
+
</picture>
|
| 91 |
|
| 92 |
+
</div>
|
| 93 |
|
| 94 |
It does not converse, reason, or write prose — it was never trained to. It reads
|
| 95 |
a catalog of typed functions and a request, and returns the calls to make or an
|
|
|
|
| 107 |
because they come from a grammar compiled out of your schemas rather than from
|
| 108 |
the weights:
|
| 109 |
|
| 110 |
+
- **Output is always well-formed JSON.** Not usually — always. Malformed output
|
| 111 |
+
is unreachable, not unlikely.
|
| 112 |
+
- **Argument keys come from your schema.** Parameter-name hallucination is
|
| 113 |
+
structurally impossible.
|
| 114 |
- **Calls to tools you did not declare cannot be emitted.**
|
| 115 |
|
| 116 |
The model is consulted at exactly five choice points: refuse-or-call, which tool,
|
| 117 |
include this optional, what value, stop or continue. Everything else — braces,
|
| 118 |
+
quotes, commas, every argument key — is determined before it runs.
|
| 119 |
+
|
| 120 |
+
Accuracy is a separate question, answered below with numbers. The contract is not
|
| 121 |
+
conditional on any of them.
|
| 122 |
+
|
| 123 |
+
## Try it in your browser
|
| 124 |
+
|
| 125 |
+
**[nikshepsvn.com/thimble](https://nikshepsvn.com/thimble/)** — the C engine
|
| 126 |
+
compiled to WebAssembly. The whole model runs in the tab (105KB engine + 48MB
|
| 127 |
+
int8 weights, no server); edit the tool catalog live and watch the grammar
|
| 128 |
+
adapt with zero retraining. ~250–650ms per call via SIMD128.
|
| 129 |
+
|
| 130 |
+
## Try it in 30 seconds
|
| 131 |
+
|
| 132 |
+
```
|
| 133 |
+
git clone https://github.com/nikshepsvn/thimble && cd thimble
|
| 134 |
+
uv venv && uv pip install -e ".[hub]"
|
| 135 |
+
|
| 136 |
+
# both files come from the HF repo; neither is in git
|
| 137 |
+
hf download flashvenom/thimble thimble-v6.pt --local-dir checkpoints/
|
| 138 |
+
hf download flashvenom/thimble tokenizer.json --local-dir data/
|
| 139 |
+
```
|
| 140 |
+
|
| 141 |
+
Then:
|
| 142 |
+
|
| 143 |
+
```
|
| 144 |
+
$ python demo.py "make a reservation at Nobu for 2 people at 7pm and text Sam saying dinner is on"
|
| 145 |
+
[
|
| 146 |
+
{"name": "createReservation",
|
| 147 |
+
"arguments": {"partySize": 2, "restaurant": "Nobu", "time": "7pm"}},
|
| 148 |
+
{"name": "sendMessage",
|
| 149 |
+
"arguments": {"body": "dinner is on", "contact": "Sam"}}
|
| 150 |
+
]
|
| 151 |
+
|
| 152 |
+
$ python demo.py "sing me a happy birthday song"
|
| 153 |
+
[] (refused: no tool applies)
|
| 154 |
+
```
|
| 155 |
+
|
| 156 |
+
Real output, not a mock — the typed integer `partySize`, the two-call
|
| 157 |
+
composition, and the refusal. Point it at your own tools with:
|
| 158 |
+
|
| 159 |
+
```
|
| 160 |
+
python scripts/eval_catalog.py --ckpt thimble-v6 \
|
| 161 |
+
--catalog my_tools.json --gold my_eval.jsonl
|
| 162 |
+
```
|
| 163 |
|
| 164 |
## Does this fit your problem?
|
| 165 |
|
| 166 |
+
**It works out of the box when** requests are command-shaped and state their
|
| 167 |
+
values: `annotate variant rs4988235 against build GRCh38`. Identifiers, codes,
|
| 168 |
+
dates, numbers, enum picks — copied, not inferred. Chains are fine: two-plus-call
|
| 169 |
+
rows score **73.5%** on a catalog it knows.
|
| 170 |
|
| 171 |
The gate is how *extractive* the request is, not what domain it belongs to. In a
|
| 172 |
pair of small probes, an unseen biomedical catalog in `dot.notation` scored 0.75
|
| 173 |
while a familiar-looking app catalog with conversational phrasing scored 0.57.
|
| 174 |
+
Unfamiliar vocabulary is survivable; phrasing that hides the values is not.
|
| 175 |
+
(Two hand-written probes, 15 rows — directional, not a measurement.)
|
| 176 |
|
| 177 |
**Adapt it when** you need conversational phrasing, disciplined handling of
|
| 178 |
+
optional arguments, or calibrated refusal. Those three are what specializing
|
| 179 |
+
buys, and they are the documented weak spots — see below.
|
| 180 |
|
| 181 |
**Use something else when** you have an open-world catalog, need Java or
|
| 182 |
+
JavaScript schema dialects, or need parallel instantiations of one schema. And
|
| 183 |
+
if you can afford 600M parameters, fine-tune Qwen instead — it will probably
|
| 184 |
+
score higher. This is for when you cannot: a memory ceiling, a latency floor, or
|
| 185 |
+
wanting a separate model per customer rather than one prompted model for all.
|
| 186 |
|
| 187 |
## Numbers
|
| 188 |
|
| 189 |
+
Ordered strict exact match — a row passes only if the function names, the call
|
| 190 |
+
order, and every argument value match. The right-hand column is a yardstick, not
|
| 191 |
+
a rival: Needle 2 (Cactus Compute, 45M params, 153B training tokens), their
|
| 192 |
+
published numbers on their metric, unmodified. It is there so the left column
|
| 193 |
+
has a scale.
|
| 194 |
|
| 195 |
**Catalog represented in training** (eval rows firewalled out):
|
| 196 |
|
|
|
|
| 209 |
| Seal-Tools out-of-domain (654) | 28.1 | 28.7 |
|
| 210 |
| BFCL v4 single-turn (3,641) | 23.5 | 42.6 |
|
| 211 |
|
| 212 |
+
The spread between those tables is visible *inside a single suite* — the cleanest
|
| 213 |
+
control in the project, because only one variable moves:
|
| 214 |
|
| 215 |
+
<picture>
|
| 216 |
+
<source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/nikshepsvn/thimble/master/assets/catalog-control-dark.png">
|
| 217 |
+
<img alt="Seal-Tools in-domain vs out-of-domain: row accuracy 33.1 vs 28.1, tool-name sequence 88.0 vs 79.0" src="https://raw.githubusercontent.com/nikshepsvn/thimble/master/assets/catalog-control-light.png" width="88%">
|
| 218 |
+
</picture>
|
| 219 |
|
| 220 |
Name-sequence accuracy tracks row accuracy exactly. The model is not failing to
|
| 221 |
+
extract arguments on unfamiliar catalogs — it is failing to pick the right
|
| 222 |
+
function.
|
| 223 |
|
| 224 |
**Disclosures.** Mobile Actions' public train split (8,693 rows, disjoint from
|
| 225 |
+
eval) is in the training mix — that is what the first table's heading means.
|
| 226 |
+
DroidCall's official split script calls `random.shuffle()` unseeded, so their
|
| 227 |
+
exact 200 rows are unreproducible by anyone; ours is a seeded split from the same
|
| 228 |
+
pool, firewalled out of training. The Seal-in margin over the yardstick is +0.5
|
| 229 |
+
on 700 rows, within sampling noise. The pre-registered dev-loss champion was a
|
| 230 |
+
sibling checkpoint scoring 28.4; that selector failure is diagnosed in
|
| 231 |
+
[FINDINGS.md](https://github.com/nikshepsvn/thimble/blob/master/FINDINGS.md) with both models' tables published.
|
| 232 |
|
| 233 |
## Adapting it to your catalog
|
| 234 |
|
| 235 |
+
```
|
| 236 |
+
python scripts/adapt.py --catalog my_tools.json --name mydomain
|
| 237 |
+
python scripts/eval_catalog.py --ckpt mydomain --baseline thimble-v6 \
|
| 238 |
+
--catalog my_tools.json --gold my_eval.jsonl
|
| 239 |
+
```
|
| 240 |
|
| 241 |
+
Three stages, each resumable with `--stage`:
|
|
|
|
|
|
|
| 242 |
|
| 243 |
+
| stage | what happens |
|
| 244 |
+
|---|---|
|
| 245 |
+
| `synth` | a teacher model writes (query → calls) rows **against your schemas**; each is validated against your parameter types and an evidence rule before it is kept |
|
| 246 |
+
| `pack` | your rows are blended with guard corpora and packed into two splits |
|
| 247 |
+
| `train` | continues from `thimble-v6`, annealing your blend into the LR-decay phase |
|
| 248 |
|
| 249 |
+
Two things about that recipe are load-bearing, both measured rather than assumed:
|
|
|
|
| 250 |
|
| 251 |
+
- **Anneal, don't retrain.** The same corrective corpus scored 28.4 fed from
|
| 252 |
+
scratch and 33.1 annealed into the decay phase. Corrective data dilutes into
|
| 253 |
+
the average when it competes with a whole corpus; it concentrates when it
|
| 254 |
+
arrives late.
|
| 255 |
+
- **Keep the guard data.** The blend deliberately carries general tool-calling
|
| 256 |
+
rows alongside yours. Annealing purely on your catalog trades away the
|
| 257 |
+
competence you are building on. `adapt.py` warns if it finds none.
|
| 258 |
+
|
| 259 |
+
Pass `--examples` if you have real gold rows; they are weighted above synthetic
|
| 260 |
+
ones. Needs `OPENROUTER_API_KEY` for synthesis and a GPU to train. For scale, the
|
| 261 |
+
v6 cycle synthesized 74,250 validated rows for $56.
|
| 262 |
+
|
| 263 |
+
**Not yet demonstrated end to end.** `adapt.py` wires together exactly the
|
| 264 |
+
machinery that produced the v6 result, but no third-party catalog has been
|
| 265 |
+
adapted and published. The recipe is measured; the ergonomics are new. If you run
|
| 266 |
+
it, the numbers are worth a pull request.
|
| 267 |
+
|
| 268 |
+
## Deploying it
|
| 269 |
+
|
| 270 |
+
The Python stack is for training, eval and adaptation. To serve or embed the
|
| 271 |
+
model there is **[cengine/](https://github.com/nikshepsvn/thimble/tree/master/cengine)** — the full decoder (tokenizer, trunk,
|
| 272 |
+
grammar walk, name head, retrieval) in one dependency-free C file:
|
| 273 |
+
|
| 274 |
+
```
|
| 275 |
+
uv run python cengine/export.py && cd cengine && make
|
| 276 |
+
./thimble -w thimble-q8.bin -t tokenizer.bin -c demo_catalog.json \
|
| 277 |
+
"make a reservation at Nobu for 2 people at 7pm"
|
| 278 |
+
```
|
| 279 |
+
|
| 280 |
+
| | weights | load | per request* |
|
| 281 |
+
|---|---:|---:|---:|
|
| 282 |
+
| Python stack (torch, CPU) | 184 MB | seconds | 582 ms |
|
| 283 |
+
| cengine fp32 | 191 MB | ~60 ms | 453 ms |
|
| 284 |
+
| cengine int8 | 48 MB | ~20 ms | 348 ms |
|
| 285 |
+
|
| 286 |
+
*mean over the first 100 Mobile Actions eval rows on an Apple M3; a request is
|
| 287 |
+
a full decode, prefill plus every choice point and rollback.
|
| 288 |
+
|
| 289 |
+
Parity is verified, not assumed: fp32 output is byte-identical to the Python
|
| 290 |
+
stack on 100/100 checked rows, and int8 differs on 2/100 — both of which
|
| 291 |
+
happened to move toward gold. Laptops, phones and edge Linux are in reach.
|
| 292 |
+
|
| 293 |
+
## Using it as an agent fast path
|
| 294 |
+
|
| 295 |
+
Semantic routers answer *which tool*; the request still pays an LLM call for
|
| 296 |
+
the arguments. **[route/](https://github.com/nikshepsvn/thimble/tree/master/route)** is the other half: a ~130ms local
|
| 297 |
+
dispatcher that returns the complete validated call plus two confidence
|
| 298 |
+
signals, so your big model is only consulted when the dispatcher abstains.
|
| 299 |
+
|
| 300 |
+
```python
|
| 301 |
+
from route.dispatch import ThimbleDispatcher
|
| 302 |
+
|
| 303 |
+
d = ThimbleDispatcher() # wraps `cengine/thimble --serve`
|
| 304 |
+
r = d.dispatch("text Sam that i'm running late", tools)
|
| 305 |
+
r.dispatched # True — confidence cleared the per-catalog gate
|
| 306 |
+
r.calls # [{"name":"sendMessage","arguments":{"body":"i'm running late","contact":"Sam"}}]
|
| 307 |
```
|
| 308 |
|
| 309 |
+
The gate is measured, not assumed. On a catalog represented in training
|
| 310 |
+
(Mobile Actions, n=300 gold rows), sweeping the value-confidence threshold:
|
|
|
|
|
|
|
| 311 |
|
| 312 |
+
| gate | requests dispatched | precision of dispatched |
|
| 313 |
+
|---|---:|---:|
|
| 314 |
+
| vlp ≥ −0.002 | 77% | 98.7% |
|
| 315 |
+
| vlp ≥ −0.001 | 65% | 99.5% |
|
| 316 |
|
| 317 |
+
On a catalog the model handles poorly, the same gate collapses coverage to
|
| 318 |
+
~17% instead of dispatching confidently wrong calls. Derive the threshold for
|
| 319 |
+
*your* catalog from a small gold set (`route.dispatch.sweep`, one command);
|
| 320 |
+
if no threshold clears your bar, adapt the model first or keep everything on
|
| 321 |
+
the fallback path. A LangGraph node example with full router traceability is
|
| 322 |
+
in [route/langgraph_fastpath.py](https://github.com/nikshepsvn/thimble/blob/master/route/langgraph_fastpath.py).
|
| 323 |
|
| 324 |
## How it works
|
| 325 |
|
|
|
|
| 342 |
**The loss is weighted by those same five decisions** — structure 1x, keys 1.5x,
|
| 343 |
names 2x, values 4x, stop-decision 6x — matched to the measured error
|
| 344 |
distribution. The model is optimized for the choices it will be asked to make,
|
| 345 |
+
not for tokens it will never emit. (A closely related weighting, without the
|
| 346 |
+
stop-decision term, appears independently in Needle's
|
| 347 |
+
[Simple Attention Networks notes](https://github.com/cactus-compute/needle/blob/main/docs/simple_attention_networks.md);
|
| 348 |
+
the scheme is not original to this project.)
|
| 349 |
|
| 350 |
**The decoder consults the model only at those points.** Everything else is
|
| 351 |
+
determined before it runs, which is where [the contract](#the-contract) comes
|
| 352 |
from. One call, start to finish — `MODEL` marks the only places the network is
|
| 353 |
asked anything:
|
| 354 |
|
|
|
|
| 407 |
tokens carry the call-sequencing signal: the model learns *when a call ends*
|
| 408 |
through structure it never has to emit. Remove them and it breaks.
|
| 409 |
|
| 410 |
+
<details>
|
| 411 |
+
<summary><b>The rest of the stack — retriever, name head, trunk</b></summary>
|
| 412 |
+
|
| 413 |
|
| 414 |
- **Retriever** — `retrieve(query, tools, emitted=...)`, a DTDR-style
|
| 415 |
(arXiv 2512.17052) refresh conditioned on the *partial plan*, so the candidate
|
|
|
|
| 427 |
(arXiv 2607.18363) finds architecture choices at this scale worth hundredths of
|
| 428 |
a nat at matched parameters. The co-design above is the part that matters.
|
| 429 |
|
| 430 |
+
</details>
|
| 431 |
+
|
| 432 |
+
<details>
|
| 433 |
+
<summary><b>How the model was built — the error-driven data loop</b></summary>
|
| 434 |
+
|
| 435 |
|
| 436 |
Row accuracy factors as `P(name sequence) x p^n`, where `p` is per-call argument
|
| 437 |
accuracy. Each version measured which factor was binding and attacked only that:
|
|
|
|
| 448 |
check: **+3.3 points at constant LR** from the corrective corpus alone. That loop
|
| 449 |
is what `adapt.py` automates for your catalog.
|
| 450 |
|
| 451 |
+
</details>
|
| 452 |
|
| 453 |
+
## What did not work
|
| 454 |
|
| 455 |
+
Eleven ideas, each killed by a measurement rather than an argument: span copying
|
| 456 |
+
(−30), pointer heads (−16), RFT-style loss down-weighting (−12), from-scratch
|
| 457 |
+
retraining (−4.7), field-set reranking (−1.4), beam/RL/best-of-N (oracle-capped
|
| 458 |
+
below target), draft-then-constrain (no tax to recover), a global optional-skip
|
| 459 |
+
prior (catalog-dependent), `MAX_CALLS` (never binding), RLOO on an annealed
|
| 460 |
+
checkpoint (diverges at every LR), and matching the benchmark's numeric typing
|
| 461 |
+
(not learnable). Plus two process failures that cost real points.
|
|
|
|
|
|
|
|
|
|
| 462 |
|
| 463 |
+
**[FINDINGS.md](https://github.com/nikshepsvn/thimble/blob/master/FINDINGS.md) has all of them** with the measurement, the reason,
|
| 464 |
+
and the takeaway. It is the most reusable part of the project.
|
| 465 |
|
| 466 |
+
## Known limits
|
| 467 |
|
| 468 |
- **Unfamiliar catalogs.** Out-of-domain name-sequence accuracy is 79% against
|
| 469 |
+
88% in-domain. Every out-of-domain deficit traces to this number — the one
|
| 470 |
+
`adapt.py` exists to move.
|
| 471 |
+
- **Optional arguments, in both directions.** The largest documented failure
|
| 472 |
+
bucket: 66 of 193 failing v5 calls added exactly one optional the query never
|
| 473 |
+
mentioned, and the model also drops optionals the query does state.
|
| 474 |
+
- **Multi-call tracks per-call accuracy, not call count.** `P(names) x p^n`, so
|
| 475 |
+
chains collapse wherever `p` is mediocre and hold where it is not: 73.5% on
|
| 476 |
+
Mobile Actions, 19.4% on Seal-Tools in-domain. The call count is not the
|
| 477 |
+
problem; the catalog is.
|
| 478 |
+
- **Parallel calls are a separate, worse failure.** `parallel` 12.0,
|
| 479 |
+
`live_parallel` 0.0 — repeated instantiations of one schema, as opposed to
|
| 480 |
+
calls the query motivates in sequence.
|
| 481 |
+
- **Schema dialects.** `simple_python` 29.3 on BFCL against `simple_java` 14.0
|
| 482 |
+
and `simple_javascript` 8.0. Those conventions are absent from a deliberately
|
| 483 |
+
extractive ~1B-token corpus.
|
| 484 |
- **768-token context.** 151 of 3,641 BFCL rows (4.1%) do not fit and score as misses.
|
| 485 |
- **Microcontrollers.** ~11.5MB at 2-bit is a property of the parameter count,
|
| 486 |
not a shippable artifact: 2-bit would need quantization-aware retraining this
|
| 487 |
model never had. The smallest thing that actually runs is the 48MB int8
|
| 488 |
+
engine ([Deploying it](#deploying-it)) — Pi-class and up, not Cortex-M.
|
| 489 |
+
|
| 490 |
+
Scale explains most of it honestly: ~1B unique tokens, no pretraining phase, a
|
| 491 |
+
corpus spent deliberately on depth rather than breadth.
|
| 492 |
+
|
| 493 |
+
## Going deeper
|
| 494 |
+
|
| 495 |
+
- **[FINDINGS.md](https://github.com/nikshepsvn/thimble/blob/master/FINDINGS.md)** — eleven negative results and two process
|
| 496 |
+
failures, with the measurement that killed each one. The most reusable part.
|
| 497 |
+
- **[REPRODUCING.md](https://github.com/nikshepsvn/thimble/blob/master/REPRODUCING.md)** — repository layout and the exact
|
| 498 |
+
pipeline that rebuilds the published numbers.
|
| 499 |
+
- **[RESULTS.md](https://github.com/nikshepsvn/thimble/blob/master/RESULTS.md)** — the full chronological experimental record.
|
| 500 |
+
- **[paper/thimble.pdf](https://github.com/nikshepsvn/thimble/blob/master/paper/thimble.pdf)** — the tech report: the co-design
|
| 501 |
+
thesis, the negative results, and the related work, in citable form.
|
| 502 |
+
|
| 503 |
+
## Honest summary
|
| 504 |
+
|
| 505 |
+
Turning a request into calls against an API you control is a smaller problem than
|
| 506 |
+
the models usually pointed at it. Treated as a translation layer rather than a
|
| 507 |
+
language model, it fits in 48M parameters, comes with guarantees a prompted model
|
| 508 |
+
cannot offer, and can be specialized to one catalog for the price of a dinner.
|
| 509 |
+
It ships as a working system, not just a checkpoint: a one-file C engine with
|
| 510 |
+
byte-verified parity, a browser demo running the whole model client-side, and a
|
| 511 |
+
measured confidence gate for fronting a larger agent. Its limits are real and
|
| 512 |
+
measured rather than described. Built by one person over a few days with AI
|
| 513 |
+
assistance, for about the price of a video game console.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 514 |
|
| 515 |
## Citation
|
| 516 |
|