flashvenom commited on
Commit
b1208f0
·
verified ·
1 Parent(s): 2fb3a39

docs: regenerate card from current README — engine, browser demo, fast-path dispatcher

Browse files
Files changed (1) hide show
  1. README.md +238 -148
README.md CHANGED
@@ -69,6 +69,9 @@ model-index:
69
  value: 23.5
70
  name: BFCL v4 single-turn
71
  ---
 
 
 
72
  # 🧵 Thimble
73
 
74
  ### A tool-calling layer, not a language model.
@@ -76,11 +79,17 @@ model-index:
76
  **Your schemas in, validated calls out, at 48M parameters.**
77
 
78
  [![License: MIT](https://img.shields.io/badge/license-MIT-1f6feb?style=flat-square)](https://github.com/nikshepsvn/thimble/blob/master/LICENSE)
79
- [![Code](https://img.shields.io/badge/GitHub-code%20%26%20adaptation%20loop-24292f?style=flat-square&logo=github)](https://github.com/nikshepsvn/thimble)
80
  [![Parameters](https://img.shields.io/badge/params-48.12M-c8324c?style=flat-square)](#numbers)
81
  [![Well-formed JSON](https://img.shields.io/badge/well--formed%20JSON-100%25%20by%20construction-2ea043?style=flat-square)](#the-contract)
 
 
 
 
 
 
82
 
83
- ![Accuracy by suite, split by whether the tool catalog appeared in training](results-light.png)
84
 
85
  It does not converse, reason, or write prose — it was never trained to. It reads
86
  a catalog of typed functions and a request, and returns the calls to make or an
@@ -98,42 +107,90 @@ Three guarantees hold on **any** catalog, with no training and no configuration,
98
  because they come from a grammar compiled out of your schemas rather than from
99
  the weights:
100
 
101
- - **Output is always well-formed JSON.** Malformed output is unreachable, not unlikely.
102
- - **Argument keys come from your schema.** Parameter-name hallucination is structurally impossible.
 
 
103
  - **Calls to tools you did not declare cannot be emitted.**
104
 
105
  The model is consulted at exactly five choice points: refuse-or-call, which tool,
106
  include this optional, what value, stop or continue. Everything else — braces,
107
- quotes, commas, every argument key — is determined before it runs. Accuracy is a
108
- separate question, answered below with numbers; the contract is not conditional
109
- on any of them.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
110
 
111
  ## Does this fit your problem?
112
 
113
- **Works out of the box when** requests are command-shaped and state their values:
114
- `annotate variant rs4988235 against build GRCh38`. Identifiers, codes, dates,
115
- numbers, enum picks — copied, not inferred. Chains are fine: two-plus-call rows
116
- score **73.5%** on a catalog it knows.
117
 
118
  The gate is how *extractive* the request is, not what domain it belongs to. In a
119
  pair of small probes, an unseen biomedical catalog in `dot.notation` scored 0.75
120
  while a familiar-looking app catalog with conversational phrasing scored 0.57.
121
- (15 rows total directional, not a measurement.)
 
122
 
123
  **Adapt it when** you need conversational phrasing, disciplined handling of
124
- optional arguments, or calibrated refusal the three documented weak spots.
 
125
 
126
  **Use something else when** you have an open-world catalog, need Java or
127
- JavaScript schema dialects, or need parallel instantiations of one schema. If you
128
- can afford 600M parameters, fine-tune Qwen instead — it will probably score
129
- higher. This is for when you cannot.
 
130
 
131
  ## Numbers
132
 
133
- Ordered strict exact match — function names, call order, and every argument value
134
- must match. The right-hand column is a yardstick, not a rival: Needle 2 (Cactus
135
- Compute, 45M params, 153B training tokens), their published numbers on their
136
- metric, so the left column has a scale.
 
137
 
138
  **Catalog represented in training** (eval rows firewalled out):
139
 
@@ -152,54 +209,117 @@ metric, so the left column has a scale.
152
  | Seal-Tools out-of-domain (654) | 28.1 | 28.7 |
153
  | BFCL v4 single-turn (3,641) | 23.5 | 42.6 |
154
 
155
- The spread is visible *inside a single suite* — the cleanest control in the
156
- project, because only one variable moves:
157
 
158
- ![Seal-Tools in-domain vs out-of-domain](catalog-control-light.png)
 
 
 
159
 
160
  Name-sequence accuracy tracks row accuracy exactly. The model is not failing to
161
- extract arguments on unfamiliar catalogs — it is failing to pick the right function.
 
162
 
163
  **Disclosures.** Mobile Actions' public train split (8,693 rows, disjoint from
164
- eval) is in the training mix — that is what the first table's heading means. The
165
- Seal-in margin over the yardstick is +0.5 on 700 rows, within sampling noise. The
166
- pre-registered selector picked a sibling checkpoint that scored worse; the
167
- failure is diagnosed with both models' tables published in
168
- [FINDINGS.md](https://github.com/nikshepsvn/thimble/blob/master/FINDINGS.md).
 
 
169
 
170
  ## Adapting it to your catalog
171
 
172
- ```bash
173
- git clone https://github.com/nikshepsvn/thimble && cd thimble
174
- uv venv && uv pip install -e ".[hub]"
 
 
175
 
176
- # both files come from this repo; neither is in git
177
- hf download flashvenom/thimble thimble-v6.pt --local-dir checkpoints/
178
- hf download flashvenom/thimble tokenizer.json --local-dir data/
179
 
180
- # where do you stand on your own tools?
181
- python scripts/eval_catalog.py --ckpt thimble-v6 \
182
- --catalog my_tools.json --gold my_eval.jsonl
 
 
183
 
184
- # synthesize against your schemas, anneal into the decay phase
185
- python scripts/adapt.py --catalog my_tools.json --name mydomain
186
 
187
- # did it help?
188
- python scripts/eval_catalog.py --ckpt mydomain --baseline thimble-v6 \
189
- --catalog my_tools.json --gold my_eval.jsonl
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
190
  ```
191
 
192
- Two things about that recipe are load-bearing, both measured rather than assumed.
193
- **Anneal, don't retrain** the same corrective corpus scored 28.4 fed from
194
- scratch and 33.1 annealed into the LR-decay phase. **Keep the guard data** —
195
- annealing purely on your catalog trades away the competence you are building on.
196
 
197
- Needs `OPENROUTER_API_KEY` for synthesis and a GPU to train. For scale, the v6
198
- cycle synthesized 74,250 validated rows for $56.
 
 
199
 
200
- `adapt.py` wires together exactly the machinery that produced the v6 result, but
201
- no third-party catalog has been adapted and published yet. The recipe is
202
- measured; the ergonomics are new.
 
 
 
203
 
204
  ## How it works
205
 
@@ -222,10 +342,13 @@ so the tokenizer's own share of the gain was never isolated.
222
  **The loss is weighted by those same five decisions** — structure 1x, keys 1.5x,
223
  names 2x, values 4x, stop-decision 6x — matched to the measured error
224
  distribution. The model is optimized for the choices it will be asked to make,
225
- not for tokens it will never emit.
 
 
 
226
 
227
  **The decoder consults the model only at those points.** Everything else is
228
- determined before it runs, which is where the contract above comes
229
  from. One call, start to finish — `MODEL` marks the only places the network is
230
  asked anything:
231
 
@@ -284,7 +407,9 @@ what the decoder will supply — cost **12 points** in a controlled twin run. Th
284
  tokens carry the call-sequencing signal: the model learns *when a call ends*
285
  through structure it never has to emit. Remove them and it breaks.
286
 
287
- ### The rest of the stack
 
 
288
 
289
  - **Retriever** — `retrieve(query, tools, emitted=...)`, a DTDR-style
290
  (arXiv 2512.17052) refresh conditioned on the *partial plan*, so the candidate
@@ -302,7 +427,11 @@ through structure it never has to emit. Remove them and it breaks.
302
  (arXiv 2607.18363) finds architecture choices at this scale worth hundredths of
303
  a nat at matched parameters. The co-design above is the part that matters.
304
 
305
- ### How the model was built
 
 
 
 
306
 
307
  Row accuracy factors as `P(name sequence) x p^n`, where `p` is per-call argument
308
  accuracy. Each version measured which factor was binding and attacked only that:
@@ -319,108 +448,69 @@ canonical date forms, ~29 were unwinnable noise in the gold. Mid-run causal
319
  check: **+3.3 points at constant LR** from the corrective corpus alone. That loop
320
  is what `adapt.py` automates for your catalog.
321
 
322
- ## What didn't work (measured, not guessed)
323
 
324
- The most reusable part of the project. Each idea was killed by an A/B, not an argument:
325
 
326
- | idea | result |
327
- |---|---|
328
- | Span-copy heads | 30 pts |
329
- | Pointer/copy head | −16 pts |
330
- | Down-weighting grammar-forced tokens (RFT-style) | −12 pts structure tokens carry call-sequencing signal |
331
- | From-scratch retrain on corrective data | −4.7 vs annealing |
332
- | Field-set reranking | −1.4 training had already fixed its target bucket |
333
- | Beam / RL / best-of-N | oracle-capped below target |
334
- | RLOO fine-tune on the annealed checkpoint | diverges at every LR — sharp minima and policy gradients don't mix |
335
- | Matching Seal's gold numeric typing | not learnable — 74% of params are mixed-convention noise |
336
 
337
- Two of these reversed conclusions that would otherwise have shipped on intuition.
 
338
 
339
- ## Where it stops working
340
 
341
  - **Unfamiliar catalogs.** Out-of-domain name-sequence accuracy is 79% against
342
- 88% in-domain. Every out-of-domain deficit traces back to this one number.
343
- - **Schema dialects.** `simple_python` scores 29.3 on BFCL, but `simple_java`
344
- 14.0 and `simple_javascript` 8.0 Java and JS schema conventions are absent
345
- from a deliberately extractive ~1B-token corpus.
346
- - **Multi-call tracks per-call accuracy, not call count.** Row accuracy is
347
- `P(names) x p^n`, so chains collapse wherever `p` is mediocre and hold up where
348
- it is not: two-plus-call rows score **73.5%** on Mobile Actions but 19.4% on
349
- Seal-Tools in-domain. The call count is not the problem; the catalog is.
350
- - **Parallel calls are a separate, worse failure.** `parallel` 12.0 and
351
- `live_parallel` 0.0 on BFCL repeated instantiations of one schema, as
352
- opposed to calls the query motivates in sequence.
 
 
 
 
353
  - **768-token context.** 151 of 3,641 BFCL rows (4.1%) do not fit and score as misses.
354
  - **Microcontrollers.** ~11.5MB at 2-bit is a property of the parameter count,
355
  not a shippable artifact: 2-bit would need quantization-aware retraining this
356
  model never had. The smallest thing that actually runs is the 48MB int8
357
- build of the C engine below — Pi-class and up, not Cortex-M.
358
-
359
- Scale is the honest explanation for most of this: ~1B unique tokens, no
360
- pretraining phase, a corpus deliberately spent on depth instead of breadth.
361
-
362
- ## Torch-free inference (C engine)
363
-
364
- **Try it in your browser: [nikshepsvn.com/thimble](https://nikshepsvn.com/thimble/)** — the engine compiled to WebAssembly, running this checkpoint client-side.
365
-
366
- The repo ships [cengine/](https://github.com/nikshepsvn/thimble/tree/master/cengine),
367
- the full decoder tokenizer, trunk, grammar walk, name head — in one
368
- dependency-free C file. 48MB int8 weights, ~20ms load, ~350ms per full request
369
- on an Apple M3, and fp32 output is byte-identical to this checkpoint run
370
- through the Python stack on 100/100 checked eval rows. Export the weights from
371
- this repo's checkpoint with `cengine/export.py`.
372
-
373
- ## Model details
374
-
375
- | | |
376
- |---|---|
377
- | Parameters | 48.12M (fp32 checkpoint; ~11.5MB at 2-bit) |
378
- | Architecture | deep-thin gated trunk: d=448, 20 layers, GQA 8/4, SwiGLU ×2.0, QK-norm, sandwich RMSNorm, tied embeddings |
379
- | Tokenizer | 16,384 BPE, digits as singletons, JSON structural chars as singletons |
380
- | Context | 768 tokens |
381
- | Decoding | grammar-constrained, five choice points, plan-conditioned retrieval between calls |
382
- | Training | Muon (trunk) + AdamW, WSD schedule, EMA, weighted CE matched to the error distribution, decay-phase data annealing |
383
-
384
- ## Files & usage
385
-
386
- - `thimble-v6.pt` — checkpoint (`torch.load(..., weights_only=False)` → `{"model": state_dict, "cfg": dict}`)
387
- - `tokenizer.json` — BPE vocab + merges
388
-
389
- The guarantees live in the decoding harness, so inference goes through the repo:
390
-
391
- ```bash
392
- git clone https://github.com/nikshepsvn/thimble
393
- cd thimble && uv venv && uv pip install -e ".[hub]"
394
-
395
- # both files come from this repo; neither is in git
396
- hf download flashvenom/thimble thimble-v6.pt --local-dir checkpoints/
397
- hf download flashvenom/thimble tokenizer.json --local-dir data/
398
-
399
- python demo.py "make a reservation at Nobu for 2 people at 7pm and text Sam saying dinner is on"
400
- # [{"name": "createReservation",
401
- # "arguments": {"partySize": 2, "restaurant": "Nobu", "time": "7pm"}},
402
- # {"name": "sendMessage",
403
- # "arguments": {"body": "dinner is on", "contact": "Sam"}}]
404
-
405
- python demo.py "sing me a happy birthday song"
406
- # [] (refused: no tool applies)
407
-
408
- python scripts/final_eval.py --ckpt thimble-v6 --suite seal-tools-in # reproduce the table
409
- ```
410
-
411
- Real output, not a mock — typed integers, two-call composition, and refusal,
412
- with structure guaranteed by the grammar.
413
-
414
- ## Integrity
415
-
416
- Public corpora (xlam, ToolACE, Dolci, Glaive, official benchmark train splits)
417
- plus stepwise-validated, evidence-filtered synthetic data. Every training row
418
- passed an **8-gram contamination firewall against every evaluation query of
419
- every reported suite** (BFCL included). Champion selection by held-out dev loss
420
- only; nothing was ever tuned on an eval set; every negative result is published.
421
-
422
- *Built by one person and an AI assistant in about a week of evenings, for about
423
- the price of a game console. The failures are the useful part.*
424
 
425
  ## Citation
426
 
 
69
  value: 23.5
70
  name: BFCL v4 single-turn
71
  ---
72
+
73
+ <div align="center">
74
+
75
  # 🧵 Thimble
76
 
77
  ### A tool-calling layer, not a language model.
 
79
  **Your schemas in, validated calls out, at 48M parameters.**
80
 
81
  [![License: MIT](https://img.shields.io/badge/license-MIT-1f6feb?style=flat-square)](https://github.com/nikshepsvn/thimble/blob/master/LICENSE)
82
+ [![Model on Hugging Face](https://img.shields.io/badge/%F0%9F%A4%97%20model-thimble--v6-ffcc4d?style=flat-square)](https://huggingface.co/flashvenom/thimble)
83
  [![Parameters](https://img.shields.io/badge/params-48.12M-c8324c?style=flat-square)](#numbers)
84
  [![Well-formed JSON](https://img.shields.io/badge/well--formed%20JSON-100%25%20by%20construction-2ea043?style=flat-square)](#the-contract)
85
+ [![Build cost](https://img.shields.io/badge/total%20build%20cost-%24260-8957e5?style=flat-square)](https://github.com/nikshepsvn/thimble/blob/master/REPRODUCING.md)
86
+
87
+ <picture>
88
+ <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/nikshepsvn/thimble/master/assets/results-dark.png">
89
+ <img alt="Accuracy by suite, split by whether the tool catalog appeared in training" src="https://raw.githubusercontent.com/nikshepsvn/thimble/master/assets/results-light.png" width="100%">
90
+ </picture>
91
 
92
+ </div>
93
 
94
  It does not converse, reason, or write prose — it was never trained to. It reads
95
  a catalog of typed functions and a request, and returns the calls to make or an
 
107
  because they come from a grammar compiled out of your schemas rather than from
108
  the weights:
109
 
110
+ - **Output is always well-formed JSON.** Not usually always. Malformed output
111
+ is unreachable, not unlikely.
112
+ - **Argument keys come from your schema.** Parameter-name hallucination is
113
+ structurally impossible.
114
  - **Calls to tools you did not declare cannot be emitted.**
115
 
116
  The model is consulted at exactly five choice points: refuse-or-call, which tool,
117
  include this optional, what value, stop or continue. Everything else — braces,
118
+ quotes, commas, every argument key — is determined before it runs.
119
+
120
+ Accuracy is a separate question, answered below with numbers. The contract is not
121
+ conditional on any of them.
122
+
123
+ ## Try it in your browser
124
+
125
+ **[nikshepsvn.com/thimble](https://nikshepsvn.com/thimble/)** — the C engine
126
+ compiled to WebAssembly. The whole model runs in the tab (105KB engine + 48MB
127
+ int8 weights, no server); edit the tool catalog live and watch the grammar
128
+ adapt with zero retraining. ~250–650ms per call via SIMD128.
129
+
130
+ ## Try it in 30 seconds
131
+
132
+ ```
133
+ git clone https://github.com/nikshepsvn/thimble && cd thimble
134
+ uv venv && uv pip install -e ".[hub]"
135
+
136
+ # both files come from the HF repo; neither is in git
137
+ hf download flashvenom/thimble thimble-v6.pt --local-dir checkpoints/
138
+ hf download flashvenom/thimble tokenizer.json --local-dir data/
139
+ ```
140
+
141
+ Then:
142
+
143
+ ```
144
+ $ python demo.py "make a reservation at Nobu for 2 people at 7pm and text Sam saying dinner is on"
145
+ [
146
+ {"name": "createReservation",
147
+ "arguments": {"partySize": 2, "restaurant": "Nobu", "time": "7pm"}},
148
+ {"name": "sendMessage",
149
+ "arguments": {"body": "dinner is on", "contact": "Sam"}}
150
+ ]
151
+
152
+ $ python demo.py "sing me a happy birthday song"
153
+ [] (refused: no tool applies)
154
+ ```
155
+
156
+ Real output, not a mock — the typed integer `partySize`, the two-call
157
+ composition, and the refusal. Point it at your own tools with:
158
+
159
+ ```
160
+ python scripts/eval_catalog.py --ckpt thimble-v6 \
161
+ --catalog my_tools.json --gold my_eval.jsonl
162
+ ```
163
 
164
  ## Does this fit your problem?
165
 
166
+ **It works out of the box when** requests are command-shaped and state their
167
+ values: `annotate variant rs4988235 against build GRCh38`. Identifiers, codes,
168
+ dates, numbers, enum picks — copied, not inferred. Chains are fine: two-plus-call
169
+ rows score **73.5%** on a catalog it knows.
170
 
171
  The gate is how *extractive* the request is, not what domain it belongs to. In a
172
  pair of small probes, an unseen biomedical catalog in `dot.notation` scored 0.75
173
  while a familiar-looking app catalog with conversational phrasing scored 0.57.
174
+ Unfamiliar vocabulary is survivable; phrasing that hides the values is not.
175
+ (Two hand-written probes, 15 rows — directional, not a measurement.)
176
 
177
  **Adapt it when** you need conversational phrasing, disciplined handling of
178
+ optional arguments, or calibrated refusal. Those three are what specializing
179
+ buys, and they are the documented weak spots — see below.
180
 
181
  **Use something else when** you have an open-world catalog, need Java or
182
+ JavaScript schema dialects, or need parallel instantiations of one schema. And
183
+ if you can afford 600M parameters, fine-tune Qwen instead — it will probably
184
+ score higher. This is for when you cannot: a memory ceiling, a latency floor, or
185
+ wanting a separate model per customer rather than one prompted model for all.
186
 
187
  ## Numbers
188
 
189
+ Ordered strict exact match — a row passes only if the function names, the call
190
+ order, and every argument value match. The right-hand column is a yardstick, not
191
+ a rival: Needle 2 (Cactus Compute, 45M params, 153B training tokens), their
192
+ published numbers on their metric, unmodified. It is there so the left column
193
+ has a scale.
194
 
195
  **Catalog represented in training** (eval rows firewalled out):
196
 
 
209
  | Seal-Tools out-of-domain (654) | 28.1 | 28.7 |
210
  | BFCL v4 single-turn (3,641) | 23.5 | 42.6 |
211
 
212
+ The spread between those tables is visible *inside a single suite* — the cleanest
213
+ control in the project, because only one variable moves:
214
 
215
+ <picture>
216
+ <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/nikshepsvn/thimble/master/assets/catalog-control-dark.png">
217
+ <img alt="Seal-Tools in-domain vs out-of-domain: row accuracy 33.1 vs 28.1, tool-name sequence 88.0 vs 79.0" src="https://raw.githubusercontent.com/nikshepsvn/thimble/master/assets/catalog-control-light.png" width="88%">
218
+ </picture>
219
 
220
  Name-sequence accuracy tracks row accuracy exactly. The model is not failing to
221
+ extract arguments on unfamiliar catalogs — it is failing to pick the right
222
+ function.
223
 
224
  **Disclosures.** Mobile Actions' public train split (8,693 rows, disjoint from
225
+ eval) is in the training mix — that is what the first table's heading means.
226
+ DroidCall's official split script calls `random.shuffle()` unseeded, so their
227
+ exact 200 rows are unreproducible by anyone; ours is a seeded split from the same
228
+ pool, firewalled out of training. The Seal-in margin over the yardstick is +0.5
229
+ on 700 rows, within sampling noise. The pre-registered dev-loss champion was a
230
+ sibling checkpoint scoring 28.4; that selector failure is diagnosed in
231
+ [FINDINGS.md](https://github.com/nikshepsvn/thimble/blob/master/FINDINGS.md) with both models' tables published.
232
 
233
  ## Adapting it to your catalog
234
 
235
+ ```
236
+ python scripts/adapt.py --catalog my_tools.json --name mydomain
237
+ python scripts/eval_catalog.py --ckpt mydomain --baseline thimble-v6 \
238
+ --catalog my_tools.json --gold my_eval.jsonl
239
+ ```
240
 
241
+ Three stages, each resumable with `--stage`:
 
 
242
 
243
+ | stage | what happens |
244
+ |---|---|
245
+ | `synth` | a teacher model writes (query → calls) rows **against your schemas**; each is validated against your parameter types and an evidence rule before it is kept |
246
+ | `pack` | your rows are blended with guard corpora and packed into two splits |
247
+ | `train` | continues from `thimble-v6`, annealing your blend into the LR-decay phase |
248
 
249
+ Two things about that recipe are load-bearing, both measured rather than assumed:
 
250
 
251
+ - **Anneal, don't retrain.** The same corrective corpus scored 28.4 fed from
252
+ scratch and 33.1 annealed into the decay phase. Corrective data dilutes into
253
+ the average when it competes with a whole corpus; it concentrates when it
254
+ arrives late.
255
+ - **Keep the guard data.** The blend deliberately carries general tool-calling
256
+ rows alongside yours. Annealing purely on your catalog trades away the
257
+ competence you are building on. `adapt.py` warns if it finds none.
258
+
259
+ Pass `--examples` if you have real gold rows; they are weighted above synthetic
260
+ ones. Needs `OPENROUTER_API_KEY` for synthesis and a GPU to train. For scale, the
261
+ v6 cycle synthesized 74,250 validated rows for $56.
262
+
263
+ **Not yet demonstrated end to end.** `adapt.py` wires together exactly the
264
+ machinery that produced the v6 result, but no third-party catalog has been
265
+ adapted and published. The recipe is measured; the ergonomics are new. If you run
266
+ it, the numbers are worth a pull request.
267
+
268
+ ## Deploying it
269
+
270
+ The Python stack is for training, eval and adaptation. To serve or embed the
271
+ model there is **[cengine/](https://github.com/nikshepsvn/thimble/tree/master/cengine)** — the full decoder (tokenizer, trunk,
272
+ grammar walk, name head, retrieval) in one dependency-free C file:
273
+
274
+ ```
275
+ uv run python cengine/export.py && cd cengine && make
276
+ ./thimble -w thimble-q8.bin -t tokenizer.bin -c demo_catalog.json \
277
+ "make a reservation at Nobu for 2 people at 7pm"
278
+ ```
279
+
280
+ | | weights | load | per request* |
281
+ |---|---:|---:|---:|
282
+ | Python stack (torch, CPU) | 184 MB | seconds | 582 ms |
283
+ | cengine fp32 | 191 MB | ~60 ms | 453 ms |
284
+ | cengine int8 | 48 MB | ~20 ms | 348 ms |
285
+
286
+ *mean over the first 100 Mobile Actions eval rows on an Apple M3; a request is
287
+ a full decode, prefill plus every choice point and rollback.
288
+
289
+ Parity is verified, not assumed: fp32 output is byte-identical to the Python
290
+ stack on 100/100 checked rows, and int8 differs on 2/100 — both of which
291
+ happened to move toward gold. Laptops, phones and edge Linux are in reach.
292
+
293
+ ## Using it as an agent fast path
294
+
295
+ Semantic routers answer *which tool*; the request still pays an LLM call for
296
+ the arguments. **[route/](https://github.com/nikshepsvn/thimble/tree/master/route)** is the other half: a ~130ms local
297
+ dispatcher that returns the complete validated call plus two confidence
298
+ signals, so your big model is only consulted when the dispatcher abstains.
299
+
300
+ ```python
301
+ from route.dispatch import ThimbleDispatcher
302
+
303
+ d = ThimbleDispatcher() # wraps `cengine/thimble --serve`
304
+ r = d.dispatch("text Sam that i'm running late", tools)
305
+ r.dispatched # True — confidence cleared the per-catalog gate
306
+ r.calls # [{"name":"sendMessage","arguments":{"body":"i'm running late","contact":"Sam"}}]
307
  ```
308
 
309
+ The gate is measured, not assumed. On a catalog represented in training
310
+ (Mobile Actions, n=300 gold rows), sweeping the value-confidence threshold:
 
 
311
 
312
+ | gate | requests dispatched | precision of dispatched |
313
+ |---|---:|---:|
314
+ | vlp ≥ −0.002 | 77% | 98.7% |
315
+ | vlp ≥ −0.001 | 65% | 99.5% |
316
 
317
+ On a catalog the model handles poorly, the same gate collapses coverage to
318
+ ~17% instead of dispatching confidently wrong calls. Derive the threshold for
319
+ *your* catalog from a small gold set (`route.dispatch.sweep`, one command);
320
+ if no threshold clears your bar, adapt the model first or keep everything on
321
+ the fallback path. A LangGraph node example with full router traceability is
322
+ in [route/langgraph_fastpath.py](https://github.com/nikshepsvn/thimble/blob/master/route/langgraph_fastpath.py).
323
 
324
  ## How it works
325
 
 
342
  **The loss is weighted by those same five decisions** — structure 1x, keys 1.5x,
343
  names 2x, values 4x, stop-decision 6x — matched to the measured error
344
  distribution. The model is optimized for the choices it will be asked to make,
345
+ not for tokens it will never emit. (A closely related weighting, without the
346
+ stop-decision term, appears independently in Needle's
347
+ [Simple Attention Networks notes](https://github.com/cactus-compute/needle/blob/main/docs/simple_attention_networks.md);
348
+ the scheme is not original to this project.)
349
 
350
  **The decoder consults the model only at those points.** Everything else is
351
+ determined before it runs, which is where [the contract](#the-contract) comes
352
  from. One call, start to finish — `MODEL` marks the only places the network is
353
  asked anything:
354
 
 
407
  tokens carry the call-sequencing signal: the model learns *when a call ends*
408
  through structure it never has to emit. Remove them and it breaks.
409
 
410
+ <details>
411
+ <summary><b>The rest of the stack — retriever, name head, trunk</b></summary>
412
+
413
 
414
  - **Retriever** — `retrieve(query, tools, emitted=...)`, a DTDR-style
415
  (arXiv 2512.17052) refresh conditioned on the *partial plan*, so the candidate
 
427
  (arXiv 2607.18363) finds architecture choices at this scale worth hundredths of
428
  a nat at matched parameters. The co-design above is the part that matters.
429
 
430
+ </details>
431
+
432
+ <details>
433
+ <summary><b>How the model was built — the error-driven data loop</b></summary>
434
+
435
 
436
  Row accuracy factors as `P(name sequence) x p^n`, where `p` is per-call argument
437
  accuracy. Each version measured which factor was binding and attacked only that:
 
448
  check: **+3.3 points at constant LR** from the corrective corpus alone. That loop
449
  is what `adapt.py` automates for your catalog.
450
 
451
+ </details>
452
 
453
+ ## What did not work
454
 
455
+ Eleven ideas, each killed by a measurement rather than an argument: span copying
456
+ (−30), pointer heads (−16), RFT-style loss down-weighting (−12), from-scratch
457
+ retraining (−4.7), field-set reranking (1.4), beam/RL/best-of-N (oracle-capped
458
+ below target), draft-then-constrain (no tax to recover), a global optional-skip
459
+ prior (catalog-dependent), `MAX_CALLS` (never binding), RLOO on an annealed
460
+ checkpoint (diverges at every LR), and matching the benchmark's numeric typing
461
+ (not learnable). Plus two process failures that cost real points.
 
 
 
462
 
463
+ **[FINDINGS.md](https://github.com/nikshepsvn/thimble/blob/master/FINDINGS.md) has all of them** with the measurement, the reason,
464
+ and the takeaway. It is the most reusable part of the project.
465
 
466
+ ## Known limits
467
 
468
  - **Unfamiliar catalogs.** Out-of-domain name-sequence accuracy is 79% against
469
+ 88% in-domain. Every out-of-domain deficit traces to this number — the one
470
+ `adapt.py` exists to move.
471
+ - **Optional arguments, in both directions.** The largest documented failure
472
+ bucket: 66 of 193 failing v5 calls added exactly one optional the query never
473
+ mentioned, and the model also drops optionals the query does state.
474
+ - **Multi-call tracks per-call accuracy, not call count.** `P(names) x p^n`, so
475
+ chains collapse wherever `p` is mediocre and hold where it is not: 73.5% on
476
+ Mobile Actions, 19.4% on Seal-Tools in-domain. The call count is not the
477
+ problem; the catalog is.
478
+ - **Parallel calls are a separate, worse failure.** `parallel` 12.0,
479
+ `live_parallel` 0.0 repeated instantiations of one schema, as opposed to
480
+ calls the query motivates in sequence.
481
+ - **Schema dialects.** `simple_python` 29.3 on BFCL against `simple_java` 14.0
482
+ and `simple_javascript` 8.0. Those conventions are absent from a deliberately
483
+ extractive ~1B-token corpus.
484
  - **768-token context.** 151 of 3,641 BFCL rows (4.1%) do not fit and score as misses.
485
  - **Microcontrollers.** ~11.5MB at 2-bit is a property of the parameter count,
486
  not a shippable artifact: 2-bit would need quantization-aware retraining this
487
  model never had. The smallest thing that actually runs is the 48MB int8
488
+ engine ([Deploying it](#deploying-it)) — Pi-class and up, not Cortex-M.
489
+
490
+ Scale explains most of it honestly: ~1B unique tokens, no pretraining phase, a
491
+ corpus spent deliberately on depth rather than breadth.
492
+
493
+ ## Going deeper
494
+
495
+ - **[FINDINGS.md](https://github.com/nikshepsvn/thimble/blob/master/FINDINGS.md)** — eleven negative results and two process
496
+ failures, with the measurement that killed each one. The most reusable part.
497
+ - **[REPRODUCING.md](https://github.com/nikshepsvn/thimble/blob/master/REPRODUCING.md)** — repository layout and the exact
498
+ pipeline that rebuilds the published numbers.
499
+ - **[RESULTS.md](https://github.com/nikshepsvn/thimble/blob/master/RESULTS.md)** the full chronological experimental record.
500
+ - **[paper/thimble.pdf](https://github.com/nikshepsvn/thimble/blob/master/paper/thimble.pdf)** the tech report: the co-design
501
+ thesis, the negative results, and the related work, in citable form.
502
+
503
+ ## Honest summary
504
+
505
+ Turning a request into calls against an API you control is a smaller problem than
506
+ the models usually pointed at it. Treated as a translation layer rather than a
507
+ language model, it fits in 48M parameters, comes with guarantees a prompted model
508
+ cannot offer, and can be specialized to one catalog for the price of a dinner.
509
+ It ships as a working system, not just a checkpoint: a one-file C engine with
510
+ byte-verified parity, a browser demo running the whole model client-side, and a
511
+ measured confidence gate for fronting a larger agent. Its limits are real and
512
+ measured rather than described. Built by one person over a few days with AI
513
+ assistance, for about the price of a video game console.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
514
 
515
  ## Citation
516