flashvenom commited on
Commit
424cfe8
·
verified ·
1 Parent(s): f2d64d6

docs: C engine exists now — retire the no-engine limitation, add torch-free inference section

Browse files
Files changed (1) hide show
  1. README.md +13 -4
README.md CHANGED
@@ -351,14 +351,23 @@ Two of these reversed conclusions that would otherwise have shipped on intuition
351
  `live_parallel` 0.0 on BFCL — repeated instantiations of one schema, as
352
  opposed to calls the query motivates in sequence.
353
  - **768-token context.** 151 of 3,641 BFCL rows (4.1%) do not fit and score as misses.
354
- - **Deployment.** 48.12M parameters is ~11.5MB at 2-bit and ~92MB at bf16, but
355
- what ships here is the 184MB fp32 checkpoint and there is no on-device
356
- inference engine. The size figure is a property of the parameter count, not of
357
- a runnable microcontroller artifact.
358
 
359
  Scale is the honest explanation for most of this: ~1B unique tokens, no
360
  pretraining phase, a corpus deliberately spent on depth instead of breadth.
361
 
 
 
 
 
 
 
 
 
 
362
  ## Model details
363
 
364
  | | |
 
351
  `live_parallel` 0.0 on BFCL — repeated instantiations of one schema, as
352
  opposed to calls the query motivates in sequence.
353
  - **768-token context.** 151 of 3,641 BFCL rows (4.1%) do not fit and score as misses.
354
+ - **Microcontrollers.** ~11.5MB at 2-bit is a property of the parameter count,
355
+ not a shippable artifact: 2-bit would need quantization-aware retraining this
356
+ model never had. The smallest thing that actually runs is the 48MB int8
357
+ build of the C engine below — Pi-class and up, not Cortex-M.
358
 
359
  Scale is the honest explanation for most of this: ~1B unique tokens, no
360
  pretraining phase, a corpus deliberately spent on depth instead of breadth.
361
 
362
+ ## Torch-free inference (C engine)
363
+
364
+ The repo ships [cengine/](https://github.com/nikshepsvn/thimble/tree/master/cengine),
365
+ the full decoder — tokenizer, trunk, grammar walk, name head — in one
366
+ dependency-free C file. 48MB int8 weights, ~20ms load, ~350ms per full request
367
+ on an Apple M3, and fp32 output is byte-identical to this checkpoint run
368
+ through the Python stack on 100/100 checked eval rows. Export the weights from
369
+ this repo's checkpoint with `cengine/export.py`.
370
+
371
  ## Model details
372
 
373
  | | |