Tetsuto/taxi-nl-3b-gguf

taxi-nl-3b — Natural Language → Taxi schema

A fine-tuned Qwen2.5-Coder-3B that translates plain-English requirements into Taxi schema code (the language used by Orbital).

Why this model exists. Strong general code models (Qwen3-Coder, Qwen2.5-Coder-32B) score 19–30% on Taxi generation in plain-prompt mode because they invent TypeScript-flavored syntax (type X = string) instead of Taxi's inherits String. Context-stuffing with a Taxi grammar primer + examples lifts those models to 67–80%, but pays the context tax on every call (multi-thousand-token prompts, ~7 sec/query at 32B). This model has Taxi baked in — single-sentence prompts produce idiomatic, compiling Taxi schemas at 1–2 sec/query.

This release (v7). A validator-in-the-loop fine-tune: each new training example was generated by a prior checkpoint and accepted only if it cleanly parsed against the strict taxilang compiler, with no undefined references and no duplicate declarations. The point of v5 is schema completeness — earlier versions sometimes referenced types they hadn't declared, or emitted patterns like @PII in syntactically invalid positions. v7 builds on v5 with 12 hand-authored anchors targeting service-with-multiple-operations, query+projection completeness, and clean @HttpOperation context — fixing schema completeness while pushing benchmark scores higher while preserving the surface-pattern coverage (annotations, polymorphism via inherits, services with @HttpOperation).

Headline benchmark (100-entry held-out set, 40 easy / 30 schema-aware / 30 open-ended)

Model	overall	easy	schema-aware	open-ended	s/query
taxi-nl-3b v7 MLX (4-bit)	94%	95	97	90	1.1
taxi-nl-3b v7 Q4 GGUF (CPU)	86%	90	90	77	2.8
qwen2.5-coder:32b + context-stuffing	80%	90	97	50	6.7
qwen3-coder-next + context-stuffing	72%	90	90	30	2.6
qwen2.5-coder:7b + context-stuffing	67%	70	87	43	1.8
qwen2.5-coder:32b plain	30%	28	63	0	5.9
qwen2.5-coder:7b plain	24%	38	30	0	1.5

The Q4 GGUF in this repo is the deployable artifact for CPU inference; based on prior quantisation runs in this lineage, expect ~5–8 pp below the MLX 4-bit number, concentrated in the open-ended bucket (multi-block compositions are most weight-precision sensitive). Even quantised, it sits well above the context-stuffed 32B bar.

Metric: compile pass rate against the strict taxilang compiler (catches both syntactic and semantic errors — unresolved type references, duplicate symbols, type mismatches).

Real-world dogfood — 20/20

Two independent sets of 10 plain-English prompts each (one canonical, one held-out and never seen during any phase of generation or validation), targeting realistic schema patterns: Stripe-style subscriptions, FIX messages, healthcare claims with @PII, REST services with @HttpOperation, polymorphic events with inherits, multi-model insurance policies, query projections, nested arrays, type reuse, RBAC. Every prompt produced compiling Taxi. v2 scored 5/10 on the canonical set; v3 scored 9/10; v5 hit 10/10 on both; v7 trades 2 dogfood for a 5 pp benchmark jump — see numbers above.

Training recipe

• Base: mlx-community/Qwen2.5-Coder-3B-Instruct-4bit
• Method: LoRA (mlx_lm.lora)
• Hyperparameters: --num-layers 16 --iters 1500 --learning-rate 2e-5 --batch-size 1 --max-seq-length 2048
• Trainable params: ~0.1% (3.3M / 3086M)
• Hardware: Apple M4, 24 GB unified memory; ~6 minutes total
• Final loss: train 0.43, val 0.32 (val-loss minimum at iter 1500; iter-2000 overfit, val 0.43 — 1500 ships)

Training data

Roughly 4,500 (description, taxi) pairs:

• Reverse-description (1,268 pairs): for each validated upstream Taxi snippet, a coder model wrote 3 NL descriptions in distinct styles (terse / task-card / doc-comment).
• Forward synthesis (5,000 pairs): generated jointly given a domain × construct bucket; each candidate validated through the strict compiler with one self-correct retry on failure.
• Hand-authored anchors (32 pairs): narrow, precise examples of the patterns earlier versions missed — @PII placement, @HttpOperation inside service, polymorphism via inherits.
• Targeted synthesis (320 pairs): Qwen-27B variations of the anchors to teach the patterns with diversity.
• Validator-in-the-loop / RFT (631 pairs): generated by a prior checkpoint, kept only when they parsed cleanly with no undefined references and no duplicate declarations.

After dedup against the held-out benchmark gold and chat formatting: ~3,700 train / ~460 valid pairs.

Inference

CLI (recommended)

pip install stackfix  # installs the `taxify` command
taxify "a Customer model with id and email"

Python (llama-cpp-python)

from llama_cpp import Llama
SYS = ("You translate natural-language requirements into idiomatic Taxi schema code. "
       "Taxi is the schema language used by Orbital (orbitalhq.com). "
       "Return ONLY the Taxi source.")
llm = Llama(model_path="taxi-nl-3b-q4.gguf", n_ctx=4096, verbose=False)
resp = llm.create_chat_completion(messages=[
    {"role": "system", "content": SYS},
    {"role": "user", "content": "Define a Customer model with id and email"},
])
print(resp["choices"][0]["message"]["content"])

With existing Taxi context

taxify "Add an Order service that fetches orders by CustomerId" --schema customer.taxi

The CLI applies a post-processor that strips any blocks the model produces whose declared symbol already exists in the in-context schema — the fine-tune occasionally replays context verbatim, and this dedup keeps output clean.

Files

• taxi-nl-3b-q4.gguf — Q4_K_M quantization, 1.8 GB, recommended for CPU.
• taxi-nl-3b-f16.gguf — full f16, 6.2 GB, for evaluation parity.

Limitations

• Out-of-distribution prompts (e.g., "design a healthcare claims system using OpenBanking conventions") fall back toward the base model's behaviour. Best for "translate this concrete schema description" rather than open-ended design.
• Reserved word handling is imperfect — names like from as parameter names occasionally trip the parser.
• TaxiQL projections (find { X[] } as { ... }) work for simple cases; complex projection rewrites are still best-effort.
• The post-process dedup is a heuristic (exact symbol-name match). If you genuinely want the schema repeated, set --no-dedup (CLI) or skip the helper.

License

Apache 2.0 (matching the base model's license).

Citation

@misc{taxi-nl-3b,
  author    = {Cloud-Gym},
  title     = {taxi-nl-3b: a small fine-tuned model for NL→Taxi translation},
  year      = {2026},
  publisher = {HuggingFace},
}