Neutrino-8B

An 8B chat model whose every transformer linear is stored five-valued (sub-2 bits per weight) in a single 2.56 GB container, and which loads as a native transformers model in one from_pretrained call.

This is the ONE published 8B SKU. Cross-model rows we publish are same-harness measurements: the rival's checkpoint run on our protocol, with thinking disabled on both sides.

• Container: neutrino-8b_v4.bin — TRTC v4, arch-3 (Qwen3 topology), 36 layers, hidden 4096, vocab 151936. 3,875,404,812 bytes, sha256 016c6f362ad237a8cf4043601efbc1e766574f76ea58691e78fdef51b4155fa0.
• Inside: 252 packed trit linears + int8 embedding lanes (untied). Nothing in the decode path is fp16/fp32 weights; scales are snapped to the runtime grid at export.
• Base: Qwen/Qwen3-8B (Apache-2.0), ternary QAT + staged post-training by Fermion Research. Tokenizer: Qwen3-8B (shipped in this repo).

Architecture

Geometry as read from the shipped container's header (bit-checked during the GGUF conversion, gguf/receipts/convert_8b.json; byte budget from the provenance-gated container walk, results_model_anatomy/ANATOMY.json):

field	value
Parameters	8,190,735,360 (6,945,767,424 ternary projection + 1,244,659,712 int8 embedding + 308,224 fp32 norm)
Decoder layers	36
Hidden width	4,096
Feed-forward width	12,288, gated (SwiGLU)
Attention	grouped-query 4:1 — 32 query heads, 8 KV heads, head_dim 128
Rotary embedding	full head width (rotary_dims 128), theta 1,000,000
Normalization	RMSNorm, eps 1e-6, plus per-head Q/K RMSNorm in attention
Context length	40,960 tokens (max_pos)
Vocabulary	151,936
Embeddings	untied — separate int8 input-embedding and lm_head records
KV cache	144 KiB/token fp16 (288 KiB fp32): 0.60 GB @ 4k, 4.83 GB @ 32k

Byte budget of the 3,875,404,812-byte container, by tensor class:

lane	bytes	share
ternary weight lane (q/k/v/o/gate/up/down × 36 layers = 252 linears)	2,604,662,784	67.2%
token embeddings, int8 (two untied 622 MB records)	1,244,659,712	32.1%
per-row metadata (dims, scales, row sums)	24,849,360	0.6%
norm vectors, fp32 (145 tensors)	1,232,896	0.03%
container header	60	—

Every layer costs exactly 72,351,744 weight-lane bytes (21.7% attention / 78.3% MLP). State occupancy across the 6.95B ternary weights: 62.63% zero / 18.68% plus / 18.69% minus (per-layer arrays in ANATOMY.json).

Formats and artifacts

artifact	bytes	sha256
neutrino-8b_v4.bin (TRTC v4 container, the file every runtime executes)	3,875,404,812	016c6f362ad237a8cf4043601efbc1e766574f76ea58691e78fdef51b4155fa0
neutrino-8b_v4.tv4z (lossless coded transport, 66.05% of raw)	2,559,836,297	83ec7a52bd6733e66fb546e89b9c6aa8b3411af7f648e9c0b2bb56ba50aac4df
gguf/neutrino-8b-fv5.gguf (llama.cpp-fork pack, bit-checked vs the container)	4,093,015,136	7dda995047dc70540d2d83f2c464dd118fff62eef1aab36ae71ceb3cc972e5d0

The tv4z transport decodes back to the container byte-exactly (md5+sha round-trip identity, results_tv4z_8b_ship/RESULTS.md; 17 s encode / 6 s decode on an M5). Three distribution surfaces read these artifacts:

1. pip engine (pip install fermion-research) — the one-command door; pulls the container and the platform-matching bin/ runtime from this repo. Measured: 24.94 tok/s CPU-only on an Apple M5, 9 threads.
2. GGUF pack + our llama.cpp fork (gguf/) — the llama.cpp-ecosystem door, CPU + CUDA. The pack uses our FV5 tensor type, so it loads through our fork only: build it once (two commands, below) and standard llama.cpp tooling works from there. Stock llama.cpp, ollama, and LM Studio builds do not include FV5 yet. Measured: 30.7 tok/s on an NVIDIA L4 at full offload, 4.68 GiB VRAM @ 4k context (fits 8 GB cards).
3. MLX pack (mlx/) — Python-native Apple-silicon runtime with custom Metal kernels. Measured: 25.0 tok/s median on a 16 GB M5 under a 6 GiB memory cap.

Quickstart

pip install fermion-research
printf 'In one sentence, why is the sky blue?\n/exit\n' | \
  fermion chat --model fermionresearch/Neutrino-8B --max-new 48

The first fermion chat downloads ~3.9 GB (container + tokenizer + native runtime) with no progress display in 0.1.5, so the terminal sits quiet for a few minutes on first run; later runs load from the local cache. (A progress display is coming in 0.1.6.)

Or pure transformers — download the repo first, because the 0.1.5 loader resolves the container relative to a local path only (passing the hub id straight to from_pretrained raises FileNotFoundError):

hf download fermionresearch/Neutrino-8B --local-dir Neutrino-8B \
    --exclude "gguf/*" --exclude "*.tv4z"   # skip the packs other runtimes use

import fermion  # registers the trtc_v4 model type
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("Neutrino-8B")   # the downloaded directory
tokenizer = AutoTokenizer.from_pretrained("Neutrino-8B")
inputs = tokenizer.apply_chat_template([{"role": "user", "content": "hi"}],
                                       add_generation_prompt=True,
                                       enable_thinking=False,
                                       return_tensors="pt", return_dict=True)
out = model.generate(**inputs, max_new_tokens=256)  # generation_config: greedy
n = inputs["input_ids"].shape[1]
print(tokenizer.decode(out[0][n:], skip_special_tokens=True))

import fermion must come first: it is what registers the trtc_v4 model type, and without it AutoTokenizer/AutoModelForCausalLM cannot read this repo's config.json. return_dict=True is not optional either on transformers 4.56 and newer, where apply_chat_template returns a mapping rather than a bare tensor.

If you skip the import you will get this, and the advice in it is a dead end:

ValueError: The checkpoint you are trying to load has model type `trtc_v4`
but Transformers does not recognize this architecture. ... You can update
Transformers with the command `pip install --upgrade transformers`.

Upgrading Transformers will never fix it, and neither will installing from source: trtc_v4 is registered at import time by the fermion-research package, so no Transformers release knows it. trust_remote_code=True does not help either, because this repo carries no auto_map. Add import fermion above the Transformers import.

generation_config.json in this repo is deliberately greedy with no repetition penalty and no top-k: that is the setting the published GSM8K, IFEval and BFCL numbers were measured at, so a bare generate(), an lm-eval run and fermion generate all reproduce each other. The conversational and long-form settings are one argument away — see Recommended settings.

GGUF surface (build our public fork once — branch fermion-fv5 at fermionresearch/llama.cpp, = upstream ggml-org/llama.cpp @ d67c0b41 + the FV5 patch — then standard llama.cpp tooling; full instructions in gguf/README.md):

hf download fermionresearch/Neutrino-8B gguf/neutrino-8b-fv5.gguf --local-dir .
git clone https://github.com/fermionresearch/llama.cpp && cd llama.cpp
git checkout fermion-fv5
cmake -B build -DCMAKE_BUILD_TYPE=Release -DLLAMA_CURL=OFF -DGGML_METAL=OFF  # no FV5 Metal kernel yet: CPU/CUDA only
cmake --build build -j --target llama-completion
./build/bin/llama-completion -m ../gguf/neutrino-8b-fv5.gguf \
    -p "Why is the sky blue?" -n 256 --temp 0 -no-cnv

MLX surface (Apple silicon — run from the mlx/ folder of the downloaded repo: the fermion_mlx package and its tokenizer files ship there, and requirements.txt installs mlx itself; full instructions in mlx/README.md):

cd Neutrino-8B/mlx          # the directory downloaded above
pip install -r requirements.txt
python -m fermion_mlx --model ../neutrino-8b_v4.bin --mode chat \
    --tokenizer . --prompt "Why is the sky blue?"

The pip path is the reference torch path: correctness-gated (bit-packed weights at rest, 0 greedy mismatches vs the native runtime over 768 tokens; receipt retained internally), honest about speed — it is NOT the fast path. The fast path ships in this repo under bin/ (see below).

Verified end-to-end in one fresh CPU container (Modal, cpu=8, 24 GiB, no GPU): wheel install -> container sha gate -> piped chat produced a real answer, verdict PASS. Peak resident memory for load+chat was under 8 GiB (bit-packed weights, max_child_rss_gib 7.87 in the receipt).

Recommended settings, by use case

Swept on a predecessor container; the sampling sweep was NOT re-run for 016c6f36… across four prompt slices — chat, factual, structured/tool-shaped, long-form — at four temperature x repetition-penalty cells. — means leave it alone; a number there would be one we did not measure.

use case	temperature	top-p	top-k	repetition penalty	penalty window	max new tokens	stop
conversation / assistant	0.01 on the C binary · 0 on torch	1.0	not exposed	1.05	256	512	EOS 151645
tool calling / structured output	0 (greedy)	—	—	1.0 (off)	—	256	EOS 151645
long-form prose	0.7	0.95	not exposed	1.05	256	1024	EOS 151645
deterministic / scriptable / benchmarks	0 (greedy)	—	—	1.0 (off)	—	256	EOS 151645

• Structured output is an 8B capability. On tool-shaped prompts this model emitted whole-reply, fence-free, parseable JSON 8/8 in every cell tried, including at temperature 0.7 — the format is not sensitive to the sampler here. Turn the repetition penalty off for it anyway: JSON requires repeating ", : and {, and a penalty over that punctuation is the one place it can actively break the format. The 0.6B-Chat scored 0/8 on the same prompts at every setting; route schema-shaped work here.
• Why 512 and 1024 token budgets. These are measured, not taste. At a 150-token cap the chat slice terminated 0/2 in every cell and every long-form generation hit the cap. The 8B is verbose; give it room.
• The repetition penalty is insurance, not a fix. Loop rate was 0.000 in all four cells, with and without it. It is in the conversation row to match what the behaviour receipts were graded at, and it is off everywhere reproducibility matters.
• top_p 0.95 on the long-form row is the one unmeasured number here — we swept top-p 1.0 only. It is the conventional pairing for temperature 0.7, flagged as convention rather than measurement.
• Top-k is not exposed on any runtime we ship, and this repo's generation_config.json sets none. Top-p is the only shortlist knob.
• --stop-id 151645 is mandatory on the C binary. Without it the binary runs in fixed-length "race mode", ignores EOS and emits exactly the token count you asked for.
• Speculative decoding and the exactness claim. Token-identical output is certified under greedy decoding (--temperature 0). With sampling, drafted output is drawn from the same distribution as undrafted output — the pip path routes through the transformers rejection-sampling correction — but individual tokens will differ. The research CUDA engine and the MLX --spec mode use argmax-match acceptance and are greedy-only.
• Exactly one repetition penalty ever applies on the torch path: the fermion CLI pins repetition_penalty=1.0 in its generate() call before installing its own windowed processor, so HF's full-context processor is never built alongside it. If you assemble your own call, do the same — two penalties of 1.05 compose to a measured 1.1025.

bin/ — prebuilt native runtimes (the fast path)

This repo bundles the prebuilt, closed-source fermion-run binaries next to the weights (the GGUF-ecosystem pattern; HF permits arbitrary binaries):

Binary	Platform	Backend
bin/fermion-run-macos-arm64	macOS arm64 (M1+)	CPU — NEON dotprod, zero dylib deps (Metal runtime is a week-one follow-up)
bin/fermion-run-linux-x64	Linux x86-64	CPU — AVX2 baseline, fully static ELF
bin/fermion-run-linux-cuda	Linux x86-64 + NVIDIA	CUDA — coming soon (not yet staged; today's CUDA fallback is the pip torch path)

Every binary ships with a <name>.sha256 sidecar (it holds a bare filename, so run the check from inside bin/); hf download writes files 0644, so mark the binary executable before running:

(cd bin && shasum -a 256 -c fermion-run-macos-arm64.sha256)
chmod +x bin/fermion-run-macos-arm64
xattr -d com.apple.quarantine bin/fermion-run-macos-arm64 2>/dev/null || true  # macOS only

pip install fermion-research remains the one door: the CLI downloads the model and the platform-matching binary from THIS repo and shells out to it for the fast path (the torch reference path stays as the fallback). Until the auto-binary CLI lands, the documented one-command binary path is in bin/README.md.

Shipped-binary speed, with venue: 24.94 tok/s CPU-only on an M5 MacBook (the macOS arm64 binary, 9 threads, 32-token greedy, banked 2026-07-25) and 25.0 tok/s median via the MLX pack on a 16 GB M5 under a 6 GiB memory cap (mlx/receipts/). These are shipped-binary numbers; our research-harness results are reported separately and never conflated with them.

Binaries are closed-source and covered by the EULA line below; speed receipts (venue+version+date on every number) are retained in our internal evaluation records. bin/MANIFEST.json in this tree is the staging manifest: it names exactly which binaries land at upload time and their provenance.

Benchmarks

Everything here is non-thinking: all our numbers, and every same-harness cell we publish for another model, are measured with thinking disabled. Release battery: the shipped brain, graded on our house harness; all five release guards (termination, IFEval floor, Redux, knowledge-delta, C4 ppl) passed before the container was baked. A receipt exists for every cell (venue, version, date) in our internal evaluation records.

Release battery of the ship brain, protocol on every row:

Meter	Score	Protocol
MMLU-Redux	67.84	generative, thinking off
IFEval, prompt-strict	73.17	generative, thinking off
IFEval, instruction-strict	80.22	same run, per-instruction grading
IFEval, prompt-loose	76.31	same run, loose extraction
BFCL v3	65.31	macro over 13 subsets, bfcl-eval 2025.10.27.1, thinking off
GSM8K, flexible-extract	51.00	0-shot generative, greedy, 256-token cap
GSM8K, strict/stated format	49.33	same run, answer must appear in the stated format

Same-harness reference points for the strongest same-byte-class open model (Ternary-Bonsai-8B, measured by us on identical protocols): MMLU-Redux 71.47 · IFEval prompt-strict 83.65 · BFCL v3 71.45 · GSM8K flexible 35.00 · GSM8K stated-format 39.67. It leads us on instruction following, tool calling and MMLU-Redux; we lead on GSM8K flexible extraction (51.00 vs 35.00) and on GSM8K stated-format extraction (49.33 vs 39.67).

Reproducing the battery

The model registers as a native transformers model (import fermion), so the public harnesses run directly. GSM8K was graded with lm-eval:

# first: hf download fermionresearch/Neutrino-8B --local-dir Neutrino-8B  (see Quickstart)
import fermion, lm_eval
from lm_eval.models.huggingface import HFLM
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("Neutrino-8B")   # the downloaded directory
tok = AutoTokenizer.from_pretrained("Neutrino-8B")
lm = HFLM(pretrained=model, tokenizer=tok)
lm_eval.simple_evaluate(model=lm, tasks=["gsm8k"], num_fewshot=0)  # flexible-extract

IFEval, MMLU-Redux, and BFCL v3 were graded with evalscope 1.4.2 against a vLLM OpenAI-compatible endpoint serving this model, with chat_template_kwargs: {"enable_thinking": false} and bfcl-eval==2025.10.27.1 pinned:

pip install evalscope==1.4.2 bfcl-eval==2025.10.27.1
evalscope eval --model Neutrino-8B --api-url http://127.0.0.1:8000/v1 \
    --datasets ifeval mmlu_redux bfcl_v3

The exact per-leg configs (context lengths 8k/16k, generation configs, the BFCL 13-subset list) and every raw report JSON are in the internal evaluation records.

Meter	Neutrino-8B	Same-harness comparison	Receipt
GSM8K flexible-extract (0-shot generative, greedy, 256-token cap, fixed 300-item subset)	51.00	Ternary-Bonsai-8B 35.0 on the byte-identical meter	internal evaluation records
GSM8K stated-format (same run, answer must land in the prescribed format; 1024-token cap)	49.33	Ternary-Bonsai-8B 39.67 on the byte-identical meter — this column is ours as of this brain	internal evaluation records

Don't-regress guards, held at release (guards, not headlines): GSM8K generative termination 0.51 — above the bf16 teacher's 0.42 anchor on the same raw-completion meter — and C4 perplexity 21.48 (base 21.40 — flat). The raw-completion protocol suppresses EOS across model families, ours and fp16 alike; chat-templated serving terminates normally.

Tool calling is a trained, measured capability of this brain, stated here as our own absolute numbers (no comparison rows on this axis): BFCL v3 13-subset macro 65.31 and IFEval prompt-strict 73.17 (bfcl-eval 2025.10.27.1, our harness, thinking off).

Weights: get them and verify them

The container ships in this repository. Fetch it and check the digest — MANIFEST.json carries the same sha256, and fermion info verifies it for you automatically:

hf download fermionresearch/Neutrino-8B neutrino-8b_v4.bin --local-dir .
shasum -a 256 neutrino-8b_v4.bin         # must print 016c6f362...b4155fa0

Product bake provenance: built from Qwen3-8B via ternary QAT and staged post-training by Fermion Research, exported to the shipped TRTC v4 container.

Runtime matrix

Single-stream decode, one artifact across every row (venue + date on each number in the internal evaluation records):

Platform	Surface	Rate	Memory
H100 80 GB	research stack, drafted (0.6B draft, certified exact)	763 tok/s (counting; 402-500 tok/s on other prompt classes)	—
H100 80 GB	research stack, plain greedy	396 tok/s	—
NVIDIA L4	GGUF pack via our CUDA fork, full offload	30.7 tok/s	4.68 GiB @ 4k ctx
Apple M5, 16 GB	MLX pack (median, 3 runs)	25.0 tok/s	3.67 GiB peak, 6 GiB cap
Apple M5	bin/ native binary, CPU only, 9 threads	24.94 tok/s	< 8 GiB resident

Speculative decoding (certified exact)

This repo's 8B pairs with Neutrino-0.6B-Chat as a draft model for speculative decoding: the 0.6B proposes tokens, the 8B verifies them, and the output is token-identical to plain 8B greedy decode — speculation changes throughput, never the answer. Both models are the same TRTC v4 container format and run on the same fermion-run binaries, so the pairing needs no extra conversion.

Meter	Value	Note
Certification (exact-match tokens)	27,648 spec tokens, 0 mismatches (receipt receipts/dspec_cert_cons3_b25.json, re-run against THIS container 2026-07-27; 54/54 arm x prompt cells identical)	draft+verify vs plain greedy, N tokens, 0 mismatches required
Counting-style prompts	×1.92 (763 tok/s vs 397 plain)	measured speedup vs plain 8B greedy
Factual / short-answer prompts	×1.26 (500 tok/s)	"
Code	×1.10 (437 tok/s)	"
Prose continuation	×1.06 (422 tok/s)	"
Conversational explanation	×1.01 (402 tok/s)	"
Venue	H100, 3-round median, dynamic-k controller, 2026-07-27	same-moment, same-machine protocol (see VERIFY)

Mechanics (measured on this brain, 2026-07-27 cert run): on counting prompts the 8B still accepts 100% of drafted tokens; on factual prompts acceptance is 86.9% and on chat-factual 90.3%; it falls to 81.4% on prose and 50.0% on technical explanation. Acceptance decays with draft length everywhere except counting, so the dynamic-k controller shortens its drafts there and the gain thins.

Receipts: receipts/dspec_cert_cons3_b25.json (banked, ship brain). Honest framing: speculative decoding is a throughput feature with a correctness certificate; it does not change quality, and the speedup is prompt-dependent — the five rows above are the honest spread, not a single headline number.

Notes

• The GGUF pack loads only through our llama.cpp fork (gguf/) until upstreaming lands; stock llama.cpp, ollama, and LM Studio binaries do not know the FV5/FV5B tensor types.
• This is a non-thinking model: it ships and is evaluated with thinking disabled (enable_thinking=False).

Model lineage and methodology

Built from Qwen3-8B via ternary QAT and staged post-training by Fermion Research; the engineering story of the format and the engine is on the research pages at fermionresearch.com/research/.

License and attribution

Weights: Apache-2.0. Derivative of Qwen/Qwen3-8B (Apache-2.0, Alibaba Cloud) — see LICENSE. Post-training data sources are disclosed in the internal evaluation records (UltraChat, MetaMath, and the other accepted-source datasets).

bin/ binaries: prebuilt, closed-source, free to use with these weights; no redistribution outside this repo; no reverse engineering. [EULA short text — final wording lands with the binaries at upload; RELEASE_RUNBOOK §6.]

Contact: contact@fermionresearch.com