FoolDev Claude Fable 5 commited on
Commit
d8ef231
Β·
1 Parent(s): 0b6774a

feat: advertise 262,144 native / 1,010,000 extensible context; bump default num_ctx

Browse files

State the Qwen 3.6 context spec precisely instead of rounding to "~1 M":
262,144 native, extensible to 1,010,000 tokens with YaRN. The baked default
num_ctx moves 1000000 -> 1010000 in params and the Modelfile (bridge-synced),
superseding the earlier "match the Fable 1M budget" value. Retune the opt-in
YaRN recipe to --rope-scale 3.853 (1010000/262144) at -c 1010000 and recompute
the theoretical footprint at the default to ~62 GB KV / ~100 GB total. Native
(262,144) figures and all override examples are unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Files changed (4) hide show
  1. CHANGELOG.md +10 -0
  2. Modelfile +7 -7
  3. README.md +7 -7
  4. params +1 -1
CHANGELOG.md CHANGED
@@ -31,6 +31,16 @@ track the **tooling and documentation**, not the underlying base model.
31
  remains in the README "Recommended sampling" table as the tighter-reasoning
32
  override.
33
 
 
 
 
 
 
 
 
 
 
 
34
  ### Fixed
35
  - **Hardware tables agree on the CPU-only RAM floor.** The README hardware
36
  table lists `β‰₯48 GB RAM (CPU-only)` while the `Modelfile` working-configs
 
31
  remains in the README "Recommended sampling" table as the tighter-reasoning
32
  override.
33
 
34
+ ### Changed (context-length ceiling raised to Qwen's 1,010,000 extensible max)
35
+ - **Advertised context is now the precise 262,144-native / 1,010,000-extensible
36
+ pair, and the baked default rises to match.** The card previously rounded the
37
+ extensible window to `~1 M` and baked `num_ctx 1000000` (a Fable-teacher budget
38
+ match); both now use the exact Qwen 3.6 figures β€” **262,144 native, extensible
39
+ to 1,010,000 tokens with YaRN**. The baked default moves `1000000 β†’ 1010000`
40
+ in `params` and the `Modelfile` (bridge-synced), the opt-in YaRN recipe is
41
+ retuned to `--rope-scale 3.853` (β‰ˆ `1010000 / 262144`) at `-c 1010000`, and the
42
+ theoretical footprint at the default is recomputed to ~62 GB KV / ~100 GB total.
43
+
44
  ### Fixed
45
  - **Hardware tables agree on the CPU-only RAM floor.** The README hardware
46
  table lists `β‰₯48 GB RAM (CPU-only)` while the `Modelfile` working-configs
Modelfile CHANGED
@@ -71,7 +71,7 @@ PARAMETER temperature 1.0
71
  PARAMETER top_p 0.95
72
  PARAMETER top_k 0
73
  PARAMETER repeat_penalty 1.05
74
- PARAMETER num_ctx 1000000
75
 
76
  # Stop tokens. Without these, Ollama only honors <|im_end|> from the GGUF
77
  # metadata; the model occasionally emits <|endoftext|> instead and Ollama
@@ -99,23 +99,23 @@ Behavior rules:
99
  # This Q4_K_M is ~19 GB on disk. Real footprint at runtime:
100
  # weights mmap ~19 GB
101
  # compute graph alloc ~19 GB (Ollama log: device.go:272 "total memory")
102
- # KV cache @ 262K ctx ~16 GB (q8_0, ~2 GB / 32K; so ~61 GB at the 1M default)
103
- # total minimum ~99 GB at the 1000000 default / ~53 GB at 262144 native
104
  # (theoretical, extrapolated from ~2 GB/32K; the
105
- # default num_ctx is 1000000 β€” a ~1M ceiling ABOVE the
106
  # 262144 native context. No YaRN rope-scaling is
107
  # baked in this GGUF, so output past ~262K degrades;
108
- # treat 1M as an advertised ceiling. Most
109
  # hosts must override num_ctx down, see below)
110
  #
111
  # Working configurations (rows assume num_ctx trimmed to a practical ~16-32K;
112
- # at the 1M default the ~99 GB footprint fits only 128 GB+ hosts β€” override down
113
  # everywhere else, see below):
114
  # βœ“ Single H100 80GB / A100 80GB β€” full GPU offload
115
  # βœ“ RTX 5090 32GB / RTX 4090 24GB β€” partial offload, ~15-25 tok/s
116
  # βœ“ Mac Studio M2/M3 Ultra 64GB+ β€” unified memory, ~20+ tok/s
117
  # βœ“ Linux box with 48GB+ RAM (CPU-only) β€” ~3-6 tok/s
118
- # ⚠ ASUS ROG Flow Z13 (Ryzen AI Max+, 32GB) β€” OOMs at the 1M default (and
119
  # above ~32K); fits with num_ctx
120
  # ≀ 4096 and num_batch ≀ 256 (verified)
121
  #
 
71
  PARAMETER top_p 0.95
72
  PARAMETER top_k 0
73
  PARAMETER repeat_penalty 1.05
74
+ PARAMETER num_ctx 1010000
75
 
76
  # Stop tokens. Without these, Ollama only honors <|im_end|> from the GGUF
77
  # metadata; the model occasionally emits <|endoftext|> instead and Ollama
 
99
  # This Q4_K_M is ~19 GB on disk. Real footprint at runtime:
100
  # weights mmap ~19 GB
101
  # compute graph alloc ~19 GB (Ollama log: device.go:272 "total memory")
102
+ # KV cache @ 262K ctx ~16 GB (q8_0, ~2 GB / 32K; so ~62 GB at the 1.01M default)
103
+ # total minimum ~100 GB at the 1010000 default / ~53 GB at 262144 native
104
  # (theoretical, extrapolated from ~2 GB/32K; the
105
+ # default num_ctx is 1010000 β€” a ~1.01M ceiling ABOVE the
106
  # 262144 native context. No YaRN rope-scaling is
107
  # baked in this GGUF, so output past ~262K degrades;
108
+ # treat 1.01M as an advertised ceiling. Most
109
  # hosts must override num_ctx down, see below)
110
  #
111
  # Working configurations (rows assume num_ctx trimmed to a practical ~16-32K;
112
+ # at the 1.01M default the ~100 GB footprint fits only 128 GB+ hosts β€” override down
113
  # everywhere else, see below):
114
  # βœ“ Single H100 80GB / A100 80GB β€” full GPU offload
115
  # βœ“ RTX 5090 32GB / RTX 4090 24GB β€” partial offload, ~15-25 tok/s
116
  # βœ“ Mac Studio M2/M3 Ultra 64GB+ β€” unified memory, ~20+ tok/s
117
  # βœ“ Linux box with 48GB+ RAM (CPU-only) β€” ~3-6 tok/s
118
+ # ⚠ ASUS ROG Flow Z13 (Ryzen AI Max+, 32GB) β€” OOMs at the 1.01M default (and
119
  # above ~32K); fits with num_ctx
120
  # ≀ 4096 and num_batch ≀ 256 (verified)
121
  #
README.md CHANGED
@@ -89,7 +89,7 @@ ollama create janus -f Modelfile && ollama run janus
89
 
90
  After either path, `ollama show janus` lists `completion`, `tools`,
91
  and `thinking` under Capabilities. Hardware: the default `num_ctx` is
92
- **1000000** β€” a ~1M ceiling above the 262144 native window (YaRN is not baked
93
  into this GGUF, so context past ~262K degrades) β€” so trim it down to fit your
94
  host (see [Hardware requirements](#hardware-requirements)).
95
 
@@ -125,7 +125,7 @@ GGUF-only release. Pull the Heretic safetensors from [`llmfan46/Qwen3.6-35B-A3B-
125
 
126
  - Qwen 3.6, 35B total / 3B active, MoE (256 experts, 8 activated per token)
127
  - 40 layers, 10 Γ— (3 Γ— DeltaNet β†’ MoE / 1 Γ— Gated Attention β†’ MoE)
128
- - 262 144 native context (extensible to ~1 M with YaRN, but YaRN is not enabled in the bundled GGUF)
129
  - Vision + video supported by upstream (mmproj not included in this release)
130
  - Vocab 248,320
131
 
@@ -295,7 +295,7 @@ needed for vision.
295
 
296
  ## Hardware requirements
297
 
298
- This is a ~19 GB Q4_K_M GGUF. Ollama's runtime footprint is **roughly 2Γ— the model file** (weights mmap + compute graph), plus a KV cache that scales ~2 GB per 32K (q8_0). The default `num_ctx` is **1000000** β€” a ~1M ceiling above the 262144 native window β€” so KV alone is ~61 GB for ~99 GB total (theoretical, extrapolated from the ~2 GB/32K rule). This GGUF ships **no YaRN rope-scaling** (`rope.freq_base 10M`, no `rope.scaling`), so positions past the 262144 native window use untrained RoPE and output degrades β€” treat 1M as an advertised ceiling and keep real work within ~262K. **Most hosts must override `num_ctx` down**: e.g. the 262144 native window β†’ ~16 GB KV / ~53 GB total, or `num_ctx 32768` β†’ ~2 GB KV / ~39 GB total. 32 GB hosts fit the model by trimming ctx + batch (see Z13 row in the table).
299
 
300
  **How to override it:** `ollama run` has no `-o` flag, and `OLLAMA_CONTEXT_LENGTH`
301
  only sets a default that the baked `num_ctx` overrides β€” so set it per-session
@@ -317,16 +317,16 @@ Programmatic callers pass the same via the API `options` field:
317
  | Single H100/A100 80 GB | Works, full offload, ~30+ tok/s |
318
  | RTX 4090 24 GB / 5090 32 GB + 32 GB RAM | Works, partial offload, ~15-25 tok/s |
319
  | Mac Studio M2/M3 Ultra 64 GB+ unified | Works, ~20+ tok/s |
320
- | 32 GB unified-memory laptops (Ryzen AI Max+, Apple M-series) | Works with `num_ctx ≀ 4096` and `num_batch ≀ 256` to fit the compute graph; the 1M default OOMs (override `num_ctx` down). Measured 28.71 tok/s on ASUS ROG Flow Z13 GZ302EA at Q4_K_M (Radeon 8060S iGPU via ROCm gfx1151). |
321
 
322
- **Reaching a coherent ~1M context (opt-in YaRN).** The bundled GGUF ships no YaRN rope-scaling, so the 1M default degrades past the 262144 native window (see above). Ollama has no rope knob, so for a genuinely coherent long context run the GGUF under llama.cpp with YaRN enabled:
323
 
324
  ```bash
325
  llama-server -m Janus-35B-A3B.Q4_K_M.gguf \
326
- --rope-scaling yarn --yarn-orig-ctx 262144 --rope-scale 3.8 -c 1000000
327
  ```
328
 
329
- `--rope-scale 3.8` β‰ˆ 1000000 / 262144; use a smaller factor for a smaller window. Static YaRN rescales *all* prompts, so enable it only when you actually need > 262K β€” it slightly degrades short-context quality otherwise.
330
 
331
  ## Chat template
332
 
 
89
 
90
  After either path, `ollama show janus` lists `completion`, `tools`,
91
  and `thinking` under Capabilities. Hardware: the default `num_ctx` is
92
+ **1010000** β€” a ~1.01M ceiling above the 262144 native window (YaRN is not baked
93
  into this GGUF, so context past ~262K degrades) β€” so trim it down to fit your
94
  host (see [Hardware requirements](#hardware-requirements)).
95
 
 
125
 
126
  - Qwen 3.6, 35B total / 3B active, MoE (256 experts, 8 activated per token)
127
  - 40 layers, 10 Γ— (3 Γ— DeltaNet β†’ MoE / 1 Γ— Gated Attention β†’ MoE)
128
+ - 262 144 native context (extensible to 1 010 000 with YaRN, but YaRN is not enabled in the bundled GGUF)
129
  - Vision + video supported by upstream (mmproj not included in this release)
130
  - Vocab 248,320
131
 
 
295
 
296
  ## Hardware requirements
297
 
298
+ This is a ~19 GB Q4_K_M GGUF. Ollama's runtime footprint is **roughly 2Γ— the model file** (weights mmap + compute graph), plus a KV cache that scales ~2 GB per 32K (q8_0). The default `num_ctx` is **1010000** β€” a ~1.01M ceiling above the 262144 native window β€” so KV alone is ~62 GB for ~100 GB total (theoretical, extrapolated from the ~2 GB/32K rule). This GGUF ships **no YaRN rope-scaling** (`rope.freq_base 10M`, no `rope.scaling`), so positions past the 262144 native window use untrained RoPE and output degrades β€” treat 1.01M as an advertised ceiling and keep real work within ~262K. **Most hosts must override `num_ctx` down**: e.g. the 262144 native window β†’ ~16 GB KV / ~53 GB total, or `num_ctx 32768` β†’ ~2 GB KV / ~39 GB total. 32 GB hosts fit the model by trimming ctx + batch (see Z13 row in the table).
299
 
300
  **How to override it:** `ollama run` has no `-o` flag, and `OLLAMA_CONTEXT_LENGTH`
301
  only sets a default that the baked `num_ctx` overrides β€” so set it per-session
 
317
  | Single H100/A100 80 GB | Works, full offload, ~30+ tok/s |
318
  | RTX 4090 24 GB / 5090 32 GB + 32 GB RAM | Works, partial offload, ~15-25 tok/s |
319
  | Mac Studio M2/M3 Ultra 64 GB+ unified | Works, ~20+ tok/s |
320
+ | 32 GB unified-memory laptops (Ryzen AI Max+, Apple M-series) | Works with `num_ctx ≀ 4096` and `num_batch ≀ 256` to fit the compute graph; the 1.01M default OOMs (override `num_ctx` down). Measured 28.71 tok/s on ASUS ROG Flow Z13 GZ302EA at Q4_K_M (Radeon 8060S iGPU via ROCm gfx1151). |
321
 
322
+ **Reaching a coherent ~1.01M context (opt-in YaRN).** The bundled GGUF ships no YaRN rope-scaling, so the 1.01M default degrades past the 262144 native window (see above). Ollama has no rope knob, so for a genuinely coherent long context run the GGUF under llama.cpp with YaRN enabled:
323
 
324
  ```bash
325
  llama-server -m Janus-35B-A3B.Q4_K_M.gguf \
326
+ --rope-scaling yarn --yarn-orig-ctx 262144 --rope-scale 3.853 -c 1010000
327
  ```
328
 
329
+ `--rope-scale 3.853` β‰ˆ 1010000 / 262144; use a smaller factor for a smaller window. Static YaRN rescales *all* prompts, so enable it only when you actually need > 262K β€” it slightly degrades short-context quality otherwise.
330
 
331
  ## Chat template
332
 
params CHANGED
@@ -3,7 +3,7 @@
3
  "top_p": 0.95,
4
  "top_k": 0,
5
  "repeat_penalty": 1.05,
6
- "num_ctx": 1000000,
7
  "stop": [
8
  "<|im_end|>",
9
  "<|endoftext|>",
 
3
  "top_p": 0.95,
4
  "top_k": 0,
5
  "repeat_penalty": 1.05,
6
+ "num_ctx": 1010000,
7
  "stop": [
8
  "<|im_end|>",
9
  "<|endoftext|>",