SPARK.md

SPARK.md — Laguna-S-2.1 on DGX Spark (GB10)

DGX Spark operator notes only.

Hardware target: NVIDIA DGX Spark only (GB10, Grace+Blackwell, ~128 GB unified).
Do not run this pack’s claimed numbers on rented cloud hosts and relabel them as Spark.

Engine pin (measured on this Spark)

Item	Value
Repo	github.com/poolsideai/llama.cpp branch laguna
Commit	04b2b72cb54048ead292884adbe11f284e3ec950
Host patch	common/speculative.cpp: math.h + ::isfinite (GNU 13.3)
CUDA arch	121a (GB10)
Bin version string	version: 1 (04b2b72)

Stack

Layer	Choice
GPU	NVIDIA GB10 (CUDA sm_121 / cmake maps to 121a)
Driver / toolkit observed	CUDA 13.0, nvcc 13.0.88
Engine (required for Laguna GGUF + DFlash)	poolsideai/llama.cpp branch laguna
Optional engine	upstream llama.cpp ≥ b10087 (Laguna support landed on trail of #25165) — prefer poolside fork for DFlash
API	llama-server OpenAI-compatible (/v1/chat/completions, /v1/completions, /v1/models)
Optional	vLLM FP8/NVFP4 paths from Poolside card — not default for this pack; GGUF+llama.cpp is the measured path

Build (CUDA, aarch64)

export PATH=/usr/local/cuda/bin:$PATH
export CUDA_HOME=/usr/local/cuda
export CC=/usr/bin/gcc CXX=/usr/bin/g++ CUDAHOSTCXX=/usr/bin/g++

git clone https://github.com/poolsideai/llama.cpp llama.cpp-laguna
cd llama.cpp-laguna
git checkout 04b2b72cb54048ead292884adbe11f284e3ec950
# pin when publishing numbers:
# git rev-parse HEAD

cmake -B build -G "Unix Makefiles" \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_C_COMPILER=/usr/bin/gcc \
  -DCMAKE_CXX_COMPILER=/usr/bin/g++ \
  -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/g++ \
  -DGGML_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES=121 \
  -DLLAMA_CURL=ON

cmake --build build -j"$(nproc)" --target llama-server llama-cli llama-bench

Notes:

• CMake rewrites 121 → 121a on this toolkit.
• NCCL missing is OK on single Spark.
• No sudo required if toolchain already installed (CUDA toolkit + g++ + make + cmake).

Weights

Default stand-behind file (official Poolside bytes from authoritative upstream):

# preferred: ./scripts/pull_official_gguf.sh
hf download poolside/Laguna-S-2.1-GGUF \
  laguna-s-2.1-Q4_K_M.gguf \
  --revision fc4e481289523cf7d0df668da6d1d391616141ca \
  --local-dir $HOME/models/laguna-s-2.1
# optional byte-identical mirror: hizrianraz/Laguna-S-2.1-GGUF

sha256sum $HOME/models/laguna-s-2.1/laguna-s-2.1-Q4_K_M.gguf
# expect a8b55c75714ea73fd90ec85de5defdc0b8d88ca0ad2108343cdd8fc22f7583e4

Optional DFlash draft (upstream only today):

hf download poolside/Laguna-S-2.1-GGUF \
  laguna-s-2.1-DFlash-BF16.gguf \
  --local-dir $HOME/models/laguna-s-2.1

Disk: Q4_K_M ≈ 90–96 GB logical (HF race); keep ≥200 GB free for download + KV headroom.

Serve — OpenAI endpoint

./build/bin/llama-server \
  -m $HOME/models/laguna-s-2.1/laguna-s-2.1-Q4_K_M.gguf \
  --host 127.0.0.1 --port 8000 \
  --ctx-size 8192 \
  -ngl -1 \
  -fa on \
  --jinja \
  --alias local-laguna \
  --metrics

Headline claims use -ngl -1 (offload all / last-green pin). Older notes with -ngl 99 are not the measured captain path.

With DFlash (poolside fork) — optional only; DO_NOT_PROMOTE on this pack:

./build/bin/llama-server \
  -m $HOME/models/laguna-s-2.1/laguna-s-2.1-Q4_K_M.gguf \
  -md $HOME/models/laguna-s-2.1/laguna-s-2.1-DFlash-BF16.gguf \
  --spec-type draft-dflash --spec-draft-n-max 7 \
  --host 127.0.0.1 --port 8000 \
  --ctx-size 8192 -ngl -1 -fa on --jinja \
  --alias local-laguna

Check:

curl -s http://127.0.0.1:8000/v1/models | head
curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"laguna","messages":[{"role":"user","content":"ping"}],"max_tokens":16}'

Point Hermes-class agents at:

base_url = http://127.0.0.1:8000/v1
api_key  = sk-local   # unused by llama-server; placeholder

See hermes/sample_client.py.

Context presets

Preset	flags	use
2k smoke	--ctx-size 2048	quick agent_smoke / latency
8k default agent	--ctx-size 8192	measured one-response tool-protocol smoke
256k card default	--ctx-size 262144	large-context experiment (unmeasured; needs RAM headroom)
1M native	--ctx-size 1048576 --rope-scaling yarn --rope-scale 128 --yarn-orig-ctx 8192	quality may drop · chat-only sampling example temp 0.7 top_p 0.95 · not agent/smoke claim default (claims use temp 0.0)

Measure — tok/s + memory

llama-bench (prompt+gen)

./build/bin/llama-bench \
  -m $HOME/models/laguna-s-2.1/laguna-s-2.1-Q4_K_M.gguf \
  -ngl -1 -fa 1 \
  -p 2048,8192 -n 128 \
  -b 512

Server wall-clock (agent-shaped)

python scripts/bench_server.py \
  --base-url http://127.0.0.1:8000/v1 \
  --ctx-mark 2k --ctx-mark 8k \
  --out results/server_bench.json

Memory

While server is up:

# unified memory pressure (Spark)
free -h
nvidia-smi  # GB10 fields may show N/A for FB; use free/rss
ps -o pid,rss,cmd -p $(pgrep -f llama-server)

Record RSS of llama-server, host free mem before/after load, and ctx size.

Measured results (2026-07-28 Spark)

Host: DGX Spark · GB10 · CUDA 13.0 · engine 04b2b72 · weight Q4_K_M @ a8b55c75…
Serve proven: -c 8192 -ngl -1 --parallel 1 --alias local-laguna --jinja -fa on · load ~6.5 min · host ~96–99 Gi used after load (121 Gi total)

Setting	prefill latency	tg tok/s	server RSS	notes
ctx 8k, ngl -1, fa on	836 prompt → 1.597 s; 3236 prompt → 4.78 s	~21.47 @ 128 gen	~2–3.4 Gi process RSS	unified mem; sole headline gen = 21.47 (results/measured.json)
gen8 short	—	10.612	same	tiny completion noise floor — not a headline
ctx 8k + DFlash (draft-dflash)	2k 2.253s · 8k 5.556s	15.286 @ 128 gen	higher (draft+target)	trial -ngl 99 vs baseline -ngl -1 · DO_NOT_PROMOTE still (gen+prefill slower) · results/dflash_2026-07-29/

One-response tool-format/routing smoke — tools were validated, not executed; live tip durations bind:

Suite	quant	pass	n	notes
agent_smoke v1 (launch bar)	Q4_K_M	40/40 · 100%	40	84.86 s ·temp 0.0 · tip 2026-07-29 13:22 WIB · runner 3bb81080… · older same-day restores ~88–98 s are historical only
hermes_agent_smoke v2	Q4_K_M	27/27 · 100%	27	100.1 s · temp 0.0 · same tip · one-response · runner 20c1e52a… · results/hermes_agent_smoke.json
hermes Layer B v3 (research)	Q4_K_M	35/35 · 100%	35	137.56 s · 2026-07-29 · not freeze bar · receipt eval/hermes_agent_smoke/layer_b_v3_live_receipt.json
Raw: results/MEASURED.md, results/measured.json, results/agent_smoke.json, results/hermes_agent_smoke.json, results/server_bench.json.

Agent-shaped benches

Not MMLU. Fixed suites under eval/:

v1 — launch bar (eval/agent_smoke/, 40 cases):

1. tool JSON — emit valid tool call objects only from the offered schema
2. multi-step fixtures — choose the next response from a pre-seeded history; no tool loop is executed
3. error repair fixtures — respond to a pre-seeded tool error payload
4. no invented tools — refuse / answer without calling names not in schema
5. short code — small pure functions, exact stdout
6. constraint-history fixtures — preserve constraints from pre-seeded prior turns in one response

v2 — Hermes-class (eval/hermes_agent_smoke/, 27 cases):

Terminal / files / web / multi_tool / multi_turn / error_repair / no_invented / browser / memory_cron / args_strict / safety.
OpenAI tool shape only — not a Nous endorsement.

python eval/agent_smoke/run_smoke.py \
  --base-url http://127.0.0.1:8000/v1 \
  --model local-laguna \
  --out results/agent_smoke.json

python eval/hermes_agent_smoke/run_hermes_smoke.py \
  --base-url http://127.0.0.1:8000/v1 \
  --model local-laguna \
  --temperature 0 \
  --out results/hermes_agent_smoke.json

Claim temperature 0.0. Both suites are one-response (tools validated, not executed).

vLLM (optional, unmeasured default)

Poolside card documents FP8 / NVFP4 Docker mirrors. This pack’s default credibility path is GGUF + llama.cpp because:

• matches official GGUF story
• OpenAI-compatible with one binary
• reproducible on Spark without container registry lag

If you measure vLLM on Spark, add a dated subsection here with image digest + tok/s — do not overwrite GGUF numbers.

Forbidden host confusion

• Do not attribute rented-cloud jobs to this file.
• Do not mix non-local / third-party gateway base URLs into sample commands.
• HF handle only for publish (hizrianraz/*).