# SPARK.md — Laguna-S-2.1 on DGX Spark (GB10)

DGX Spark operator notes only.

Hardware target: **NVIDIA DGX Spark only** (GB10, Grace+Blackwell, ~128 GB unified).  
Do not run this pack’s claimed numbers on rented cloud hosts and relabel them as Spark.

## Engine pin (measured on this Spark)

| Item | Value |
|------|--------|
| Repo | `github.com/poolsideai/llama.cpp` branch `laguna` |
| Commit | `04b2b72cb54048ead292884adbe11f284e3ec950` |
| Host patch | `common/speculative.cpp`: `math.h` + `::isfinite` (GNU 13.3) |
| CUDA arch | `121a` (GB10) |
| Bin version string | `version: 1 (04b2b72)` |


## Stack

| Layer | Choice |
|-------|--------|
| GPU | NVIDIA GB10 (CUDA sm_121 / cmake maps to `121a`) |
| Driver / toolkit observed | CUDA 13.0, nvcc 13.0.88 |
| Engine (required for Laguna GGUF + DFlash) | [poolsideai/llama.cpp](https://github.com/poolsideai/llama.cpp) branch **`laguna`** |
| Optional engine | upstream llama.cpp **≥ b10087** (Laguna support landed on trail of #25165) — prefer poolside fork for DFlash |
| API | `llama-server` OpenAI-compatible (`/v1/chat/completions`, `/v1/completions`, `/v1/models`) |
| Optional | vLLM FP8/NVFP4 paths from Poolside card — **not** default for this pack; GGUF+llama.cpp is the measured path |

## Build (CUDA, aarch64)

```bash
export PATH=/usr/local/cuda/bin:$PATH
export CUDA_HOME=/usr/local/cuda
export CC=/usr/bin/gcc CXX=/usr/bin/g++ CUDAHOSTCXX=/usr/bin/g++

git clone https://github.com/poolsideai/llama.cpp llama.cpp-laguna
cd llama.cpp-laguna
git checkout 04b2b72cb54048ead292884adbe11f284e3ec950
# pin when publishing numbers:
# git rev-parse HEAD

cmake -B build -G "Unix Makefiles" \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_C_COMPILER=/usr/bin/gcc \
  -DCMAKE_CXX_COMPILER=/usr/bin/g++ \
  -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/g++ \
  -DGGML_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES=121 \
  -DLLAMA_CURL=ON

cmake --build build -j"$(nproc)" --target llama-server llama-cli llama-bench
```

Notes:

- CMake rewrites `121` → `121a` on this toolkit.
- NCCL missing is OK on single Spark.
- No sudo required if toolchain already installed (CUDA toolkit + g++ + make + cmake).

## Weights

Default stand-behind file (official Poolside bytes from authoritative upstream):

```bash
# preferred: ./scripts/pull_official_gguf.sh
hf download poolside/Laguna-S-2.1-GGUF \
  laguna-s-2.1-Q4_K_M.gguf \
  --revision fc4e481289523cf7d0df668da6d1d391616141ca \
  --local-dir $HOME/models/laguna-s-2.1
# optional byte-identical mirror: hizrianraz/Laguna-S-2.1-GGUF

sha256sum $HOME/models/laguna-s-2.1/laguna-s-2.1-Q4_K_M.gguf
# expect a8b55c75714ea73fd90ec85de5defdc0b8d88ca0ad2108343cdd8fc22f7583e4
```

Optional DFlash draft (upstream only today):

```bash
hf download poolside/Laguna-S-2.1-GGUF \
  laguna-s-2.1-DFlash-BF16.gguf \
  --local-dir $HOME/models/laguna-s-2.1
```

Disk: Q4_K_M ≈ 90–96 GB logical (HF race); keep ≥200 GB free for download + KV headroom.

## Serve — OpenAI endpoint

```bash
./build/bin/llama-server \
  -m $HOME/models/laguna-s-2.1/laguna-s-2.1-Q4_K_M.gguf \
  --host 127.0.0.1 --port 8000 \
  --ctx-size 8192 \
  -ngl -1 \
  -fa on \
  --jinja \
  --alias local-laguna \
  --metrics
```

Headline claims use **`-ngl -1`** (offload all / last-green pin). Older notes with `-ngl 99` are not the measured captain path.

With DFlash (poolside fork) — optional only; **DO_NOT_PROMOTE** on this pack:

```bash
./build/bin/llama-server \
  -m $HOME/models/laguna-s-2.1/laguna-s-2.1-Q4_K_M.gguf \
  -md $HOME/models/laguna-s-2.1/laguna-s-2.1-DFlash-BF16.gguf \
  --spec-type draft-dflash --spec-draft-n-max 7 \
  --host 127.0.0.1 --port 8000 \
  --ctx-size 8192 -ngl -1 -fa on --jinja \
  --alias local-laguna
```

Check:

```bash
curl -s http://127.0.0.1:8000/v1/models | head
curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"laguna","messages":[{"role":"user","content":"ping"}],"max_tokens":16}'
```

Point Hermes-class agents at:

```text
base_url = http://127.0.0.1:8000/v1
api_key  = sk-local   # unused by llama-server; placeholder
```

See `hermes/sample_client.py`.

### Context presets

| Preset | flags | use |
|--------|-------|-----|
| 2k smoke | `--ctx-size 2048` | quick agent_smoke / latency |
| 8k default agent | `--ctx-size 8192` | measured one-response tool-protocol smoke |
| 256k card default | `--ctx-size 262144` | large-context experiment (unmeasured; needs RAM headroom) |
| 1M native | `--ctx-size 1048576 --rope-scaling yarn --rope-scale 128 --yarn-orig-ctx 8192` | quality may drop · **chat-only** sampling example temp 0.7 top_p 0.95 · **not** agent/smoke claim default (claims use temp **0.0**) |

## Measure — tok/s + memory

### llama-bench (prompt+gen)

```bash
./build/bin/llama-bench \
  -m $HOME/models/laguna-s-2.1/laguna-s-2.1-Q4_K_M.gguf \
  -ngl -1 -fa 1 \
  -p 2048,8192 -n 128 \
  -b 512
```

### Server wall-clock (agent-shaped)

```bash
python scripts/bench_server.py \
  --base-url http://127.0.0.1:8000/v1 \
  --ctx-mark 2k --ctx-mark 8k \
  --out results/server_bench.json
```

### Memory

While server is up:

```bash
# unified memory pressure (Spark)
free -h
nvidia-smi  # GB10 fields may show N/A for FB; use free/rss
ps -o pid,rss,cmd -p $(pgrep -f llama-server)
```

Record **RSS** of `llama-server`, host free mem before/after load, and ctx size.

## Measured results (2026-07-28 Spark)

Host: DGX Spark · GB10 · CUDA 13.0 · engine `04b2b72` · weight Q4_K_M @ `a8b55c75…`  
Serve proven: `-c 8192 -ngl -1 --parallel 1 --alias local-laguna --jinja -fa on` · load ~6.5 min · host ~96–99 Gi used after load (121 Gi total)

| Setting | prefill latency | tg tok/s | server RSS | notes |
|---------|-----------------|----------|------------|-------|
| ctx 8k, ngl -1, fa on | 836 prompt → 1.597 s; 3236 prompt → 4.78 s | **~21.47** @ 128 gen | ~2–3.4 Gi process RSS | unified mem; **sole headline gen = 21.47** (`results/measured.json`) |
| gen8 short | — | 10.612 | same | tiny completion noise floor — not a headline |
| ctx 8k + DFlash (`draft-dflash`) | 2k 2.253s · 8k 5.556s | **15.286** @ 128 gen | higher (draft+target) | trial `-ngl 99` vs baseline `-ngl -1` · **DO_NOT_PROMOTE** still (gen+prefill slower) · `results/dflash_2026-07-29/` |

One-response tool-format/routing smoke — tools were validated, **not executed**; **live tip durations bind**:

| Suite | quant | pass | n | notes |
|-------|-------|------|---|-------|
| agent_smoke v1 (launch bar) | Q4_K_M | **40/40 · 100%** | 40 | **84.86 s** ·temp 0.0 · tip 2026-07-29 13:22 WIB · runner `3bb81080…` · older same-day restores ~88–98 s are historical only |
| hermes_agent_smoke v2 | Q4_K_M | **27/27 · 100%** | 27 | **100.1 s** · temp 0.0 · same tip · one-response · runner `20c1e52a…` · `results/hermes_agent_smoke.json` |
| hermes Layer B v3 (research) | Q4_K_M | **35/35 · 100%** | 35 | **137.56 s** · 2026-07-29 · **not freeze bar** · receipt `eval/hermes_agent_smoke/layer_b_v3_live_receipt.json` |
Raw: `results/MEASURED.md`, `results/measured.json`, `results/agent_smoke.json`, `results/hermes_agent_smoke.json`, `results/server_bench.json`.

## Agent-shaped benches

Not MMLU. Fixed suites under `eval/`:

**v1 — launch bar** (`eval/agent_smoke/`, 40 cases):

1. **tool JSON** — emit valid tool call objects only from the offered schema  
2. **multi-step fixtures** — choose the next response from a pre-seeded history; no tool loop is executed
3. **error repair fixtures** — respond to a pre-seeded tool error payload
4. **no invented tools** — refuse / answer without calling names not in schema  
5. **short code** — small pure functions, exact stdout  
6. **constraint-history fixtures** — preserve constraints from pre-seeded prior turns in one response

**v2 — Hermes-class** (`eval/hermes_agent_smoke/`, 27 cases):

Terminal / files / web / multi_tool / multi_turn / error_repair / no_invented / browser / memory_cron / args_strict / safety.  
OpenAI tool shape only — not a Nous endorsement.

```bash
python eval/agent_smoke/run_smoke.py \
  --base-url http://127.0.0.1:8000/v1 \
  --model local-laguna \
  --out results/agent_smoke.json

python eval/hermes_agent_smoke/run_hermes_smoke.py \
  --base-url http://127.0.0.1:8000/v1 \
  --model local-laguna \
  --temperature 0 \
  --out results/hermes_agent_smoke.json
```

Claim temperature **0.0**. Both suites are **one-response** (tools validated, not executed).

## vLLM (optional, unmeasured default)

Poolside card documents FP8 / NVFP4 Docker mirrors. This pack’s **default credibility path is GGUF + llama.cpp** because:

- matches official GGUF story  
- OpenAI-compatible with one binary  
- reproducible on Spark without container registry lag  

If you measure vLLM on Spark, add a dated subsection here with image digest + tok/s — do not overwrite GGUF numbers.

## Forbidden host confusion

- Do **not** attribute rented-cloud jobs to this file.  
- Do **not** mix non-local / third-party gateway base URLs into sample commands.  
- HF handle only for publish (`hizrianraz/*`).