JohnBirks/llama-405b-multilora-production

Training 7 Specialized LoRA Adapters for Llama 3.1 405B on a Single 8-GPU Node — All Hot Simultaneously, 55+ Days of Uptime

---

What This Post Is

This is a production engineering report, not a benchmark demo.

Everything documented here has been running continuously since April 29, 2026 — 55+ days of uninterrupted uptime at the time of publication. The numbers below are not peak results captured during optimal conditions. They are the steady-state performance of a live system serving internal production workloads for a private pre-launch platform.

The hardware is a single AWS p4de.24xlarge: one node, 8× NVIDIA A100-SXM4-80GB GPUs. Not a multi-node cluster. Not H100s.

---

What This Is Not

This is not full-model fine-tuning of 405B parameters.

This is not a claim that every workload can run this way.

This is not a replacement for multi-node training at frontier-lab scale.

This is a documented path for training and serving multiple specialized LoRA adapters on a 405B-class Instruct model using one 8× A100 node — a configuration previously assumed to be impractical or cost-prohibitive for this class of work.

---

The Achievement

Three things demonstrated together on a single 8-GPU node:

1. Trained 7 domain-specialized LoRA adapters on Meta's Llama 3.1 405B Instruct — sequentially, on the same box, in a single session, using on-the-fly per-parameter NF4 quantization during model loading. No pre-quantized checkpoint required. No multi-node. Total wall-clock time: approximately 10h 42m for all 7 adapters, including model loading, setup between runs, and training (April 28, 2026 02:17 → 12:59 UTC).

2. Served all 7 fine-tuned adapters simultaneously hot via vLLM's multi-LoRA scheduler — one shared 405B base, 7 adapters resident in VRAM at the same time, sub-200ms switching between any pair. No cold-swap latency. No 90–150 second model loads. All 7 available instantly.

3. Migrated to AWQ-int4 with Marlin kernels on top of the existing multi-LoRA topology — 5–7× throughput increase without retraining a single adapter, without modifying the serving architecture.

---

Hardware

Instance	AWS p4de.24xlarge
GPUs	8× NVIDIA A100-SXM4-80GB (SXM4, NVLink)
Total VRAM	640 GB
System RAM	1.1 TB
CUDA	12.8 / Driver 570.133.20
Cost	$32.77/hr ($24K/month)

---

Software Stack

Package	Version
vLLM	0.10.2
PyTorch	2.8.0
Transformers	4.55.4
bitsandbytes	0.49.2
Axolotl	training framework
CUDA	12.8

---

Base Model

• Model: meta-llama/Meta-Llama-3.1-405B-Instruct
• Serving: AWQ-int4 via hugging-quants/Meta-Llama-3.1-405B-Instruct-AWQ-INT4
• Quantization kernel: awq_marlin (Marlin fused kernels)
• Training quantization: On-the-fly NF4 via bitsandbytes during model load — no pre-quantized checkpoint

Why Instruct, not Base: With 444–2,819 examples per adapter, there is enough signal for domain specialization and style transfer, but nowhere near enough to teach instruction-following, stop token behavior, or conversational structure from scratch. The Instruct variant's RLHF investment is preserved and specialized on top of — not discarded.

On adapters and quantization: The adapters were trained while the base model was loaded with NF4 quantization. The resulting LoRA adapter weights were then served on top of the AWQ-int4 base without retraining. Existing guidance suggested adapters would need to be retrained against the new base quantization. In this case, they did not.

---

The 7 Adapters

All 7 trained sequentially on the same node in a single session. All 7 served simultaneously hot.

Adapter	Domain	Training Examples	LoRA Rank	LoRA Alpha	Adapter Size	Train Loss
domain-1	Conversational	444	r=16	α=32	2.4 GB	2.952
seo	SEO strategy	541	r=16	α=32	2.4 GB	2.986
cro	Conversion optimization	1,089	r=32	α=64	4.7 GB	2.446
builder	System architecture	1,252	r=32	α=64	4.7 GB	2.661
librarian	Knowledge retrieval	1,677	r=32	α=64	4.7 GB	2.500
domain-2	Strategic analysis	1,875	r=32	α=64	4.7 GB	2.461
legal	Legal reasoning	2,819	r=32	α=64	4.7 GB	2.539

Total training examples: 9,697 across all 7 adapters Target modules (all adapters): q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj Base model commit: be673f326cab4cd22ccfef76109faf68e41aa5f1

On Training Data Quality

Each adapter was trained on three layers of curated data:

1. Deep domain knowledge — the field itself (SEO best practices, legal frameworks, conversion psychology, etc.)
2. Application methodology — how that knowledge gets applied in practice
3. Organizational context — how the domain is applied within a specific operating model

This layered approach is what separates domain adapter training from generic instruction tuning. The base model knows SEO from the internet. A fine-tuned adapter knows SEO the way a two-year employee knows it — including the application layer that no amount of general training can provide.

Training Methodology — The Three-Step Data Pipeline

Many LoRA fine-tuning examples stop at generic public or instruction-style data. This deployment used a three-step pipeline that meaningfully changes the quality ceiling of what a small adapter can do:

Step 1 — Domain saturation. Each adapter was fed a curated knowledge base for its domain. For SEO: established frameworks, ranking factors, content structure principles, and technical SEO considerations available from public sources. The goal was to give the base model nothing new — it already knows SEO from pretraining — but to concentrate and organize that knowledge into the adapter's weight space rather than leaving it distributed across 405B parameters.

Step 2 — Organizational context injection. The same adapter then received everything about how that domain operates within this specific company — the operating model, the priorities, the constraints, the application layer that cannot exist in any public dataset because it is specific to this organization. This is the step that transforms a domain expert into a domain expert who works here. No public training data can provide this. It has to be built.

Step 3 — Frontier model distillation. Domain knowledge and organizational context were fed to frontier API models (GPT-4o, Gemini). The outputs — high-quality, domain-accurate, organizationally-aware responses — were captured and used as training examples for the LoRA adapter. The adapter was not trained on raw human-written data alone. It was trained on distilled frontier model output, scoped to the exact domain and organizational context from Steps 1 and 2.

The result: a small adapter (2.4–4.7 GB) that can approach frontier-model quality on narrow, well-scoped domain tasks — because it was trained on frontier model output, not raw internet data alone.

Why this matters beyond this deployment: This pipeline is repeatable across domains and organizations with sufficient domain data, context, and evaluation discipline. The infrastructure to run it at 405B scale is documented above. The methodology to build the training data is: define the domain, build the organizational context layer, run distillation against a frontier API model, train the adapter. Organizations that need private, specialized AI — and cannot send data through external APIs — now have a documented path toward a production-grade domain specialist running on their own infrastructure.

Blind benchmark: In one internal 4-way blind evaluation judged by GPT-4o (all responses presented without labels), these fine-tuned adapters outscored Gemini 2.5 Pro on CRO and Legal domains — at 220 tokens output vs Gemini's 2,000+ tokens. GPT-4o scored our adapters 8/10 domain accuracy vs Gemini's 6–7/10. In this evaluation, specialization beat general scale on narrowly defined domain tasks.

---

VRAM — All 7 Adapters Hot

Live snapshot captured June 8, 2026 during active inference:

GPU	Total	Used	Free
GPU 0	81,920 MiB	61,975 MiB	19,179 MiB
GPU 1	81,920 MiB	61,975 MiB	19,179 MiB
GPU 2	81,920 MiB	61,975 MiB	19,179 MiB
GPU 3	81,920 MiB	61,975 MiB	19,179 MiB
GPU 4	81,920 MiB	61,975 MiB	19,179 MiB
GPU 5	81,920 MiB	61,975 MiB	19,179 MiB
GPU 6	81,920 MiB	61,975 MiB	19,179 MiB
GPU 7	81,920 MiB	61,975 MiB	19,179 MiB
TOTAL	640 GB	~484 GB (76%)	~150 GB (23%)

VRAM breakdown (approximate):

• AWQ-int4 405B base model: ~202.5 GB (405B params × 0.5 bytes/param)
• KV cache (pre-allocated): ~25.6 GB
• All 7 LoRA adapters: ~29 GB total
• CUDA/PyTorch workspace, NCCL buffers, allocator reservations: ~227 GB
• Free headroom: ~150 GB

The ~227 GB workspace/allocator delta is standard PyTorch behavior — memory is reserved in large blocks to avoid mid-inference reallocation. It is not wasted; it is pre-allocated capacity the allocator holds for burst usage.

150 GB of free VRAM remaining after all 7 adapters are hot. The system was not at its limit.

---

Production vLLM Configuration

vllm serve /path/to/Meta-Llama-3.1-405B-Instruct-AWQ-INT4/snapshots/<commit>/ \
  --quantization awq_marlin \
  --tensor-parallel-size 8 \
  --enable-lora \
  --max-loras 7 \
  --max-cpu-loras 7 \
  --max-lora-rank 32 \
  --lora-modules \
    domain-1=/data/finetune/domain1/adapter \
    domain-2=/data/finetune/domain2/adapter \
    librarian=/data/finetune/librarian/adapter \
    builder=/data/finetune/builder/adapter \
    seo=/data/finetune/seo/adapter \
    cro=/data/finetune/cro/adapter \
    legal=/data/finetune/legal/adapter \
  --host 127.0.0.1 \
  --port 11434 \
  --max-model-len 4096 \
  --max-num-seqs 8 \
  --kv-cache-memory <tuned-per-system> \
  --disable-custom-all-reduce \
  --enable-prefix-caching \
  --served-model-name llama405b

Environment:

HF_HOME=/path/to/hf-cache
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
NCCL_DEBUG=WARN

Note on --kv-cache-memory: This value requires per-system tuning based on available VRAM after model and adapter loading. The default gpu_memory_utilization=0.90 will OOM with 7 adapters loaded. Tune explicitly. See migration bugs section for context.

---

Benchmark Methodology

Benchmarks were run against the live production vLLM endpoint at 42 days 17 hours of uptime. Unless otherwise stated, all measurements include request overhead, prompt processing, time-to-first-token, and generation. Tests were repeated across all adapters using fixed output lengths. Concurrent tests were dispatched via asyncio at the same wall-clock time. Results reflect steady-state behavior under active internal workloads, not isolated laboratory conditions.

---

Benchmark Results

All measurements taken June 8, 2026 on the live system at 42 days 17 hours of uptime.

Single-Adapter Throughput

Adapter	100 tokens	Latency	Tok/sec
domain-1	100	5,359ms	18.66
domain-2	100	5,227ms	19.13
librarian	100	5,239ms	19.09
builder	100	5,226ms	19.14
seo	100	5,228ms	19.13
cro	100	5,234ms	19.11
legal	100	5,221ms	19.15

Average: 19.06 tok/sec across all 7 adapters.

18–19 tok/sec vs 25 tok/sec peak: the 100-token test starts cold (full prompt encode + generation). The 25 tok/sec peak is measured under warmed KV cache and prefix cache hits, which is the steady-state condition.

Time-To-First-Token

Every adapter: 63–66ms. All adapters are permanently resident in VRAM — no warm-up required.

Adapter Switching Latency

Measured as wall-clock time for a complete end-to-end request immediately after switching to a different adapter:

Switch	Latency
domain-1 → domain-2	168ms
domain-2 → librarian	169ms
librarian → builder	170ms
builder → seo	169ms
seo → cro	169ms
cro → legal	170ms
legal → domain-1	169ms

Average: 169ms end-to-end (includes TTFT + generation of 3 tokens). The adapter swap itself: ~0ms. It is a pointer operation in the vLLM multi-LoRA scheduler — not a memory transfer. All 7 adapters are already in VRAM.

Compare to previous architecture (Ollama, one model at a time): 90–150 seconds per swap.

7 Adapters Simultaneously — True Concurrent Load

All 7 requests dispatched at the exact same instant via asyncio, each targeting a different adapter with a real domain-specific prompt:

Adapter	Tokens	Latency	Tok/sec
domain-1	80	6,680ms	11.98
domain-2	80	6,756ms	11.84
librarian	80	6,756ms	11.84
builder	80	6,756ms	11.84
seo	80	6,756ms	11.84
cro	80	6,756ms	11.84
legal	80	6,756ms	11.84

Wall-clock: 6.76 seconds for all 7 to complete. Combined: 560 tokens in 6.76s = 82.9 tok/sec across the cluster. Zero failures.

Per-adapter throughput drops from 19.1 tok/sec sequential to 11.84 tok/sec under full concurrent load — expected, as the GPU compute budget is shared. All 7 responded simultaneously with coherent domain-specific output.

Higher Concurrency — 14 and 21 Simultaneous Requests

Measured on the live system at 55 days uptime. 80 tokens per request.

14 concurrent requests (2 per adapter, all 7 adapters):

Metric	Result
Wall-clock time	32.41s
Total tokens	3,584
Combined throughput	110.6 tok/sec
Failures	0

21 concurrent requests (3 per adapter, all 7 adapters):

Metric	Result
Wall-clock time	46.75s
Total tokens	5,376
Combined throughput	115.0 tok/sec
Failures	0

Combined throughput increases from 7 → 14 → 21 concurrent requests (82.9 → 110.6 → 115.0 tok/sec). This is the vLLM continuous batching scheduler filling batch slots more efficiently as queue depth increases — the system becomes more compute-efficient under higher load, not less. No OOM events at any concurrency level. Zero failures across all three concurrency tiers.

Output Length Scaling

Output Tokens	Latency	Tok/sec
32	1,671ms	19.15
64	3,334ms	19.20
128	6,671ms	19.19
256	13,440ms	19.05

Near-perfect linear scaling from 32 → 256 tokens. No KV cache degradation. No pipeline bottleneck.

Extended Context Throughput (Single Adapter: Legal)

Measured on the live system at 55 days uptime. Single adapter (legal), fixed 80-token output cap, varying input context length:

Input Context	Total Request Time
512 tokens	27.35s
1,024 tokens	54.21s
2,048 tokens	108.66s

Scaling is roughly linear as context length increases. No OOM events, no KV cache cliff, no failure at any tested context length. Request time scales predictably with input size — the system handles longer enterprise documents without degradation.

Temperature Effect on Throughput

Temperature	Tok/sec
0.0 (greedy)	19.14
0.5	19.15
1.0 (stochastic)	19.16

Temperature has no measurable effect on throughput. The inference pipeline is compute-bound by the Marlin kernel execution, not sampling logic.

Throughput vs Previous Architecture

Phase	Quantization	Throughput
Phase 1 — Ollama (single model)	NF4	3–5 tok/sec
Phase 2 — vLLM NF4	NF4 eager	3–5 tok/sec
Phase 3 — vLLM AWQ-int4	AWQ Marlin	~19 tok/sec sustained / 25 tok/sec peak

5–7× multiplier. No adapters retrained. No architecture changes. Base model swapped from NF4 to AWQ-int4; existing NF4-trained LoRA adapters loaded directly on the new base.

---

GPU Utilization and Thermal Profile Under Load

Measured during the concurrent 7-adapter benchmark:

GPU	Utilization	Temp	Throttle Point	Headroom
GPU 0 (tensor parallel coordinator)	94%	69°C	~83°C	14°C
GPU 1	20%	58°C	~83°C	25°C
GPU 2	5%	67°C	~83°C	16°C
GPU 3	15%	57°C	~83°C	26°C
GPU 4	0%	63°C	~83°C	20°C
GPU 5	0%	53°C	~83°C	30°C
GPU 6	0%	64°C	~83°C	19°C
GPU 7	0%	52°C	~83°C	31°C

GPU utilization figures are point-in-time nvidia-smi samples and should be read as operational telemetry, not a full profiler trace. The distribution reflects tensor parallel coordination: GPU 0 handles scheduling under this workload configuration. Throughput, latency, thermal headroom, and error-free uptime are the primary stability indicators.

Every GPU ran 14–31°C below thermal throttle under maximum concurrent load. The system was not thermally stressed.

---

Production Health Audit — 47 Days

Metric	Result
System uptime	47 days, 23 hours, 37 minutes
vLLM service started	2026-04-29 21:55:09 UTC
vLLM service restarts	0
Boot sessions	1 — booted once, ran continuously
OOM kill events (kernel ring buffer)	0
NCCL errors or timeouts	0
GPU ECC corrected errors (all 8 GPUs)	0
GPU ECC uncorrected errors (all 8 GPUs)	0
Swap memory used	0 kB
System RAM available	932 GB of 1,117 GB

Zero hardware errors across 8 GPUs over 47 days. Zero memory errors. Zero network errors. Zero service restarts. One boot.

System remains live and uninterrupted at 55+ days at time of publication. Audit snapshot above was captured at 47 days 23 hours. Additional benchmarks (extended context, higher concurrency) captured at 55 days 6 hours.

This is not a benchmark-only deployment. This is a system that has been running under real internal workloads since April 29, 2026.

---

Migration Journey — Key Bugs Encountered

These are the walls I hit while building this. Documenting them so others do not have to rediscover them the hard way.

Bug 1 — The venv incident: First vLLM install (0.7.3) succeeded silently but downgraded PyTorch from 2.8.0 to 2.5.1 system-wide, breaking the entire training pipeline. Root cause: vLLM 0.7.3 has a hard torch<=2.5.1 pin. Fix: isolated venv at /opt/vllm-venv. Always install vLLM into a dedicated venv on shared GPU boxes.

Bug 2 — OOM on first boot: First vLLM 0.10.2 boot with all 7 LoRA adapters loaded crashed OOM. Root cause: default gpu_memory_utilization=0.90 combined with 7 adapter overhead and KV cache left no headroom. Fix: explicit memory budget tuning. Specific constants are not published — this requires iterative per-system tuning based on your exact VRAM, adapter count, and context length configuration.

Bug 3 — KV cache / max_num_seqs balance: Default vLLM context length settings allocated too much KV cache per sequence, leaving insufficient memory for concurrent requests. Iterative tuning required across --max-model-len, --max-num-seqs, and --kv-cache-memory. These three interact and must be balanced together — there is no universal formula.

Bug 4 — Backend routing: Production app routes were passing a caller-supplied field allowing requests to bypass intended request handling. Discovered during architectural review. All routes locked. Never expose adapter selection directly to the caller in a production API.

Bug 5 — Bearer token rotation gap: Daily bearer token rotation briefly created a window where the old token had expired but the new token hadn't propagated. Fixed with an overlap window. Exact duration not published.

Bug 6 — max_tokens cap: --max-model-len=4096 means prompt + generated tokens combined cannot exceed 4096. Requests with long prompts must cap max_tokens accordingly. Add an application-layer cap.

---

Prior Guidance and What This Result Shows

Meta's published guidance for 405B-class training assumes infrastructure far beyond a single 8× A100 node. Multi-specialist serving at this scale was commonly assumed to require multi-node clusters at $100K–$200K per month. AWQ-int4 migration was assumed to require retraining adapters against the new base quantization. Running training and serving on the same single node was considered operationally risky.

This deployment shows those assumptions are not universal. The specific result — LoRA adapters, quantized loading, single node, multi-adapter simultaneous serving — documents a narrower, cheaper path for this specific workload. It does not replace multi-node training at frontier-lab scale. It shows that for organizations running specialized LoRA adapters on a 405B-class Instruct model, a single p4de.24xlarge is a viable and stable path.

---

What This Unlocks

Teams that need:

• Data sovereignty — regulated industries (healthcare, finance, legal, defense) that cannot use external APIs due to compliance requirements — this suggests a viable path to frontier-scale AI on their own infrastructure
• Cost structure — $24K/month for a dedicated 405B multi-specialist node vs $100K–$200K/month for multi-node alternatives; at sustained production volume, marginal per-token infrastructure cost drops because the node cost is fixed
• Multi-agent workflows — sub-200ms specialist switching makes multi-agent orchestration commercially viable; the previous 90–150 second swap latency made it unusable in real products
• Frontier access for small teams — a single high-memory GPU node available from major cloud providers, no specialized networking, no multi-node complexity, no InfiniBand required

---

Hardware Scaling Projections

All benchmarks above were measured on 8× A100-SXM4-80GB. The following are directional estimates based on documented memory bandwidth ratios applied to the measured baseline — not measured benchmarks. The architecture should require minimal modification to run on newer hardware.

Why bandwidth is the bottleneck: LLM inference throughput is bound by memory bandwidth — how fast weights move from VRAM to compute. TTFT is bound by compute FLOPS. Because the adapter swap is already a ~0ms pointer operation, switching latency is simply TTFT + generation time — both improve directly with hardware generation.

Hardware	Mem BW	VRAM (8×)	Throughput	TTFT	Switch Latency	Status
A100 SXM4	2.0 TB/s	640 GB	19 tok/sec	63–66ms	168–170ms	✅ Measured
H100 SXM5	3.35 TB/s	640 GB	~35–45 tok/sec	~25–35ms	~90–100ms	Projected
H200 SXM	4.8 TB/s	1,128 GB	~50–60 tok/sec	~18–25ms	~70ms	Projected
AMD MI300X	5.3 TB/s	1,536 GB	~55–65 tok/sec	~15–20ms	~60ms	Projected

On H200: 1,128 GB total VRAM. Our 484 GB payload leaves 644 GB free. Estimated capacity: 50–60 simultaneous specialized adapters on a single node.

The core point: The hardest problems — VRAM allocation, multi-LoRA routing, AWQ-int4 migration, production stability — were addressed on the most constrained hardware in this tier. The architecture benefits automatically from hardware improvements. No retraining required.

---

Benchmark Summary

Metric	Value
Single-adapter throughput	18.7–19.2 tok/sec
Peak sustained throughput	25 tok/sec
Time-to-first-token	63–66ms
Adapter switch (end-to-end)	168–170ms
Adapter swap overhead	~0ms
7 concurrent adapters combined	82.9 tok/sec
Output scaling degradation (32→256 tok)	<1%
Temperature effect on throughput	<0.02 tok/sec
Uptime at benchmark	42 days 17 hours
Extended context / concurrency benchmarks	55 days 6 hours
Current uptime at publication	55+ days
OOM events	0
NCCL errors	0
Service restarts	0
GPU hardware errors (all 8)	0

---

Closing

This infrastructure powers a platform currently in final pre-launch. Everything documented here reflects active internal production use — not a research experiment or benchmark-only deployment. The system has been serving real workloads continuously since April 29, 2026.

Built by a non-ML-engineer while building the company it serves.

---

Related Work

This infrastructure was built to support a private AI-powered health and wellness platform.

The operating philosophy behind the build is documented in Operator Doctrine: JohnBirksOfficial.com

---

Questions, replications, and corrections welcome.

Sanitized configs: gist.github.com/JohnMBirks/8de22d2face739d3a518a25fb59a864a