---
license: apache-2.0
base_model:
- kel-jp/llm-jp-4-8b-instruct-NVFP4
library_name: speculators
inference: false
language:
- ja
- en
tags:
- speculative-decoding
- speculative-decoding-draft
- dflash
- draft-model
- speculator
- speculators
- vllm
- llm-jp
- japanese
- nvfp4
- dgx-spark
---

# llm-jp-4-8b-instruct-NVFP4 DFlash Drafter

This repository contains a DFlash draft model for speculative decoding with
[`kel-jp/llm-jp-4-8b-instruct-NVFP4`](https://huggingface.co/kel-jp/llm-jp-4-8b-instruct-NVFP4).

This is not a standalone text-generation model. Use it as the draft/speculator
model while serving the NVFP4 model as the verifier in stock vLLM.

## Intended Use

- Verifier model: `kel-jp/llm-jp-4-8b-instruct-NVFP4`
- Drafting method: `dflash`
- Draft architecture: block size 4, 1 transformer layer, 28k draft vocabulary
- Proposal setting used in the released config: greedy, `speculative_tokens=3`
- Runtime target: stock vLLM with DFlash support

## vLLM Example

```bash
vllm serve kel-jp/llm-jp-4-8b-instruct-NVFP4 \
  --trust-remote-code \
  --reasoning-parser llmjp4 \
  --speculative-config '{"method":"dflash","model":"kel-jp/llm-jp-4-8b-instruct-NVFP4-speculator.dflash","num_speculative_tokens":3}'
```

If you use an environment wrapper, set the same JSON as `SPECULATIVE_CONFIG`.
The verifier model still needs the serving requirements documented on the NVFP4
model page, including the LLM-jp remote code/plugin setup.

## Benchmark

Representative Japanese streaming decode benchmark on ELYZA-100 prompts:

| Setup | Mean decode tok/s | Notes |
| --- | ---: | --- |
| Baseline NVFP4 | 29.73 | no speculative decoding |
| NVFP4 + this DFlash drafter | 48.76 paired / 49.04 standalone run | stock vLLM, DFlash |

Paired speedup summary:

- Ratio of mean decode throughput: 1.64x
- Mean paired request speedup: 1.64x
- Median paired request speedup: 1.63x
- p05/p95 paired speedup: 1.37x / 1.97x
- Requests at least 1.5x faster: 70 / 100
- Requests at least 2.0x faster: 5 / 100
- Requests slower than baseline: 0 / 100

The raw benchmark artifacts are included under [`benchmark/`](benchmark/):

- `dflash20k-b4-l1-v28k-elyza100-streaming.json`
- `dflash20k-b4-l1-v28k-elyza100-streaming.csv`
- `elyza100-baseline-vs-dflash-summary.json`
- `elyza100-baseline-vs-dflash-speedup.png`
- `dflash20k-b4-l1-elyza100-tps-histogram.png`

![Decode speedup distribution](benchmark/elyza100-baseline-vs-dflash-speedup.png)

## Validation Metrics

The released checkpoint's held-out validation metrics:

```json
{
  "loss_epoch": 2.047496609403255,
  "full_acc_epoch": 0.4021220020978507,
  "position_1_acc_epoch": 0.5539027137301571,
  "position_2_acc_epoch": 0.38056573750461736,
  "position_3_acc_epoch": 0.27111557473966125
}
```

## Files

- `model.safetensors`: released drafter weights
- `config.json`: DFlash/speculators configuration consumed by vLLM
- `config.py`: custom config class required by the drafter
- `val_metrics.json`: validation metrics from the selected checkpoint
- `benchmark/`: benchmark summaries, per-request CSV/JSON, and plots

## Limitations

- This artifact is useful only with a compatible verifier model and DFlash-capable
  vLLM runtime.
- The published benchmark is throughput-oriented and uses Japanese ELYZA-100
  prompts. It is not a general quality evaluation of the verifier model.
- Speedup depends on prompt mix, max token settings, batching, hardware, and vLLM
  version.

## License

Apache-2.0, following the verifier model release.