Nemotron-3.5-30B-A3B-Antislop-FTPO-V2

A merged, ready-to-serve BF16 checkpoint of NVIDIA's Nemotron 3.5 30B-A3B, tuned with Antislop and FTPO to suppress the overused lexical patterns that make LLM prose recognisable as LLM prose.

43.2% of overused patterns suppressed across six domains, 67.8% on creative writing, with no capability regression on any benchmark measured.

The LoRA adapter that produced this checkpoint is released separately at thoughtworks/Nemotron-3.5-30B-A3B-Antislop-FTPO-V2-LoRA.

What changed from V1

V1 suppressed 66.41% on creative writing prompts and was measured only there. V2 keeps the method and widens the scope:

	V1	V2
Prompt domains	creative writing only	six (creative, real-user, factual QA, explanatory, instruction-following, conversational)
Held-out eval	400 prompts	952 prompts
Human baseline	Reddit writing	3.13B prose chars, 12 registers
Pipeline iterations	2	5
Banlist	4,267 patterns	8,033 patterns
Typography profiling	none	28 features
Preference pairs	~13,000	15,000

V1 and V2 headline numbers are not directly comparable. V1 scored against its own 4,267-pattern banlist; V2 scores against a banlist 1.9x larger. On V1's exact 388 creative prompt IDs, V2 measures 67.78% — level with V1 while also covering five more domains. V1's benchmarks were also run with thinking on and V2's with thinking off, so the absolute MMLU and IFEval figures differ between cards for protocol reasons, not model reasons.

What this is

Language models overuse a characteristic vocabulary: stock dialogue tags, a fixed set of atmospheric props, and a long tail of default names. Antislop identifies those patterns for this specific model by comparing its output against a human-written baseline, then uses a backtracking sampler to catch the model reaching for one and record what it should have reached for instead. Those records become preference pairs, and FTPO (Final Token Preference Optimization) trains the preference into the weights so it persists with the sampler switched off.

V2 adds a second axis: typography and punctuation. V1's n-gram analysis dropped every non-letter token before counting, so it was structurally blind to em dashes, ellipses, curly quotes, emoji, and markdown. V2 profiles 28 such features against human rates and fires bans proportionally — with probability p = 1 - human_rate / model_rate, so the residual lands at human parity rather than zero. Suppressing a tic to far below human usage is its own detectable signature.

Results

952 held-out prose prompts, identical sampling for both models (temp 1.0, top_p 1.0, top_k 50, min_p 0.01), Antislop sampler off. These numbers reflect what training changed in the weights, not what a sampler suppresses at inference.

Metric	Baseline	FTPO V2	Δ
Banlist suppression (overall)	0%	43.21%	+43.21
Banlist suppression (creative, n=388)	0%	67.78%	+67.78
Banned-pattern rate per 100k chars	203.28	115.45	−43.2%
MMLU (600 q)	0.7583	0.7567	−0.16pp
GSM8K (250 q)	0.9040	0.9160	+1.20pp
MMLU-Pro (600 q)	0.7000	0.7200	+2.00pp
IFEval prompt-strict (323)	0.7802	0.7709	−0.93pp
IFEval instruction-strict (323)	0.8299	0.8209	−0.90pp
HumanEval+ pass@1 (164)	0.8598	0.8537	−0.61pp
Lexical diversity (index, baseline=100)	100.00	99.98	−0.02
MATTR-500	0.5567	0.5634	+0.0067

No benchmark difference is meaningfully distinguishable from zero at these sample sizes. HumanEval+ is worth calling out: V2 deliberately removed all code and math prompts from the training pool, and code ability is unchanged (one problem out of 164).

Suppression by domain

Domain	n	Baseline /100k	V2 /100k	Suppression
creative	388	218.13	70.29	67.78%
real_user	275	199.10	142.56	28.40%
instruction_following	71	49.80	36.00	27.71%
factual_qa	106	230.63	197.41	14.40%
explanatory	82	237.60	213.54	10.13%
conversational	30	249.92	238.30	4.65%
overall	952	203.28	115.45	43.21%

Suppression is strongest where the slop is densest and the register is furthest from the model's assistant default. Conversational is the weak case at 4.65%.

Typography: a split result

This is the new capability in V2, and it works for some feature classes and fails for others. Rates per 100k prose chars against the human baseline, code stripped:

Improved:

Feature	Human	Baseline	V2	Before	After
not just X but Y	1.10	2.17	1.12	1.97x	1.02x
curly apostrophe	33.83	122.61	29.58	3.62x	0.87x
curly double quotes	40.19	46.56	14.32	1.16x	0.36x
ellipsis character	2.20	3.68	0.28	1.67x	0.13x
...	15.31	24.63	18.06	1.61x	1.18x
markdown hrule	10.25	12.97	11.10	1.26x	1.08x

Made worse:

Feature	Human	Baseline	V2	Before	After
unspaced em dash	2.07	80.58	108.38	38.9x	52.3x
markdown header	1.31	60.05	61.07	45.9x	46.7x
bullet list item	3.89	140.54	148.83	36.1x	38.3x
emoji	0.14	2.59	2.85	18.6x	20.4x

The split is not random. Features with an exact substitute get fixed; features requiring a structural choice get worse. A curly apostrophe maps to ', so FTPO's "chosen" token is well defined and learnable. An em dash's replacement is a comma, a colon, a period, or a rewritten clause depending on the sentence — the chosen token is effectively arbitrary and the gradient carries no consistent signal. Markdown headers, bullets and emoji fail the same way.

If you need em dashes suppressed, use the Antislop sampler at inference. Proportional firing held them at ~1.0x human during generation; it is the weight-baking step that fails.

Note also that several of the wins overshoot to below human rates. Structural diagnostics moved toward human on five of six measures (sentence length mean/sd/cv, conjunction openers, single-sentence paragraphs) and away on one (paragraph length).

What gets banned

The banlist is not a hand-written style guide. It is 8,033 patterns measured as overused in this model's own output relative to human prose: 4,342 n-grams (2,218 bigrams, 2,124 trigrams) and 3,990 slop phrases.

Genuine prose slop dominates: fingers hovering, voice echoes, impossibly large, eyes wide terror, short sharp sound, bruised purple choked, critics argue, particular kind magic, chuckled sound like.

Known limitations

• Em dashes and markdown structure get worse, as above.
• The 43.21% figure understates true slop suppression. The banlist contains prompt echoes that were never slop: character names (elara, kaelen, huck, sansa), code tokens (pygame, cloudformation), and non-English function words. These inflate the denominator. A prompt-echo filter is the main outstanding fix.
• No writing-quality judge was run. V1 reported an LLM-judge comparison; V2's judge and τ-bench runs were blocked by an invalid API key and are simply absent, not null results.
• Conversational and explanatory registers barely move (4.65% and 10.13%).
• Suppression is a reduction, not a hard filter.

Usage

Requires trust_remote_code=True for the nemotron_h architecture.

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "thoughtworks/Nemotron-3.5-30B-A3B-Antislop-FTPO-V2"
tok = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id, torch_dtype="bfloat16", device_map="auto", trust_remote_code=True
)

messages = [{"role": "user", "content": "Write the opening of a story about a lighthouse keeper."}]
text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
out = model.generate(**tok(text, return_tensors="pt").to(model.device), max_new_tokens=512)
print(tok.decode(out[0], skip_special_tokens=True))

Note: on transformers 5.14.x, apply_chat_template(..., tokenize=True) under the TokenizersBackend tokenizer class returns a truncated sequence. Render with tokenize=False and tokenize the resulting string, as shown above. This affects the upstream base model identically and is not specific to this checkpoint.

The chat template defaults to thinking on, emitting an open <think> block. Pass enable_thinking=False to apply_chat_template (or chat_template_kwargs={"enable_thinking": false} over an OpenAI-compatible API) for direct prose. All prose evaluation above was run with thinking off.

vLLM

Stable vLLM 0.27.1 and later support NemotronHForCausalLM natively.

vllm serve thoughtworks/Nemotron-3.5-30B-A3B-Antislop-FTPO-V2 \
    --trust-remote-code \
    --moe-backend auto \
    --mamba-cache-mode align \
    --enable-prefix-caching \
    --max-num-batched-tokens 4096 \
    --max-num-seqs 64 \
    --max-model-len 65536

For tool calling, add --enable-auto-tool-choice --tool-call-parser qwen3_xml, which handles this model's XML tool syntax correctly.

llama.cpp

Supported from build b9610 (nemotron_h_moe). When quantizing, keep the output tensor at f16:

llama-quantize --output-tensor-type f16 model-bf16.gguf model-Q4_K_M.gguf Q4_K_M

The entire fine-tune lives in lm_head as a perturbation of mean 3.3e-05. A default Q4_K_M puts the output tensor at Q6_K, whose quantization noise is large enough to swamp a delta that small and quietly undo the training. Expect ~24.9 GB at 6.29 BPW rather than the usual ~19 GB, partly for this reason and partly because 134 of 401 tensors fall back to q5_0 (moe_intermediate_size 1856 is not a multiple of the Q4_K block size).

Training

Method	Antislop, then FTPO (Final Token Preference Optimization)
Target modules	lm_head only
LoRA rank / alpha / dropout	256 / 256 / 0.05
Steps / epochs	434 / 2
Learning rate	5.45e-05 (auto-scaled)
Final train loss	2.065
Pipeline iterations	5
Generation	4,511 prompts per iteration
Preference pairs	15,000 (12,000 lexical, 3,000 typography)
Banlist size	8,033 patterns + 28 typography features
Hardware	1x H200, about 6 h for the FTPO stage

Training-set filtering removed 11,900 pairs of rare content vocabulary (Zipf < 2.5, e.g. proper nouns mistaken for slop), 5,315 non-Latin-script pairs, and 2,089 displacement-only pairs where the model had merely rerouted one suppressed tic into another.

Model architecture

Inherited unchanged from the base model. A hybrid Latent Mixture-of-Experts design with interleaved Mamba-2 and MoE layers plus select attention layers, and Multi-Token Prediction (MTP) layers for speculative decoding. 52 layers (23 Mamba-2, 23 MoE, 6 attention), 128 experts with 6 active plus 1 shared, 30B total parameters, 3B active. Context length up to 262,144.

License

Released under OpenMDW-1.1, matching the license NVIDIA applies to the public Nemotron 3.5 Lightning releases. The Antislop framework itself is MIT-licensed.

Citation

The method was published at ICLR 2026:

@inproceedings{paech2026antislop,
  title     = {Antislop: A Comprehensive Framework for Identifying and Eliminating
               Repetitive Patterns in Language Models},
  author    = {Paech, Samuel and Roush, Allen and Goldfeder, Judah and Shwartz-Ziv, Ravid},
  booktitle = {The Fourteenth International Conference on Learning Representations},
  year      = {2026},
  url       = {https://openreview.net/forum?id=gLcyM1khyp},
  eprint    = {2510.15061},
  archivePrefix = {arXiv},
  primaryClass = {cs.CL}
}

• Paper: ICLR 2026 poster · OpenReview · arXiv:2510.15061
• Code: github.com/sam-paech/auto-antislop (MIT)

Acknowledgements

Built on NVIDIA's Nemotron 3.5, with thanks to the NVIDIA team for the benchmark guidance that shaped the evaluation.