Qwen3.5-27B-Latent-Reasoning

A latent-reasoning adapter for Qwen/Qwen3.5-27B. Instead of producing an explicit textual chain-of-thought inside the <think>...</think> block, the model emits a sequence of latent reasoning vectors that are consumed internally and never appear in the visible output. At inference time this lets the model "think" while emitting fewer visible tokens, trading textual scratchpad for a dense continuous representation.

The contribution of this repository is only the trained latent-reasoning head (≈451M params, ≈900 MB in bfloat16). The base Qwen3.5-27B is referenced and must be downloaded separately from the original Qwen repository.

Method

The adapter is one extra Qwen-style transformer block plus a back-projection layer. During the <think> phase it operates on the concatenation of the current token embedding and the previous-step hidden state; its output is projected back into the model's hidden space and fed into the frozen Qwen as the next embedding (replacing the token embedding for that position). When the model samples </think>, generation switches back to ordinary visible token-by-token decoding.

This style of "thinking in latent space" is closest in spirit to COCONUT (Chain of Continuous Thought) and to the latent reasoning literature; the training objective and head architecture follow the MiMo family of papers (latent token prediction on top of a frozen LLM).

Files

file	purpose
latent_head.safetensors	16 tensors, bfloat16. The trained adapter weights.
adapter_config.json	Base-model reference, training metadata, vLLM weight-name remap.
tensor_index.json	Per-tensor shape/dtype manifest.
test_inference.py	Minimal smoke-test script (also serves as a usage example).

The base-model embed_tokens matrix is shared with the frozen Qwen and is intentionally not stored here.

Inference

This adapter is consumed by a fork of vLLM with first-class support for latent-reasoning heads (the feature/latent-mtp-integration branch). The fork registers the architecture Qwen3_5LatentMTP and accepts the adapter through several reference forms:

/path/to/latent_head.safetensors                                # local file
/path/to/checkpoint.pt                                          # local .pt (legacy)
junglepy/Qwen3.5-27B-Latent-Reasoning                           # HF repo (default file)
junglepy/Qwen3.5-27B-Latent-Reasoning:latent_head.safetensors   # HF repo + filename
hf://junglepy/Qwen3.5-27B-Latent-Reasoning/latent_head.safetensors  # explicit hf:// scheme

The fork remaps the adapter's tensor names into the standard vLLM model namespace:

core.layer.*                  → model.layers.0.*
core.fc.*                     → model.fc.*
core.pre_fc_norm_embedding.*  → model.pre_fc_norm_embedding.*
core.pre_fc_norm_hidden.*     → model.pre_fc_norm_hidden.*
core.norm.*                   → model.norm.*
to_embed.*                    → to_embed.*

Minimal invocation (the adapter is fetched directly from this repo):

from transformers import AutoTokenizer
from vllm import LLM, SamplingParams

base = "Qwen/Qwen3.5-27B"
adapter = "junglepy/Qwen3.5-27B-Latent-Reasoning:latent_head.safetensors"

tok = AutoTokenizer.from_pretrained(base, trust_remote_code=True)
llm = LLM(
    model=base,
    dtype="bfloat16",
    trust_remote_code=True,
    max_model_len=3072,
    async_scheduling=False,
)

prompt = tok.apply_chat_template(
    [{"role": "user", "content": "What is 17 × 23?"}],
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=True,
)

params = SamplingParams(
    temperature=0.0,
    max_tokens=512,
    extra_args={
        "latent_reasoning": {
            "backend": "qwen35_mtp",
            "checkpoint": adapter,
            "think_close_token_id": 248069,   # </think>
            "max_internal_tokens": 320,
        }
    },
)

out = llm.generate([prompt], params)[0].outputs[0]
print(out.text)
print("latent steps used:", out.latent_internal_token_count)

The accompanying test_inference.py runs the same path end-to-end on two example prompts.

If you sit behind an HTTP proxy, the first run may need env -u http_proxy -u https_proxy -u HTTP_PROXY -u HTTPS_PROXY -u all_proxy -u ALL_PROXY NO_PROXY='*' … to let huggingface_hub reach huggingface.co for the initial download.

Training summary

• Base model (frozen): Qwen/Qwen3.5-27B (Apache-2.0)
• Trainable params: 451M (the additional embed_tokens matrix stays frozen and equal to the base)
• Best validation loss: 0.7034
• Optimizer step: 6000 (≈48k samples seen)
• Training mix: "targeted v2" — gsm8k, MATH (subset), ECQA, WorldTree, EntailmentBank, plus R1-style chain-of-thought traces (NuminaMath-1.5, OpenCodeReasoning).
• Reasoning width (w): 8
• Latent steps per chunk (chunk_size = 2): each latent vector is duplicated to two reasoning positions in the cached sequence.

A 5-example sanity run on gsm8k with this exact adapter (loaded directly from this HF repo) reached accuracy 4/5 (think_tokens_mean 320, visible_tokens_mean 287, self_exit_rate 0.00, maxed_rate 1.00 — at max_internal_tokens=320).

License & attribution

This adapter is released under Apache 2.0, mirroring the base model's license. By using it together with Qwen/Qwen3.5-27B, you agree to the base model's license terms as well.

When citing this work please also cite the underlying Qwen3.5 base model.