mosesman/LFM2.5-2.6B-openvino-int4-npu

LFM2.5-2.6B - OpenVINO INT4 (Intel AI Boost / NPU)

OpenVINO IR export of LiquidAI/LFM2.5-2.6B quantized for Intel AI Boost NPU, using the same NPU-correct recipe as the sibling mosesman/LFM2-2.6B-openvino-int4-npu build.

--weight-format int4 --sym --group-size 128 --backup-precision int8_sym

Why this recipe

Default INT4 exports (INT8 asymmetric backup layers, i.e. zero-points) produce garbled tokens on the Intel NPU. Forcing the backup layers to symmetric INT8 (--backup-precision int8_sym) keeps NPU arithmetic zero-point-free, which the NPU executes cleanly.

• ~95% of weights stay INT4 (group size 128)
• Only layers that cannot stay INT4 (embeddings, LM head, etc.) use INT8-symmetric backup
• Validated on the sibling v2 build on: CPU (Core Ultra 7 256V), NPU (Intel AI Boost / NPU4000), iGPU (Arc 140V)

This export uses the identical recipe and tooling versions as the validated v2 build, so NPU behavior is expected to match. NPU smoke-test on your own Lunar Lake machine recommended after first load (see notes below).

What changed vs. the v2 export

LFM2.5-2.6B is the agentic post-trained successor:

• Vocabulary doubled to 128K (from 64K) for non-Latin script support - larger embedding/LM head tables
• Context window 131,072 (was 128,000)
• Same hybrid backbone: 30 layers (22 double-gated short conv blocks + 8 GQA), 2.69B params
• Post-trained with agentic RL inside agent harnesses (tool use, instruction following, multi-step tasks)
• Same ChatML-like chat template (<|im_start|> / <|im_end|>), tool-call support

Quick start (NPU) - OpenVINO GenAI

pip install openvino openvino-genai transformers

import openvino_genai as ov_genai
from transformers import AutoTokenizer

model_id = "mosesman/LFM2.5-2.6B-openvino-int4-npu"
device = "NPU"  # or "GPU" / "CPU"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "What is 17*19? Reply with only the number."}],
    tokenize=False,
    add_generation_prompt=True,
)

pipe = ov_genai.LLMPipeline(model_id, device)
config = ov_genai.GenerationConfig()
config.max_new_tokens = 128
config.do_sample = False

print(pipe.generate(prompt, config))

NPU-only tip: pass "NPU" (not AUTO / HETERO). Tokenizer/detokenizer glue may still use CPU; model compute stays on NPU.

Liquid recommends these generation params for LFM2.5: temperature 0.1, top_k 50, repetition_penalty 1.1.

First-time NPU compile vs later starts

The first NPU load can take ~1-2 minutes while OpenVINO / the NPU driver compiles the graph. OpenVINO caches the compiled blob (.blob / driver caches under local temp / cache dirs), so subsequent cold starts usually launch in seconds. Later generates in the same process are much faster either way.

Alternative: Optimum-Intel (OVModelForCausalLM)

pip install "optimum[openvino]" transformers

from optimum.intel.openvino import OVModelForCausalLM
from transformers import AutoTokenizer, pipeline

model_id = "mosesman/LFM2.5-2.6B-openvino-int4-npu"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = OVModelForCausalLM.from_pretrained(model_id, device="NPU")

pipe = pipeline("text-generation", model=model, tokenizer=tokenizer)
messages = [{"role": "user", "content": "What is 17*19? Reply with only the number."}]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
print(pipe(prompt, max_new_tokens=32, do_sample=False)[0]["generated_text"])

Local helper script

This folder includes run_lfm_device.py (NPU -> GPU -> CPU fallback):

python run_lfm_device.py --device NPU --greedy

Reproduce the export

optimum-cli export openvino \
  --model LiquidAI/LFM2.5-2.6B \
  --trust-remote-code \
  --weight-format int4 \
  --sym \
  --group-size 128 \
  --backup-precision int8_sym \
  ./LFM2.5-2.6B-openvino-int4-npu

Tooling used for this build (must match - see note below):

• optimum-intel 2.1.0
• nncf 3.3.0
• openvino / openvino-genai 2026.3.0
• transformers 5.4.0 (LFM2.5's config rejects newer Transformers - ValueError: Maximum required is 5.4.0)

Benchmark notes

Greedy decode measurements from the sibling v2 build on the same machine (identical recipe; expect similar figures):

Device	Approx. tok/s	Notes
Arc 140V (GPU)	~59	fastest
CPU	~35	quick load
AI Boost (NPU)	~20	lowest power; slow first compile

Exact numbers vary by driver, prompt length, and power mode.

Requirements

• Intel Core Ultra with Intel AI Boost (for NPU)
• Recent Intel NPU driver (v2 build tested on 32.0.100.4841; OpenVINO notes recommend >= 32.0.100.4621 on Windows)
• OpenVINO 2026.3+ recommended (LFM2 NPU / NPUW support)

Also runs on Intel GPU / CPU via the same IR if needed.

Files

OpenVINO GenAI layout:

• openvino_model.xml / .bin
• openvino_tokenizer.xml / .bin
• openvino_detokenizer.xml / .bin
• tokenizer + chat template configs

License / attribution

Derivative of LiquidAI/LFM2.5-2.6B under the LFM Open License v1.0 (lfm1.0).

• License text: see LICENSE (copied from the base model)
• Upstream: https://huggingface.co/LiquidAI/LFM2.5-2.6B
• License FAQ: https://www.liquid.ai/lfm-license

This repository redistributes a quantized derivative. Downstream commercial use remains subject to Liquid AI's license terms (including the commercial revenue threshold).

Disclaimer

Provided as-is for interoperability with Intel NPU / OpenVINO. Not affiliated with Liquid AI or Intel. Validate quality for your own workload before production use.

Citation

If you use LFM2.5, please cite Liquid AI's model card / paper for LiquidAI/LFM2.5-2.6B.