kashif3314/nemotron-3.5-asr-streaming-0.6b-gguf

nemotron-3.5-asr-streaming-0.6b — GGUF (unofficial)

Unofficial conversion. Converted with parakeet.cpp from nvidia/nemotron-3.5-asr-streaming-0.6b. Not parity-validated — it has been smoke-tested (loads, transcribes English correctly, cache-aware streaming works on CPU and Apple Metal) but has not been run through parakeet.cpp's full WER-0-vs-NeMo validation across languages. For a validated, official build, watch mudler/parakeet-cpp-gguf.

A GGUF build of NVIDIA's multilingual, prompt-conditioned cache-aware streaming RNN-T (40+ locales) for running locally with parakeet.cpp — no Python at inference, CPU or any ggml GPU backend (Metal/CUDA/Vulkan).

Files

File	Recipe	Size	Stock parakeet.cpp?
nemotron-3.5-asr-streaming-0.6b-q4_k.gguf	q4_k linear, F32 rest	~0.67 GB	✅ yes
nemotron-3.5-asr-streaming-0.6b-q4k-f16.gguf	q4_k linear, f16 rest	~0.49 GB	⚠️ patched build only
nemotron-3.5-asr-streaming-0.6b-q4k-q8.gguf	q4_k linear, q8_0 rest	~0.46 GB ⭐	⚠️ patched build only

All three quantize the linear (mul_mat) weights to q4_k. They differ in the rest (conv/LSTM/featurizer/norm/bias):

• q4_k keeps them F32 — loads on any released parakeet.cpp.
• q4k-f16 / q4k-q8 store them f16 / q8_0 and rely on the loader dequantizing them back to F32 at load (compute path unchanged). That dequant is not yet in released parakeet.cpp, so these need a patched build (below).
• q4k-q8 is the recommended one — smallest, and WER 0 vs NeMo, byte-identical decode to the q4_k file on tested clips. (q4k-f16 is kept for reference.) q8_0 needs each tensor's leading dim divisible by 32; tensors that don't fit fall back to f16, which is why q8 is only a little smaller than f16 here.

Loading the patched-build files (q4k-q8 / q4k-f16)

Apply the patch in this repo — parakeet-rest-loader.patch — which adds the load-time dequant (plus a quantize --rest f16|q8_0 flag). It loads both files and leaves all existing models unaffected:

git clone --recursive https://github.com/mudler/parakeet.cpp && cd parakeet.cpp
git apply /path/to/parakeet-rest-loader.patch
cmake -B build -DPARAKEET_GGML_METAL=ON && cmake --build build -j   # or your backend
./build/examples/cli/parakeet-cli transcribe \
    --model nemotron-3.5-asr-streaming-0.6b-q4k-q8.gguf --input audio.wav --lang en

The patch modifies parakeet.cpp, which is MIT-licensed, and is provided under that same MIT License (copyright and license notice retained in the patch header). It does not change parakeet.cpp's license. If/when this lands upstream the patch becomes unnecessary.

Usage

# Offline
parakeet-cli transcribe --model nemotron-3.5-asr-streaming-0.6b-q4_k.gguf \
    --input audio.wav --lang en        # or --lang auto / es / fr ...

# Cache-aware streaming
parakeet-cli transcribe --model nemotron-3.5-asr-streaming-0.6b-q4_k.gguf \
    --input audio.wav --stream --lang en

License

The model weights are governed by OpenMDW-1.1, inherited from the upstream NVIDIA model. See the original model card for the full license and terms. This repository redistributes a format-converted copy of those weights under the same license.