README.md

---
license: mit
base_model: fixie-ai/ultravox-v0_5-llama-3_1-8b
base_model_relation: quantized
tags:
- ultravox 
- Llama 3.1 8b
- GGUF
- quantized
- 6-bit
---

## mmproj GGUF ultravox-v0_5-llama-3_1-8b by fixie-ai

Original model: https://huggingface.co/fixie-ai/ultravox-v0_5-llama-3_1-8b

This is a F16 mmproj file intented to be used in conjunction with Llama-3.1-8B-Instruct. A
high performance hybrid quant of Llama-3.1-8B-Instruct is available here: https://huggingface.co/steampunque/Llama-3.1-8B-Instruct-MP-GGUF

Usage:

Llama-3.1-8B-Instruct is made into an audio capable model using the fixie-ai audio multimedia projector tuned to work with it.  This
enables the model to input both audio (.mp3 and .wav files) and text inputs and generate text outputs.  The mmproj file is made available in this repository
and the hybrid quant model file is linked above and below. More information about running multimedia may be found in the docs in the mtmd readme in the tools directory
of the llama.cpp source tree https://github.com/ggml-org/llama.cpp/blob/master/tools/mtmd/README.md.

Extensive testing show that as of 7/1/2025 this is the only useful/usable small audio model available for llama.cpp.  The fixie 1b model based
on Llama 3.2 1b and qwen omni 7b were both found to be extremely unreliable at the task of transcribing audio streams which is one of the most useful
practical applications of an audio model.  This model can transcribe audio streams extremely accurately as long as the audio is broken up into ~60s
chunks maximum.  ffmpeg can easily handle breaking up the audio into chunks and formatting it to a desired type and sample rate.  Recommendeded
sample rate is 16000, single audio channel, .wav format with 16 bits per sample with input audio broken up into 30s to 60s chunks for transcription.
This config was tested to work well over a wide range of test audio clips ranging in duration up to 20m, with extremely accurate transcription
found for 3 different tested voices.

The Deepseek R1 distill of Llama 3.1 8b is also compatible with this mmproj.  A hybrid quant of this model is available here :
https://huggingface.co/steampunque/Deepseek-R1-Distill-Llama-8B-Hybrid-GGUF

Note that a file ultravox-v0_5-llama-3_1-8b.Q6_K_H.gguf was made available in this repository, but its use is deprecated in favor of
Llama-3.1-8B-Instruct naming to avoid confusion.  Future ultravox hybrid quant releases will only provide the original model name in a
separate model repository to avoid duplicating the exact file with a different name in the repository.

Benchmarks:

Audio benchmarks for the model will eventually be given here: https://huggingface.co/spaces/steampunque/benchlm

## Download the file from below:
| Link | Type | Size/e9 B | Notes |
|------|------|-----------|-------|
| [Llama-3.1-8B-Instruct.Q6_K_H.gguf](https://huggingface.co/steampunque/Llama-3.1-8B-Instruct-MP-GGUF/resolve/main/Llama-3.1-8B-Instruct.Q6_K_H.gguf) | Q6_K_H | 6e9 B | 0.6B smaller than Q6_K |
| [ultravox-v0_5-llama-3_1-8b.mmproj.gguf](https://huggingface.co/steampunque/ultravox-v0_5-llama-3_1-8b-MP-GGUF/resolve/main/ultravox-v0_5-llama-3_1-8b.mmproj.gguf) | mmproj | 1.38e9 B | multimedia projector |
 
A discussion thread about the hybrid layer quant approach can be found here on the llama.cpp git repository:

https://github.com/ggml-org/llama.cpp/discussions/13040