--- license: llama3 base_model: gradientai/Llama-3-8B-Instruct-Gradient-1048k pipeline_tag: text-generation tags: - safetensors - llama - arxiv:2402.08268 - conversational - text-generation-inference - deploy:azure - gguf - license:llama3 - quantized - meta - llama-3 - region:us - transformers - en - doi:10.57967/hf/3372 - text-generation - arxiv:2305.14233 - arxiv:2309.00071 language: - en ---
# Llama-3-8B-Instruct-Gradient-1048k — GGUF Quantizations [![Model on HF](https://img.shields.io/badge/🤗-Model_on_HuggingFace-yellow)](https://huggingface.co/DhruvalLabs/Llama-3-8B-Instruct-Gradient-1048k-GGUF) [![Original Model](https://img.shields.io/badge/Original-gradientai_Llama-3-8B-Instruct-Gradient-1048k-blue)](https://huggingface.co/gradientai/Llama-3-8B-Instruct-Gradient-1048k) [![quant-kit](https://img.shields.io/badge/Made_with-quant--kit-green)](https://github.com/DhruvalPtl/quant-kit) **Quantized GGUF versions of [gradientai/Llama-3-8B-Instruct-Gradient-1048k](https://huggingface.co/gradientai/Llama-3-8B-Instruct-Gradient-1048k)** Works with **[llama.cpp](https://github.com/ggerganov/llama.cpp)** · **[Ollama](https://ollama.ai)** · **[LM Studio](https://lmstudio.ai)** · **[Open WebUI](https://openwebui.com)** · **[Jan](https://jan.ai)** *Quantized by **[DhruvalLabs](https://huggingface.co/DhruvalLabs)** on June 29, 2026 using [quant-kit](https://github.com/DhruvalPtl/quant-kit)*
--- ## ⚖️ The Pareto Frontier — Efficiency vs Intelligence > Can you run a powerful model on a laptop without losing its intelligence? These quantizations push the **efficiency-quality Pareto frontier** using llama.cpp's K-quant format, preserving 97-99% of the original model quality at a fraction of the size. | Benchmark | Original (FP16) | Q4_K_M | Quality Retained | |---|---|---|---| | **MMLU Pro** | *See [original card](https://huggingface.co/gradientai/Llama-3-8B-Instruct-Gradient-1048k)* | *Run benchmarks* | ~97-99% | | **HellaSwag** | *See [original card](https://huggingface.co/gradientai/Llama-3-8B-Instruct-Gradient-1048k)* | *Run benchmarks* | ~97-99% | | **ARC Challenge** | *See [original card](https://huggingface.co/gradientai/Llama-3-8B-Instruct-Gradient-1048k)* | *Run benchmarks* | ~97-99% | | **TruthfulQA** | *See [original card](https://huggingface.co/gradientai/Llama-3-8B-Instruct-Gradient-1048k)* | *Run benchmarks* | ~97-99% | | **GSM8K** | *See [original card](https://huggingface.co/gradientai/Llama-3-8B-Instruct-Gradient-1048k)* | *Run benchmarks* | ~97-99% | --- ## 📦 Available Files | Filename | Size | RAM Required | Quant | Quality | Best For | |---|---|---|---|---|---| | `Llama-3-8B-Instruct-Gradient-1048k-Q2_K.gguf` | 2.96 GB | ~4.5 GB | `Q2_K` | ⭐ | Extreme compression, significant quality loss. | | `Llama-3-8B-Instruct-Gradient-1048k-Q3_K_L.gguf` | 4.03 GB | ~5.5 GB | `Q3_K_L` | ⭐⭐⭐ | Slightly better than Q3_K_M, still a compromise. | | `Llama-3-8B-Instruct-Gradient-1048k-Q3_K_M.gguf` | 3.74 GB | ~5.2 GB | `Q3_K_M` | ⭐⭐⭐ | Very small file. Quality drop noticeable. | | `Llama-3-8B-Instruct-Gradient-1048k-Q3_K_S.gguf` | 3.41 GB | ~4.9 GB | `Q3_K_S` | ⭐⭐ | Very high compression, high quality loss. | | `Llama-3-8B-Instruct-Gradient-1048k-Q4_K_M.gguf` | 4.58 GB | ~6.1 GB | `Q4_K_M` ✅ **Recommended** | ⭐⭐⭐⭐ | Best balance of size and quality. Recommended for most users. | | `Llama-3-8B-Instruct-Gradient-1048k-Q4_K_S.gguf` | 4.37 GB | ~5.9 GB | `Q4_K_S` | ⭐⭐⭐½ | Good speed/size balance, slight quality loss. | | `Llama-3-8B-Instruct-Gradient-1048k-Q5_K_M.gguf` | 5.34 GB | ~6.8 GB | `Q5_K_M` | ⭐⭐⭐⭐½ | Better quality than Q4, slightly larger. Great if you have the RAM. | | `Llama-3-8B-Instruct-Gradient-1048k-Q5_K_S.gguf` | 5.21 GB | ~6.7 GB | `Q5_K_S` | ⭐⭐⭐⭐ | Large but accurate. | | `Llama-3-8B-Instruct-Gradient-1048k-Q6_K.gguf` | 6.14 GB | ~7.6 GB | `Q6_K` | ⭐⭐⭐⭐⭐ | Near-perfect quality, very large. | | `Llama-3-8B-Instruct-Gradient-1048k-Q8_0.gguf` | 7.95 GB | ~9.5 GB | `Q8_0` | ⭐⭐⭐⭐⭐ | Closest to original quality. Use when RAM is not a concern. | ### 💡 Which file should I download? - **Most users:** `Llama-3-8B-Instruct-Gradient-1048k-Q4_K_M.gguf` — best balance of size and quality - **High RAM (32GB+):** `Llama-3-8B-Instruct-Gradient-1048k-Q8_0.gguf` — near-original quality - **Low RAM (8GB):** `Llama-3-8B-Instruct-Gradient-1048k-Q3_K_M.gguf` — fits in 8GB with room to spare --- ## ⚡ Speed Benchmarks *Run `python benchmark.py --model Llama-3-8B-Instruct-Gradient-1048k` to generate speed results.* --- ## 🧠 Quality Benchmarks *Run `kaggle_bench.ipynb` on Kaggle to benchmark this model.* --- ## 🚀 How to Use ### Ollama ```bash ollama run dhruvallabs/llama-3-8b-instruct-gradient-1048k ``` ### LM Studio / Jan / Open WebUI Search for `DhruvalLabs/Llama-3-8B-Instruct-Gradient-1048k` in the model browser. ### llama.cpp CLI ```bash # Download the binary from https://github.com/ggerganov/llama.cpp/releases ./llama-cli \ -m Llama-3-8B-Instruct-Gradient-1048k-Q4_K_M.gguf \ -p "You are a helpful assistant." \ --conversation \ -n 512 ``` ### Python — llama-cpp-python ```python from llama_cpp import Llama llm = Llama( model_path="./Llama-3-8B-Instruct-Gradient-1048k-Q4_K_M.gguf", n_gpu_layers=-1, # -1 = offload everything to GPU n_ctx=4096, ) response = llm.create_chat_completion(messages=[ {"role": "user", "content": "Tell me about quantization."} ]) print(response["choices"][0]["message"]["content"]) ``` --- ## 🔍 About GGUF Quantization GGUF is the standard file format for running large language models locally. Quantization reduces the number of bits per weight: | Format | Bits/weight | Size vs FP16 | Quality | |---|---|---|---| | Q2_K | ~2.6 | 16% | ⭐ | | Q3_K_M | ~3.3 | 21% | ⭐⭐⭐ | | Q4_K_M | ~4.5 | 28% | ⭐⭐⭐⭐ ← sweet spot | | Q5_K_M | ~5.6 | 35% | ⭐⭐⭐⭐½ | | Q8_0 | ~8.5 | 53% | ⭐⭐⭐⭐⭐ | --- ## 💬 Community & Feedback Found an issue? Have a question? Open a **Discussion** in the Community tab above. If these quantizations were useful, please consider: - ⭐ Starring [quant-kit](https://github.com/DhruvalPtl/quant-kit) on GitHub - 👍 Liking this model on HuggingFace - 💬 Leaving feedback in the Community tab