--- title: GGUF Chat emoji: 🧠 colorFrom: indigo colorTo: purple sdk: docker app_port: 7860 pinned: false license: apache-2.0 --- # GGUF Chat (Docker Space) A self-contained Hugging Face **Docker Space** that downloads a GGUF quant of [`bartowski/google_gemma-3-2b-it-GGUF`](https://huggingface.co/bartowski/google_gemma-3-2b-it-GGUF) and serves it through a Gradio chat UI — with a live download/load progress bar on first request, then token-by-token streaming — powered by [`llama-cpp-python`](https://github.com/abetlen/llama-cpp-python). The default model is a small (~2B parameter) instruct model chosen specifically to run comfortably on the **free CPU-basic Spaces tier**. Swap in a different repo any time via the `GGUF_REPO_ID` environment variable — no code changes needed. --- ## What's in this repo | File | Purpose | |-------------------|-----------------------------------------------------------------------| | `Dockerfile` | Multi-stage build: tries a prebuilt CPU wheel for `llama-cpp-python` first, falling back to compiling it in a builder stage; ships a slim runtime image with no compilers. | | `requirements.txt`| Pure-Python runtime deps (`gradio`, `huggingface_hub`). | | `app.py` | Downloads/caches the GGUF, loads it with `llama-cpp-python`, and serves the Gradio UI (progress bar + streaming chat). | | `README.md` | This file (also the Space's metadata card, via the YAML frontmatter above). | This repo is ready to push directly to a new **Docker** Space with no further edits. --- ## A note on the free Spaces tier This project is tuned to actually work on the **free CPU basic** tier end-to-end, which comes with real constraints worth knowing up front: - **No persistent disk by default.** Model weights re-download on every full restart/rebuild (see "Model caching" below) unless you pay for the Persistent Storage add-on. - **2 vCPUs, 16GB RAM.** `N_THREADS` defaults to `2` to match the actual core count rather than over-subscribing; `N_CTX` defaults to a modest `4096` tokens, well within what a ~1B `Q4_K_M` model needs on this hardware. - **Build minutes are shared/limited infrastructure**, so the Dockerfile tries a prebuilt wheel for `llama-cpp-python` before ever compiling from source (see "Technical notes" below) — this is the single biggest lever for keeping first-time build time low on the free tier. - If you outgrow any of this — a bigger model, longer context — the fixes are all opt-in: upgrade to a paid CPU/GPU hardware tier, or add Persistent Storage, and adjust `N_CTX`/`N_THREADS` accordingly. No code changes required, since everything is environment-variable driven. ## Deploying to Hugging Face Spaces 1. Create a new Space at . 2. Choose **Docker** as the Space SDK (not "Gradio" or "Streamlit" — this project builds and runs its own Dockerfile). 3. Pick the **CPU basic (free)** hardware tier — this project is built to run entirely on CPU. 4. Push these four files to the Space repo: ```bash git clone https://huggingface.co/spaces// cd cp /path/to/Dockerfile /path/to/requirements.txt /path/to/app.py /path/to/README.md . git add . git commit -m "Deploy GGUF chat Space" git push ``` 5. The Space will build the Docker image (typically well under a minute for `llama-cpp-python` if a prebuilt wheel is available — see "Technical notes" — otherwise 15-30+ minutes compiling from source) and then start the container. On first chat message, `app.py` downloads the selected GGUF file from the Hub and shows a live progress bar in the UI itself (in addition to the Space's **Logs** tab). 6. Once the model finishes loading, replies stream in token-by-token. No secrets or tokens are required for the default (public) repo. If you point this at a **gated/private** GGUF repo, add an `HF_TOKEN` secret in the Space's **Settings → Variables and secrets**; `huggingface_hub` picks it up automatically. --- ## Configuration (environment variables) All of these are set with sensible defaults in the `Dockerfile` and can be overridden per-Space under **Settings → Variables and secrets** without touching any code: | Variable | Default | Description | |-----------------------|-------------------------------------------------------|--------------| | `GGUF_REPO_ID` | `bartowski/google_gemma-3-2b-it-GGUF` | Hub repo to pull the GGUF from. | | `GGUF_FILENAME` | *(empty = auto-select)* | Force an exact filename instead of auto-selecting by quant. | | `PREFERRED_QUANT` | `Q4_K_M` | Preferred quantization. Falls back automatically (`Q4_K_S` → `Q5_K_M` → ... → smallest available `.gguf`) if not present. `mmproj` (vision) and `-MTP-` (speculative-decoding draft head) files are skipped by the auto-selector in favor of a plain text-chat quant. | | `MODEL_CACHE_DIR` | `/data/models` | Local cache directory for downloaded model weights. | | `HF_HOME` | `/data/hf_home` | Cache directory for Hub metadata. | | `N_CTX` | `4096` | Context window (tokens) allocated at load time. Raise this if you swap in a model that needs more headroom and you have the RAM to back it. | | `N_THREADS` | `2` | CPU threads for inference. Matches the free tier's 2 vCPUs by default; raise it if you upgrade hardware. | | `N_BATCH` | `256` | Prompt processing batch size. | | `MAX_NEW_TOKENS` | `899` | Max tokens generated per reply. | | `TEMPERATURE` | `0.7` | Sampling temperature. | | `TOP_P` | `0.9` | Nucleus sampling. | | `TOP_K` | `40` | Top-k sampling. | | `REPEAT_PENALTY` | `1.1` | Repetition penalty. | | `SYSTEM_PROMPT` | *(a short "be concise, adapt to the request" prompt — see `app.py`)* | System prompt prepended to every conversation. Override to customize the assistant's behavior. | | `DOWNLOAD_MAX_RETRIES` | `5` | Retry attempts (exponential backoff) for the model download. | --- ## Model caching & the free tier's storage caveat `app.py` downloads the model once into `MODEL_CACHE_DIR` and reuses the cached file for every subsequent chat request — it will **not** re-download on every message, and it survives the container going to sleep/waking back up from inactivity. However, **the free Spaces tier has no *persistent* storage**: the container's disk (including `/data`) is rebuilt from scratch whenever the Space is fully **restarted or rebuilt** (e.g. after a `git push`, a factory reboot, or an infrastructure migration). In that case, the model will be re-downloaded once on the next startup — this is a platform limitation, not a bug in this app. If you need the cache to survive restarts, enable **Persistent Storage** for the Space (a paid add-on) and point `MODEL_CACHE_DIR`/`HF_HOME` at the mounted persistent volume (typically `/data`, which is already the default here). --- ## Local development (outside Docker) ```bash # Build llama-cpp-python with a build appropriate for your machine: CMAKE_ARGS="-DGGML_NATIVE=ON" pip install llama-cpp-python pip install -r requirements.txt python app.py # then open http://localhost:7860 ``` ## Building/running the Docker image locally ```bash docker build -t gguf-chat-space . docker run -it -p 7860:7860 gguf-chat-space # then open http://localhost:7860 ``` --- ## Technical notes - **`llama-cpp-python` install strategy**: the builder stage first tries to fetch a **prebuilt CPU wheel** from the maintainer's custom index (`https://abetlen.github.io/llama-cpp-python/whl/cpu`) using `pip wheel --only-binary=:all:`, which fails fast (rather than silently falling back to a slow source build) if no matching wheel exists for the image's Python ABI/platform. This turns the install from a **15-30+ minute from-source compile into a roughly one-minute download** on the common case. Only if no prebuilt wheel is available does the Dockerfile fall back to building from source, with `CMAKE_ARGS="-DGGML_NATIVE=OFF -DGGML_AVX2=ON -DGGML_FMA=ON -DGGML_F16C=ON"`. `GGML_NATIVE` is deliberately disabled because the machine that *builds* the Docker image is not guaranteed to be the same CPU that *runs* it; auto-detected "native" builds can otherwise crash with `SIGILL` on the Space's actual runner. AVX2/FMA/F16C are supported by essentially all modern cloud x86_64 CPUs and give good performance without that risk. The compiler toolchain (`build-essential`, `cmake`, `ninja-build`, `git`) is only installed in this fallback branch, and the fallback build caps itself at 4 parallel compile jobs to avoid getting OOM-killed on the free Spaces builder, which can report more CPU cores than it has RAM to back a fully parallel C++ build. - **Chat template**: the `Llama` object is created without a hardcoded `chat_format`, so `llama-cpp-python` auto-detects and applies the Jinja chat template embedded in the GGUF's own metadata. This is what lets the same `app.py` work correctly across different model families (Gemma, Llama, Qwen, ...) if you change `GGUF_REPO_ID`, without hardcoding any model-specific prompt formatting. - **Progress bar UI**: the model downloads/loads lazily on the *first* chat request rather than at container startup, and `app.py` streams a live HTML progress bar (download %, then an animated layer-load indicator) into the response pane while that happens, in addition to logging progress to the Space's **Logs** tab. The layer-load bar is a smooth "still working" animation rather than an exact per-layer readout, since `llama-cpp-python`'s Python API doesn't expose real-time load progress. - **Streaming**: implemented via `llm.create_chat_completion(..., stream=True)`, yielding incrementally-growing text into the UI for token-by-token display. - **GPU layers**: `n_gpu_layers=0` — this Space is CPU-only by design, matching the free Spaces hardware tier. - **File selection**: uses `huggingface_hub.HfApi().model_info(..., files_metadata=True)` to inspect all files with sizes, filters out `mmproj` (vision projector) and `-MTP-` (speculative decoding draft-head) variants by default, then picks the smallest file matching `PREFERRED_QUANT`, falling back through a quant-quality-ordered list if needed.