Fine-tuning Gemma 4 for Hindi on a Free GPU — and shipping it to run on a CPU

I run a weekly Hindi LLM Series: I take small open models, fine-tune them to follow instructions in Hindi, and ship GGUF quants so they run on a laptop — no GPU, no API, no internet.

This post is the full walkthrough of the latest one: Gemma 4 E4B, fine-tuned for Hindi, trained for ₹0 on a free cloud GPU, running locally on CPU. It's part tutorial, part field notes — including the one bug that quietly destroyed an earlier model of mine, and the eval that kept me honest about what the fine-tune actually does.

If you want to skip to the result: 16-bit model · GGUF · live demo.

The setup: free GPU that actually supports bf16

I trained on Lightning.ai's free tier, which gives an on-demand L4 (24 GB). The L4 matters for one specific reason over the usual free Colab/Kaggle T4: it supports bfloat16. Gemma models are prone to float16 overflow during training, and bf16 sidesteps a whole class of NaN/instability problems. On a T4 you fight that; on an L4 it just works.

The stack:

• Unsloth for the fine-tuning (2x faster, lower VRAM, and it ships ready-made Gemma 4 notebooks)
• LoRA (r=16) — only ~0.46% of parameters are trained
• Gemma 4 E4B as the base

A quick note on E4B, because it surprised me: it runs at roughly 4B speed but has ~8B total parameters (it's a Matformer / per-layer-embeddings architecture). So the GGUF files are sized like an 8B model — the Q4_K_M lands at ~5.3 GB, not the ~2.5 GB you'd expect from a "4B." Worth knowing before you promise people a tiny download.

Loading it with Unsloth:

from unsloth import FastModel

model, tokenizer = FastModel.from_pretrained(
    model_name     = "unsloth/gemma-4-E4B-it",
    dtype          = None,        # auto → bf16 on L4
    max_seq_length = 2048,
    load_in_4bit   = True,        # QLoRA
    full_finetuning= False,
)

The data: two datasets, two different schemas

I used AI4Bharat's indic-instruct-data-v0.1 — specifically the anudesh and dolly subsets, Hindi splits, ~10k examples combined.

Here's a gotcha that cost me a few minutes: the two subsets have different schemas.

• anudesh (hi split) already has a messages column in chat format — [{role, content}, ...]. Ready to use.
• dolly (hi split) does not — it has separate instruction / context / response columns (plus quality_metrics with chrF++ scores from the back-translation).

So you can't just concatenate them. You map each into a common conversation format:

from datasets import load_dataset, concatenate_datasets

REPO = "ai4bharat/indic-instruct-data-v0.1"
anudesh = load_dataset(REPO, "anudesh", split="hi").rename_column("messages", "conversations").select_columns(["conversations"])
dolly   = load_dataset(REPO, "dolly",   split="hi")

def dolly_to_msgs(ex):
    ctx = (ex.get("context") or "").strip()
    user = ex["instruction"] if not ctx else f'{ex["instruction"]}\n\n{ctx}'
    return {"conversations": [
        {"role": "user",      "content": user},
        {"role": "assistant", "content": ex["response"]},
    ]}
dolly = dolly.map(dolly_to_msgs, remove_columns=dolly.column_names)

data = concatenate_datasets([anudesh, dolly]).shuffle(seed=3407).select(range(10000))

The one bug that matters: double-BOS

This is the part I most want people to read, because it cost me an entire model.

A few weeks earlier I fine-tuned Gemma 3 1B on Hindi. The result was broken in a fascinating way: it produced multilingual gibberish — it would start a Hindi sentence and drift into French, then Russian, then Thai, never actually answering. I shipped the whole package around it (model card, GGUF, demo Space) before realizing the model itself was cooked, then had to make everything private.

The likely culprit: a double BOS token.

Gemma's chat template already prepends a <bos> token. If your formatting code also adds one — or the data collator adds another on top of the template's — the model trains on inputs with two BOS tokens, and the base model's representations get scrambled. On a fragile 1B that's catastrophic.

The fix is to make sure exactly one BOS survives. When you apply the chat template during formatting, strip the leading <bos> so the collator's one is the only one:

def fmt(ex):
    texts = [tokenizer.apply_chat_template(c, tokenize=False,
             add_generation_prompt=False).removeprefix("<bos>")   # ← the fix
             for c in ex["conversations"]]
    return {"text": texts}

dataset = data.map(fmt, batched=True)
# sanity check: print(dataset[0]["text"][:200]) — you should see ONE <bos>, not two

I now print a sample after formatting every single time and eyeball the BOS count. Cheap insurance.

(One more E4B-specific landmine: setting use_cache=False corrupts E2B/E4B generations into garbage, and gradient checkpointing can force that flag. Recent Unsloth patches this — so just make sure you're on the latest unsloth / unsloth_zoo. This was another way to reproduce the "gibberish" symptom, unrelated to BOS.)

The training recipe

Nothing exotic — the recipe matters less than getting the template right.

model = FastModel.get_peft_model(
    model,
    finetune_vision_layers = False,    # text-only Hindi → freeze the vision tower
    finetune_language_layers = True,
    finetune_attention_modules = True,
    finetune_mlp_modules = True,
    r = 16, lora_alpha = 16, lora_dropout = 0, bias = "none",
)

from unsloth.chat_templates import get_chat_template, train_on_responses_only
tokenizer = get_chat_template(tokenizer, chat_template="gemma-4")

Two choices worth calling out:

1. finetune_vision_layers=False — E4B is multimodal, but this is a text-only Hindi model, so the vision tower is frozen dead weight we don't touch.
2. train_on_responses_only — loss is computed only on the assistant's reply, not the user's prompt. For instruction tuning this makes a real difference.

Trainer: batch size 2, grad accumulation 4, 2 epochs, LR 1e-4 (cosine), bf16. On the L4 this took a couple of hours and landed at a final train loss of ~0.29, falling smoothly the whole way.

The principle I actually learned: verify before you package

After the 1B disaster, I made one rule: confirm the model is good before building anything around it.

So this time, the order was:

1. Load the base E4B and ask it two Hindi questions → confirm the base is clean.
2. Train.
3. Ask the fine-tune a few Hindi questions → confirm clean, complete answers, no language drift.
4. Only then build cards, quants, the demo, and the posts.

This sounds obvious. It is obvious. But last time I did it backwards and burned a session polishing a broken model. This time the order saved me — every downstream step was built on something I'd already verified worked.

An honest eval: does the fine-tune actually beat the base?

Here's where it gets uncomfortable, in a good way. I ran a side-by-side eval: base Gemma 4 vs my fine-tune, on 25 Hindi prompts (facts, reasoning, creative, instructions).

The honest finding: base Gemma 4 already speaks Hindi well. It's a 140-language model. I did not make it "smarter." But two clear, repeatable differences emerged:

1. The base constantly code-switches to English. Ask it about health and you get:

संतुलित आहार लें **(Eat a Balanced Diet)**… साबुत अनाज **(whole grains)**… अस्वास्थ्यकर वसा **(unhealthy fats)**…

My fine-tune stays in clean, native Hindi — no parenthetical English.

2. The fine-tune follows instructions more tightly. Ask for "three tips" and the base writes a 1,200-character essay with markdown headers; the fine-tune gives exactly three clean lines. Ask for a "short message" and the base offers a menu of five options; the fine-tune writes one.

And to be fair — the base is more detailed and comprehensive. If you want depth, base wins. I optimized for concise, Hindi-native, instruction-following, and edge-deployable, not for raw capability.

Saying that out loud — "here's where my model is worse" — is, I think, the most useful thing in the whole writeup. A model card that only lists wins isn't an evaluation, it's an ad.

Shipping it: GGUF for the edge

The whole point of the series is local use, so I exported GGUF quants with Unsloth:

HF = "pankajpandey-dev/gemma-4-e4b-hindi-instruct"
model.push_to_hub_gguf(HF+"-GGUF", tokenizer, quantization_method="q4_k_m")
model.push_to_hub_gguf(HF+"-GGUF", tokenizer, quantization_method="q5_k_m")
model.push_to_hub_gguf(HF+"-GGUF", tokenizer, quantization_method="q8_0")
model.push_to_hub_merged(HF, tokenizer, save_method="merged_16bit")  # 16-bit
model.push_to_hub(HF+"-lora")                                        # adapter

Gotcha: because E4B is multimodal, the GGUF export also produced a BF16-mmproj.gguf — the vision projector. For a text-only model this is dead weight (~1 GB) that confuses users. Delete it from the repo; you only ship the text quants.

Sizes (remember, ~8B total params): Q4_K_M ≈ 5.3 GB, Q5_K_M ≈ 5.8 GB, Q8_0 ≈ 8.0 GB. Q4_K_M runs comfortably on a laptop with 8 GB+ RAM, on CPU.

The demo: a Gradio Space on free CPU

I built a demo Space that runs the Q4 GGUF via llama-cpp-python on a free CPU Space. Three things that bit me, so you don't have to:

• Use the prebuilt CPU wheel. Building llama-cpp-python from source on a Space takes forever and can fail. Pin the Space to Python 3.11 and use --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cpu so pip pulls a prebuilt wheel.
• Pin Gradio. A recent Gradio version crashed on schema parsing (argument of type 'bool' is not iterable). Pinning to a known-good version fixed it.
• Build the Gemma prompt manually. Relying on create_chat_completion's template gave me one-word replies. Constructing the turn format explicitly (<start_of_turn>user … <start_of_turn>model) with <bos> and proper stop tokens (<end_of_turn>) gave full, clean answers.

On CPU each token takes ~1–2 seconds, so I added an animated typing indicator — on a slow backend, showing the model is working matters as much as the output.

What didn't work: Ollama

I wanted a one-line ollama run for the model. It doesn't work yet — ollama create fails with llama-quantize failed: exit status 1 on Gemma 4 E4B GGUFs. This is an upstream Ollama bug (tracked in ollama/ollama#15447, the gemma4 architecture), not a problem with the model — the same GGUF runs fine in llama.cpp, LM Studio, and llama-cpp-python.

I tried updating Ollama, deleting the mmproj, and the workaround template from the issue thread. No luck. So I documented it honestly on the model card and moved on. The model is fully usable today; Ollama will catch up.

Takeaways

If you're starting your own small fine-tune:

1. Use a bf16-capable free GPU (L4 on Lightning) over a T4 — it removes a whole class of training instability.
2. Get the chat template right before anything else. Double-BOS will silently wreck a small model. Print a formatted sample and count your BOS tokens.
3. Verify the model before you package it. Cards, quants, demos, and posts are all wasted effort on a broken model. Confirm first.
4. Evaluate honestly, including where you lose. "Here's what my model is worse at" is more credible than a list of wins.
5. GGUF + a CPU demo is what makes a model usable by people without GPUs — that's the whole value of small models.

The model is live: 16-bit · GGUF · demo. It's part of my ongoing Hindi LLM Series — small, open, edge-friendly Indic models, one a week.

Built with Unsloth. Data by AI4Bharat. 🙏