Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,337 @@
+---
+language:
+  - en
+license: gemma
+library_name: gguf
+tags:
+  - gemma3n
+  - document-qa
+  - extractive-qa
+  - rag
+  - gguf
+  - ollama
+  - cpu-compatible
+  - no-hallucination
+  - abstention
+pipeline_tag: question-answering
+base_model: google/gemma-3n-E4B-it
+datasets:
+  - adorosario/gemma3n-qa-synthetic
+model-index:
+  - name: gemma3n-qa-v4-fixed
+    results:
+      - task:
+          type: question-answering
+          name: Document-Grounded QA
+        dataset:
+          name: SimpleQA-Verified Synthetic Test
+          type: custom
+        metrics:
+          - type: exact_match
+            value: 83.2
+            name: Exact Match
+          - type: f1
+            value: 90.0
+            name: Token F1
+          - type: f1
+            value: 98.9
+            name: Abstention F1
+---
+
+# gemma3n-qa-v4-fixed
+
+**A fine-tuned Gemma 3n model for document-grounded question answering that eliminates hallucination and knows when to say "I don't know."**
+
+| Metric | This Model | Baseline | Improvement |
+|--------|------------|----------|-------------|
+| Exact Match | **83.2%** | 22.0% | **+61.2 pts** |
+| Token F1 | **90.0%** | 34.8% | **+55.2 pts** |
+| Abstention F1 | **98.9%** | ~0% | **+98.9 pts** |
+
+## TL;DR
+
+This model answers questions **only** from provided context. When the answer isn't there, it says `NOT FOUND IN DOCUMENTS` instead of making things up.
+
+**The problem it solves:** The baseline Gemma 3n hallucinates answers not in the context. Ask "Who is the president of France?" with context about the Eiffel Tower, and baseline confidently says "Emmanuel Macron" - information it made up. This fine-tuned version correctly responds "NOT FOUND IN DOCUMENTS."
+
+---
+
+## Quick Start
+
+### With Ollama
+
+```bash
+# Download the model
+curl -L -o gemma3n-qa-v4-fixed.gguf https://huggingface.co/adorosario/gemma3n-qa-v4-fixed/resolve/main/gemma3n-qa-v4-fixed-q4_k_m.gguf
+
+# Create Modelfile
+cat > Modelfile << 'EOF'
+FROM ./gemma3n-qa-v4-fixed.gguf
+TEMPLATE """<bos><start_of_turn>user
+{{ .System }}
+
+{{ .Prompt }}<end_of_turn>
+<start_of_turn>model
+{{ .Response }}<end_of_turn>"""
+PARAMETER stop <end_of_turn>
+PARAMETER stop <eos>
+PARAMETER temperature 0
+EOF
+
+# Create and run
+ollama create gemma3n-qa-v4-fixed -f Modelfile
+ollama run gemma3n-qa-v4-fixed
+```
+
+### Python API (Ollama)
+
+```python
+import requests
+
+def ask_document(question: str, context: str) -> str:
+    prompt = f"""You are a helpful assistant that answers questions based on provided context.
+If the answer is not found in the context, respond with "NOT FOUND IN DOCUMENTS".
+
+Question: {question}
+
+Context:
+{context}"""
+
+    response = requests.post(
+        "http://localhost:11434/api/generate",
+        json={
+            "model": "gemma3n-qa-v4-fixed",
+            "prompt": prompt,
+            "stream": False
+        }
+    )
+    return response.json()["response"]
+
+# Example
+answer = ask_document(
+    question="When was the Eiffel Tower built?",
+    context="The Eiffel Tower was built from 1887 to 1889 by Gustave Eiffel."
+)
+print(answer)  # Output: "from 1887 to 1889"
+```
+
+---
+
+## The Hallucination Problem (Why This Model Exists)
+
+### Baseline Behavior (Bad)
+
+```
+Question: Who is the president of France?
+Context: The Eiffel Tower is in Paris. It was built by Gustave Eiffel.
+
+Baseline Response: "Emmanuel Macron"  ← HALLUCINATED! Not in context!
+```
+
+### Fine-tuned Behavior (Good)
+
+```
+Question: Who is the president of France?
+Context: The Eiffel Tower is in Paris. It was built by Gustave Eiffel.
+
+Fine-tuned Response: "NOT FOUND IN DOCUMENTS"  ← Correct abstention!
+```
+
+This is critical for RAG applications where you need the model to be **honest about what it doesn't know**.
+
+---
+
+## Prompt Format (Required)
+
+The model requires this specific prompt format to work correctly:
+
+```
+You are a helpful assistant that answers questions based on provided context.
+If the answer is not found in the context, respond with "NOT FOUND IN DOCUMENTS".
+
+Question: {your question}
+
+Context:
+{your context}
+```
+
+**Without the abstention instruction**, the model may not properly refuse to answer questions outside the context.
+
+---
+
+## Performance
+
+### Benchmark Results (6,046 test examples)
+
+| Metric | Value | Description |
+|--------|-------|-------------|
+| **Exact Match** | 83.2% | Answer exactly matches gold standard |
+| **Token F1** | 90.0% | Token overlap with gold answer |
+| **Abstention Precision** | 98.2% | When it abstains, it's correct |
+| **Abstention Recall** | 99.7% | It catches almost all unanswerable questions |
+| **Abstention F1** | 98.9% | Combined abstention performance |
+
+### Comparison with Baseline
+
+| Metric | Fine-tuned | Baseline (gemma3n:e4b) | Improvement |
+|--------|------------|------------------------|-------------|
+| Exact Match | 83.2% | 22.0% | +61.2 pts (+278%) |
+| Token F1 | 90.0% | 34.8% | +55.2 pts (+159%) |
+| Abstention F1 | 98.9% | ~0% | Model learned abstention |
+
+### Statistical Significance
+
+- **p-value**: < 0.00001 (highly significant)
+- **95% CI**: 82.3% - 84.1% (fine-tuned) vs 13.9% - 30.1% (baseline)
+- Confidence intervals don't overlap
+
+---
+
+## Hardware Requirements
+
+| Hardware | Supported | Latency | Notes |
+|----------|-----------|---------|-------|
+| **CPU only** (8 cores, 32GB RAM) | Yes | 4-6 sec | Validated on n2-standard-8 |
+| NVIDIA T4 (16GB) | Yes | <1 sec | Recommended |
+| Consumer GPU (8GB) | Yes | 1-2 sec | Works with Q4_K_M |
+| Apple Silicon | Yes | 1-3 sec | Via llama.cpp |
+
+**Memory requirement**: ~10 GB RAM for inference
+
+---
+
+## Training Details
+
+### Base Model
+- **Model**: Google Gemma 3n E4B (4B effective parameters)
+- **Source**: `unsloth/gemma-3n-E4B-it-unsloth-bnb-4bit`
+
+### Fine-tuning Configuration
+
+| Parameter | Value |
+|-----------|-------|
+| Method | LoRA (Low-Rank Adaptation) |
+| Rank (r) | 32 |
+| Alpha | 64 |
+| Dropout | 0.05 |
+| Learning Rate | 2e-4 |
+| Epochs | 3 |
+| Batch Size | 4 (effective: 16 with grad accum) |
+| Precision | bfloat16 |
+| Training Time | ~20 hours on A100 40GB |
+
+### Training Data
+
+- **Dataset**: [adorosario/gemma3n-qa-synthetic](https://huggingface.co/datasets/adorosario/gemma3n-qa-synthetic)
+- **Size**: 57,081 examples (45,220 train / 5,815 val / 6,046 test)
+- **Composition**: 73% answerable QA, 27% abstention examples
+- **Source**: Synthetic generation from SimpleQA-Verified knowledge base
+- **Generation**: GPT-4o-mini
+- **Cost**: ~$15-20 USD
+
+### Critical Implementation Detail
+
+The v4 success came from **manual label masking** - training only on model responses, not on the prompt. Previous versions (v1, v3) failed because this wasn't properly implemented.
+
+---
+
+## How-To Guides
+
+### Use with llama.cpp
+
+```bash
+# Download
+wget https://huggingface.co/adorosario/gemma3n-qa-v4-fixed/resolve/main/gemma3n-qa-v4-fixed-q4_k_m.gguf
+
+# Run
+./llama-cli -m gemma3n-qa-v4-fixed-q4_k_m.gguf \
+  -p "You are a helpful assistant...\n\nQuestion: ...\n\nContext:\n..." \
+  --temp 0
+```
+
+### Use in a RAG Pipeline
+
+```python
+from langchain.llms import Ollama
+
+llm = Ollama(model="gemma3n-qa-v4-fixed", temperature=0)
+
+def rag_query(question: str, retrieved_docs: list) -> str:
+    context = "\n\n".join(retrieved_docs)
+    prompt = f"""You are a helpful assistant that answers questions based on provided context.
+If the answer is not found in the context, respond with "NOT FOUND IN DOCUMENTS".
+
+Question: {question}
+
+Context:
+{context}"""
+    return llm.invoke(prompt)
+```
+
+### Use with AnythingLLM
+
+1. Import the GGUF into Ollama (see Quick Start)
+2. In AnythingLLM, select `gemma3n-qa-v4-fixed` as the model
+3. Set system prompt to include the abstention instruction
+4. Set temperature to 0
+
+---
+
+## Limitations
+
+### What This Model Does Well
+- Extracting answers from provided context
+- Knowing when to abstain ("NOT FOUND IN DOCUMENTS")
+- Running on CPU-only hardware
+- Fast inference (4-6 seconds on CPU)
+
+### What This Model Does NOT Do
+- **Generate answers** beyond the context (by design)
+- **Multi-hop reasoning** requiring external knowledge
+- **Non-English languages** (trained on English only)
+- **Long contexts** beyond 4096 tokens
+- **Multi-turn conversation** (single-turn QA only)
+
+### Known Issues
+- Requires specific prompt format for abstention
+- ~2% quality loss from Q4_K_M quantization
+- May struggle with heavily paraphrased answers
+
+---
+
+## Files
+
+| File | Size | Description |
+|------|------|-------------|
+| `gemma3n-qa-v4-fixed-q4_k_m.gguf` | 7.68 GB | Main model (Q4_K_M quantization) |
+
+---
+
+## Citation
+
+```bibtex
+@misc{gemma3n-qa-v4-fixed-2025,
+  author = {Do Rosario, Alden},
+  title = {gemma3n-qa-v4-fixed: Fine-tuned Gemma 3n for Document-Grounded QA with Abstention},
+  year = {2025},
+  publisher = {HuggingFace},
+  url = {https://huggingface.co/adorosario/gemma3n-qa-v4-fixed},
+  note = {Fine-tuned for extractive QA with learned abstention behavior}
+}
+```
+
+---
+
+## Related Resources
+
+- **Training Dataset**: [adorosario/gemma3n-qa-synthetic](https://huggingface.co/datasets/adorosario/gemma3n-qa-synthetic)
+- **Base Model**: [Google Gemma 3n](https://ai.google.dev/gemma/docs/gemma-3n)
+- **Training Framework**: [Unsloth](https://github.com/unslothai/unsloth)
+
+---
+
+## Acknowledgments
+
+- Google for the Gemma 3n base model
+- Unsloth team for efficient fine-tuning tools
+- OpenAI for GPT-4o-mini used in synthetic data generation