rag-spec-evaluation.md

# RAG Spec Evaluation Report

## Context

This is a gap analysis of the Doc-Ingestion app against a production-grade RAG spec across three phases. The goal is to identify what is built, what partially meets the spec, and what is missing or misaligned.

---

## Spec vs. Implementation: Detailed Evaluation

### Phase 1 — Fundamentals

| Requirement | Status | Detail |
|---|---|---|
| Ingest documents | ✅ Built | PDF, DOCX, TXT, MD, HTML via `src/core/document_processor.py` |
| 500–800 **token** chunks | ⚠️ Partial / Misaligned | Chunking uses **characters** (default: 1000 chars, 200 overlap), not tokens. Spec requires token-based chunking (500–800 tokens). No tokenizer is applied during chunking. |
| 100-token overlap | ⚠️ Partial / Misaligned | Overlap is 200 characters, not 100 tokens. Same root issue: characters vs. tokens. |
| Vector store (Chroma or Qdrant) | ✅ Built | ChromaDB for dev (`data/embeddings/chroma`), Qdrant optional for prod. Both present in `src/utils/database.py`. |

**Gap to fix:**
- `src/core/document_processor.py` `chunk_text()` method uses character sliding window. Needs to be replaced with a tokenizer-aware splitter (e.g., `tiktoken` or `transformers` tokenizer) targeting 500–800 tokens with 100-token overlap.
- `config.yaml` `chunk_size: 1000` and `overlap: 200` need to change to token units.

---

### Phase 2 — Hybrid Retrieval + Re-ranking

| Requirement | Status | Detail |
|---|---|---|
| BM25 keyword search | ✅ Built | `src/core/bm25_search.py`, `BM25Search` using `rank-bm25` |
| Vector semantic search | ✅ Built | `src/core/vector_search.py`, `VectorSearch` with ChromaDB |
| Hybrid retrieval combining both | ✅ Built | `src/core/hybrid_retriever.py`, `HybridRetriever` with Reciprocal Rank Fusion (RRF), parallel execution |
| Cross-encoder re-ranker | ✅ Built | `src/core/reranker.py`, `CrossEncoderReranker` using `cross-encoder/ms-marco-MiniLM-L-6-v2` |

**Phase 2 is fully built and exceeds the spec** (adds RRF fusion, LRU caching, confidence scoring, configurable weights).

---

### Phase 3 — Evaluation Dataset + CI/CD

| Requirement | Status | Detail |
|---|---|---|
| Golden dataset of 50–200 Q&A pairs | ⚠️ Missing | Only `evals/datasets/smoke.jsonl` (~few entries, 1 KB) and `evals/datasets/sample.jsonl` (~6 KB, estimated ~10–15 pairs). Neither meets the 50–200 pair threshold. |
| Offline evaluation script | ✅ Built | `evals/run_evals.py` (504 lines) with 8+ metrics: answer_relevancy, context_precision, context_recall, ROUGE-L, citation_rate, faithfulness |
| CI/CD pipeline integration | ⚠️ Partial | `.github/workflows/ci.yml` has `evals-smoke` job, but it runs with `--mock` flag (MockPipeline). It does **not** measure real faithfulness — it tests the eval harness, not the RAG pipeline. |
| Measure faithfulness | ⚠️ Partial | NLI faithfulness via `src/evaluation/truthfulness.py` exists. RAGAS integration exists in `evals/adapters/ragas_llm_adapter.py` but is optional (requires `ragas>=0.2`, `langchain-core` extra deps) and not wired into CI. |

---

### Recommended Tech Stack Alignment

| Recommendation | Status | Detail |
|---|---|---|
| LangChain or LangGraph | ❌ Not used | Core pipeline uses direct HTTP API calls (`src/core/llm_provider.py`). LangChain only appears as a thin adapter in `evals/adapters/ragas_llm_adapter.py` to satisfy RAGAS interface — it is not the orchestration framework. |
| ChromaDB or Qdrant | ✅ Built | Both present |
| Ragas for evaluation | ⚠️ Optional / Incomplete | Present as optional adapter, not enforced. CI runs MockPipeline without RAGAS. |

---

## Summary: What Is Missing

### Must-Fix (spec violations)

1. **Token-based chunking** — `src/core/document_processor.py:chunk_text()` uses character counts, not tokens. Replace with tokenizer-aware chunking (e.g., `tiktoken`) targeting 500–800 tokens, 100-token overlap. Update `config.yaml` units accordingly.

2. **Golden evaluation dataset (50–200 pairs)** — `evals/datasets/` only has smoke (~few entries) and sample (~10–15 pairs). Need to create a curated dataset of at least 50 ground-truth Q&A pairs with reference contexts, authored against real ingested documents.

3. **CI/CD runs real faithfulness evaluation** — The `evals-smoke` GitHub Actions job uses `--mock`. A CI job that runs `LivePipeline` against the golden dataset (or a representative subset) and gates on a faithfulness threshold is required by the spec.

### Should-Fix (partial alignment)

4. **RAGAS made non-optional** — RAGAS faithfulness should be a hard dependency in `evals/`, not a conditional import behind `try/except`. The eval report should always include RAGAS faithfulness score.

5. **LangChain / LangGraph adoption** — The spec recommends LangChain/LangGraph as the orchestration layer. Currently the pipeline is custom HTTP. This is a tech stack deviation, not a functional gap — worth noting but lower priority than items 1–3.

---

## What Is Already Production-Grade (exceeds spec)

- Full hybrid retrieval with RRF fusion, parallel execution, and LRU caching
- Cross-encoder reranking with batch scoring and threshold filtering
- Multi-provider LLM routing (Ollama, OpenAI, Anthropic, Gemini) with streaming
- NLI-based inline truthfulness scoring in the serving path
- Citation tracking and verification
- Response caching (Redis + in-memory fallback)
- Rate limiting, API key auth, audit logging
- Comprehensive IR metrics module (P@K, R@K, MRR, MAP, NDCG)
- Unit + integration test coverage with CI

---

## Verification Steps (after fixes)

1. After token-based chunking: ingest a known document, query it, verify chunk boundaries fall within 500–800 token range using `tiktoken` inspection script.
2. After golden dataset: run `python -m evals.run_evals --dataset evals/datasets/golden.jsonl --live` and confirm dataset size ≥ 50.
3. After CI wiring: confirm GitHub Actions `evals-golden` job fails when faithfulness drops below threshold (e.g., `nli_faithfulness < 0.7`).