Initial scaffold: gitignore, env, CLAUDE.md

.gitignore

@@ -0,0 +1,3 @@
+.env
+__pycache__/
+.venv/

CLAUDE.md

@@ -0,0 +1,171 @@
+# CLAUDE.md — AusRegBench
+
+> This file is read by Claude Code at the start of every session.
+> It is the source of truth for scope and rules. If a request conflicts with
+> this file, **stop and ask the human** before proceeding.
+
+## What this project is
+
+**AusRegBench** is an open, reproducible **benchmark + diagnostic tool** that measures
+where RAG systems get **Australian financial-services and prudential law** wrong —
+specifically how often they (a) cite provisions that don't exist, and (b) cite real
+provisions but misstate the obligation — and pinpoints **which pipeline stage**
+(retrieve / augment / generate) caused each failure.
+
+It is a **measurement instrument**, not a product. Its entire value is that its
+ground truth is trustworthy. Treat it that way in every decision.
+
+## Frozen scope — DO NOT EXPAND
+
+- **One domain:** Australian financial-services & prudential regulation. No other verticals.
+- **Five configs** (the only configs — no sixth): naive, hybrid, rerank, KG-augmented, grounded/citation-forced.
+- **One finding:** a faithfulness/stage-attribution result, e.g. "the provision was retrieved but the obligation was misstated, so the failure lives in generation, not retrieval."
+
+The temptation to add a vertical, a sixth config, MCP, multi-agent orchestration, or
+cloud deployment **mid-build** is the single biggest risk to this project shipping.
+All of those are **post-ship extensions only**, added one at a time, and only if they
+sharpen the finding. **If a session would add any of them, STOP and ask the human.**
+
+## The two human-verification non-negotiables
+
+1. **The gold set is human-verified.** The LLM may do the heavy lifting — surface the
+   relevant provision, summarise it, draft a candidate obligation, propose the citation.
+   But a human verifies every gold item against the actual provision text. The LLM is
+   never the source of truth for what the law says. (This is the whole point: we are
+   testing LLMs, so an LLM cannot define the correct answers — that's circular.)
+2. **The headline numbers are human-verified.** Fabrication rate, misstatement rate,
+   Cohen's κ, stage-attribution — the human reads the code that produces these and
+   spot-checks the outputs. These numbers are the findings; they cannot be vibe-checked.
+
+## The 90/10 coding rule
+
+- **Vibe-code the ~90% where a bug looks wrong** (Streamlit dashboard, config plumbing,
+  DeepEval wiring, run scripts). Go fast, hands-off.
+- **Human-verify the ~10% where a bug looks correct.** Specifically:
+  - **Clause-aware chunking that preserves paragraph IDs.** A misaligned ID silently
+    corrupts every downstream citation check. Spot-check chunks against the source by eye.
+  - **The deterministic citation-checker (regex).** A parsing edge-case bug prints a
+    clean, wrong number.
+  - **The κ computation.** Verify label arrays are aligned before trusting the value.
+
+## Locked stack
+
+- **Language:** Python 3.11. Pin dependencies (lockfile). Set seeds for reproducibility.
+- **Orchestration:** LangChain (pipelines) + LangGraph (the agentic / KG-augmented config).
+- **Vector DB:** **Neon** (managed serverless Postgres) + **pgvector**. No local Docker.
+- **Embeddings:** default `text-embedding-3-large` (OpenAI); plus **BGE-large** (open) for
+  the "embedding choice changes results" comparison.
+- **Retrieval:** BM25 (Postgres FTS or `rank_bm25`) + dense; combine with Reciprocal Rank Fusion.
+- **Reranker:** a cross-encoder (e.g. `bge-reranker`) or Cohere Rerank.
+- **Generation models (systems under test) — pick current versions at build time:**
+  - Frontier closed: GPT-5.x (OpenAI API)
+  - Strong open-weight, via API (not local): DeepSeek V4 (DeepSeek API or OpenRouter)
+  - Cheap/small tier: a budget model, to establish the cost floor
+- **Judge:** **Claude (Anthropic API)** — a different family from every generator, to avoid
+  self-preference bias. Never use the same model to generate and to judge.
+- **Eval metrics:** DeepEval (faithfulness, answer relevancy, contextual recall).
+- **Tracing:** LangSmith — span-level retrieve→augment→generate, for stage-attribution.
+- **Dashboard:** Streamlit, deployed on Streamlit Community Cloud or Hugging Face Spaces
+  (NOT AWS — over-engineering for a static benchmark; cloud/MLOps skills live in coursework).
+
+> Note: Claude Pro funds Claude Code + chat only. All pipeline model calls (OpenAI,
+> Anthropic judge, DeepSeek) bill separately on API credits. Use prompt caching and
+> batch endpoints to control cost and rate limits.
+
+## The five configs
+
+1. **Naive** — embed → top-k → generate. Baseline.
+2. **Hybrid** — BM25 + dense + RRF. Hypothesis: lexical matching helps on exact citation tokens.
+3. **Rerank** — retrieve 20 → cross-encoder → top 5. Hypothesis: higher precision cuts wrong-provision errors.
+4. **KG-augmented** — entity/relationship graph over provisions (the differentiator).
+   Hypothesis: helps where obligations cross-reference each other.
+5. **Grounded / citation-forced** — model must quote the provision verbatim + cite the
+   paragraph ID, and refuse if unsupported. Hypothesis: cuts fabrication/misstatement most.
+
+## Failure taxonomy (exactly one bucket per (query, config))
+
+1. **Fabricated citation** — provision doesn't exist. (Caught deterministically by regex.)
+2. **Real-but-irrelevant** — provision exists but doesn't support the claim.
+3. **Correct citation / misstated obligation** — exists and relevant, but the obligation is
+   wrong or distorted. ← the wedge.
+4. **Missing citation** — answer needed authority and gave none.
+5. **Correct & faithful** — the pass bucket.
+
+## Ground-truth pipeline (two layers)
+
+- **Layer 1 — deterministic.** Regex-parse every citation; check against the corpus citation
+  index. Catches bucket 1 with zero judgment. Fabrication rate is publishable on its own.
+- **Layer 2 — entailment, validated.** LLM judge sees the *actual retrieved provision text*
+  + the model's claim; decides entailment (buckets 2 & 3). **Validate the judge:** hand-label
+  ≥100 items, run the judge on the same 100, report **Cohen's κ** + accuracy. Judge model
+  must differ from the generator. This validation is the biggest credibility differentiator.
+
+## Gold set design (quality over count — aim 80–120)
+
+- Stratify by the failure mode each query targets: retrievability stress, distractor stress,
+  obligation-fidelity stress (the largest stratum), cross-reference stress.
+- Per query, record: question, gold answer, gold citation(s) with paragraph IDs, the supported
+  proposition, the stratum, and a one-line rationale.
+- **Anti-leakage:** NEVER put the gold answer or gold citation text into any prompt.
+- Favour concrete obligations ("an APRA-regulated entity must…") over vague principles.
+
+## Data
+
+- **Statutory:** Open Australian Legal Corpus (`isaacus/open-australian-legal-corpus`, HF).
+  Pull subset: Corporations Act 2001 (Cth), Banking Act 1959 (Cth), relevant regulations.
+- **Regulator instruments:** APRA Prudential Standards (start CPS 220, CPS 230, CPS 234);
+  a coherent handful of ASIC Regulatory Guides.
+- **Keep clause/paragraph IDs as metadata on every chunk.** Most important ingestion decision.
+- Attribute OALC (CC BY 4.0). Note provenance/licensing — it's on-theme for a compliance project.
+
+## Repo structure
+
+```
+ausregbench/
+  data/            # ingested corpus + provenance notes
+  src/
+    ingest.py      # clause-aware chunking, keeps paragraph IDs
+    retrieval.py   # bm25, dense, hybrid (RRF), rerank
+    configs/       # the 5 RAG configs
+    eval.py        # taxonomy scoring + DeepEval metrics
+    judge.py       # LLM-judge + kappa validation
+  queries/         # gold set as versioned .jsonl
+  results/         # raw runs + summary tables
+  dashboard/       # Streamlit app
+  README.md        # what/why/how + headline result + how to reproduce
+  methodology.md   # taxonomy, judge protocol, kappa, limitations
+  LICENSE
+```
+
+## Build phases
+
+1. **Skeleton:** Neon + pgvector, OALC ingest + clause-aware chunking (IDs preserved),
+   naive config end-to-end on ~10 queries.
+2. **Breadth:** the other 4 configs + eval harness + LangSmith tracing.
+3. **Ground truth:** full 80–120 gold set + two-layer judge + **validate the judge (κ)**.
+4. **Ship:** run everything, build the Streamlit dashboard, write the report.
+5. **(Optional, post-ship):** MCP wrapper, additional model, etc. — one at a time, only if it sharpens the finding.
+
+## Prior art to engage (cite correctly)
+
+- **Monash AusLaw Citation Benchmark** (Han, Burgess & Shareghi) — case-law/legislation
+  **citation prediction**; found standalone generative models fail almost entirely and BM25
+  beats dense retrieval. Cite as corroboration for the hybrid config AND as the "how is this
+  different" contrast (they predict citations; we measure obligation faithfulness + stage).
+- **Deakin "Seven Failure Points"** (Barnett et al., 2024, arXiv 2401.05856) — an engineering
+  **experience report** of RAG failure points across research/education/biomedical. NOT
+  case-law citation prediction. Contrast: they catalogued failures qualitatively; we built a
+  reproducible quantitative benchmark for one high-stakes domain.
+
+## When to STOP and ask the human
+
+- Any request to add a vertical, a 6th config, MCP, multi-agent orchestration, or AWS/cloud deploy.
+- Any change to chunking logic or how paragraph IDs are tracked.
+- Anything that touches the gold set's correctness.
+- Using the same model as both generator and judge.
+- Putting gold answers/citations into a prompt (leakage).
+
+## Done means done
+
+When the report is written, the repo is clean, the dashboard runs, and the LinkedIn post is
+drafted — **stop.** Resist infinite polishing. Shipped at 90% beats perfect-and-abandoned.