The Artha Personal-Finance Reasoning Benchmark

A reproducible, model-agnostic benchmark for evaluating LLM-based personal-finance agents on two axes that existing financial-QA benchmarks do not jointly exercise:

1. Answer quality — aggregative and multi-hop reasoning over a user's full transaction ledger (banking and investments), scored with a four-dimension rubric by a three-model-family LLM judge panel.
2. Grounding / hallucination resistance — queries that embed a false premise (an account, merchant, goal, or time period the user does not have), each verified false by construction.

The underlying user population is synthetic but BLS-calibrated (U.S. Bureau of Labor Statistics Consumer Expenditure Survey PUMD 2024, 20,793 households), so spending distributions are realistic rather than arbitrary. Any agent or LLM can be run against the queries and scored with the included harness and rubric — the benchmark does not depend on the Artha system.

See DATASHEET.md for the full Gebru-style datasheet (composition, calibration, labeling, uses, limitations).

Dataset structure

Note on the Dataset Viewer. The transaction population ships as a nested per-user JSON map (data/artha_users_v2.json) rather than a flat table, so the automatic HF viewer may not render it. Load it directly:

import json
from huggingface_hub import hf_hub_download
path = hf_hub_download("Tej-Katika/artha-benchmark",
                       "data/artha_users_v2.json", repo_type="dataset")
users = json.load(open(path))

Quick start

# 1. Regenerate the calibrated population (or use data/artha_users_v2.json).
python data_generation/generate_artha_data_v2.py   # reads calibration/archetype_profiles.json

# 2. Point your agent at the queries in benchmarks/*.yml and collect responses.
#    Each YAML pins the question and varies the user; answer one query per (category, user).

# 3. Score with the rubric. The harness uses Claude Opus as the primary judge:
python harness/artha_eval.py --domain banking --n-banking 10
#    Cross-family judges (re-score the same responses):
python harness/judge_openai.py  --input <results>.json --model gpt-5 --out-tag openai_gpt5
python harness/judge_gemini.py  --input <results>.json --model gemini-2.5-flash

Report per-judge pass rates (Opus / GPT-5 / Flash), not a single fused number — same-family bias is real and the panel exists to expose it.

Reference results (same data, same judges)

Provided to calibrate new submissions, not as a leaderboard. The improvement over the baseline is significant under all three judges (paired McNemar, n=599).

Reproducing the paper's exact numbers. The reference results were computed on a later regeneration of 509 users / 127,952 transactions held in the evaluation database, not on the 500-user snapshot shipped here. Both come from the same generator and calibration. The exact evaluation-population export will be added as data/artha_eval_population_509.json; until then the 500-user snapshot is a calibration-faithful example for trying the harness.

License

Benchmark specifications and derived calibration data: CC BY 4.0. Code (harness, generator, calibration scripts): MIT. The BLS source is U.S. Government public-domain data. See LICENSE.

Citation

If you use this benchmark, please cite both the artifact and the accompanying paper (see CITATION.cff):

@misc{katika2026arthabenchmark,
  title        = {The Artha Personal-Finance Reasoning Benchmark},
  author       = {Katika, Tejashwar Reddy},
  year         = {2026},
  howpublished = {Hugging Face Datasets},
  url          = {https://huggingface.co/datasets/Tej-Katika/artha-benchmark},
  note         = {DOI: 10.57967/hf/9387}
}

@article{katika2026artha,
  title   = {Artha: A Domain Ontology-Driven Agentic Framework for LLM-Based Personal Finance Reasoning},
  author  = {Katika, Tejashwar Reddy},
  year    = {2026},
  journal = {SSRN Electronic Journal},
  note    = {Preprint},
  url     = {https://ssrn.com/abstract=6885058}
}