Inference scaling paper — data release (with eval logs)

This package contains the analysis tables and the Inspect AI .eval trajectory logs behind the paper's main analysis.

Benchmarks: FrontierMath (frontiermath), SWE-Bench Pro (swebenchpro), HealthBench (healthbench), Humanity's Last Exam (hle), Terminal-Bench (terminalbench). The paper's cyber CTF, TLO and CT evaluations are not part of this release.

Models: claude-opus-4-0, claude-opus-4-5, claude-opus-4-6, gpt-5-2025-08-07, gpt-5.2-2025-12-11, gpt-5.4-2026-03-05.

Conditions: S-adaptive (no feedback), S-adaptive+C (oracle feedback).

Files

data/trajectory_data.csv — one row per trajectory

The unit of analysis: a single agent run on a single benchmark task under one (model, condition, token limit). Adds stopping_reason, error_message, and trajectory-level token totals to the key columns.

data/submission_data.csv — one row per answer submission

Each trajectory may submit multiple candidate answers. submit_number is the 1-based submission index; score is the submission's score (benchmark-specific scale; 0/1 for pass–fail benchmarks, continuous for HealthBench). The unprefixed token columns (output_tokens_target_model, input_tokens_all_models, ...) are cumulative counts at the moment of the submission, so they give the inference spend up to each answer.

data/turn_data.csv — one row per agent turn

turn_number is the 1-based turn index; working_time / wall_time are seconds since the trajectory started. The unprefixed token columns are cumulative counts up to and including the turn.

Note on loading: the CSVs carry full double precision. To reproduce the source values bit-exactly with pandas, pass float_precision="round_trip" to read_csv — the default fast parser can be one unit-in-the-last-place off on a handful of values.

Key columns shared by all three tables:

stopping_reason (trajectory table) takes: completed_with_submit, completed_no_submit, token_limit (budget exhausted — a normal outcome under inference-scaling conditions, not an error), repetition_guard (stopped after repeated identical submissions). Trajectories that errored at the framework level were excluded upstream and re-run, so they do not appear here (error_message is always empty).

Provenance and filtering

Trajectories were produced with Inspect AI and post-processed as follows:

1. Trajectories that failed at the framework level were excluded and re-run. A sample was treated as errored/incomplete if (a) it carried a framework error (sample.error), (b) its event stream contained an ErrorEvent (retry-wrapped or sandbox-adapter failures), or (c) any of its submissions received no valid score ("silent scorer"). Failing tool calls (non-zero exits etc.) are normal agent behaviour and were not treated as errors.
2. Trajectories were aggregated across runs, capped to the target number of epochs per task, and restricted to a canonical task subset per benchmark; submissions without a valid score were dropped.
3. Rows for evals not included in this release (cyber CTFs, TLO, CT) were removed.

Notes for exact reproduction:

• These tables are byte-for-byte the analysis inputs used for the paper (only the excluded evals' rows were removed and log_file paths were made package-relative). Analyses that pool across the full seven-eval set in the paper will differ where the excluded evals contributed.
• A small number of turn/submission rows reference trajectories that have no trajectory_data.csv row: they come from superseded logs whose per-turn rows remained in the analysis cache and contributed to the paper's typical-token-budget calculation. They are retained to keep the shared data identical to the paper's inputs.
• One referenced log (logs/hle/flow_S-adaptive_HLE_gpt-5.2_5M_e1_s3.eval, 21 rows) no longer exists and is absent from the full package.

Eval logs (logs/)

Every .eval file referenced by the tables' log_file column, grouped into one directory per benchmark (logs/frontiermath/, logs/healthbench/, logs/hle/, logs/swebenchpro/, logs/terminalbench/). Each filename begins with a prefix recording where the run originally lived: flow_completed_ / flow_partial_ = the main experiment run (partial logs were salvaged from interrupted runs and carry no header-level results); from_ole_ = supplementary SWE-Bench Pro runs; iterations_<timestamp>_ = pilot runs whose data feeds the analysis. Files are Inspect AI eval logs; read them with

from inspect_ai.log import read_eval_log, read_eval_log_samples
log = read_eval_log("logs/hle/<name>.eval")   # full log
samples = read_eval_log_samples("logs/hle/<name>.eval",
                                all_samples_required=False)  # streaming

or browse them with inspect view --log-dir logs/.

Errored/incomplete samples (definition above) were removed from the shared logs:

• Logs consisting only of errored samples are excluded entirely.
• Logs mixing complete and errored samples were rewritten with the errored samples removed. Rewritten logs' header-level results/stats blocks predate the removal and may overcount — per-file original/kept/removed counts are in logs_manifest.csv.
• Untouched logs were copied byte-for-byte.

logs_manifest.csv lists every referenced log with its action (copied / rewritten / excluded_all_errored / missing_at_source) and per-reason removed-sample counts.

Match a table row to its log sample via (log_file, sample_id, original_epoch) — original_epoch corresponds to the sample's epoch field inside the log. Because logs are shared per-file, they may also contain samples (extra epochs, non-canonical tasks) that the filtered tables do not use.