Reproducing the eval.md numbers

Step-by-step guide to reproduce the FID / CLIP Score numbers in eval.md from scratch. Unlike v0 (whose eval scripts lived in a throwaway env), everything needed is in this repo: fetch_coco_subset.py builds the data, eval/run_eval.py computes the metrics with torchmetrics.

CPU-only is fine (that's how the published numbers were produced). Budget ~7 GB disk: ~5 GB dataset shards + ~2 GB python deps and model weights (InceptionV3, CLIP ViT-B/32).

1. Environment

python3.13 -m venv eval-venv
# Windows: eval-venv\Scripts\activate    macOS/Linux: source eval-venv/bin/activate

pip install torch torchvision --index-url https://download.pytorch.org/whl/cpu
pip install torchmetrics torch-fidelity transformers "datasets<3" safetensors pillow numpy

torch-fidelity is the backend torchmetrics uses for the FID InceptionV3 features; transformers backs the CLIPScore metric.

2. Build the data (train + eval split)

python fetch_coco_subset.py --out ../pm-work

This streams sayakpaul/coco-30-val-2014 (a public 30K-pair subset of MS-COCO val2014) and produces:

eval_real/ — stream rows 0–4999: real images, 256×256 center-crop JPEG (FID real set)
eval_captions.json — the 5,000 captions for those rows (generation + CLIP Score)
coco_train.npz + coco_train.captions.json — training pairs from stream rows 10,000+, at 64×64

Leakage control: the same COCO image appears in multiple rows with different captions, so training rows are dropped if their image content hash (md5 of a 64×64 bilinear thumbnail) appears in the eval set, and deduped against each other. The counts printed at the end of the script report both.

Streaming row-by-row from the Hub is slow (~~hours on a normal connection). The much faster path is to download the 10 parquet shards first (~~4.9 GB), then:

python fetch_coco_subset.py --out ../pm-work --parquet-dir ../pm-work/shards

Sorted shard filename order equals hub streaming order, so both paths produce identical splits.

3. (Optional) retrain

python train.py --data ../pm-work/coco_train.npz --epochs 50

Deterministic given --seed (default 0): same data + same seed reproduces the published model.png up to floating-point ordering. Skip this step to evaluate the shipped model.png as-is.

4. Run the eval

python eval/run_eval.py --work ../pm-work --model model.png --n 5000

This:

Generates one 64×64 image per eval caption (native resolution — no upscaling before metrics).
FID via torchmetrics.image.fid.FrechetInceptionDistance (feature=2048, InceptionV3 pool3): real = 5,000 COCO val2014 images at 256×256 center-crop, fake = the 5,000 generated 64×64 images. The metric's Inception backbone resizes both sets to 299×299 internally.
CLIP Score via torchmetrics.multimodal.CLIPScore with openai/clip-vit-base-patch32 (the leaderboard default): each generated image against the caption that produced it, averaged.

Results are printed and written to eval_results.json in the --work dir. First run downloads InceptionV3 (~~95 MB) and CLIP ViT-B/32 (~~600 MB).

Notes

n=5000 vs the standard n=30000: FID's 2048-dim covariance estimate benefits from more samples; n=5000 was chosen as a compromise that is far past the singular-covariance regime that made v0's n=40 number directional-only, while staying CPU-friendly. Raising --n (and EVAL_N in fetch_coco_subset.py) toward 30K tightens the estimate at proportional cost — but note this dataset has 30K rows, not 30K unique images, and rows past index 10,000 are used for training, so a larger eval set requires re-splitting.
The generated set's resolution (64×64) is reported as the native generation resolution per the leaderboard rules; FID/CLIP preprocessing upsamples internally as part of the metric, not as part of generation.
INFERENCE.py (safetensors) and main.py (model.png) are verified byte-identical, so either weights file reproduces the same metrics.