PixelModel v3: SIREN+FiLM CPPN, 919K params, beats v1 FID

63a1291 verified 3 days ago

9.88 kB

license: mit
pipeline_tag: text-to-image
language:
  - en
tags:
  - image
  - t2i
  - text-to-image
  - custom-code
  - tiny
  - siren
  - cppn
model-index:
  - name: PixelModel-v3
    results:
      - task:
          type: text-to-image
        dataset:
          name: fid
          type: fid
        metrics:
          - name: fid
            type: fid
            value: 392.1
      - task:
          type: text-to-image
        dataset:
          name: clip
          type: clip
        metrics:
          - name: clip
            type: clip
            value: 19.88

PixelModel v3 🖼️

A neural network where the weights are the image. Same gimmick as v1 and v2: model.png is not a picture of the model, it is the model. Every pixel encodes a weight. At inference the PNG is decoded back into weight matrices, the prompt is embedded, and a coordinate network paints an image at any resolution.

The difference this time is the network. v2 was v1 with wider layers and nothing else (its own card says "that's the only change"). v3 keeps the PNG idea and rebuilds the architecture: learned word embeddings instead of a fixed hash, sine activations instead of tanh, and FiLM conditioning instead of concatenation. It is 919,427 parameters, and it beats v1's FID.

What changed from v1

v1's real limits were three specific things, and v3 replaces each one.

Learned word embeddings. v1 hashed the prompt into a fixed vector with zero learnable parameters, so the text path could never learn what a word means. v3 has a real embedding table over the common COCO caption words (lowercase, whitespace tokenised), mean pooled into the latent.
A SIREN decoder. tanh smears high frequencies. v3 uses sine activations with the proper SIREN initialisation (first layer uniform on plus or minus 1/fan_in, later layers uniform on plus or minus sqrt(6/fan_in)/w0, w0 of 30). Getting that init wrong silently kills training, so train.py prints the per layer activation statistics in the first few steps as a check.
FiLM conditioning. Instead of gluing the latent onto the coordinates, the latent generates a per layer scale and shift that modulate each decoder layer. The FiLM generator is zero initialised, so at step 0 the scale is 1 and the shift is 0 and the SIREN statistics are left intact.

Results

The headline number: on the same eval pipeline, v3 beats v1 on FID and lands a hair short on CLIP.

Metric	v0	v1	v2	v3
Parameters	202,752	23,747	200,000	919,427
FID, published	566.84	439.46	not published	see below
CLIP, published	18.60	20.02	not published	see below
FID, this pipeline (lower better)		420.75		392.10
CLIP, this pipeline (higher better)		20.10		19.88
`model.png`	64x3200	160x149	scaled	959x959 (1.69 MB)
Native resolution	32	64	64	128

Read this carefully, because the fair comparison is not v3 against v1's published number. It is v3 against v1 re run through the exact same code.

Re scoring v1 here gave FID 420.75, not the published 439.46, and CLIP 20.10, not 20.02. That roughly 19 point FID gap is library and subset drift, not a quality change. Comparing v3's 392.10 to the published 439.46 would have overstated the win by about 40 percent, which is why the regression is not optional.
On the same pipeline, v3 vs v1 is FID 392.10 vs 420.75 (v3 wins by 28.6, about 7 percent) and CLIP 19.88 vs 20.10 (v3 is 0.22 behind, essentially tied). So the honest claim is: v3 clearly improves FID and matches CLIP, at about 39 times v1's parameter count but less than half of v2's.
v2 has no published FID or CLIP (its card lists both as "NOT DONE") and its weights were not re scored here, so it contributes no number.

Everything labelled "this pipeline" is n=5000, MS COCO val2014 with a 256 center crop, scored with torchmetrics (FrechetInceptionDistance and CLIPScore with openai/clip-vit-base-patch32), same eval subset and same library versions. The eval images are hash checked to be disjoint from training.

Training

80 epochs on a single RTX 3090, about 40 minutes, batch 32, 128x128 crops, cosine learning rate from 2e-4, mixed precision. Training loss goes from 0.0741 to 0.0641 and is still drifting down at the end.

On scaling up

I also trained a larger variant (about 3.4M parameters, wider and deeper, at 256x256) to see whether the extra capacity helps. It does not. At the base learning rate it would not descend at all, and even after dropping the learning rate its best loss (0.070) never reached base's 0.064 before it destabilised. A deeper SIREN with a large FiLM hypernetwork is simply harder to optimise here, and the extra parameters buy nothing for this task. So the released model is the 919K base. That is the honest outcome, not a hidden one.

Architecture

prompt string
  lowercase / whitespace tokenize -> token ids (up to 20)
  learned embedding table (3923 x 64), mean pool over real tokens   [251,072]
  Linear(64 -> 256) -> sin
  Linear(256 -> 192)                                        = latent z (192)

FiLM generator (zero init, so scale=1 / shift=0 at start):
  z -> Linear(192 -> 2 x 256 per sine layer)                = (scale, shift) x4  [395,264]

for every pixel (x, y), decoded as one batched tensor (no python loop):
  Fourier features of (x, y): [x, y] plus sin/cos over 8 octaves = 34 dims
  SineLayer 0: Linear(34  -> 256) -> FiLM -> sin(30 * .)    (SIREN first layer)
  SineLayer 1: Linear(256 -> 256) -> FiLM -> sin(30 * .)
  SineLayer 2: Linear(256 -> 256) -> FiLM -> sin(30 * .)
  SineLayer 3: Linear(256 -> 256) -> FiLM -> sin(30 * .)
  head: Linear(256 -> 3) -> sigmoid                         = RGB

Because pixels come from coordinates, the parameter count does not depend on resolution: --res 256 runs the same 919,427 weights as --res 64.

block	tensors	parameters
embedding table	`embed.weight`	251,072
text encoder	`text_fc1`, `text_fc2`	65,984
FiLM generator	`film`	395,264
SIREN decoder	`sine_layers.0..3`	206,336
RGB head	`head`	771
total		919,427

Weight encoding

Same scheme as v1, sized up. Each pixel stores one weight at 16 bit precision, mapped linearly from the range [-2, 2]:

R channel is the high byte
G channel is the low byte
B channel is reserved

At 919,427 weights the PNG is a roughly square 959x959 image of 1.69 MB that loads in about 30 ms. Round trip quantisation error is about 3e-5 per weight, and no trained weight fell outside [-2, 2]. model.safetensors holds the identical weights in standard format with the full parameter breakdown in its header.

Usage

# build the train and eval data plus the vocabulary from COCO
python fetch_coco_subset.py --out ../pm-work

# train (writes model.png and config.json every epoch)
python train.py --data ../pm-work/coco_train.npz --vocab ../pm-work/vocab.json \
    --crop 128 --batch-size 32

# inference from model.png (the canonical model)
python main.py "a red double decker bus" --out bus.png

# standalone inference from model.safetensors (needs only that file plus torch)
python convert_to_safetensors.py
python INFERENCE.py "a red double decker bus" --out bus.png

# any resolution from the same weights
python main.py "a beach with palm trees" --res 256

# eval: FID and CLIP
python eval/run_eval.py --arch v3 --work ../pm-work --n 5000

# regression: re score v1 through the identical pipeline for a fair comparison
python eval/run_eval.py --arch precomputed --work ../pm-work \
    --images-dir v1_render/ --n 5000

main.py (from model.png) and INFERENCE.py (from model.safetensors) produce byte identical output for the same prompt and resolution.

What to expect

This is a 919K parameter coordinate network. It is far more capable than v1's 23K, and it tracks caption conditioned colour and coarse layout better, but it does not draw a sharp bus. The point is the size to quality ratio and the honesty of the PNG as model idea, measured by eval/run_eval.py rather than asserted.

Files

model.png                   THE MODEL (959x959), produced by train.py
model.safetensors           same weights, standard format plus param metadata
config.json                 architecture and parameter count metadata
vocab.json                  word level tokenizer vocabulary, from fetch
main.py                     inference, loads model.png
INFERENCE.py                inference, loads model.safetensors (standalone)
convert_to_safetensors.py   model.png to model.safetensors
train.py                    training (AMP, cosine LR, per epoch PNG write)
model.py                    architecture, tokenizer and PNG weight codec
fetch_coco_subset.py        builds train and eval data plus vocab from COCO
eval/run_eval.py            FID and CLIP Score via torchmetrics

Environment

torch has no wheels for Python 3.14 yet, so use Python 3.11 or 3.12.

uv venv --python 3.11 .venv
# training
uv pip install torch numpy pillow safetensors datasets
# eval (FID and CLIP). torchmetrics CLIPScore breaks on transformers 5.x, so pin
# transformers to 4.49.0 or the metric raises inside torchmetrics.
uv pip install torchmetrics torchvision torch-fidelity scipy "transformers==4.49.0"

Training and eval here ran on a rented RTX 3090, torch 2.13.0+cu130. fetch_coco_subset.py downloads the split before iterating rather than streaming it, because a long streaming loop dies to dropped connections partway through.

Bench Labs. Simple, Reliable, Open sourced.