README.md

---
license: mit
pipeline_tag: text-to-image
language:
- en
tags:
- image
- t2i
- text-to-image
- custom-code
- tiny
- siren
- cppn
model-index:
- name: PixelModel-v3
  results:
  - task:
      type: text-to-image
    dataset:
      name: fid
      type: fid
    metrics:
    - name: fid
      type: fid
      value: 392.10
  - task:
      type: text-to-image
    dataset:
      name: clip
      type: clip
    metrics:
    - name: clip
      type: clip
      value: 19.88
---

# PixelModel v3 🖼️

<img src="model.png" alt="The entire PixelModel v3: a 959x959 PNG holding all 919,427 weights" style="image-rendering: pixelated; width: 320px; max-width: 100%;">

A neural network where the weights **are** the image. Same gimmick as v1 and v2:
`model.png` is not a picture of the model, it is the model. Every pixel encodes a
weight. At inference the PNG is decoded back into weight matrices, the prompt is
embedded, and a coordinate network paints an image at any resolution.

The difference this time is the network. v2 was v1 with wider layers and nothing
else (its own card says "that's the only change"). v3 keeps the PNG idea and
rebuilds the architecture: learned word embeddings instead of a fixed hash, sine
activations instead of tanh, and FiLM conditioning instead of concatenation. It
is 919,427 parameters, and it beats v1's FID.

## What changed from v1

v1's real limits were three specific things, and v3 replaces each one.

1. **Learned word embeddings.** v1 hashed the prompt into a fixed vector with zero
   learnable parameters, so the text path could never learn what a word means. v3
   has a real embedding table over the common COCO caption words (lowercase,
   whitespace tokenised), mean pooled into the latent.
2. **A SIREN decoder.** tanh smears high frequencies. v3 uses sine activations
   with the proper SIREN initialisation (first layer uniform on plus or minus
   1/fan_in, later layers uniform on plus or minus sqrt(6/fan_in)/w0, w0 of 30).
   Getting that init wrong silently kills training, so `train.py` prints the
   per layer activation statistics in the first few steps as a check.
3. **FiLM conditioning.** Instead of gluing the latent onto the coordinates, the
   latent generates a per layer scale and shift that modulate each decoder layer.
   The FiLM generator is zero initialised, so at step 0 the scale is 1 and the
   shift is 0 and the SIREN statistics are left intact.

## Results

The headline number: on the same eval pipeline, v3 beats v1 on FID and lands a
hair short on CLIP.

![v1 vs v3 benchmark](pixelmodel-v3-benchmark.png)

| Metric | v0 | v1 | v2 | v3 |
|---|---|---|---|---|
| Parameters | 202,752 | 23,747 | 200,000 | **919,427** |
| FID, published | 566.84 | 439.46 | not published | see below |
| CLIP, published | 18.60 | 20.02 | not published | see below |
| FID, this pipeline (lower better) | | 420.75 | | **392.10** |
| CLIP, this pipeline (higher better) | | 20.10 | | 19.88 |
| `model.png` | 64x3200 | 160x149 | scaled | 959x959 (1.69 MB) |
| Native resolution | 32 | 64 | 64 | 128 |

Read this carefully, because the fair comparison is not v3 against v1's published
number. It is v3 against v1 re run through the exact same code.

- Re scoring v1 here gave FID **420.75**, not the published 439.46, and CLIP
  **20.10**, not 20.02. That roughly 19 point FID gap is library and subset drift,
  not a quality change. Comparing v3's 392.10 to the published 439.46 would have
  overstated the win by about 40 percent, which is why the regression is not
  optional.
- On the same pipeline, v3 vs v1 is FID **392.10 vs 420.75** (v3 wins by 28.6,
  about 7 percent) and CLIP **19.88 vs 20.10** (v3 is 0.22 behind, essentially
  tied). So the honest claim is: v3 clearly improves FID and matches CLIP, at
  about 39 times v1's parameter count but less than half of v2's.
- v2 has no published FID or CLIP (its card lists both as "NOT DONE") and its
  weights were not re scored here, so it contributes no number.

Everything labelled "this pipeline" is n=5000, MS COCO val2014 with a 256 center
crop, scored with `torchmetrics` (`FrechetInceptionDistance` and `CLIPScore` with
`openai/clip-vit-base-patch32`), same eval subset and same library versions. The
eval images are hash checked to be disjoint from training.

## Training

![training loss](pixelmodel-v3-loss.png)

80 epochs on a single RTX 3090, about 40 minutes, batch 32, 128x128 crops, cosine
learning rate from 2e-4, mixed precision. Training loss goes from 0.0741 to
0.0641 and is still drifting down at the end.

### On scaling up

I also trained a larger variant (about 3.4M parameters, wider and deeper, at
256x256) to see whether the extra capacity helps. It does not. At the base
learning rate it would not descend at all, and even after dropping the learning
rate its best loss (0.070) never reached base's 0.064 before it destabilised. A
deeper SIREN with a large FiLM hypernetwork is simply harder to optimise here, and
the extra parameters buy nothing for this task. So the released model is the
919K base. That is the honest outcome, not a hidden one.

## Architecture

```text
prompt string
  lowercase / whitespace tokenize -> token ids (up to 20)
  learned embedding table (3923 x 64), mean pool over real tokens   [251,072]
  Linear(64 -> 256) -> sin
  Linear(256 -> 192)                                        = latent z (192)

FiLM generator (zero init, so scale=1 / shift=0 at start):
  z -> Linear(192 -> 2 x 256 per sine layer)                = (scale, shift) x4  [395,264]

for every pixel (x, y), decoded as one batched tensor (no python loop):
  Fourier features of (x, y): [x, y] plus sin/cos over 8 octaves = 34 dims
  SineLayer 0: Linear(34  -> 256) -> FiLM -> sin(30 * .)    (SIREN first layer)
  SineLayer 1: Linear(256 -> 256) -> FiLM -> sin(30 * .)
  SineLayer 2: Linear(256 -> 256) -> FiLM -> sin(30 * .)
  SineLayer 3: Linear(256 -> 256) -> FiLM -> sin(30 * .)
  head: Linear(256 -> 3) -> sigmoid                         = RGB
```

Because pixels come from coordinates, the parameter count does not depend on
resolution: `--res 256` runs the same 919,427 weights as `--res 64`.

| block | tensors | parameters |
|---|---|---|
| embedding table | `embed.weight` | 251,072 |
| text encoder | `text_fc1`, `text_fc2` | 65,984 |
| FiLM generator | `film` | 395,264 |
| SIREN decoder | `sine_layers.0..3` | 206,336 |
| RGB head | `head` | 771 |
| **total** | | **919,427** |

## Weight encoding

Same scheme as v1, sized up. Each pixel stores one weight at 16 bit precision,
mapped linearly from the range [-2, 2]:

- R channel is the high byte
- G channel is the low byte
- B channel is reserved

At 919,427 weights the PNG is a roughly square 959x959 image of 1.69 MB that loads
in about 30 ms. Round trip quantisation error is about 3e-5 per weight, and no
trained weight fell outside [-2, 2]. `model.safetensors` holds the identical
weights in standard format with the full parameter breakdown in its header.

## Usage

```bash
# build the train and eval data plus the vocabulary from COCO
python fetch_coco_subset.py --out ../pm-work

# train (writes model.png and config.json every epoch)
python train.py --data ../pm-work/coco_train.npz --vocab ../pm-work/vocab.json \
    --crop 128 --batch-size 32

# inference from model.png (the canonical model)
python main.py "a red double decker bus" --out bus.png

# standalone inference from model.safetensors (needs only that file plus torch)
python convert_to_safetensors.py
python INFERENCE.py "a red double decker bus" --out bus.png

# any resolution from the same weights
python main.py "a beach with palm trees" --res 256

# eval: FID and CLIP
python eval/run_eval.py --arch v3 --work ../pm-work --n 5000

# regression: re score v1 through the identical pipeline for a fair comparison
python eval/run_eval.py --arch precomputed --work ../pm-work \
    --images-dir v1_render/ --n 5000
```

`main.py` (from `model.png`) and `INFERENCE.py` (from `model.safetensors`) produce
byte identical output for the same prompt and resolution.

## What to expect

This is a 919K parameter coordinate network. It is far more capable than v1's 23K,
and it tracks caption conditioned colour and coarse layout better, but it does not
draw a sharp bus. The point is the size to quality ratio and the honesty of the
PNG as model idea, measured by `eval/run_eval.py` rather than asserted.

## Files

```text
model.png                   THE MODEL (959x959), produced by train.py
model.safetensors           same weights, standard format plus param metadata
config.json                 architecture and parameter count metadata
vocab.json                  word level tokenizer vocabulary, from fetch
main.py                     inference, loads model.png
INFERENCE.py                inference, loads model.safetensors (standalone)
convert_to_safetensors.py   model.png to model.safetensors
train.py                    training (AMP, cosine LR, per epoch PNG write)
model.py                    architecture, tokenizer and PNG weight codec
fetch_coco_subset.py        builds train and eval data plus vocab from COCO
eval/run_eval.py            FID and CLIP Score via torchmetrics
```

## Environment

torch has no wheels for Python 3.14 yet, so use Python 3.11 or 3.12.

```bash
uv venv --python 3.11 .venv
# training
uv pip install torch numpy pillow safetensors datasets
# eval (FID and CLIP). torchmetrics CLIPScore breaks on transformers 5.x, so pin
# transformers to 4.49.0 or the metric raises inside torchmetrics.
uv pip install torchmetrics torchvision torch-fidelity scipy "transformers==4.49.0"
```

Training and eval here ran on a rented RTX 3090, torch 2.13.0+cu130.
`fetch_coco_subset.py` downloads the split before iterating rather than streaming
it, because a long streaming loop dies to dropped connections partway through.

Bench Labs. Simple, Reliable, Open sourced.