PixelModel v3: SIREN+FiLM CPPN, 919K params, beats v1 FID

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+model.png filter=lfs diff=lfs merge=lfs -text

INFERENCE.py

@@ -0,0 +1,63 @@
+"""PixelModel v3 inference from model.safetensors (standalone).
+
+Needs only: model.safetensors, model.py, vocab.json, torch. No model.png,
+no config.json - the architecture config is read from the safetensors header.
+Produces output identical to main.py for the same prompt and resolution.
+
+    python convert_to_safetensors.py
+    python INFERENCE.py "a red double decker bus" --out bus.png
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+
+import numpy as np
+import torch
+from PIL import Image
+from safetensors import safe_open
+
+from model import (
+    ModelConfig, PixelModelV3, load_vocab, encode_caption, make_coord_grid,
+)
+
+
+def load_from_safetensors(path, device):
+    with safe_open(path, framework="pt", device="cpu") as f:
+        meta = f.metadata() or {}
+        cfg = ModelConfig.from_json(json.loads(meta["config"]))
+        model = PixelModelV3(cfg)
+        state = {k: f.get_tensor(k) for k in f.keys()}
+    model.load_state_dict(state)
+    model.to(device).eval()
+    return model, cfg
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("prompt")
+    ap.add_argument("--out", default="out.png")
+    ap.add_argument("--res", type=int, default=128)
+    ap.add_argument("--safetensors", default="model.safetensors")
+    ap.add_argument("--vocab", default="vocab.json")
+    ap.add_argument("--device", default="cpu")
+    args = ap.parse_args()
+
+    model, cfg = load_from_safetensors(args.safetensors, args.device)
+    vocab = load_vocab(args.vocab)
+
+    tokens = encode_caption(args.prompt, vocab, cfg.max_tokens)
+    tokens = torch.from_numpy(tokens).long().unsqueeze(0).to(args.device)
+    coords = make_coord_grid(args.res, args.res, device=args.device,
+                             dtype=torch.float32).unsqueeze(0)
+    with torch.no_grad():
+        rgb = model(tokens, coords)
+    img = (rgb.clamp(0, 1).reshape(args.res, args.res, 3).cpu().numpy() * 255.0
+           ).round().astype(np.uint8)
+    Image.fromarray(img, "RGB").save(args.out)
+    print(f'[inference] "{args.prompt}" @ {args.res}x{args.res} -> {args.out}')
+
+
+if __name__ == "__main__":
+    main()

README.md

@@ -0,0 +1,243 @@
+---
+license: mit
+pipeline_tag: text-to-image
+language:
+- en
+tags:
+- image
+- t2i
+- text-to-image
+- custom-code
+- tiny
+- siren
+- cppn
+model-index:
+- name: PixelModel-v3
+  results:
+  - task:
+      type: text-to-image
+    dataset:
+      name: fid
+      type: fid
+    metrics:
+    - name: fid
+      type: fid
+      value: 392.10
+  - task:
+      type: text-to-image
+    dataset:
+      name: clip
+      type: clip
+    metrics:
+    - name: clip
+      type: clip
+      value: 19.88
+---
+
+# PixelModel v3 🖼️
+
+<img src="model.png" alt="The entire PixelModel v3: a 959x959 PNG holding all 919,427 weights" style="image-rendering: pixelated; width: 320px; max-width: 100%;">
+
+A neural network where the weights **are** the image. Same gimmick as v1 and v2:
+`model.png` is not a picture of the model, it is the model. Every pixel encodes a
+weight. At inference the PNG is decoded back into weight matrices, the prompt is
+embedded, and a coordinate network paints an image at any resolution.
+
+The difference this time is the network. v2 was v1 with wider layers and nothing
+else (its own card says "that's the only change"). v3 keeps the PNG idea and
+rebuilds the architecture: learned word embeddings instead of a fixed hash, sine
+activations instead of tanh, and FiLM conditioning instead of concatenation. It
+is 919,427 parameters, and it beats v1's FID.
+
+## What changed from v1
+
+v1's real limits were three specific things, and v3 replaces each one.
+
+1. **Learned word embeddings.** v1 hashed the prompt into a fixed vector with zero
+   learnable parameters, so the text path could never learn what a word means. v3
+   has a real embedding table over the common COCO caption words (lowercase,
+   whitespace tokenised), mean pooled into the latent.
+2. **A SIREN decoder.** tanh smears high frequencies. v3 uses sine activations
+   with the proper SIREN initialisation (first layer uniform on plus or minus
+   1/fan_in, later layers uniform on plus or minus sqrt(6/fan_in)/w0, w0 of 30).
+   Getting that init wrong silently kills training, so `train.py` prints the
+   per layer activation statistics in the first few steps as a check.
+3. **FiLM conditioning.** Instead of gluing the latent onto the coordinates, the
+   latent generates a per layer scale and shift that modulate each decoder layer.
+   The FiLM generator is zero initialised, so at step 0 the scale is 1 and the
+   shift is 0 and the SIREN statistics are left intact.
+
+## Results
+
+The headline number: on the same eval pipeline, v3 beats v1 on FID and lands a
+hair short on CLIP.
+
+![v1 vs v3 benchmark](pixelmodel-v3-benchmark.png)
+
+| Metric | v0 | v1 | v2 | v3 |
+|---|---|---|---|---|
+| Parameters | 202,752 | 23,747 | 200,000 | **919,427** |
+| FID, published | 566.84 | 439.46 | not published | see below |
+| CLIP, published | 18.60 | 20.02 | not published | see below |
+| FID, this pipeline (lower better) | | 420.75 | | **392.10** |
+| CLIP, this pipeline (higher better) | | 20.10 | | 19.88 |
+| `model.png` | 64x3200 | 160x149 | scaled | 959x959 (1.69 MB) |
+| Native resolution | 32 | 64 | 64 | 128 |
+
+Read this carefully, because the fair comparison is not v3 against v1's published
+number. It is v3 against v1 re run through the exact same code.
+
+- Re scoring v1 here gave FID **420.75**, not the published 439.46, and CLIP
+  **20.10**, not 20.02. That roughly 19 point FID gap is library and subset drift,
+  not a quality change. Comparing v3's 392.10 to the published 439.46 would have
+  overstated the win by about 40 percent, which is why the regression is not
+  optional.
+- On the same pipeline, v3 vs v1 is FID **392.10 vs 420.75** (v3 wins by 28.6,
+  about 7 percent) and CLIP **19.88 vs 20.10** (v3 is 0.22 behind, essentially
+  tied). So the honest claim is: v3 clearly improves FID and matches CLIP, at
+  about 39 times v1's parameter count but less than half of v2's.
+- v2 has no published FID or CLIP (its card lists both as "NOT DONE") and its
+  weights were not re scored here, so it contributes no number.
+
+Everything labelled "this pipeline" is n=5000, MS COCO val2014 with a 256 center
+crop, scored with `torchmetrics` (`FrechetInceptionDistance` and `CLIPScore` with
+`openai/clip-vit-base-patch32`), same eval subset and same library versions. The
+eval images are hash checked to be disjoint from training.
+
+## Training
+
+![training loss](pixelmodel-v3-loss.png)
+
+80 epochs on a single RTX 3090, about 40 minutes, batch 32, 128x128 crops, cosine
+learning rate from 2e-4, mixed precision. Training loss goes from 0.0741 to
+0.0641 and is still drifting down at the end.
+
+### On scaling up
+
+I also trained a larger variant (about 3.4M parameters, wider and deeper, at
+256x256) to see whether the extra capacity helps. It does not. At the base
+learning rate it would not descend at all, and even after dropping the learning
+rate its best loss (0.070) never reached base's 0.064 before it destabilised. A
+deeper SIREN with a large FiLM hypernetwork is simply harder to optimise here, and
+the extra parameters buy nothing for this task. So the released model is the
+919K base. That is the honest outcome, not a hidden one.
+
+## Architecture
+
+```text
+prompt string
+  lowercase / whitespace tokenize -> token ids (up to 20)
+  learned embedding table (3923 x 64), mean pool over real tokens   [251,072]
+  Linear(64 -> 256) -> sin
+  Linear(256 -> 192)                                        = latent z (192)
+
+FiLM generator (zero init, so scale=1 / shift=0 at start):
+  z -> Linear(192 -> 2 x 256 per sine layer)                = (scale, shift) x4  [395,264]
+
+for every pixel (x, y), decoded as one batched tensor (no python loop):
+  Fourier features of (x, y): [x, y] plus sin/cos over 8 octaves = 34 dims
+  SineLayer 0: Linear(34  -> 256) -> FiLM -> sin(30 * .)    (SIREN first layer)
+  SineLayer 1: Linear(256 -> 256) -> FiLM -> sin(30 * .)
+  SineLayer 2: Linear(256 -> 256) -> FiLM -> sin(30 * .)
+  SineLayer 3: Linear(256 -> 256) -> FiLM -> sin(30 * .)
+  head: Linear(256 -> 3) -> sigmoid                         = RGB
+```
+
+Because pixels come from coordinates, the parameter count does not depend on
+resolution: `--res 256` runs the same 919,427 weights as `--res 64`.
+
+| block | tensors | parameters |
+|---|---|---|
+| embedding table | `embed.weight` | 251,072 |
+| text encoder | `text_fc1`, `text_fc2` | 65,984 |
+| FiLM generator | `film` | 395,264 |
+| SIREN decoder | `sine_layers.0..3` | 206,336 |
+| RGB head | `head` | 771 |
+| **total** | | **919,427** |
+
+## Weight encoding
+
+Same scheme as v1, sized up. Each pixel stores one weight at 16 bit precision,
+mapped linearly from the range [-2, 2]:
+
+- R channel is the high byte
+- G channel is the low byte
+- B channel is reserved
+
+At 919,427 weights the PNG is a roughly square 959x959 image of 1.69 MB that loads
+in about 30 ms. Round trip quantisation error is about 3e-5 per weight, and no
+trained weight fell outside [-2, 2]. `model.safetensors` holds the identical
+weights in standard format with the full parameter breakdown in its header.
+
+## Usage
+
+```bash
+# build the train and eval data plus the vocabulary from COCO
+python fetch_coco_subset.py --out ../pm-work
+
+# train (writes model.png and config.json every epoch)
+python train.py --data ../pm-work/coco_train.npz --vocab ../pm-work/vocab.json \
+    --crop 128 --batch-size 32
+
+# inference from model.png (the canonical model)
+python main.py "a red double decker bus" --out bus.png
+
+# standalone inference from model.safetensors (needs only that file plus torch)
+python convert_to_safetensors.py
+python INFERENCE.py "a red double decker bus" --out bus.png
+
+# any resolution from the same weights
+python main.py "a beach with palm trees" --res 256
+
+# eval: FID and CLIP
+python eval/run_eval.py --arch v3 --work ../pm-work --n 5000
+
+# regression: re score v1 through the identical pipeline for a fair comparison
+python eval/run_eval.py --arch precomputed --work ../pm-work \
+    --images-dir v1_render/ --n 5000
+```
+
+`main.py` (from `model.png`) and `INFERENCE.py` (from `model.safetensors`) produce
+byte identical output for the same prompt and resolution.
+
+## What to expect
+
+This is a 919K parameter coordinate network. It is far more capable than v1's 23K,
+and it tracks caption conditioned colour and coarse layout better, but it does not
+draw a sharp bus. The point is the size to quality ratio and the honesty of the
+PNG as model idea, measured by `eval/run_eval.py` rather than asserted.
+
+## Files
+
+```text
+model.png                   THE MODEL (959x959), produced by train.py
+model.safetensors           same weights, standard format plus param metadata
+config.json                 architecture and parameter count metadata
+vocab.json                  word level tokenizer vocabulary, from fetch
+main.py                     inference, loads model.png
+INFERENCE.py                inference, loads model.safetensors (standalone)
+convert_to_safetensors.py   model.png to model.safetensors
+train.py                    training (AMP, cosine LR, per epoch PNG write)
+model.py                    architecture, tokenizer and PNG weight codec
+fetch_coco_subset.py        builds train and eval data plus vocab from COCO
+eval/run_eval.py            FID and CLIP Score via torchmetrics
+```
+
+## Environment
+
+torch has no wheels for Python 3.14 yet, so use Python 3.11 or 3.12.
+
+```bash
+uv venv --python 3.11 .venv
+# training
+uv pip install torch numpy pillow safetensors datasets
+# eval (FID and CLIP). torchmetrics CLIPScore breaks on transformers 5.x, so pin
+# transformers to 4.49.0 or the metric raises inside torchmetrics.
+uv pip install torchmetrics torchvision torch-fidelity scipy "transformers==4.49.0"
+```
+
+Training and eval here ran on a rented RTX 3090, torch 2.13.0+cu130.
+`fetch_coco_subset.py` downloads the split before iterating rather than streaming
+it, because a long streaming loop dies to dropped connections partway through.
+
+Bench Labs. Simple, Reliable, Open sourced.

config.json

@@ -0,0 +1,156 @@
+{
+  "total_parameters": 919427,
+  "param_breakdown": [
+    {
+      "name": "embed.weight",
+      "shape": [
+        3923,
+        64
+      ],
+      "params": 251072
+    },
+    {
+      "name": "text_fc1.weight",
+      "shape": [
+        256,
+        64
+      ],
+      "params": 16384
+    },
+    {
+      "name": "text_fc1.bias",
+      "shape": [
+        256
+      ],
+      "params": 256
+    },
+    {
+      "name": "text_fc2.weight",
+      "shape": [
+        192,
+        256
+      ],
+      "params": 49152
+    },
+    {
+      "name": "text_fc2.bias",
+      "shape": [
+        192
+      ],
+      "params": 192
+    },
+    {
+      "name": "film.weight",
+      "shape": [
+        2048,
+        192
+      ],
+      "params": 393216
+    },
+    {
+      "name": "film.bias",
+      "shape": [
+        2048
+      ],
+      "params": 2048
+    },
+    {
+      "name": "sine_layers.0.linear.weight",
+      "shape": [
+        256,
+        34
+      ],
+      "params": 8704
+    },
+    {
+      "name": "sine_layers.0.linear.bias",
+      "shape": [
+        256
+      ],
+      "params": 256
+    },
+    {
+      "name": "sine_layers.1.linear.weight",
+      "shape": [
+        256,
+        256
+      ],
+      "params": 65536
+    },
+    {
+      "name": "sine_layers.1.linear.bias",
+      "shape": [
+        256
+      ],
+      "params": 256
+    },
+    {
+      "name": "sine_layers.2.linear.weight",
+      "shape": [
+        256,
+        256
+      ],
+      "params": 65536
+    },
+    {
+      "name": "sine_layers.2.linear.bias",
+      "shape": [
+        256
+      ],
+      "params": 256
+    },
+    {
+      "name": "sine_layers.3.linear.weight",
+      "shape": [
+        256,
+        256
+      ],
+      "params": 65536
+    },
+    {
+      "name": "sine_layers.3.linear.bias",
+      "shape": [
+        256
+      ],
+      "params": 256
+    },
+    {
+      "name": "head.weight",
+      "shape": [
+        3,
+        256
+      ],
+      "params": 768
+    },
+    {
+      "name": "head.bias",
+      "shape": [
+        3
+      ],
+      "params": 3
+    }
+  ],
+  "png_width": 959,
+  "png_height": 959,
+  "png_pixels": 919681,
+  "png_bytes": 1769719,
+  "png_write_seconds": 0.4894,
+  "weights_clamped": 0,
+  "config": {
+    "vocab_size": 3923,
+    "embed_dim": 64,
+    "max_tokens": 20,
+    "text_hidden": 256,
+    "z_dim": 192,
+    "decoder_width": 256,
+    "num_sine_layers": 4,
+    "num_freq": 8,
+    "w0_first": 30.0,
+    "w0_hidden": 30.0,
+    "weight_range": [
+      -2.0,
+      2.0
+    ],
+    "coord_dim": 34
+  }
+}

convert_to_safetensors.py

@@ -0,0 +1,59 @@
+"""Export the canonical model.png to model.safetensors (standard format).
+
+model.png is the canonical model. This produces the exact same weights in a
+standard container, with verifiable parameter-count metadata baked into the
+safetensors header - so INFERENCE.py can run from safetensors alone and match
+main.py output bit for bit.
+
+    python convert_to_safetensors.py            # model.png -> model.safetensors
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+
+import torch
+from safetensors.torch import save_file
+
+from model import load_config, load_model_png, param_breakdown, PAD_ID
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--png", default="model.png")
+    ap.add_argument("--config", default="config.json")
+    ap.add_argument("--out", default="model.safetensors")
+    args = ap.parse_args()
+
+    cfg = load_config(args.config)
+    model = load_model_png(args.png, cfg, map_location="cpu")
+
+    info = param_breakdown(model)
+    total = info["total_parameters"]
+    n_bias = sum(int(t.numel()) for n, t in model.named_parameters() if n.endswith("bias"))
+
+    state = {k: v.contiguous().cpu() for k, v in model.state_dict().items()}
+
+    metadata = {
+        "format": "pt",
+        "model": "PixelModel-v3",
+        "total_parameters": str(total),
+        "param_breakdown": json.dumps(info["param_breakdown"]),
+        "has_bias": "true" if n_bias > 0 else "false",
+        "bias_parameters": str(n_bias),
+        "text_encoder_parameters": str(
+            sum(int(t.numel()) for n, t in model.named_parameters()
+                if n.startswith("embed") or n.startswith("text_"))),
+        "vae_parameters": "0",
+        "config": json.dumps(cfg.to_json()),
+    }
+
+    save_file(state, args.out, metadata=metadata)
+    print(f"[convert] {args.png} -> {args.out}")
+    print(f"[convert] total_parameters = {total:,}  (bias={n_bias:,})")
+    print(f"[convert] config + param_breakdown embedded in safetensors metadata")
+
+
+if __name__ == "__main__":
+    main()

eval/run_eval.py

@@ -0,0 +1,140 @@
+"""FID + CLIP Score for PixelModel v3 - identical protocol to v1.
+
+Real set: MS-COCO val2014 pairs (coco_eval.npz from fetch_coco_subset.py),
+256x256 center-crop, n=5000. Metrics via torchmetrics:
+  FID  -> torchmetrics.image.fid.FrechetInceptionDistance
+  CLIP -> torchmetrics.multimodal.CLIPScore, openai/clip-vit-base-patch32
+
+Two modes, scored by the *same* code so numbers are comparable:
+
+  # score v3 directly from model.png
+  python eval/run_eval.py --arch v3 --work ../pm-work \
+      --png model.png --config config.json --vocab vocab.json --n 5000
+
+  # regression check: score any OTHER model (v1, v2) from pre-rendered PNGs
+  # (files named 00000.png, 00001.png, ... aligned to eval order)
+  python eval/run_eval.py --arch precomputed --work ../pm-work \
+      --images-dir v1_render/ --n 5000
+
+This is what makes the README's v1-vs-v3 comparison apples-to-apples: both go
+through this script, on the same reals, with the same torchmetrics versions.
+"""
+
+from __future__ import annotations
+
+import argparse
+import glob
+import json
+import os
+import sys
+
+import numpy as np
+import torch
+from PIL import Image
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+from model import load_config, load_model_png, load_vocab, encode_caption, make_coord_grid
+
+
+def resize_u8(arr_u8, size):
+    return np.asarray(Image.fromarray(arr_u8, "RGB").resize((size, size), Image.BICUBIC),
+                      dtype=np.uint8)
+
+
+def render_v3(args, captions, device):
+    cfg = load_config(args.config)
+    model = load_model_png(args.png, cfg, map_location=device)
+    vocab = load_vocab(args.vocab)
+    coords = make_coord_grid(args.render_res, args.render_res, device=device,
+                             dtype=torch.float32).unsqueeze(0)
+    outs = []
+    for i in range(0, len(captions), args.batch):
+        batch_caps = captions[i:i + args.batch]
+        toks = np.stack([encode_caption(c, vocab, cfg.max_tokens) for c in batch_caps])
+        toks = torch.from_numpy(toks).long().to(device)
+        c = coords.expand(len(batch_caps), -1, -1)
+        with torch.no_grad():
+            rgb = model(toks, c)
+        rgb = (rgb.clamp(0, 1).reshape(len(batch_caps), args.render_res, args.render_res, 3)
+               .cpu().numpy() * 255.0).round().astype(np.uint8)
+        for im in rgb:
+            outs.append(resize_u8(im, args.fid_size))
+        if i % (args.batch * 20) == 0:
+            print(f"[eval] rendered {i+len(batch_caps)}/{len(captions)}")
+    return np.stack(outs)
+
+
+def load_precomputed(images_dir, n, fid_size):
+    files = sorted(glob.glob(os.path.join(images_dir, "*.png")))[:n]
+    if not files:
+        raise FileNotFoundError(f"no PNGs in {images_dir}")
+    outs = [resize_u8(np.asarray(Image.open(f).convert("RGB"), dtype=np.uint8), fid_size)
+            for f in files]
+    return np.stack(outs)
+
+
+def to_nchw_u8(arr):
+    return torch.from_numpy(arr).permute(0, 3, 1, 2).contiguous()
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--arch", choices=["v3", "precomputed"], default="v3")
+    ap.add_argument("--work", default="../pm-work", help="dir with coco_eval.npz")
+    ap.add_argument("--png", default="model.png")
+    ap.add_argument("--config", default="config.json")
+    ap.add_argument("--vocab", default="vocab.json")
+    ap.add_argument("--images-dir", default=None, help="precomputed renders")
+    ap.add_argument("--n", type=int, default=5000)
+    ap.add_argument("--render-res", type=int, default=128)
+    ap.add_argument("--fid-size", type=int, default=256, help="match v1 (256 crop reals)")
+    ap.add_argument("--batch", type=int, default=32)
+    ap.add_argument("--clip-model", default="openai/clip-vit-base-patch32")
+    ap.add_argument("--device", default="cuda")
+    ap.add_argument("--out", default="eval_results.json")
+    args = ap.parse_args()
+
+    device = args.device if torch.cuda.is_available() else "cpu"
+    from torchmetrics.image.fid import FrechetInceptionDistance
+    from torchmetrics.multimodal.clip_score import CLIPScore
+
+    ev = np.load(os.path.join(args.work, "coco_eval.npz"), allow_pickle=True)
+    reals = ev["images"][:args.n]
+    captions = [str(c) for c in ev["captions"][:args.n]]
+    n = min(len(reals), len(captions), args.n)
+    reals, captions = reals[:n], captions[:n]
+    reals = np.stack([resize_u8(im, args.fid_size) for im in reals])
+    print(f"[eval] arch={args.arch}  n={n}  fid_size={args.fid_size}  device={device}")
+
+    if args.arch == "v3":
+        fakes = render_v3(args, captions, device)
+    else:
+        fakes = load_precomputed(args.images_dir, n, args.fid_size)
+        if len(fakes) != n:
+            raise ValueError(f"precomputed count {len(fakes)} != {n}")
+
+    fid = FrechetInceptionDistance(feature=2048, normalize=False).to(device)
+    for i in range(0, n, args.batch):
+        fid.update(to_nchw_u8(reals[i:i + args.batch]).to(device), real=True)
+        fid.update(to_nchw_u8(fakes[i:i + args.batch]).to(device), real=False)
+    fid_val = float(fid.compute())
+
+    clip = CLIPScore(model_name_or_path=args.clip_model).to(device)
+    for i in range(0, n, args.batch):
+        imgs = to_nchw_u8(fakes[i:i + args.batch]).to(device)
+        clip.update(imgs, captions[i:i + args.batch])
+    clip_val = float(clip.compute())
+
+    results = {
+        "arch": args.arch, "n": n, "fid": round(fid_val, 2),
+        "clip_score": round(clip_val, 2), "render_res": args.render_res,
+        "fid_size": args.fid_size, "clip_model": args.clip_model,
+    }
+    with open(args.out, "w") as f:
+        json.dump(results, f, indent=2)
+    print(f"[eval] FID = {fid_val:.2f}   CLIP Score = {clip_val:.2f}   (n={n})")
+    print(f"[eval] wrote {args.out}")
+
+
+if __name__ == "__main__":
+    main()

eval_base.json

@@ -0,0 +1,9 @@
+{
+  "arch": "v3",
+  "n": 5000,
+  "fid": 392.1,
+  "clip_score": 19.88,
+  "render_res": 128,
+  "fid_size": 256,
+  "clip_model": "openai/clip-vit-base-patch32"
+}

eval_v1_regression.json

@@ -0,0 +1,9 @@
+{
+  "arch": "precomputed",
+  "n": 5000,
+  "fid": 420.75,
+  "clip_score": 20.1,
+  "render_res": 128,
+  "fid_size": 256,
+  "clip_model": "openai/clip-vit-base-patch32"
+}

fetch_coco_subset.py

@@ -0,0 +1,214 @@
+"""Build the PixelModel v3 train/eval data from MS-COCO.
+
+Same source as v1: `sayakpaul/coco-30-val-2014` (MS-COCO val2014 caption/image
+pairs). The set is split *deterministically by image hash* into a train subset
+and an eval subset so the two are provably disjoint and reproducible - exactly
+the regression-safe protocol v1 used.
+
+Outputs (into --out, default ../pm-work):
+  coco_train.npz   images (uint8, N x S x S x 3) + token ids (int32, N x T)
+  coco_eval.npz    eval images + raw caption strings (for FID/CLIP)
+  vocab.json       word-level tokenizer vocabulary (built from TRAIN only)
+  split_manifest.json  hashes + counts, so disjointness is auditable
+
+Run on the training machine (needs `datasets` + network):
+    python fetch_coco_subset.py --out ../pm-work
+"""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import io
+import json
+import os
+import time
+
+import numpy as np
+from PIL import Image
+
+from model import build_vocab, encode_caption, PAD_ID
+
+
+def center_square_resize(img: Image.Image, size: int) -> np.ndarray:
+    """Center-crop to a square, resize to size x size, return uint8 HWC RGB."""
+    img = img.convert("RGB")
+    w, h = img.size
+    s = min(w, h)
+    left = (w - s) // 2
+    top = (h - s) // 2
+    img = img.crop((left, top, left + s, top + s)).resize((size, size), Image.BICUBIC)
+    return np.asarray(img, dtype=np.uint8)
+
+
+def image_hash(img: Image.Image) -> str:
+    """Stable content hash of an image, independent of dataset row order."""
+    buf = io.BytesIO()
+    img.convert("RGB").resize((64, 64), Image.BICUBIC).save(buf, format="PNG")
+    return hashlib.md5(buf.getvalue()).hexdigest()
+
+
+def caption_of(example) -> str:
+    """COCO rows expose captions under a few different keys across mirrors."""
+    for key in ("caption", "captions", "text", "sentences", "annotations_captions"):
+        if key in example and example[key]:
+            v = example[key]
+            if isinstance(v, (list, tuple)):
+                return str(v[0])
+            return str(v)
+    raise KeyError(f"no caption field found; row keys = {list(example.keys())}")
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--out", default="../pm-work", help="output directory")
+    ap.add_argument("--dataset", default="sayakpaul/coco-30-val-2014",
+                    help="HF dataset id (MS-COCO caption/image pairs)")
+    ap.add_argument("--split", default="train", help="HF split to pull from")
+    ap.add_argument("--n-train", type=int, default=20000)
+    ap.add_argument("--n-eval", type=int, default=5000)
+    ap.add_argument("--store-size", type=int, default=144,
+                    help="square size images are stored at (must be >= train crop)")
+    ap.add_argument("--vocab-size", type=int, default=8192, help="base model, incl. PAD+UNK")
+    ap.add_argument("--vocab-size-xl", type=int, default=16384, help="XL variant vocab cap")
+    ap.add_argument("--min-freq", type=int, default=2)
+    ap.add_argument("--max-tokens", type=int, default=20)
+    ap.add_argument("--eval-hash-mod", type=int, default=6,
+                    help="1/eval-hash-mod of images are reserved for eval")
+    ap.add_argument("--streaming", action="store_true",
+                    help="stream the split instead of downloading it first. "
+                         "Faster to start, but a long collection loop can die "
+                         "to a dropped connection; off by default.")
+    ap.add_argument("--load-retries", type=int, default=6)
+    ap.add_argument("--max-reconnects", type=int, default=8,
+                    help="resume attempts if iteration dies mid-collection")
+    ap.add_argument("--seed", type=int, default=0)
+    args = ap.parse_args()
+
+    os.makedirs(args.out, exist_ok=True)
+    from datasets import load_dataset
+
+    def open_dataset():
+        """Load the split, retrying the network part. Default is NON-streaming:
+        the shards are downloaded once (HF retries/resumes those internally) and
+        iteration is then purely local, so a long collection loop cannot die
+        halfway to a dropped HTTP connection."""
+        last = None
+        for t in range(args.load_retries):
+            try:
+                return load_dataset(args.dataset, split=args.split,
+                                    streaming=args.streaming)
+            except Exception as e:
+                last = e
+                wait = 5 * (t + 1)
+                print(f"[data] load attempt {t + 1}/{args.load_retries} failed "
+                      f"({type(e).__name__}: {e}); retrying in {wait}s", flush=True)
+                time.sleep(wait)
+        raise RuntimeError(f"could not load {args.dataset}: {last}")
+
+    train_imgs, train_caps_raw, train_hashes = [], [], []
+    eval_imgs, eval_caps, eval_hashes = [], [], []
+    seen = set()
+
+    def need_more():
+        return len(train_imgs) < args.n_train or len(eval_imgs) < args.n_eval
+
+    last_report = 0
+    reconnects = 0
+    print(f"[data] loading {args.dataset} split={args.split} "
+          f"(streaming={args.streaming})", flush=True)
+    while need_more():
+        ds = open_dataset()
+        try:
+            for ex in ds:
+                if not need_more():
+                    break
+                try:
+                    img = ex.get("image") or ex.get("img")
+                    if img is None:
+                        continue
+                    cap = caption_of(ex)
+                    h = image_hash(img)
+                    if h in seen:
+                        continue
+                    seen.add(h)
+                    is_eval = (int(h, 16) % args.eval_hash_mod == 0)
+                    if is_eval:
+                        if len(eval_imgs) >= args.n_eval:
+                            continue
+                        eval_imgs.append(center_square_resize(img, 256))
+                        eval_caps.append(cap)
+                        eval_hashes.append(h)
+                    else:
+                        if len(train_imgs) >= args.n_train:
+                            continue
+                        train_imgs.append(center_square_resize(img, args.store_size))
+                        train_caps_raw.append(cap)
+                        train_hashes.append(h)
+                except Exception:
+                    continue
+                total = len(train_imgs) + len(eval_imgs)
+                if total - last_report >= 1000:
+                    last_report = total
+                    print(f"[data] train={len(train_imgs)} eval={len(eval_imgs)}", flush=True)
+            break
+        except Exception as e:
+            reconnects += 1
+            print(f"[data] iteration died ({type(e).__name__}: {e}); "
+                  f"reconnect {reconnects}/{args.max_reconnects} with "
+                  f"train={len(train_imgs)} eval={len(eval_imgs)}", flush=True)
+            if reconnects >= args.max_reconnects:
+                print("[data] giving up on reconnects; proceeding with what we have",
+                      flush=True)
+                break
+            time.sleep(5)
+
+    assert set(train_hashes).isdisjoint(set(eval_hashes)), "train/eval overlap!"
+    print(f"[data] collected train={len(train_imgs)} eval={len(eval_imgs)} "
+          f"(disjoint: {set(train_hashes).isdisjoint(set(eval_hashes))})")
+
+    vocab = build_vocab(train_caps_raw, args.vocab_size, args.min_freq)
+    with open(os.path.join(args.out, "vocab.json"), "w") as f:
+        json.dump(vocab, f)
+    vocab_xl = build_vocab(train_caps_raw, args.vocab_size_xl, args.min_freq)
+    with open(os.path.join(args.out, "vocab_xl.json"), "w") as f:
+        json.dump(vocab_xl, f)
+    print(f"[data] vocab size = {len(vocab)} (cap {args.vocab_size}), "
+          f"vocab_xl = {len(vocab_xl)} (cap {args.vocab_size_xl})")
+
+    train_tokens = np.stack(
+        [encode_caption(c, vocab, args.max_tokens) for c in train_caps_raw])
+    np.savez_compressed(
+        os.path.join(args.out, "coco_train.npz"),
+        images=np.stack(train_imgs),
+        tokens=train_tokens,
+        captions=np.array(train_caps_raw, dtype=object),
+        store_size=args.store_size,
+        max_tokens=args.max_tokens,
+    )
+    np.savez_compressed(
+        os.path.join(args.out, "coco_eval.npz"),
+        images=np.stack(eval_imgs),
+        captions=np.array(eval_caps, dtype=object),
+    )
+    with open(os.path.join(args.out, "split_manifest.json"), "w") as f:
+        json.dump({
+            "dataset": args.dataset,
+            "n_train": len(train_imgs),
+            "n_eval": len(eval_imgs),
+            "eval_hash_mod": args.eval_hash_mod,
+            "vocab_size": len(vocab),
+            "store_size": args.store_size,
+            "max_tokens": args.max_tokens,
+            "disjoint": bool(set(train_hashes).isdisjoint(set(eval_hashes))),
+            "train_hash_sample": train_hashes[:8],
+            "eval_hash_sample": eval_hashes[:8],
+        }, f, indent=2)
+
+    mb = os.path.getsize(os.path.join(args.out, "coco_train.npz")) / 1e6
+    print(f"[data] wrote coco_train.npz ({mb:.1f} MB), coco_eval.npz, vocab.json")
+    print(f"[data] done -> {args.out}")
+
+
+if __name__ == "__main__":
+    main()

main.py

@@ -0,0 +1,55 @@
+"""PixelModel v3 inference from the canonical model.png.
+
+    python main.py "a red double decker bus" --out bus.png
+    python main.py "a beach with palm trees" --res 256
+
+model.png is the model. This script decodes it, tokenises the prompt with the
+shipped vocab.json, and paints an image at any resolution. Fully deterministic,
+so it matches INFERENCE.py (which loads model.safetensors) bit for bit.
+"""
+
+from __future__ import annotations
+
+import argparse
+
+import numpy as np
+import torch
+from PIL import Image
+
+from model import (
+    load_config, load_model_png, load_vocab, encode_caption, make_coord_grid,
+)
+
+
+def render(model, cfg, vocab, prompt, res, device):
+    tokens = encode_caption(prompt, vocab, cfg.max_tokens)
+    tokens = torch.from_numpy(tokens).long().unsqueeze(0).to(device)
+    coords = make_coord_grid(res, res, device=device, dtype=torch.float32).unsqueeze(0)
+    with torch.no_grad():
+        rgb = model(tokens, coords)
+    img = (rgb.clamp(0, 1).reshape(res, res, 3).cpu().numpy() * 255.0).round().astype(np.uint8)
+    return img
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("prompt")
+    ap.add_argument("--out", default="out.png")
+    ap.add_argument("--res", type=int, default=128, help="output resolution (native 128)")
+    ap.add_argument("--png", default="model.png", help="the model")
+    ap.add_argument("--config", default="config.json")
+    ap.add_argument("--vocab", default="vocab.json")
+    ap.add_argument("--device", default="cpu")
+    args = ap.parse_args()
+
+    cfg = load_config(args.config)
+    model = load_model_png(args.png, cfg, map_location=args.device)
+    vocab = load_vocab(args.vocab)
+
+    img = render(model, cfg, vocab, args.prompt, args.res, args.device)
+    Image.fromarray(img, "RGB").save(args.out)
+    print(f'[main] "{args.prompt}" @ {args.res}x{args.res} -> {args.out}')
+
+
+if __name__ == "__main__":
+    main()

model.py

@@ -0,0 +1,402 @@
+"""PixelModel v3 - architecture + PNG weight codec.
+
+A text-to-image CPPN whose entire weight set is stored as pixels in a PNG.
+
+v3 changes relative to v1/v2 (see README):
+  1. Learned word-embedding table (mean-pooled) replaces the 0-param hash embed.
+  2. SIREN (sine) decoder with the real SIREN init instead of tanh.
+  3. Wider/deeper decoder (128-256 wide, 4-5 layers).
+  4. FiLM conditioning (per-layer gamma/beta from z) replaces concat conditioning.
+  5. PNG weight-encoding scheme extended to the new param count.
+  6/7. Fully batched coordinate decoding (no per-pixel python loop).
+
+The PNG codec is deliberately identical in spirit to v1: one weight per pixel,
+16-bit precision spread across the R (high byte) and G (low byte) channels,
+B reserved, values linearly mapped from a fixed [-2, 2] range.
+"""
+
+from __future__ import annotations
+
+import json
+import math
+import os
+import time
+from dataclasses import dataclass, asdict, field
+
+import numpy as np
+
+try:
+    import torch
+    import torch.nn as nn
+    import torch.nn.functional as F
+except Exception:
+    torch = None
+    nn = object
+
+
+
+WEIGHT_RANGE = (-2.0, 2.0)
+
+PAD_ID = 0
+UNK_ID = 1
+
+
+@dataclass
+class ModelConfig:
+    vocab_size: int = 8192
+    embed_dim: int = 64
+    max_tokens: int = 20
+    text_hidden: int = 256
+    z_dim: int = 192
+
+    decoder_width: int = 256
+    num_sine_layers: int = 4
+    num_freq: int = 8
+    w0_first: float = 30.0
+    w0_hidden: float = 30.0
+
+    weight_range: tuple = WEIGHT_RANGE
+
+    @property
+    def coord_dim(self) -> int:
+        return 2 + 4 * self.num_freq
+
+    def to_json(self) -> dict:
+        d = asdict(self)
+        d["weight_range"] = list(self.weight_range)
+        d["coord_dim"] = self.coord_dim
+        return d
+
+    @staticmethod
+    def from_json(d: dict) -> "ModelConfig":
+        keys = {f for f in ModelConfig.__dataclass_fields__}
+        clean = {k: v for k, v in d.items() if k in keys}
+        if "weight_range" in clean:
+            clean["weight_range"] = tuple(clean["weight_range"])
+        return ModelConfig(**clean)
+
+
+
+import re as _re
+
+_TOKEN_RE = _re.compile(r"[a-z0-9]+")
+
+
+def simple_tokenize(text: str) -> list[str]:
+    """Lowercase, split on non-alphanumeric runs. Deterministic and reversible
+    enough for a bag-of-words embedding."""
+    return _TOKEN_RE.findall(text.lower())
+
+
+def build_vocab(captions, vocab_size: int, min_freq: int = 1) -> dict:
+    """captions: iterable of strings -> {token: id}. Ids 0/1 are PAD/UNK."""
+    from collections import Counter
+    counter = Counter()
+    for cap in captions:
+        counter.update(simple_tokenize(cap))
+    vocab = {"<pad>": PAD_ID, "<unk>": UNK_ID}
+    for tok, freq in counter.most_common():
+        if len(vocab) >= vocab_size:
+            break
+        if freq < min_freq:
+            break
+        if tok not in vocab:
+            vocab[tok] = len(vocab)
+    return vocab
+
+
+def load_vocab(path: str) -> dict:
+    with open(path) as f:
+        return json.load(f)
+
+
+def encode_caption(text: str, vocab: dict, max_tokens: int) -> np.ndarray:
+    """text -> int32 array of length max_tokens (PAD-filled, UNK-mapped)."""
+    ids = [vocab.get(t, UNK_ID) for t in simple_tokenize(text)][:max_tokens]
+    out = np.full(max_tokens, PAD_ID, dtype=np.int32)
+    out[:len(ids)] = ids
+    return out
+
+
+
+def fourier_features(coords, num_freq: int):
+    """coords: (..., 2) in [-1, 1] -> (..., 2 + 4*num_freq).
+
+    One explicit batched tensor op over the whole pixel set; no python loop
+    over pixels. Frequencies are the octaves pi * 2^k, k = 0..num_freq-1.
+    """
+    freqs = (2.0 ** torch.arange(num_freq, device=coords.device, dtype=coords.dtype)) * math.pi
+    scaled = coords.unsqueeze(-1) * freqs
+    sin = torch.sin(scaled)
+    cos = torch.cos(scaled)
+    enc = torch.cat([coords, sin.flatten(-2), cos.flatten(-2)], dim=-1)
+    return enc
+
+
+
+class FiLMSineLayer(nn.Module):
+    """Linear -> FiLM(gamma, beta) -> sin(w0 * .).
+
+    FiLM is applied *after* the linear and *before* the sine, so the
+    conditioning warps the pre-activation the same way a learned bias would,
+    while the SIREN frequency scaling still governs the activation spectrum.
+    """
+
+    def __init__(self, in_features: int, out_features: int, w0: float, is_first: bool):
+        super().__init__()
+        self.in_features = in_features
+        self.out_features = out_features
+        self.w0 = w0
+        self.is_first = is_first
+        self.linear = nn.Linear(in_features, out_features)
+        self.siren_init()
+
+    def siren_init(self):
+        with torch.no_grad():
+            if self.is_first:
+                bound = 1.0 / self.in_features
+            else:
+                bound = math.sqrt(6.0 / self.in_features) / self.w0
+            self.linear.weight.uniform_(-bound, bound)
+            if self.linear.bias is not None:
+                self.linear.bias.uniform_(-bound, bound)
+
+    def forward(self, x, gamma, beta):
+        h = self.linear(x)
+        h = gamma * h + beta
+        return torch.sin(self.w0 * h)
+
+
+
+class PixelModelV3(nn.Module):
+    def __init__(self, cfg: ModelConfig):
+        super().__init__()
+        self.cfg = cfg
+
+        self.embed = nn.Embedding(cfg.vocab_size, cfg.embed_dim, padding_idx=PAD_ID)
+        nn.init.normal_(self.embed.weight, std=0.05)
+        with torch.no_grad():
+            self.embed.weight[PAD_ID].zero_()
+
+        self.text_fc1 = nn.Linear(cfg.embed_dim, cfg.text_hidden)
+        self.text_fc2 = nn.Linear(cfg.text_hidden, cfg.z_dim)
+
+        self.film = nn.Linear(cfg.z_dim, cfg.num_sine_layers * 2 * cfg.decoder_width)
+        nn.init.zeros_(self.film.weight)
+        nn.init.zeros_(self.film.bias)
+
+        layers = []
+        in_dim = cfg.coord_dim
+        for i in range(cfg.num_sine_layers):
+            is_first = (i == 0)
+            w0 = cfg.w0_first if is_first else cfg.w0_hidden
+            layers.append(FiLMSineLayer(in_dim, cfg.decoder_width, w0=w0, is_first=is_first))
+            in_dim = cfg.decoder_width
+        self.sine_layers = nn.ModuleList(layers)
+
+        self.head = nn.Linear(cfg.decoder_width, 3)
+        with torch.no_grad():
+            bound = math.sqrt(6.0 / cfg.decoder_width) / cfg.w0_hidden
+            self.head.weight.uniform_(-bound, bound)
+            nn.init.zeros_(self.head.bias)
+
+        self.num_freq = cfg.num_freq
+
+        self.grad_checkpoint = False
+
+    def encode_text(self, token_ids):
+        """token_ids: (B, L) long -> z: (B, z_dim)."""
+        emb = self.embed(token_ids)
+        mask = (token_ids != PAD_ID).float().unsqueeze(-1)
+        summed = (emb * mask).sum(dim=1)
+        count = mask.sum(dim=1).clamp(min=1.0)
+        pooled = summed / count
+        h = torch.sin(self.text_fc1(pooled))
+        z = self.text_fc2(h)
+        return z
+
+    def film_params(self, z):
+        """z: (B, z_dim) -> list of (gamma, beta), each (B, 1, width)."""
+        raw = self.film(z)
+        raw = raw.view(-1, self.cfg.num_sine_layers, 2, self.cfg.decoder_width)
+        out = []
+        for i in range(self.cfg.num_sine_layers):
+            gamma = 1.0 + raw[:, i, 0, :].unsqueeze(1)
+            beta = raw[:, i, 1, :].unsqueeze(1)
+            out.append((gamma, beta))
+        return out
+
+    def decode(self, coords, z, return_stats: bool = False):
+        """coords: (B, N, 2) in [-1, 1]; z: (B, z_dim) -> rgb (B, N, 3).
+
+        The whole pixel set N is decoded as one batched tensor op per layer.
+        """
+        x = fourier_features(coords, self.num_freq)
+        films = self.film_params(z)
+        use_ckpt = self.grad_checkpoint and self.training and not return_stats and x.requires_grad
+        stats = []
+        for layer, (gamma, beta) in zip(self.sine_layers, films):
+            if use_ckpt:
+                x = torch.utils.checkpoint.checkpoint(layer, x, gamma, beta, use_reentrant=False)
+            else:
+                x = layer(x, gamma, beta)
+            if return_stats:
+                stats.append((float(x.mean()), float(x.std())))
+        rgb = torch.sigmoid(self.head(x))
+        if return_stats:
+            return rgb, stats
+        return rgb
+
+    def forward(self, token_ids, coords, return_stats: bool = False):
+        z = self.encode_text(token_ids)
+        return self.decode(coords, z, return_stats=return_stats)
+
+
+
+def make_coord_grid(height: int, width: int, device=None, dtype=None):
+    """-> (H*W, 2) coordinate grid in [-1, 1]. One tensor op, no python loop."""
+    ys = torch.linspace(-1.0, 1.0, height, device=device, dtype=dtype)
+    xs = torch.linspace(-1.0, 1.0, width, device=device, dtype=dtype)
+    gy, gx = torch.meshgrid(ys, xs, indexing="ij")
+    return torch.stack([gx, gy], dim=-1).reshape(-1, 2)
+
+
+
+def param_breakdown(model: "PixelModelV3") -> dict:
+    """Per-parameter-tensor breakdown, in the exact order used by the PNG codec."""
+    breakdown = []
+    total = 0
+    for name, p in model.named_parameters():
+        n = int(p.numel())
+        breakdown.append({"name": name, "shape": list(p.shape), "params": n})
+        total += n
+    return {"total_parameters": total, "param_breakdown": breakdown}
+
+
+
+def png_dims_for(n_params: int) -> tuple[int, int]:
+    """Roughly-square PNG that holds n_params pixels (one weight per pixel)."""
+    width = int(math.ceil(math.sqrt(n_params)))
+    height = int(math.ceil(n_params / width))
+    return width, height
+
+
+def flatten_params(model: "PixelModelV3") -> np.ndarray:
+    chunks = [p.detach().cpu().float().reshape(-1).numpy() for p in model.parameters()]
+    return np.concatenate(chunks)
+
+
+def load_flat_params(model: "PixelModelV3", flat: np.ndarray) -> None:
+    i = 0
+    for p in model.parameters():
+        n = p.numel()
+        chunk = torch.from_numpy(np.ascontiguousarray(flat[i:i + n])).to(p.dtype).view_as(p)
+        p.data.copy_(chunk)
+        i += n
+    if i != flat.shape[0]:
+        raise ValueError(f"param/flat size mismatch: consumed {i}, have {flat.shape[0]}")
+
+
+def weights_to_png(flat: np.ndarray, path: str, weight_range=WEIGHT_RANGE):
+    """Write a float weight vector to a PNG. Returns (width, height, n_clamped)."""
+    from PIL import Image
+
+    n = int(flat.size)
+    width, height = png_dims_for(n)
+    total = width * height
+    padded = np.zeros(total, dtype=np.float64)
+    padded[:n] = flat.astype(np.float64)
+
+    lo, hi = weight_range
+    n_clamped = int(np.count_nonzero((padded[:n] < lo) | (padded[:n] > hi)))
+    clamped = np.clip(padded, lo, hi)
+    u16 = np.round((clamped - lo) / (hi - lo) * 65535.0).astype(np.uint16)
+
+    img = np.zeros((height, width, 3), dtype=np.uint8)
+    img[..., 0] = (u16 >> 8).astype(np.uint8).reshape(height, width)
+    img[..., 1] = (u16 & 0xFF).astype(np.uint8).reshape(height, width)
+    Image.fromarray(img, "RGB").save(path, optimize=True)
+    return width, height, n_clamped
+
+
+def png_to_weights(path: str, n_params: int, weight_range=WEIGHT_RANGE) -> np.ndarray:
+    """Decode a PNG back into the first n_params weight values (float32)."""
+    from PIL import Image
+
+    arr = np.asarray(Image.open(path).convert("RGB"))
+    hi_byte = arr[..., 0].astype(np.uint16)
+    lo_byte = arr[..., 1].astype(np.uint16)
+    u16 = ((hi_byte << 8) | lo_byte).reshape(-1)[:n_params]
+    lo, hi = weight_range
+    vals = u16.astype(np.float64) / 65535.0 * (hi - lo) + lo
+    return vals.astype(np.float32)
+
+
+
+def save_model_png(model: "PixelModelV3", png_path: str, config_path: str | None = None,
+                   verbose: bool = True) -> dict:
+    """Write model.png (and optionally config.json). Returns a stats dict."""
+    flat = flatten_params(model)
+    n = int(flat.size)
+    t0 = time.time()
+    width, height, n_clamped = weights_to_png(flat, png_path, model.cfg.weight_range)
+    write_s = time.time() - t0
+    size_bytes = os.path.getsize(png_path)
+
+    info = param_breakdown(model)
+    info.update({
+        "png_width": width,
+        "png_height": height,
+        "png_pixels": width * height,
+        "png_bytes": size_bytes,
+        "png_write_seconds": round(write_s, 4),
+        "weights_clamped": n_clamped,
+        "config": model.cfg.to_json(),
+    })
+
+    if config_path is not None:
+        with open(config_path, "w") as f:
+            json.dump(info, f, indent=2)
+
+    if verbose:
+        print(f"[png] total params      : {n:,}")
+        print(f"[png] resolution        : {width} x {height}  ({width*height:,} pixels)")
+        print(f"[png] file size on disk : {size_bytes/1024:.1f} KiB ({size_bytes/1e6:.3f} MB)")
+        print(f"[png] write time        : {write_s*1000:.1f} ms")
+        if n_clamped:
+            frac = 100.0 * n_clamped / n
+            print(f"[png] WARNING clamped   : {n_clamped:,} weights ({frac:.3f}%) outside {model.cfg.weight_range}")
+        if size_bytes > 4_000_000:
+            print(f"[png] WARNING size      : {size_bytes/1e6:.2f} MB is larger than expected")
+    return info
+
+
+def load_model_png(png_path: str, cfg: ModelConfig, map_location="cpu",
+                   verbose: bool = True) -> "PixelModelV3":
+    """Rebuild the model architecture from cfg, then fill weights from the PNG."""
+    model = PixelModelV3(cfg)
+    n_params = sum(p.numel() for p in model.parameters())
+    t0 = time.time()
+    flat = png_to_weights(png_path, n_params, cfg.weight_range)
+    load_flat_params(model, flat)
+    model.to(map_location)
+    model.eval()
+    if verbose:
+        print(f"[png] loaded {n_params:,} weights from {png_path} in {(time.time()-t0)*1000:.1f} ms")
+    return model
+
+
+def load_config(config_path: str) -> ModelConfig:
+    with open(config_path) as f:
+        info = json.load(f)
+    cfg_dict = info.get("config", info)
+    return ModelConfig.from_json(cfg_dict)
+
+
+
+def roundtrip_error(model: "PixelModelV3", tmp_png: str) -> float:
+    """Max abs weight error after a PNG encode/decode round trip."""
+    before = flatten_params(model)
+    weights_to_png(before, tmp_png, model.cfg.weight_range)
+    after = png_to_weights(tmp_png, before.size, model.cfg.weight_range)
+    return float(np.max(np.abs(before - after)))

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4a996ad168f97a48f500fa5b78752e42018bfa871216d903d0640be8a9a2c075
+size 3680932

train.py

@@ -0,0 +1,276 @@
+"""Train PixelModel v3. The canonical output is model.png (weights-as-pixels).
+
+Designed for a single 4 GB RTX 3050: mixed precision, a configurable batch and
+crop size, a cosine LR schedule, and a peak-VRAM budget it actively watches.
+
+The default run is a multi-hour job that meaningfully uses the GPU - not v1's
+30-minute CPU toy. Shorten it with --epochs for a quick sanity pass.
+
+    python train.py --data ../pm-work/coco_train.npz --vocab ../pm-work/vocab.json
+"""
+
+from __future__ import annotations
+
+import argparse
+import math
+import os
+import time
+
+import numpy as np
+
+import torch
+import torch.nn.functional as F
+
+from model import (
+    ModelConfig, PixelModelV3, make_coord_grid, save_model_png, load_vocab,
+)
+
+
+def get_args():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--data", default="../pm-work/coco_train.npz")
+    ap.add_argument("--vocab", default="../pm-work/vocab.json")
+    ap.add_argument("--out-png", default="model.png")
+    ap.add_argument("--out-config", default="config.json")
+
+    ap.add_argument("--epochs", type=int, default=80,
+                    help="default is a multi-hour run; lower it for a quick pass")
+    ap.add_argument("--batch-size", type=int, default=16)
+    ap.add_argument("--crop", type=int, default=128, help="training crop size")
+    ap.add_argument("--pixels-per-step", type=int, default=0,
+                    help="decode only this many random pixels per image each step "
+                         "(0 = the full crop). Decouples VRAM from crop size, so "
+                         "high-res crops train with a usable batch (NeRF-style).")
+    ap.add_argument("--lr", type=float, default=2e-4)
+    ap.add_argument("--min-lr", type=float, default=1e-6, help="cosine floor")
+    ap.add_argument("--warmup-steps", type=int, default=500)
+    ap.add_argument("--grad-clip", type=float, default=1.0)
+    ap.add_argument("--steps-per-epoch", type=int, default=0,
+                    help="0 = one pass over the data per epoch")
+
+    ap.add_argument("--embed-dim", type=int, default=None)
+    ap.add_argument("--text-hidden", type=int, default=None)
+    ap.add_argument("--z-dim", type=int, default=None)
+    ap.add_argument("--decoder-width", type=int, default=None)
+    ap.add_argument("--num-sine-layers", type=int, default=None)
+    ap.add_argument("--num-freq", type=int, default=None)
+    ap.add_argument("--w0-first", type=float, default=None)
+    ap.add_argument("--w0-hidden", type=float, default=None)
+
+    ap.add_argument("--device", default="cuda")
+    ap.add_argument("--data-device", choices=["auto", "gpu", "cpu"], default="auto",
+                    help="where images live for cropping. 'gpu' keeps the whole "
+                         "image tensor on-device and crops there (removes the CPU "
+                         "bottleneck on fast GPUs); 'auto' = gpu when CUDA is used")
+    ap.add_argument("--amp", dest="amp", action="store_true", default=True)
+    ap.add_argument("--no-amp", dest="amp", action="store_false")
+    ap.add_argument("--grad-checkpoint", action="store_true",
+                    help="checkpoint sine layers to cut activation memory "
+                         "(lets XL use a bigger batch on a small/shared GPU)")
+    ap.add_argument("--vram-budget-gb", type=float, default=3.5)
+    ap.add_argument("--log-interval", type=int, default=50)
+    ap.add_argument("--save-every-epochs", type=int, default=1)
+    ap.add_argument("--limit", type=int, default=0, help="use only N samples (debug)")
+    ap.add_argument("--seed", type=int, default=0)
+    return ap.parse_args()
+
+
+def build_config(args, vocab_size, max_tokens) -> ModelConfig:
+    cfg = ModelConfig(vocab_size=vocab_size, max_tokens=max_tokens)
+    for name in ("embed_dim", "text_hidden", "z_dim", "decoder_width",
+                 "num_sine_layers", "num_freq", "w0_first", "w0_hidden"):
+        v = getattr(args, name)
+        if v is not None:
+            setattr(cfg, name, v)
+    return cfg
+
+
+def cosine_lr(step, total_steps, base_lr, min_lr, warmup):
+    if step < warmup:
+        return base_lr * (step + 1) / max(1, warmup)
+    t = (step - warmup) / max(1, total_steps - warmup)
+    t = min(1.0, t)
+    return min_lr + 0.5 * (base_lr - min_lr) * (1 + math.cos(math.pi * t))
+
+
+def random_crops(images_u8, idx, crop, rng):
+    """images_u8: (N,S,S,3) uint8 -> batch (B, crop, crop, 3) float32 in [0,1]."""
+    B = len(idx)
+    S = images_u8.shape[1]
+    out = np.empty((B, crop, crop, 3), dtype=np.float32)
+    for i, j in enumerate(idx):
+        top = rng.integers(0, S - crop + 1)
+        left = rng.integers(0, S - crop + 1)
+        out[i] = images_u8[j, top:top + crop, left:left + crop, :].astype(np.float32) / 255.0
+    return out
+
+
+def gpu_random_crops(images_u8, idx, crop, gen):
+    """Fully vectorised random crop on-device. images_u8: (N,S,S,3) uint8 on GPU,
+    idx: (B,) long on GPU -> (B, crop, crop, 3) float32 in [0,1]. No python loop,
+    no host<->device copy per step, so a fast GPU is not left waiting on the CPU."""
+    B = idx.shape[0]
+    S = images_u8.shape[1]
+    sel = images_u8[idx]
+    span = S - crop + 1
+    top = torch.randint(0, span, (B,), generator=gen, device=sel.device)
+    left = torch.randint(0, span, (B,), generator=gen, device=sel.device)
+    ar = torch.arange(crop, device=sel.device)
+    rows = (top[:, None] + ar)[:, :, None]
+    cols = (left[:, None] + ar)[:, None, :]
+    b = torch.arange(B, device=sel.device)[:, None, None]
+    out = sel[b, rows, cols]
+    return out.float() / 255.0
+
+
+def print_activation_stats(model, tokens, coords, tag):
+    """SIREN sanity check: per-layer sine-output mean/std. Healthy SIREN layers
+    sit near mean~0, std~0.5-0.7. Dead (~0 std) or exploding (>>1) means the
+    init is wrong - catch it here, not three hours into a bad loss curve."""
+    model.eval()
+    with torch.no_grad():
+        _, stats = model(tokens, coords, return_stats=True)
+    model.train()
+    line = " | ".join(f"L{i}: mean={m:+.3f} std={s:.3f}" for i, (m, s) in enumerate(stats))
+    print(f"[siren-stats {tag}] {line}")
+
+
+def main():
+    args = get_args()
+    torch.manual_seed(args.seed)
+    rng = np.random.default_rng(args.seed)
+    device = args.device if torch.cuda.is_available() or args.device == "cpu" else "cpu"
+    if device == "cpu":
+        print("[warn] CUDA not available -> running on CPU (AMP disabled)")
+        args.amp = False
+
+    from model import encode_caption
+    data = np.load(args.data, allow_pickle=True)
+    images = data["images"]
+    max_tokens = int(data["max_tokens"]) if "max_tokens" in data else 20
+    if args.limit:
+        images = images[:args.limit]
+    N, S = images.shape[0], images.shape[1]
+    assert S >= args.crop, f"stored size {S} < crop {args.crop}"
+
+    vocab = load_vocab(args.vocab)
+    cfg = build_config(args, vocab_size=len(vocab), max_tokens=max_tokens)
+
+    if "captions" in data:
+        caps = [str(c) for c in data["captions"]]
+        if args.limit:
+            caps = caps[:args.limit]
+        tokens_all = np.stack([encode_caption(c, vocab, max_tokens) for c in caps]).astype(np.int64)
+    else:
+        tokens_all = data["tokens"].astype(np.int64)
+        if args.limit:
+            tokens_all = tokens_all[:args.limit]
+
+    data_on_gpu = args.data_device == "gpu" or (args.data_device == "auto" and device == "cuda")
+    gen = torch.Generator(device=device).manual_seed(args.seed)
+    if data_on_gpu:
+        images_dev = torch.from_numpy(np.ascontiguousarray(images)).to(device)
+        tokens_dev = torch.from_numpy(tokens_all).to(device)
+        gb = images_dev.numel() / 1e9
+        print(f"[train] images resident on {device}: {gb:.2f} GB uint8")
+    else:
+        images_dev = tokens_dev = None
+
+    print(f"[train] {N} images @ {S}px, crop={args.crop}, vocab={len(vocab)}, "
+          f"device={device}, amp={args.amp}, data_on_gpu={data_on_gpu}")
+
+    model = PixelModelV3(cfg).to(device)
+    model.grad_checkpoint = args.grad_checkpoint
+    n_params = sum(p.numel() for p in model.parameters())
+    print(f"[train] model params = {n_params:,}, grad_checkpoint={args.grad_checkpoint}")
+
+    opt = torch.optim.Adam(model.parameters(), lr=args.lr, betas=(0.9, 0.99))
+    scaler = torch.amp.GradScaler("cuda", enabled=args.amp)
+
+    steps_per_epoch = args.steps_per_epoch or max(1, N // args.batch_size)
+    total_steps = steps_per_epoch * args.epochs
+
+    coords = make_coord_grid(args.crop, args.crop, device=device, dtype=torch.float32)
+    coords = coords.unsqueeze(0)
+    HW = args.crop * args.crop
+    if args.pixels_per_step:
+        print(f"[train] pixel subsampling: {args.pixels_per_step}/{HW} pixels per image per step")
+
+    print(f"[train] {steps_per_epoch} steps/epoch x {args.epochs} epochs "
+          f"= {total_steps} steps")
+
+    step = 0
+    t_start = time.time()
+    for epoch in range(args.epochs):
+        perm = rng.permutation(N)
+        model.train()
+        running = 0.0
+        for it in range(steps_per_epoch):
+            batch_idx = perm[(it * args.batch_size) % N:
+                             (it * args.batch_size) % N + args.batch_size]
+            if len(batch_idx) < args.batch_size:
+                batch_idx = rng.integers(0, N, size=args.batch_size)
+
+            B = len(batch_idx)
+            if data_on_gpu:
+                idx_t = torch.as_tensor(batch_idx, device=device, dtype=torch.long)
+                crops = gpu_random_crops(images_dev, idx_t, args.crop, gen)
+                target_full = crops.reshape(B, -1, 3)
+                tokens = tokens_dev[idx_t]
+            else:
+                crops = random_crops(images, batch_idx, args.crop, rng)
+                target_full = torch.from_numpy(crops.reshape(B, -1, 3)).to(device)
+                tokens = torch.from_numpy(tokens_all[batch_idx]).to(device)
+
+            if 0 < args.pixels_per_step < HW:
+                pix = torch.randint(0, HW, (B, args.pixels_per_step), device=device, generator=gen)
+                bar = torch.arange(B, device=device)[:, None]
+                coords_b = coords[0][pix]
+                target = target_full[bar, pix]
+            else:
+                coords_b = coords.expand(B, -1, -1)
+                target = target_full
+
+            lr = cosine_lr(step, total_steps, args.lr, args.min_lr, args.warmup_steps)
+            for g in opt.param_groups:
+                g["lr"] = lr
+
+            opt.zero_grad(set_to_none=True)
+            with torch.amp.autocast("cuda", enabled=args.amp):
+                pred = model(tokens, coords_b)
+                loss = F.mse_loss(pred, target)
+            scaler.scale(loss).backward()
+            if args.grad_clip:
+                scaler.unscale_(opt)
+                torch.nn.utils.clip_grad_norm_(model.parameters(), args.grad_clip)
+            scaler.step(opt)
+            scaler.update()
+
+            running += loss.item()
+
+            if step in (0, 1, 5, 20, 100) or (epoch == 0 and it % 200 == 0):
+                print_activation_stats(model, tokens, coords_b, tag=f"step{step}")
+
+            if step % args.log_interval == 0:
+                msg = f"[e{epoch:03d} s{step:06d}] loss={loss.item():.5f} lr={lr:.2e}"
+                if device == "cuda":
+                    peak = torch.cuda.max_memory_allocated() / 1e9
+                    msg += f" vram_peak={peak:.2f}GB"
+                    if peak > args.vram_budget_gb:
+                        msg += f"  !! over {args.vram_budget_gb}GB budget"
+                print(msg)
+            step += 1
+
+        if (epoch + 1) % args.save_every_epochs == 0 or epoch == args.epochs - 1:
+            info = save_model_png(model, args.out_png, args.out_config, verbose=(epoch == 0))
+            elapsed = (time.time() - t_start) / 60.0
+            print(f"[e{epoch:03d}] avg_loss={running/steps_per_epoch:.5f} "
+                  f"-> wrote {args.out_png} ({info['png_bytes']/1e6:.3f} MB) "
+                  f"| {elapsed:.1f} min elapsed")
+
+    print(f"[train] done in {(time.time()-t_start)/60.0:.1f} min. "
+          f"Canonical model -> {args.out_png}")
+
+
+if __name__ == "__main__":
+    main()

vocab.json

@@ -0,0 +1 @@
+{"<pad>": 0, "<unk>": 1, "a": 2, "on": 3, "the": 4, "of": 5, "in": 6, "with": 7, "and": 8, "is": 9, "man": 10, "to": 11, "an": 12, "sitting": 13, "two": 14, "are": 15, "at": 16, "people": 17, "standing": 18, "white": 19, "next": 20, "woman": 21, "street": 22, "that": 23, "it": 24, "table": 25, "some": 26, "holding": 27, "large": 28, "person": 29, "his": 30, "down": 31, "up": 32, "tennis": 33, "train": 34, "black": 35, "top": 36, "group": 37, "small": 38, "field": 39, "near": 40, "cat": 41, "dog": 42, "front": 43, "room": 44, "has": 45, "by": 46, "water": 47, "red": 48, "plate": 49, "young": 50, "bathroom": 51, "riding": 52, "while": 53, "sign": 54, "playing": 55, "baseball": 56, "kitchen": 57, "walking": 58, "building": 59, "blue": 60, "there": 61, "other": 62, "looking": 63, "food": 64, "bus": 65, "green": 66, "clock": 67, "pizza": 68, "parked": 69, "grass": 70, "beach": 71, "side": 72, "bed": 73, "snow": 74, "three": 75, "toilet": 76, "road": 77, "city": 78, "ball": 79, "for": 80, "boy": 81, "her": 82, "several": 83, "picture": 84, "player": 85, "wearing": 86, "out": 87, "girl": 88, "flying": 89, "men": 90, "couple": 91, "over": 92, "sits": 93, "bench": 94, "one": 95, "from": 96, "computer": 97, "area": 98, "skateboard": 99, "laptop": 100, "bear": 101, "game": 102, "wooden": 103, "their": 104, "horse": 105, "as": 106, "yellow": 107, "phone": 108, "sink": 109, "outside": 110, "laying": 111, "board": 112, "eating": 113, "cake": 114, "through": 115, "many": 116, "giraffe": 117, "frisbee": 118, "wall": 119, "brown": 120, "this": 121, "motorcycle": 122, "around": 123, "under": 124, "window": 125, "umbrella": 126, "living": 127, "desk": 128, "old": 129, "trees": 130, "park": 131, "air": 132, "truck": 133, "tree": 134, "very": 135, "s": 136, "elephant": 137, "open": 138, "each": 139, "stop": 140, "car": 141, "child": 142, "behind": 143, "fire": 144, "close": 145, "background": 146, "little": 147, "together": 148, "its": 149, "stands": 150, "inside": 151, "court": 152, "glass": 153, "big": 154, "surfboard": 155, "cars": 156, "boat": 157, "different": 158, "hand": 159, "into": 160, "light": 161, "kite": 162, "orange": 163, "skis": 164, "ocean": 165, "mirror": 166, "cell": 167, "floor": 168, "hydrant": 169, "view": 170, "shirt": 171, "sheep": 172, "bunch": 173, "filled": 174, "photo": 175, "chair": 176, "back": 177, "sky": 178, "covered": 179, "bird": 180, "sidewalk": 181, "bowl": 182, "sandwich": 183, "counter": 184, "flowers": 185, "tower": 186, "racket": 187, "zebra": 188, "teddy": 189, "tracks": 190, "wave": 191, "stand": 192, "tall": 193, "parking": 194, "couch": 195, "women": 196, "being": 197, "vase": 198, "another": 199, "baby": 200, "day": 201, "zebras": 202, "fence": 203, "head": 204, "traffic": 205, "lot": 206, "hill": 207, "bat": 208, "bananas": 209, "station": 210, "along": 211, "elephants": 212, "plane": 213, "vegetables": 214, "taking": 215, "ground": 216, "dirt": 217, "image": 218, "ready": 219, "giraffes": 220, "tie": 221, "stuffed": 222, "off": 223, "horses": 224, "airplane": 225, "skiing": 226, "middle": 227, "above": 228, "sit": 229, "full": 230, "grassy": 231, "empty": 232, "camera": 233, "cows": 234, "signs": 235, "four": 236, "broccoli": 237, "long": 238, "holds": 239, "beside": 240, "boats": 241, "children": 242, "pole": 243, "slope": 244, "doing": 245, "during": 246, "grazing": 247, "ski": 248, "hanging": 249, "across": 250, "corner": 251, "going": 252, "snowy": 253, "soccer": 254, "wine": 255, "umbrellas": 256, "hot": 257, "smiling": 258, "piece": 259, "hat": 260, "track": 261, "dogs": 262, "mountain": 263, "refrigerator": 264, "using": 265, "who": 266, "luggage": 267, "keyboard": 268, "them": 269, "skate": 270, "body": 271, "buildings": 272, "stove": 273, "driving": 274, "fruit": 275, "watching": 276, "brick": 277, "colorful": 278, "surf": 279, "bike": 280, "cutting": 281, "set": 282, "pair": 283, "birds": 284, "guy": 285, "against": 286, "someone": 287, "pink": 288, "oven": 289, "talking": 290, "chairs": 291, "lady": 292, "tv": 293, "herd": 294, "looks": 295, "posing": 296, "suit": 297, "wii": 298, "cup": 299, "trick": 300, "paper": 301, "items": 302, "television": 303, "night": 304, "cow": 305, "jumping": 306, "video": 307, "display": 308, "surfing": 309, "door": 310, "box": 311, "animals": 312, "glasses": 313, "kites": 314, "motorcycles": 315, "carrying": 316, "be": 317, "swinging": 318, "crowd": 319, "bedroom": 320, "airport": 321, "intersection": 322, "hit": 323, "getting": 324, "face": 325, "dressed": 326, "house": 327, "way": 328, "various": 329, "shower": 330, "something": 331, "colored": 332, "wood": 333, "about": 334, "walk": 335, "bears": 336, "double": 337, "skier": 338, "banana": 339, "restaurant": 340, "sleeping": 341, "tray": 342, "traveling": 343, "like": 344, "hands": 345, "meat": 346, "remote": 347, "bottle": 348, "running": 349, "lying": 350, "surfer": 351, "lights": 352, "passenger": 353, "river": 354, "coffee": 355, "busy": 356, "waiting": 357, "showing": 358, "bicycle": 359, "kids": 360, "preparing": 361, "screen": 362, "male": 363, "all": 364, "rides": 365, "plates": 366, "scissors": 367, "ramp": 368, "enclosure": 369, "cheese": 370, "play": 371, "book": 372, "seat": 373, "high": 374, "metal": 375, "walks": 376, "line": 377, "slice": 378, "donuts": 379, "lined": 380, "bridge": 381, "racquet": 382, "carrots": 383, "decorated": 384, "painted": 385, "home": 386, "he": 387, "skateboarder": 388, "made": 389, "him": 390, "mouse": 391, "grey": 392, "players": 393, "bag": 394, "mouth": 395, "few": 396, "adult": 397, "waves": 398, "past": 399, "silver": 400, "shown": 401, "knife": 402, "bread": 403, "microwave": 404, "look": 405, "zoo": 406, "batter": 407, "jet": 408, "walls": 409, "dining": 410, "outdoor": 411, "snowboard": 412, "gray": 413, "half": 414, "dark": 415, "animal": 416, "store": 417, "number": 418, "suitcase": 419, "hair": 420, "monitor": 421, "toy": 422, "throwing": 423, "lake": 424, "lots": 425, "surrounded": 426, "scene": 427, "cabinets": 428, "meal": 429, "runway": 430, "skiers": 431, "public": 432, "sand": 433, "topped": 434, "purple": 435, "coming": 436, "row": 437, "have": 438, "buses": 439, "attached": 440, "drink": 441, "windows": 442, "fork": 443, "tub": 444, "resting": 445, "displayed": 446, "tables": 447, "watch": 448, "cut": 449, "beer": 450, "stopped": 451, "bright": 452, "jacket": 453, "female": 454, "decker": 455, "surfboards": 456, "no": 457, "girls": 458, "birthday": 459, "can": 460, "bikes": 461, "oranges": 462, "onto": 463, "sun": 464, "shelf": 465, "between": 466, "nice": 467, "statue": 468, "rocks": 469, "salad": 470, "cross": 471, "or": 472, "been": 473, "towards": 474, "boys": 475, "basket": 476, "shows": 477, "plants": 478, "edge": 479, "kid": 480, "eat": 481, "market": 482, "drinking": 483, "toppings": 484, "pulling": 485, "modern": 486, "underneath": 487, "pile": 488, "lit": 489, "chocolate": 490, "helmet": 491, "crossing": 492, "clean": 493, "pitch": 494, "multiple": 495, "stone": 496, "seen": 497, "sunny": 498, "apples": 499, "moving": 500, "swing": 501, "pictures": 502, "flower": 503, "shot": 504, "watches": 505, "rain": 506, "fries": 507, "mountains": 508, "working": 509, "ride": 510, "others": 511, "clear": 512, "hitting": 513, "catch": 514, "leaning": 515, "hotel": 516, "teeth": 517, "furniture": 518, "plastic": 519, "rack": 520, "older": 521, "chicken": 522, "skateboarding": 523, "cats": 524, "not": 525, "making": 526, "dress": 527, "catcher": 528, "books": 529, "photograph": 530, "single": 531, "wet": 532, "perched": 533, "uniform": 534, "platform": 535, "office": 536, "slices": 537, "takes": 538, "pan": 539, "pitcher": 540, "blanket": 541, "fruits": 542, "graffiti": 543, "laptops": 544, "setting": 545, "shore": 546, "cloudy": 547, "left": 548, "post": 549, "tiled": 550, "pizzas": 551, "meter": 552, "beautiful": 553, "doughnut": 554, "snowboarder": 555, "rail": 556, "plant": 557, "donut": 558, "leaves": 559, "well": 560, "computers": 561, "shoes": 562, "variety": 563, "passengers": 564, "containing": 565, "yard": 566, "time": 567, "country": 568, "sofa": 569, "cart": 570, "tile": 571, "suitcases": 572, "taken": 573, "branch": 574, "path": 575, "control": 576, "motor": 577, "types": 578, "bags": 579, "town": 580, "forest": 581, "engine": 582, "rock": 583, "brushing": 584, "bathtub": 585, "passing": 586, "sandy": 587, "boxes": 588, "cooking": 589, "appliances": 590, "staring": 591, "having": 592, "they": 593, "vehicle": 594, "placed": 595, "pieces": 596, "shop": 597, "crowded": 598, "pillows": 599, "trying": 600, "where": 601, "style": 602, "cooked": 603, "take": 604, "trucks": 605, "boards": 606, "case": 607, "tricks": 608, "nearby": 609, "serve": 610, "fenced": 611, "brush": 612, "concrete": 613, "smiles": 614, "fresh": 615, "trains": 616, "phones": 617, "pot": 618, "swimming": 619, "go": 620, "umpire": 621, "enjoying": 622, "railroad": 623, "arm": 624, "apple": 625, "dock": 626, "police": 627, "garden": 628, "airplanes": 629, "lamp": 630, "right": 631, "lays": 632, "surface": 633, "equipment": 634, "dish": 635, "bath": 636, "benches": 637, "gear": 638, "including": 639, "polar": 640, "asian": 641, "match": 642, "candles": 643, "pool": 644, "fridge": 645, "center": 646, "beds": 647, "eaten": 648, "cellphone": 649, "jump": 650, "flies": 651, "container": 652, "pretty": 653, "catching": 654, "guys": 655, "reading": 656, "serving": 657, "bottles": 658, "school": 659, "drinks": 660, "sinks": 661, "wedding": 662, "doors": 663, "same": 664, "cute": 665, "feeding": 666, "vases": 667, "surfers": 668, "alone": 669, "bowls": 670, "games": 671, "docked": 672, "gathered": 673, "lap": 674, "carriage": 675, "swings": 676, "rice": 677, "run": 678, "five": 679, "dirty": 680, "pen": 681, "feet": 682, "cement": 683, "she": 684, "shaped": 685, "closed": 686, "mid": 687, "performing": 688, "harbor": 689, "pulled": 690, "tomatoes": 691, "controller": 692, "t": 693, "get": 694, "dry": 695, "shorts": 696, "pasture": 697, "away": 698, "wild": 699, "prepares": 700, "trunk": 701, "after": 702, "square": 703, "facing": 704, "dinner": 705, "trash": 706, "which": 707, "sandwiches": 708, "desert": 709, "pose": 710, "stall": 711, "putting": 712, "curb": 713, "toothbrush": 714, "vintage": 715, "beneath": 716, "woods": 717, "striped": 718, "both": 719, "family": 720, "stacked": 721, "assorted": 722, "kind": 723, "bride": 724, "toward": 725, "reflection": 726, "turn": 727, "french": 728, "tied": 729, "poles": 730, "just": 731, "bar": 732, "sticking": 733, "lush": 734, "rocky": 735, "cluttered": 736, "colors": 737, "served": 738, "messy": 739, "hay": 740, "work": 741, "picnic": 742, "fireplace": 743, "subway": 744, "kitten": 745, "breakfast": 746, "neck": 747, "show": 748, "throw": 749, "pillow": 750, "object": 751, "blender": 752, "place": 753, "pots": 754, "fancy": 755, "dessert": 756, "cage": 757, "mounted": 758, "reaching": 759, "cattle": 760, "himself": 761, "rider": 762, "fly": 763, "clouds": 764, "mother": 765, "ice": 766, "cream": 767, "event": 768, "here": 769, "paddle": 770, "foreground": 771, "flat": 772, "round": 773, "low": 774, "boarding": 775, "among": 776, "broken": 777, "nintendo": 778, "gold": 779, "coat": 780, "closeup": 781, "drives": 782, "winter": 783, "curtain": 784, "business": 785, "skateboards": 786, "seated": 787, "potatoes": 788, "commuter": 789, "fashioned": 790, "poses": 791, "vehicles": 792, "electronic": 793, "assortment": 794, "dishes": 795, "eyes": 796, "below": 797, "atop": 798, "ceiling": 799, "painting": 800, "church": 801, "distance": 802, "soup": 803, "vegetable": 804, "lone": 805, "outfit": 806, "multi": 807, "ship": 808, "race": 809, "space": 810, "narrow": 811, "bicycles": 812, "writing": 813, "steel": 814, "christmas": 815, "overlooking": 816, "pedestrians": 817, "snowboarding": 818, "wooded": 819, "monitors": 820, "features": 821, "professional": 822, "bottom": 823, "leather": 824, "lid": 825, "commercial": 826, "plays": 827, "groom": 828, "rests": 829, "doughnuts": 830, "team": 831, "flock": 832, "trail": 833, "transit": 834, "wide": 835, "pans": 836, "clocks": 837, "see": 838, "suits": 839, "huge": 840, "backpack": 841, "flag": 842, "legs": 843, "type": 844, "pointing": 845, "restroom": 846, "blurry": 847, "smaller": 848, "streets": 849, "haired": 850, "disc": 851, "sauce": 852, "you": 853, "spoon": 854, "sort": 855, "tarmac": 856, "gate": 857, "bacon": 858, "bow": 859, "outdoors": 860, "van": 861, "adults": 862, "urban": 863, "garage": 864, "appears": 865, "does": 866, "sized": 867, "bun": 868, "put": 869, "skies": 870, "happy": 871, "sliced": 872, "party": 873, "eggs": 874, "make": 875, "tabby": 876, "towel": 877, "makes": 878, "but": 879, "farm": 880, "floors": 881, "course": 882, "was": 883, "rest": 884, "skating": 885, "holder": 886, "carpet": 887, "foot": 888, "construction": 889, "roof": 890, "hole": 891, "used": 892, "sheet": 893, "pictured": 894, "mustard": 895, "leans": 896, "decorative": 897, "action": 898, "american": 899, "ear": 900, "towels": 901, "giant": 902, "machine": 903, "clothes": 904, "end": 905, "says": 906, "new": 907, "toys": 908, "planes": 909, "cover": 910, "pie": 911, "apartment": 912, "tan": 913, "log": 914, "rug": 915, "decorations": 916, "desktop": 917, "wheel": 918, "pastry": 919, "urinals": 920, "floating": 921, "flip": 922, "bushes": 923, "cabinet": 924, "steps": 925, "stick": 926, "milk": 927, "baskets": 928, "plain": 929, "graze": 930, "land": 931, "antique": 932, "rural": 933, "model": 934, "hotdog": 935, "locomotive": 936, "clothing": 937, "couches": 938, "jumps": 939, "shirtless": 940, "mound": 941, "friends": 942, "sides": 943, "hold": 944, "toilets": 945, "wetsuit": 946, "part": 947, "curled": 948, "touching": 949, "name": 950, "structure": 951, "sculpture": 952, "hard": 953, "pond": 954, "lettuce": 955, "landing": 956, "paved": 957, "advertisement": 958, "drawn": 959, "ornate": 960, "tomato": 961, "relaxing": 962, "prepared": 963, "foods": 964, "pulls": 965, "reads": 966, "held": 967, "ties": 968, "persons": 969, "overhead": 970, "houses": 971, "we": 972, "stairs": 973, "photos": 974, "barn": 975, "leash": 976, "herself": 977, "faces": 978, "2": 979, "stadium": 980, "smoke": 981, "passes": 982, "riders": 983, "spectators": 984, "bull": 985, "stack": 986, "approaching": 987, "wire": 988, "beverage": 989, "veggies": 990, "snowboards": 991, "controllers": 992, "chips": 993, "alongside": 994, "grill": 995, "shelves": 996, "shopping": 997, "flags": 998, "before": 999, "lunch": 1000, "frosting": 1001, "residential": 1002, "mushrooms": 1003, "selfie": 1004, "arranged": 1005, "purse": 1006, "headphones": 1007, "nose": 1008, "glove": 1009, "cloth": 1010, "directions": 1011, "chili": 1012, "racing": 1013, "uniforms": 1014, "sailing": 1015, "beige": 1016, "pants": 1017, "onions": 1018, "bite": 1019, "reaches": 1020, "signal": 1021, "piled": 1022, "pepperoni": 1023, "stuck": 1024, "electric": 1025, "power": 1026, "eats": 1027, "falling": 1028, "arms": 1029, "includes": 1030, "sheets": 1031, "cups": 1032, "booth": 1033, "block": 1034, "meters": 1035, "direction": 1036, "heads": 1037, "shoe": 1038, "sill": 1039, "flown": 1040, "what": 1041, "boarder": 1042, "do": 1043, "tea": 1044, "pavement": 1045, "noodles": 1046, "growing": 1047, "drive": 1048, "terminal": 1049, "only": 1050, "ledge": 1051, "elderly": 1052, "quiet": 1053, "fall": 1054, "partially": 1055, "indoor": 1056, "pick": 1057, "things": 1058, "sea": 1059, "costume": 1060, "itself": 1061, "fish": 1062, "bank": 1063, "military": 1064, "island": 1065, "tiny": 1066, "pasta": 1067, "beans": 1068, "hillside": 1069, "if": 1070, "walkway": 1071, "baking": 1072, "lines": 1073, "skiis": 1074, "motorcyclist": 1075, "potted": 1076, "pastries": 1077, "without": 1078, "pack": 1079, "carrot": 1080, "rainy": 1081, "art": 1082, "mixed": 1083, "calf": 1084, "waits": 1085, "cupboards": 1086, "plaid": 1087, "rackets": 1088, "loaded": 1089, "amongst": 1090, "practicing": 1091, "garbage": 1092, "tent": 1093, "kneeling": 1094, "palm": 1095, "opposite": 1096, "loading": 1097, "ripe": 1098, "deck": 1099, "throws": 1100, "hits": 1101, "wait": 1102, "written": 1103, "hats": 1104, "leading": 1105, "stainless": 1106, "shade": 1107, "patio": 1108, "turned": 1109, "railing": 1110, "design": 1111, "petting": 1112, "port": 1113, "highway": 1114, "pedestrian": 1115, "vanity": 1116, "mitt": 1117, "toddler": 1118, "ketchup": 1119, "posed": 1120, "short": 1121, "goats": 1122, "glazed": 1123, "soap": 1124, "goes": 1125, "mattress": 1126, "how": 1127, "stream": 1128, "fighter": 1129, "jets": 1130, "asleep": 1131, "collar": 1132, "airliner": 1133, "wheels": 1134, "papers": 1135, "cakes": 1136, "gather": 1137, "range": 1138, "ducks": 1139, "electronics": 1140, "break": 1141, "fried": 1142, "steep": 1143, "built": 1144, "wind": 1145, "opened": 1146, "kinds": 1147, "curtains": 1148, "helmets": 1149, "upside": 1150, "hotdogs": 1151, "apron": 1152, "travel": 1153, "lawn": 1154, "lies": 1155, "polka": 1156, "rear": 1157, "waters": 1158, "attempts": 1159, "library": 1160, "sunset": 1161, "fishing": 1162, "uses": 1163, "sunglasses": 1164, "pier": 1165, "storm": 1166, "following": 1167, "ladies": 1168, "biting": 1169, "downhill": 1170, "rolling": 1171, "blow": 1172, "covering": 1173, "tongue": 1174, "corn": 1175, "roll": 1176, "trailer": 1177, "bunches": 1178, "entertainment": 1179, "pancakes": 1180, "doll": 1181, "dryer": 1182, "so": 1183, "rowing": 1184, "collection": 1185, "i": 1186, "also": 1187, "digital": 1188, "device": 1189, "bending": 1190, "use": 1191, "wears": 1192, "egg": 1193, "net": 1194, "these": 1195, "wires": 1196, "color": 1197, "sticks": 1198, "microphone": 1199, "base": 1200, "thrown": 1201, "sail": 1202, "duck": 1203, "mini": 1204, "containers": 1205, "brightly": 1206, "helping": 1207, "gets": 1208, "telephone": 1209, "dot": 1210, "tail": 1211, "delicious": 1212, "arrangement": 1213, "focus": 1214, "audience": 1215, "attempting": 1216, "featuring": 1217, "hangs": 1218, "marina": 1219, "siting": 1220, "platter": 1221, "position": 1222, "carries": 1223, "baggage": 1224, "formal": 1225, "toaster": 1226, "cap": 1227, "brushes": 1228, "rows": 1229, "aerial": 1230, "hospital": 1231, "hiding": 1232, "roadway": 1233, "cones": 1234, "soda": 1235, "lighting": 1236, "utensils": 1237, "pipe": 1238, "sweater": 1239, "handle": 1240, "ingredients": 1241, "iron": 1242, "tour": 1243, "butter": 1244, "bush": 1245, "fallen": 1246, "simple": 1247, "located": 1248, "shoulder": 1249, "covers": 1250, "slopes": 1251, "leg": 1252, "puppy": 1253, "frisbees": 1254, "load": 1255, "controls": 1256, "pushing": 1257, "bookshelf": 1258, "contains": 1259, "baked": 1260, "skateboarders": 1261, "paint": 1262, "skyline": 1263, "flight": 1264, "strawberries": 1265, "league": 1266, "six": 1267, "candle": 1268, "lift": 1269, "neon": 1270, "scooter": 1271, "missing": 1272, "cargo": 1273, "boots": 1274, "males": 1275, "jersey": 1276, "dresser": 1277, "displaying": 1278, "fake": 1279, "someones": 1280, "fun": 1281, "checking": 1282, "vest": 1283, "pickup": 1284, "wings": 1285, "surfs": 1286, "interior": 1287, "giving": 1288, "folded": 1289, "pouring": 1290, "stares": 1291, "ben": 1292, "industrial": 1293, "mirrors": 1294, "leafy": 1295, "larger": 1296, "bars": 1297, "words": 1298, "spread": 1299, "youth": 1300, "catches": 1301, "balloons": 1302, "moves": 1303, "beak": 1304, "balcony": 1305, "personal": 1306, "good": 1307, "enclosed": 1308, "ring": 1309, "resort": 1310, "cupcakes": 1311, "fans": 1312, "circular": 1313, "packed": 1314, "motion": 1315, "more": 1316, "travelling": 1317, "images": 1318, "chefs": 1319, "dishwasher": 1320, "trays": 1321, "forward": 1322, "supplies": 1323, "silly": 1324, "stuff": 1325, "matching": 1326, "lobby": 1327, "runs": 1328, "thin": 1329, "sleeps": 1330, "neatly": 1331, "freshly": 1332, "laughing": 1333, "branches": 1334, "fixtures": 1335, "bent": 1336, "comforter": 1337, "hang": 1338, "costumes": 1339, "tooth": 1340, "licking": 1341, "guitar": 1342, "trolley": 1343, "ceramic": 1344, "stool": 1345, "peppers": 1346, "fast": 1347, "panda": 1348, "almost": 1349, "blankets": 1350, "mobile": 1351, "paw": 1352, "shining": 1353, "sale": 1354, "deep": 1355, "feeder": 1356, "eye": 1357, "toast": 1358, "steam": 1359, "wicker": 1360, "landscape": 1361, "smile": 1362, "batting": 1363, "selling": 1364, "tank": 1365, "worker": 1366, "seagull": 1367, "sunlight": 1368, "sausage": 1369, "talks": 1370, "lounging": 1371, "shallow": 1372, "hallway": 1373, "owner": 1374, "string": 1375, "officers": 1376, "pickle": 1377, "dump": 1378, "crosswalk": 1379, "weather": 1380, "evening": 1381, "attire": 1382, "ham": 1383, "item": 1384, "smart": 1385, "pocket": 1386, "safety": 1387, "chasing": 1388, "watering": 1389, "tunnel": 1390, "remotes": 1391, "kicking": 1392, "whole": 1393, "steak": 1394, "savannah": 1395, "semi": 1396, "burger": 1397, "rails": 1398, "rainbow": 1399, "cuts": 1400, "entrance": 1401, "grabbing": 1402, "bucket": 1403, "toothbrushes": 1404, "mug": 1405, "soldiers": 1406, "friend": 1407, "posted": 1408, "performs": 1409, "competition": 1410, "fountain": 1411, "buns": 1412, "clay": 1413, "diner": 1414, "balls": 1415, "neighborhood": 1416, "turkey": 1417, "stir": 1418, "tools": 1419, "hooked": 1420, "lane": 1421, "shrimp": 1422, "collage": 1423, "pet": 1424, "visible": 1425, "floral": 1426, "cabin": 1427, "patterned": 1428, "naked": 1429, "officer": 1430, "tusks": 1431, "parent": 1432, "motorcyclists": 1433, "marble": 1434, "counters": 1435, "sports": 1436, "paws": 1437, "tops": 1438, "leaping": 1439, "pattern": 1440, "mostly": 1441, "smoking": 1442, "countryside": 1443, "section": 1444, "hose": 1445, "gathering": 1446, "rusty": 1447, "move": 1448, "rusted": 1449, "heavy": 1450, "framed": 1451, "neat": 1452, "kitty": 1453, "quilt": 1454, "surrounding": 1455, "golf": 1456, "pass": 1457, "bamboo": 1458, "cobblestone": 1459, "story": 1460, "comes": 1461, "fan": 1462, "slightly": 1463, "blinds": 1464, "grapes": 1465, "sharing": 1466, "bearded": 1467, "wrapped": 1468, "foil": 1469, "london": 1470, "cold": 1471, "chain": 1472, "shirts": 1473, "medium": 1474, "doorway": 1475, "fluffy": 1476, "countertop": 1477, "human": 1478, "balancing": 1479, "homemade": 1480, "sliding": 1481, "basketball": 1482, "herding": 1483, "plugged": 1484, "onion": 1485, "parade": 1486, "speakers": 1487, "statues": 1488, "unfinished": 1489, "adorable": 1490, "still": 1491, "frame": 1492, "stage": 1493, "habitat": 1494, "valley": 1495, "travels": 1496, "grilled": 1497, "jumbo": 1498, "hills": 1499, "cellular": 1500, "trough": 1501, "map": 1502, "checkered": 1503, "liquid": 1504, "o": 1505, "driveway": 1506, "funny": 1507, "gloves": 1508, "stickers": 1509, "approaches": 1510, "awaiting": 1511, "objects": 1512, "steeple": 1513, "speed": 1514, "blowing": 1515, "shrubbery": 1516, "shops": 1517, "asphalt": 1518, "suite": 1519, "expanse": 1520, "cycle": 1521, "busses": 1522, "lining": 1523, "extended": 1524, "warning": 1525, "washing": 1526, "fully": 1527, "sailboat": 1528, "storage": 1529, "roman": 1530, "love": 1531, "boarders": 1532, "elaborate": 1533, "silverware": 1534, "twin": 1535, "chef": 1536, "spray": 1537, "freezer": 1538, "urinal": 1539, "jeans": 1540, "roses": 1541, "heading": 1542, "dried": 1543, "wading": 1544, "displays": 1545, "chinese": 1546, "frosted": 1547, "headboard": 1548, "grown": 1549, "individual": 1550, "grazes": 1551, "juice": 1552, "married": 1553, "beard": 1554, "cookies": 1555, "st": 1556, "museum": 1557, "ears": 1558, "weeds": 1559, "bay": 1560, "pitching": 1561, "portrait": 1562, "miniature": 1563, "hugging": 1564, "such": 1565, "devices": 1566, "tin": 1567, "warehouse": 1568, "climbing": 1569, "nearly": 1570, "napkin": 1571, "poster": 1572, "printer": 1573, "abandoned": 1574, "backdrop": 1575, "racquets": 1576, "bicyclists": 1577, "key": 1578, "size": 1579, "tourists": 1580, "stripes": 1581, "anchored": 1582, "connected": 1583, "odd": 1584, "far": 1585, "vendor": 1586, "stoplight": 1587, "indoors": 1588, "closet": 1589, "porch": 1590, "ribbon": 1591, "bin": 1592, "order": 1593, "desserts": 1594, "states": 1595, "level": 1596, "chewing": 1597, "cans": 1598, "sleep": 1599, "split": 1600, "typing": 1601, "propped": 1602, "dressing": 1603, "potato": 1604, "dead": 1605, "moped": 1606, "wear": 1607, "muffin": 1608, "fry": 1609, "muddy": 1610, "canoe": 1611, "spraying": 1612, "mixer": 1613, "saddle": 1614, "straw": 1615, "waterway": 1616, "kissing": 1617, "when": 1618, "scattered": 1619, "circle": 1620, "oriental": 1621, "decor": 1622, "spot": 1623, "foreign": 1624, "grocery": 1625, "scale": 1626, "besides": 1627, "softball": 1628, "organized": 1629, "horseback": 1630, "plaza": 1631, "rolls": 1632, "mans": 1633, "prepare": 1634, "amount": 1635, "patch": 1636, "goat": 1637, "squatting": 1638, "bats": 1639, "examining": 1640, "chopped": 1641, "seems": 1642, "fingers": 1643, "courtyard": 1644, "carts": 1645, "sad": 1646, "wheelchair": 1647, "cooks": 1648, "tuxedo": 1649, "lay": 1650, "lamb": 1651, "gentleman": 1652, "peanut": 1653, "point": 1654, "alley": 1655, "d": 1656, "kneels": 1657, "help": 1658, "hydrants": 1659, "cigarette": 1660, "candy": 1661, "ave": 1662, "exhibit": 1663, "opening": 1664, "delivery": 1665, "beef": 1666, "card": 1667, "crust": 1668, "return": 1669, "seats": 1670, "sprinkles": 1671, "foggy": 1672, "cauliflower": 1673, "backs": 1674, "sets": 1675, "returning": 1676, "somewhat": 1677, "shape": 1678, "fixing": 1679, "peeled": 1680, "designs": 1681, "tattoo": 1682, "paintings": 1683, "skirt": 1684, "apart": 1685, "snacks": 1686, "magazine": 1687, "boogie": 1688, "places": 1689, "pitchers": 1690, "hamburger": 1691, "raft": 1692, "holiday": 1693, "ways": 1694, "screens": 1695, "champagne": 1696, "stopping": 1697, "hood": 1698, "kick": 1699, "cubicle": 1700, "those": 1701, "roaming": 1702, "furry": 1703, "belt": 1704, "multicolored": 1705, "life": 1706, "peeking": 1707, "interesting": 1708, "granite": 1709, "sailboats": 1710, "classroom": 1711, "bunk": 1712, "cords": 1713, "radio": 1714, "frisbe": 1715, "helps": 1716, "desks": 1717, "outfield": 1718, "eight": 1719, "figurine": 1720, "propeller": 1721, "world": 1722, "mat": 1723, "dispenser": 1724, "coast": 1725, "arch": 1726, "ramps": 1727, "mom": 1728, "syrup": 1729, "keep": 1730, "icing": 1731, "played": 1732, "jetliner": 1733, "views": 1734, "roads": 1735, "flooring": 1736, "fighting": 1737, "mud": 1738, "jungle": 1739, "teenagers": 1740, "adjusts": 1741, "sneakers": 1742, "workers": 1743, "swim": 1744, "talk": 1745, "major": 1746, "greens": 1747, "snack": 1748, "toasted": 1749, "indicating": 1750, "temple": 1751, "pens": 1752, "army": 1753, "lime": 1754, "shades": 1755, "silhouette": 1756, "ones": 1757, "produce": 1758, "tow": 1759, "terrain": 1760, "suspended": 1761, "teams": 1762, "students": 1763, "celebrating": 1764, "washer": 1765, "bigger": 1766, "cruise": 1767, "hour": 1768, "contents": 1769, "cheesy": 1770, "cook": 1771, "stripe": 1772, "elegant": 1773, "railway": 1774, "texting": 1775, "jackets": 1776, "investigating": 1777, "planter": 1778, "star": 1779, "third": 1780, "groups": 1781, "logs": 1782, "cluster": 1783, "crane": 1784, "english": 1785, "tossing": 1786, "biker": 1787, "turning": 1788, "hung": 1789, "sugar": 1790, "drying": 1791, "transportation": 1792, "elevated": 1793, "rough": 1794, "raised": 1795, "gentle": 1796, "york": 1797, "come": 1798, "really": 1799, "tissue": 1800, "hind": 1801, "pipes": 1802, "japanese": 1803, "pigeons": 1804, "wilderness": 1805, "fat": 1806, "blonde": 1807, "coleslaw": 1808, "either": 1809, "kept": 1810, "grassland": 1811, "pears": 1812, "nothing": 1813, "females": 1814, "bare": 1815, "seagulls": 1816, "cardboard": 1817, "most": 1818, "dimly": 1819, "n": 1820, "ahead": 1821, "members": 1822, "listening": 1823, "mopeds": 1824, "flooded": 1825, "metro": 1826, "aged": 1827, "knees": 1828, "cafe": 1829, "unique": 1830, "gas": 1831, "hilly": 1832, "second": 1833, "possibly": 1834, "themed": 1835, "tasty": 1836, "refrigerators": 1837, "photographed": 1838, "piano": 1839, "wing": 1840, "pickles": 1841, "mall": 1842, "cubs": 1843, "bell": 1844, "stare": 1845, "dual": 1846, "smelling": 1847, "uncooked": 1848, "parasol": 1849, "freight": 1850, "british": 1851, "beverages": 1852, "depicting": 1853, "guide": 1854, "wrapper": 1855, "jar": 1856, "ferry": 1857, "rope": 1858, "cartoon": 1859, "illuminated": 1860, "early": 1861, "tiered": 1862, "aircraft": 1863, "stops": 1864, "iphone": 1865, "magnets": 1866, "backpacks": 1867, "diving": 1868, "condiments": 1869, "stunts": 1870, "tasting": 1871, "mesh": 1872, "site": 1873, "lie": 1874, "4": 1875, "v": 1876, "placing": 1877, "plains": 1878, "first": 1879, "snowboarders": 1880, "fashion": 1881, "posters": 1882, "bricked": 1883, "products": 1884, "driver": 1885, "language": 1886, "hall": 1887, "mural": 1888, "caption": 1889, "castle": 1890, "sparse": 1891, "populated": 1892, "curved": 1893, "meeting": 1894, "natural": 1895, "grinding": 1896, "outfits": 1897, "rooms": 1898, "blocks": 1899, "motorbike": 1900, "captive": 1901, "enjoy": 1902, "den": 1903, "bulls": 1904, "word": 1905, "areas": 1906, "divider": 1907, "shelter": 1908, "advertising": 1909, "smiley": 1910, "gourmet": 1911, "call": 1912, "mask": 1913, "horns": 1914, "speeding": 1915, "outstretched": 1916, "music": 1917, "stars": 1918, "process": 1919, "teenager": 1920, "handles": 1921, "signals": 1922, "midair": 1923, "photographs": 1924, "touch": 1925, "barbed": 1926, "sharp": 1927, "angle": 1928, "entering": 1929, "ipod": 1930, "president": 1931, "learning": 1932, "cereal": 1933, "service": 1934, "formation": 1935, "turns": 1936, "cards": 1937, "sweet": 1938, "airborne": 1939, "blocked": 1940, "dusk": 1941, "athlete": 1942, "united": 1943, "forks": 1944, "overpass": 1945, "wandering": 1946, "teaching": 1947, "consisting": 1948, "artistic": 1949, "strewn": 1950, "calm": 1951, "parents": 1952, "breaking": 1953, "scooters": 1954, "cupcake": 1955, "canoes": 1956, "parasailing": 1957, "bouquet": 1958, "moored": 1959, "spots": 1960, "west": 1961, "carpeted": 1962, "rams": 1963, "shines": 1964, "shoreline": 1965, "too": 1966, "special": 1967, "strawberry": 1968, "piles": 1969, "points": 1970, "bean": 1971, "pregnant": 1972, "chopsticks": 1973, "everyone": 1974, "sniffing": 1975, "were": 1976, "cowboy": 1977, "wildlife": 1978, "stunt": 1979, "peas": 1980, "plated": 1981, "countertops": 1982, "layer": 1983, "sideways": 1984, "bagel": 1985, "installed": 1986, "figures": 1987, "grain": 1988, "shiny": 1989, "honey": 1990, "shrubs": 1991, "skillet": 1992, "conference": 1993, "jockey": 1994, "suburban": 1995, "curve": 1996, "barrier": 1997, "cellphones": 1998, "bunny": 1999, "3": 2000, "sub": 2001, "winds": 2002, "ends": 2003, "powered": 2004, "racks": 2005, "african": 2006, "pin": 2007, "presents": 2008, "surround": 2009, "rounded": 2010, "crates": 2011, "shots": 2012, "stacks": 2013, "obstacle": 2014, "note": 2015, "speaking": 2016, "packaged": 2017, "lemons": 2018, "sport": 2019, "tiles": 2020, "headed": 2021, "tries": 2022, "jars": 2023, "china": 2024, "coach": 2025, "bikers": 2026, "teal": 2027, "numerals": 2028, "roller": 2029, "preparation": 2030, "button": 2031, "electrical": 2032, "cases": 2033, "handing": 2034, "vending": 2035, "looked": 2036, "slicing": 2037, "cleaned": 2038, "lighted": 2039, "leaf": 2040, "slide": 2041, "tidy": 2042, "hanger": 2043, "ladder": 2044, "gown": 2045, "donkey": 2046, "furnishings": 2047, "berries": 2048, "reflecting": 2049, "furnished": 2050, "backyard": 2051, "nicely": 2052, "coin": 2053, "customers": 2054, "cliff": 2055, "straight": 2056, "hardwood": 2057, "tractor": 2058, "monkey": 2059, "walled": 2060, "security": 2061, "pepper": 2062, "fabric": 2063, "towers": 2064, "rustic": 2065, "falls": 2066, "yacht": 2067, "newspaper": 2068, "asparagus": 2069, "tags": 2070, "guard": 2071, "change": 2072, "spinach": 2073, "enter": 2074, "slanted": 2075, "metropolitan": 2076, "chest": 2077, "gated": 2078, "leads": 2079, "photographer": 2080, "paddling": 2081, "laid": 2082, "shake": 2083, "canopy": 2084, "lovely": 2085, "complete": 2086, "tropical": 2087, "setup": 2088, "peaceful": 2089, "decoration": 2090, "village": 2091, "stalls": 2092, "strange": 2093, "unmade": 2094, "carnival": 2095, "guitars": 2096, "morning": 2097, "intently": 2098, "flavored": 2099, "wake": 2100, "somewhere": 2101, "within": 2102, "trunks": 2103, "operating": 2104, "bundled": 2105, "conversation": 2106, "inflatable": 2107, "crouches": 2108, "extra": 2109, "washington": 2110, "shelving": 2111, "stretching": 2112, "own": 2113, "us": 2114, "mean": 2115, "feed": 2116, "breads": 2117, "war": 2118, "unusual": 2119, "pony": 2120, "directly": 2121, "blond": 2122, "sprinkled": 2123, "staircase": 2124, "chained": 2125, "lamps": 2126, "nest": 2127, "caught": 2128, "sails": 2129, "wagon": 2130, "clearing": 2131, "treat": 2132, "cool": 2133, "toothpaste": 2134, "lego": 2135, "podium": 2136, "speech": 2137, "mugs": 2138, "tee": 2139, "may": 2140, "tourist": 2141, "playfully": 2142, "mama": 2143, "ostrich": 2144, "external": 2145, "referee": 2146, "college": 2147, "cone": 2148, "keyboards": 2149, "information": 2150, "expressing": 2151, "closely": 2152, "grilling": 2153, "dresses": 2154, "landed": 2155, "message": 2156, "avenue": 2157, "conveyor": 2158, "technology": 2159, "cherry": 2160, "knives": 2161, "filming": 2162, "golden": 2163, "tag": 2164, "topping": 2165, "embedded": 2166, "dough": 2167, "moon": 2168, "stating": 2169, "don": 2170, "catchers": 2171, "finished": 2172, "opponent": 2173, "stationary": 2174, "faucet": 2175, "grasses": 2176, "scenic": 2177, "bells": 2178, "flamingos": 2179, "wife": 2180, "skates": 2181, "seven": 2182, "cane": 2183, "alarm": 2184, "ducklings": 2185, "frying": 2186, "snowsuit": 2187, "paneled": 2188, "dinning": 2189, "eagle": 2190, "finger": 2191, "metallic": 2192, "vegetation": 2193, "crosses": 2194, "selection": 2195, "local": 2196, "slowly": 2197, "unit": 2198, "facility": 2199, "sword": 2200, "daughter": 2201, "1": 2202, "sculptures": 2203, "tape": 2204, "commode": 2205, "tip": 2206, "parts": 2207, "galloping": 2208, "olives": 2209, "fuzzy": 2210, "yet": 2211, "cuddling": 2212, "sight": 2213, "pottery": 2214, "cd": 2215, "cloudless": 2216, "thick": 2217, "striking": 2218, "craft": 2219, "celery": 2220, "healthy": 2221, "series": 2222, "feathers": 2223, "excited": 2224, "parachute": 2225, "unoccupied": 2226, "santa": 2227, "cupboard": 2228, "perches": 2229, "nuts": 2230, "heard": 2231, "stance": 2232, "wrought": 2233, "observing": 2234, "gym": 2235, "sesame": 2236, "tents": 2237, "measuring": 2238, "ridden": 2239, "rocking": 2240, "pillars": 2241, "winding": 2242, "downtown": 2243, "vests": 2244, "let": 2245, "rodeo": 2246, "nightstand": 2247, "skinny": 2248, "dotted": 2249, "ten": 2250, "lounge": 2251, "deserts": 2252, "pairs": 2253, "deli": 2254, "gesturing": 2255, "deckered": 2256, "infant": 2257, "dashboard": 2258, "blurred": 2259, "combing": 2260, "private": 2261, "foliage": 2262, "evergreen": 2263, "bullet": 2264, "legged": 2265, "earphones": 2266, "themselves": 2267, "sticker": 2268, "flipping": 2269, "factory": 2270, "sleek": 2271, "print": 2272, "tying": 2273, "disk": 2274, "menu": 2275, "worn": 2276, "lodge": 2277, "class": 2278, "engaged": 2279, "swans": 2280, "pelicans": 2281, "cabinetry": 2282, "interstate": 2283, "faced": 2284, "nap": 2285, "fixed": 2286, "numbers": 2287, "progress": 2288, "halfway": 2289, "bundle": 2290, "letters": 2291, "pajamas": 2292, "paddock": 2293, "appliance": 2294, "pug": 2295, "cages": 2296, "wool": 2297, "icy": 2298, "dust": 2299, "try": 2300, "sells": 2301, "buttons": 2302, "ornaments": 2303, "protest": 2304, "pit": 2305, "diaper": 2306, "meadow": 2307, "autumn": 2308, "notes": 2309, "drivers": 2310, "mashed": 2311, "digging": 2312, "cucumbers": 2313, "vast": 2314, "quite": 2315, "draped": 2316, "read": 2317, "robot": 2318, "pushes": 2319, "stomach": 2320, "overcast": 2321, "bit": 2322, "arriving": 2323, "basin": 2324, "club": 2325, "cutter": 2326, "mantle": 2327, "incoming": 2328, "shaggy": 2329, "handed": 2330, "midst": 2331, "care": 2332, "clutter": 2333, "loaf": 2334, "stretch": 2335, "flowered": 2336, "saying": 2337, "penguin": 2338, "material": 2339, "stretched": 2340, "wiping": 2341, "intricate": 2342, "zone": 2343, "european": 2344, "mp3": 2345, "tethered": 2346, "thing": 2347, "crown": 2348, "trip": 2349, "cushions": 2350, "boot": 2351, "11": 2352, "rising": 2353, "strip": 2354, "vandalized": 2355, "pantry": 2356, "shadow": 2357, "scratching": 2358, "bald": 2359, "gravy": 2360, "bathing": 2361, "lifting": 2362, "console": 2363, "swan": 2364, "pedestal": 2365, "banner": 2366, "oval": 2367, "layered": 2368, "provide": 2369, "robe": 2370, "tire": 2371, "cord": 2372, "workstation": 2373, "puts": 2374, "led": 2375, "tulips": 2376, "drawers": 2377, "folding": 2378, "parmesan": 2379, "capped": 2380, "dancing": 2381, "markers": 2382, "leaned": 2383, "frozen": 2384, "weathered": 2385, "much": 2386, "exit": 2387, "spoons": 2388, "german": 2389, "races": 2390, "knee": 2391, "pads": 2392, "media": 2393, "bins": 2394, "quiche": 2395, "skyscrapers": 2396, "bidet": 2397, "towering": 2398, "directing": 2399, "somebody": 2400, "policeman": 2401, "distant": 2402, "bug": 2403, "tablecloth": 2404, "beginning": 2405, "barrel": 2406, "hovers": 2407, "serves": 2408, "stocked": 2409, "claim": 2410, "pad": 2411, "son": 2412, "cloud": 2413, "pine": 2414, "heart": 2415, "seating": 2416, "self": 2417, "torn": 2418, "shaking": 2419, "else": 2420, "rectangular": 2421, "waffle": 2422, "blocking": 2423, "pacifier": 2424, "dip": 2425, "competing": 2426, "junk": 2427, "wit": 2428, "whipped": 2429, "include": 2430, "buckets": 2431, "sniffs": 2432, "powdered": 2433, "father": 2434, "crossed": 2435, "overgrown": 2436, "flowing": 2437, "learn": 2438, "regular": 2439, "appear": 2440, "newly": 2441, "twilight": 2442, "tablet": 2443, "woolly": 2444, "levels": 2445, "designed": 2446, "runner": 2447, "omelet": 2448, "thru": 2449, "utility": 2450, "raw": 2451, "border": 2452, "kayaks": 2453, "mote": 2454, "lighthouse": 2455, "glide": 2456, "airlines": 2457, "wonderful": 2458, "animated": 2459, "serious": 2460, "start": 2461, "peach": 2462, "rounds": 2463, "flavors": 2464, "billboards": 2465, "navigating": 2466, "remodeled": 2467, "union": 2468, "blazer": 2469, "crooked": 2470, "skiier": 2471, "visitors": 2472, "share": 2473, "tender": 2474, "moment": 2475, "daytime": 2476, "mustache": 2477, "goalie": 2478, "logo": 2479, "cheesecake": 2480, "your": 2481, "brand": 2482, "settings": 2483, "celebrate": 2484, "parks": 2485, "homes": 2486, "ferris": 2487, "steer": 2488, "projection": 2489, "mac": 2490, "cruising": 2491, "link": 2492, "save": 2493, "frog": 2494, "sold": 2495, "backside": 2496, "headlights": 2497, "leave": 2498, "pull": 2499, "seem": 2500, "training": 2501, "attractive": 2502, "threw": 2503, "want": 2504, "boar": 2505, "varieties": 2506, "bad": 2507, "lemon": 2508, "garnish": 2509, "contemporary": 2510, "peoples": 2511, "trainer": 2512, "better": 2513, "embracing": 2514, "heavily": 2515, "decadent": 2516, "football": 2517, "cop": 2518, "architectural": 2519, "clown": 2520, "indian": 2521, "choppy": 2522, "plenty": 2523, "presentation": 2524, "husband": 2525, "dozen": 2526, "shavings": 2527, "exiting": 2528, "tshirt": 2529, "elder": 2530, "cloths": 2531, "indicates": 2532, "carousel": 2533, "religious": 2534, "amazed": 2535, "ina": 2536, "crib": 2537, "plumbing": 2538, "hockey": 2539, "poodle": 2540, "refridgerator": 2541, "tide": 2542, "transparent": 2543, "district": 2544, "walsk": 2545, "woven": 2546, "waving": 2547, "queen": 2548, "relaxes": 2549, "dragon": 2550, "dome": 2551, "limb": 2552, "captured": 2553, "dusty": 2554, "pressing": 2555, "smoothie": 2556, "sidecar": 2557, "athletes": 2558, "paddles": 2559, "th": 2560, "yummy": 2561, "privacy": 2562, "tips": 2563, "brief": 2564, "soaring": 2565, "couples": 2566, "nursing": 2567, "teenaged": 2568, "grizzly": 2569, "teammates": 2570, "dreary": 2571, "casual": 2572, "signage": 2573, "upward": 2574, "modified": 2575, "locked": 2576, "tier": 2577, "hours": 2578, "machines": 2579, "hollywood": 2580, "peers": 2581, "blueberry": 2582, "best": 2583, "cookie": 2584, "vines": 2585, "garlic": 2586, "ropes": 2587, "stranded": 2588, "coats": 2589, "opens": 2590, "sweatshirt": 2591, "scarf": 2592, "leaving": 2593, "barred": 2594, "mixture": 2595, "pillar": 2596, "fellow": 2597, "arches": 2598, "racer": 2599, "cel": 2600, "cafeteria": 2601, "opposing": 2602, "angles": 2603, "tanks": 2604, "starts": 2605, "salads": 2606, "cranes": 2607, "already": 2608, "online": 2609, "amidst": 2610, "napkins": 2611, "nature": 2612, "shed": 2613, "lambs": 2614, "stormy": 2615, "complex": 2616, "glitter": 2617, "windsurfing": 2618, "taxi": 2619, "cab": 2620, "mountainside": 2621, "rise": 2622, "arena": 2623, "herbs": 2624, "forehand": 2625, "alcohol": 2626, "lean": 2627, "hundreds": 2628, "ponies": 2629, "mail": 2630, "ambulance": 2631, "styled": 2632, "flips": 2633, "organic": 2634, "partner": 2635, "goggles": 2636, "puppies": 2637, "mountainous": 2638, "biscuit": 2639, "meats": 2640, "halfpipe": 2641, "than": 2642, "band": 2643, "iwth": 2644, "bleachers": 2645, "gigantic": 2646, "crafts": 2647, "dairy": 2648, "compartment": 2649, "need": 2650, "bicyclist": 2651, "africa": 2652, "combination": 2653, "pigeon": 2654, "buliding": 2655, "idle": 2656, "maker": 2657, "gifts": 2658, "san": 2659, "francisco": 2660, "blind": 2661, "knocked": 2662, "conversing": 2663, "comfortably": 2664, "cozy": 2665, "beanie": 2666, "household": 2667, "accessories": 2668, "younger": 2669, "sunshine": 2670, "horned": 2671, "amp": 2672, "flanked": 2673, "barbeque": 2674, "scenery": 2675, "picked": 2676, "balloon": 2677, "seas": 2678, "gadget": 2679, "barefoot": 2680, "entree": 2681, "memorabilia": 2682, "tucked": 2683, "peak": 2684, "grasslands": 2685, "historical": 2686, "sizes": 2687, "money": 2688, "positioned": 2689, "dots": 2690, "pumpkin": 2691, "basil": 2692, "sandals": 2693, "nine": 2694, "finely": 2695, "entry": 2696, "windshield": 2697, "rises": 2698, "pineapple": 2699, "hoodie": 2700, "styrofoam": 2701, "lettering": 2702, "system": 2703, "spreading": 2704, "heels": 2705, "muscular": 2706, "gazing": 2707, "cushion": 2708, "reveals": 2709, "recliner": 2710, "microwaves": 2711, "claw": 2712, "peel": 2713, "festival": 2714, "creek": 2715, "towed": 2716, "know": 2717, "stockings": 2718, "victorian": 2719, "speaker": 2720, "england": 2721, "scape": 2722, "stems": 2723, "pointed": 2724, "slides": 2725, "passed": 2726, "dipping": 2727, "wrestling": 2728, "starting": 2729, "artificial": 2730, "kayak": 2731, "upright": 2732, "socks": 2733, "patrons": 2734, "demolished": 2735, "appearing": 2736, "numerous": 2737, "blood": 2738, "geese": 2739, "necklace": 2740, "panel": 2741, "extremely": 2742, "attention": 2743, "then": 2744, "forefront": 2745, "am": 2746, "nypd": 2747, "drawer": 2748, "photographing": 2749, "hello": 2750, "moustache": 2751, "edges": 2752, "protective": 2753, "pepsi": 2754, "dad": 2755, "real": 2756, "darth": 2757, "vader": 2758, "carefully": 2759, "letting": 2760, "pop": 2761, "touches": 2762, "angel": 2763, "hiking": 2764, "camping": 2765, "spotted": 2766, "multitude": 2767, "6": 2768, "graphic": 2769, "character": 2770, "main": 2771, "smartphone": 2772, "sleeve": 2773, "needs": 2774, "buffet": 2775, "chrome": 2776, "noodle": 2777, "picking": 2778, "easy": 2779, "now": 2780, "peeks": 2781, "junction": 2782, "grand": 2783, "figure": 2784, "herds": 2785, "should": 2786, "30": 2787, "fours": 2788, "newborn": 2789, "handlebars": 2790, "zipper": 2791, "marker": 2792, "graduates": 2793, "guiding": 2794, "wheeled": 2795, "glassware": 2796, "hound": 2797, "barely": 2798, "fill": 2799, "removed": 2800, "kettle": 2801, "stew": 2802, "forested": 2803, "characters": 2804, "environment": 2805, "speeds": 2806, "stores": 2807, "wander": 2808, "similar": 2809, "float": 2810, "state": 2811, "coke": 2812, "gives": 2813, "ridding": 2814, "wrapping": 2815, "hauling": 2816, "backed": 2817, "bookshelves": 2818, "upon": 2819, "separate": 2820, "expression": 2821, "viewing": 2822, "naps": 2823, "g": 2824, "grapefruit": 2825, "plethora": 2826, "toiletries": 2827, "marked": 2828, "commuters": 2829, "daily": 2830, "otherwise": 2831, "grafitti": 2832, "beers": 2833, "canyon": 2834, "strapped": 2835, "ad": 2836, "radishes": 2837, "barrels": 2838, "deer": 2839, "student": 2840, "sided": 2841, "protection": 2842, "paperwork": 2843, "crest": 2844, "cobbled": 2845, "windmills": 2846, "navy": 2847, "smack": 2848, "studio": 2849, "necks": 2850, "posts": 2851, "slaw": 2852, "follows": 2853, "snuggling": 2854, "everywhere": 2855, "necktie": 2856, "bikini": 2857, "certain": 2858, "situated": 2859, "bumper": 2860, "iced": 2861, "cheeses": 2862, "10": 2863, "farmers": 2864, "porcelain": 2865, "prey": 2866, "cast": 2867, "fair": 2868, "waling": 2869, "pinned": 2870, "rod": 2871, "doubles": 2872, "attachment": 2873, "buds": 2874, "broadway": 2875, "fur": 2876, "page": 2877, "athletic": 2878, "peace": 2879, "crashes": 2880, "recording": 2881, "thames": 2882, "spices": 2883, "stretches": 2884, "grandfather": 2885, "holes": 2886, "profile": 2887, "fluorescent": 2888, "vane": 2889, "boating": 2890, "diamond": 2891, "formally": 2892, "relax": 2893, "theme": 2894, "angled": 2895, "transported": 2896, "fort": 2897, "raising": 2898, "blueberries": 2899, "lab": 2900, "give": 2901, "spring": 2902, "farmer": 2903, "keeping": 2904, "perform": 2905, "carried": 2906, "motorists": 2907, "partly": 2908, "bowling": 2909, "picks": 2910, "plunger": 2911, "airways": 2912, "equipped": 2913, "projected": 2914, "meals": 2915, "helmeted": 2916, "prairie": 2917, "banquet": 2918, "beyond": 2919, "bricks": 2920, "starbucks": 2921, "knight": 2922, "balance": 2923, "sat": 2924, "portable": 2925, "keys": 2926, "drawing": 2927, "chandelier": 2928, "raquet": 2929, "walked": 2930, "enthusiast": 2931, "monument": 2932, "treats": 2933, "gravel": 2934, "horn": 2935, "teenage": 2936, "hearty": 2937, "ox": 2938, "musical": 2939, "lifts": 2940, "grab": 2941, "marks": 2942, "lonely": 2943, "identical": 2944, "splashing": 2945, "pay": 2946, "cooling": 2947, "greenery": 2948, "wooly": 2949, "carriages": 2950, "lifted": 2951, "para": 2952, "flops": 2953, "oxygen": 2954, "surrounds": 2955, "motorbikes": 2956, "steamed": 2957, "glides": 2958, "lunging": 2959, "stump": 2960, "mushroom": 2961, "teen": 2962, "great": 2963, "blends": 2964, "cracked": 2965, "warm": 2966, "pours": 2967, "glow": 2968, "roundabout": 2969, "package": 2970, "valve": 2971, "stepping": 2972, "tube": 2973, "taxis": 2974, "rabbit": 2975, "halves": 2976, "submerged": 2977, "latest": 2978, "barge": 2979, "bookcase": 2980, "cookware": 2981, "happening": 2982, "theirs": 2983, "tugboat": 2984, "rundown": 2985, "stripped": 2986, "alien": 2987, "fist": 2988, "fabrics": 2989, "hides": 2990, "mexican": 2991, "raining": 2992, "would": 2993, "hut": 2994, "goal": 2995, "liquor": 2996, "crouched": 2997, "wheeler": 2998, "career": 2999, "highlight": 3000, "comb": 3001, "businesses": 3002, "atv": 3003, "trapped": 3004, "cyclists": 3005, "architecture": 3006, "belongings": 3007, "laundry": 3008, "guinness": 3009, "product": 3010, "retriever": 3011, "curly": 3012, "puddle": 3013, "random": 3014, "bobble": 3015, "diced": 3016, "futon": 3017, "extending": 3018, "standard": 3019, "mosquito": 3020, "netting": 3021, "lizard": 3022, "oversized": 3023, "supports": 3024, "11th": 3025, "shaving": 3026, "adjacent": 3027, "will": 3028, "makeup": 3029, "patterns": 3030, "coconut": 3031, "tabletop": 3032, "pliers": 3033, "dachshund": 3034, "stamps": 3035, "handled": 3036, "stamp": 3037, "carry": 3038, "pita": 3039, "vibrant": 3040, "uk": 3041, "grow": 3042, "garb": 3043, "cuckoo": 3044, "teens": 3045, "removes": 3046, "elders": 3047, "tagging": 3048, "dorm": 3049, "prep": 3050, "melons": 3051, "bakery": 3052, "keychain": 3053, "jumped": 3054, "efficiency": 3055, "macaroni": 3056, "fir": 3057, "emo": 3058, "hipster": 3059, "wallet": 3060, "doesn": 3061, "roadside": 3062, "eatery": 3063, "goose": 3064, "michigan": 3065, "heron": 3066, "figurines": 3067, "charger": 3068, "chunks": 3069, "fog": 3070, "underground": 3071, "kiss": 3072, "ultimate": 3073, "wrangling": 3074, "untouched": 3075, "wetsuits": 3076, "maneuvering": 3077, "sprayed": 3078, "current": 3079, "bovine": 3080, "burgers": 3081, "foamy": 3082, "enclose": 3083, "puffy": 3084, "idly": 3085, "lever": 3086, "sandwhich": 3087, "wedges": 3088, "kiwi": 3089, "might": 3090, "domestic": 3091, "resemble": 3092, "boardwalk": 3093, "blooming": 3094, "hamster": 3095, "shuttle": 3096, "sampling": 3097, "cant": 3098, "boxing": 3099, "specials": 3100, "occasion": 3101, "blvd": 3102, "sideburns": 3103, "grimaces": 3104, "garnished": 3105, "avocado": 3106, "clutched": 3107, "bulbs": 3108, "marketplace": 3109, "wakeboarder": 3110, "spears": 3111, "entire": 3112, "pencil": 3113, "freeze": 3114, "attending": 3115, "probably": 3116, "lavish": 3117, "estate": 3118, "firetruck": 3119, "awkward": 3120, "squirrel": 3121, "lecture": 3122, "theater": 3123, "bushel": 3124, "grasps": 3125, "dumping": 3126, "autograph": 3127, "underpass": 3128, "ruler": 3129, "videogame": 3130, "lighter": 3131, "basic": 3132, "watched": 3133, "patiently": 3134, "mens": 3135, "solider": 3136, "cheek": 3137, "tending": 3138, "fisherman": 3139, "needle": 3140, "saddles": 3141, "sled": 3142, "arrives": 3143, "ditch": 3144, "mats": 3145, "oncoming": 3146, "strike": 3147, "competitors": 3148, "wok": 3149, "burner": 3150, "wax": 3151, "yawning": 3152, "stain": 3153, "massive": 3154, "congregation": 3155, "crock": 3156, "oil": 3157, "sigh": 3158, "rains": 3159, "organizer": 3160, "hoagie": 3161, "relish": 3162, "waterfront": 3163, "bl": 3164, "spire": 3165, "mark": 3166, "fishermen": 3167, "packs": 3168, "snuggled": 3169, "baseline": 3170, "detour": 3171, "torso": 3172, "cinnamon": 3173, "signposts": 3174, "region": 3175, "unattended": 3176, "kitchens": 3177, "blossoms": 3178, "steering": 3179, "filth": 3180, "hurdle": 3181, "expired": 3182, "tell": 3183, "armchair": 3184, "commodes": 3185, "fitted": 3186, "galley": 3187, "bale": 3188, "grove": 3189, "planters": 3190, "numeral": 3191, "releasing": 3192, "geared": 3193, "accompanied": 3194, "gazelles": 3195, "rapids": 3196, "parrots": 3197, "allow": 3198, "practice": 3199, "pretend": 3200, "check": 3201, "supply": 3202, "chased": 3203, "fedex": 3204, "overturned": 3205, "array": 3206, "procession": 3207, "snuggle": 3208, "fetching": 3209, "lip": 3210, "balances": 3211, "cops": 3212, "freestanding": 3213, "buy": 3214, "futuristic": 3215, "toss": 3216, "alert": 3217, "llama": 3218, "shares": 3219, "favorite": 3220, "custom": 3221, "filling": 3222, "telephones": 3223, "rooster": 3224, "pawing": 3225, "penguins": 3226, "veggie": 3227, "proceeding": 3228, "pace": 3229, "photographers": 3230, "sunrise": 3231, "pork": 3232, "cilantro": 3233, "recently": 3234, "why": 3235, "fie": 3236, "deserted": 3237, "amtrak": 3238, "peddling": 3239, "brocolli": 3240, "cuddled": 3241, "whit": 3242, "lava": 3243, "dangling": 3244, "batsman": 3245, "shoulders": 3246, "descends": 3247, "controlling": 3248, "stoplights": 3249, "shapes": 3250, "tortilla": 3251, "pets": 3252, "cube": 3253, "focused": 3254, "shaved": 3255, "bet": 3256, "teakettle": 3257, "wares": 3258, "jeep": 3259, "free": 3260, "halloween": 3261, "skyscraper": 3262, "simulator": 3263, "smartphones": 3264, "affixed": 3265, "carpets": 3266, "antennae": 3267, "sandles": 3268, "tint": 3269, "upscale": 3270, "uniformed": 3271, "lotion": 3272, "completely": 3273, "cathedral": 3274, "interact": 3275, "bill": 3276, "gras": 3277, "text": 3278, "mop": 3279, "wheelie": 3280, "sanding": 3281, "toddlers": 3282, "coupe": 3283, "tattoos": 3284, "primitive": 3285, "assembly": 3286, "strings": 3287, "dim": 3288, "muddied": 3289, "squat": 3290, "highchair": 3291, "bits": 3292, "manner": 3293, "peculiar": 3294, "remodel": 3295, "warming": 3296, "peeler": 3297, "mercedes": 3298, "mustached": 3299, "intense": 3300, "kicks": 3301, "clover": 3302, "huddle": 3303, "folks": 3304, "frothy": 3305, "peering": 3306, "bulldog": 3307, "wipes": 3308, "ascending": 3309, "celebration": 3310, "fed": 3311, "foam": 3312, "locations": 3313, "lanes": 3314, "discussion": 3315, "pirate": 3316, "mansion": 3317, "hitter": 3318, "nestled": 3319, "cylindrical": 3320, "sheer": 3321, "prince": 3322, "cleaning": 3323, "dig": 3324, "bearing": 3325, "tinted": 3326, "nighttime": 3327, "valves": 3328, "vans": 3329, "sauces": 3330, "grounded": 3331, "competitive": 3332, "vanilla": 3333, "splash": 3334, "cutout": 3335, "firemen": 3336, "whine": 3337, "arrive": 3338, "activities": 3339, "curling": 3340, "unbaked": 3341, "caution": 3342, "sewing": 3343, "server": 3344, "stored": 3345, "shark": 3346, "barren": 3347, "chubby": 3348, "chilies": 3349, "motocycle": 3350, "meatball": 3351, "installation": 3352, "chickens": 3353, "italian": 3354, "pretending": 3355, "rolled": 3356, "push": 3357, "stations": 3358, "bred": 3359, "paying": 3360, "arid": 3361, "dvd": 3362, "gliding": 3363, "studying": 3364, "bends": 3365, "cameras": 3366, "quaint": 3367, "marsh": 3368, "wrappers": 3369, "letter": 3370, "retail": 3371, "markings": 3372, "51": 3373, "crosstown": 3374, "refueling": 3375, "batters": 3376, "seeds": 3377, "mantel": 3378, "tether": 3379, "species": 3380, "caps": 3381, "pierced": 3382, "ale": 3383, "daughters": 3384, "greenville": 3385, "cemetery": 3386, "pant": 3387, "liner": 3388, "lightning": 3389, "hoses": 3390, "radiator": 3391, "youngest": 3392, "instruments": 3393, "upper": 3394, "portion": 3395, "crystal": 3396, "20": 3397, "miles": 3398, "toliet": 3399, "23": 3400, "parrot": 3401, "gamer": 3402, "seashells": 3403, "freeway": 3404, "magenta": 3405, "oceans": 3406, "carving": 3407, "noses": 3408, "begin": 3409, "daylight": 3410, "university": 3411, "sing": 3412, "actors": 3413, "struck": 3414, "price": 3415, "indolently": 3416, "thumb": 3417, "jam": 3418, "kale": 3419, "even": 3420, "vulture": 3421, "columns": 3422, "screaming": 3423, "confused": 3424, "broccolli": 3425, "article": 3426, "cabana": 3427, "cobblestones": 3428, "stools": 3429, "rodent": 3430, "calico": 3431, "rushing": 3432, "junky": 3433, "croissant": 3434, "trolly": 3435, "summer": 3436, "lamppost": 3437, "snowing": 3438, "doubled": 3439, "poolside": 3440, "smooth": 3441, "wallpaper": 3442, "smashed": 3443, "watermelon": 3444, "concealed": 3445, "olive": 3446, "connecting": 3447, "tiger": 3448, "straps": 3449, "baker": 3450, "soaked": 3451, "luncheon": 3452, "buggy": 3453, "peeling": 3454, "forrest": 3455, "aisle": 3456, "occupied": 3457, "travelers": 3458, "beetle": 3459, "pecking": 3460, "today": 3461, "uncut": 3462, "dance": 3463, "sunlit": 3464, "eagerly": 3465, "renovated": 3466, "windowsill": 3467, "hairy": 3468, "related": 3469, "individuals": 3470, "steaming": 3471, "skills": 3472, "reflects": 3473, "mint": 3474, "crouching": 3475, "melted": 3476, "stoves": 3477, "roofs": 3478, "sipping": 3479, "tongs": 3480, "basement": 3481, "tracking": 3482, "inviting": 3483, "explore": 3484, "western": 3485, "alaska": 3486, "think": 3487, "contain": 3488, "dvds": 3489, "thermometer": 3490, "menus": 3491, "ups": 3492, "oblong": 3493, "streetlight": 3494, "tool": 3495, "merging": 3496, "dine": 3497, "dilapidated": 3498, "me": 3499, "access": 3500, "came": 3501, "nike": 3502, "facial": 3503, "present": 3504, "cigarettes": 3505, "chews": 3506, "hairbrush": 3507, "hooded": 3508, "sidelines": 3509, "globe": 3510, "structures": 3511, "patchwork": 3512, "wrap": 3513, "sparkler": 3514, "costumed": 3515, "beaten": 3516, "arugula": 3517, "thumbs": 3518, "blade": 3519, "bushels": 3520, "elbow": 3521, "dying": 3522, "maneuvers": 3523, "fondant": 3524, "dental": 3525, "floss": 3526, "brownie": 3527, "scrolls": 3528, "clusters": 3529, "poking": 3530, "daisies": 3531, "add": 3532, "transport": 3533, "butterflys": 3534, "cramped": 3535, "quarters": 3536, "congregate": 3537, "twp": 3538, "docks": 3539, "payphone": 3540, "steaks": 3541, "bubble": 3542, "tagged": 3543, "kitchenette": 3544, "camp": 3545, "wades": 3546, "tangle": 3547, "depict": 3548, "isn": 3549, "least": 3550, "twig": 3551, "sock": 3552, "label": 3553, "file": 3554, "shrub": 3555, "spins": 3556, "motes": 3557, "member": 3558, "unload": 3559, "c": 3560, "parasols": 3561, "raises": 3562, "yak": 3563, "creme": 3564, "plowing": 3565, "oxen": 3566, "halved": 3567, "squash": 3568, "days": 3569, "reclines": 3570, "rested": 3571, "graveyard": 3572, "spilled": 3573, "waterfall": 3574, "canes": 3575, "straightening": 3576, "chin": 3577, "bedspread": 3578, "mice": 3579, "banners": 3580, "snowshoes": 3581, "prop": 3582, "clearly": 3583, "picker": 3584, "incredible": 3585, "gazelle": 3586, "minimal": 3587, "renovation": 3588, "kiosk": 3589, "spreads": 3590, "litter": 3591, "never": 3592, "consumption": 3593, "captivity": 3594, "transporting": 3595, "exhaust": 3596, "crackers": 3597, "coated": 3598, "parchment": 3599, "coral": 3600, "bra": 3601, "weathervane": 3602, "relics": 3603, "maintenance": 3604, "elvis": 3605, "dozens": 3606, "courts": 3607, "revealing": 3608, "reflective": 3609, "motel": 3610, "casserole": 3611, "humming": 3612, "retro": 3613, "turquoise": 3614, "roam": 3615, "freely": 3616, "wanting": 3617, "speak": 3618, "job": 3619, "live": 3620, "compete": 3621, "lo": 3622, "huddled": 3623, "parallel": 3624, "upwards": 3625, "cotton": 3626, "elegantly": 3627, "flatbed": 3628, "poured": 3629, "billboard": 3630, "overlook": 3631, "listing": 3632, "liberty": 3633, "reach": 3634, "ally": 3635, "dumpster": 3636, "magnifying": 3637, "bales": 3638, "kits": 3639, "fading": 3640, "protruding": 3641, "cyclist": 3642, "biking": 3643, "convention": 3644, "sorry": 3645, "baguette": 3646, "rose": 3647, "gun": 3648, "aimed": 3649, "did": 3650, "observe": 3651, "featured": 3652, "positions": 3653, "purses": 3654, "greeting": 3655, "aboard": 3656, "department": 3657, "adorn": 3658, "jug": 3659, "milkshake": 3660, "tiling": 3661, "coca": 3662, "cola": 3663, "yarn": 3664, "grounds": 3665, "provides": 3666, "popular": 3667, "lounges": 3668, "assist": 3669, "performers": 3670, "nuzzling": 3671, "bud": 3672, "downward": 3673, "gothic": 3674, "oar": 3675, "pineapples": 3676, "hug": 3677, "eleven": 3678, "interested": 3679, "shells": 3680, "poised": 3681, "kayaking": 3682, "works": 3683, "iin": 3684, "checker": 3685, "flavor": 3686, "any": 3687, "wants": 3688, "hazard": 3689, "wildebeest": 3690, "copper": 3691, "hear": 3692, "homeless": 3693, "lidded": 3694, "vein": 3695, "trotting": 3696, "trade": 3697, "surprised": 3698, "glimpse": 3699, "reclining": 3700, "debris": 3701, "streetlights": 3702, "lavatory": 3703, "curbside": 3704, "bib": 3705, "snout": 3706, "makeshift": 3707, "tight": 3708, "rim": 3709, "perfect": 3710, "livestock": 3711, "limes": 3712, "afro": 3713, "future": 3714, "tires": 3715, "sandwhiches": 3716, "gorilla": 3717, "caged": 3718, "pooh": 3719, "dribbling": 3720, "vessel": 3721, "trey": 3722, "carrier": 3723, "jail": 3724, "swung": 3725, "inlet": 3726, "practices": 3727, "railings": 3728, "fragile": 3729, "sporting": 3730, "glaring": 3731, "got": 3732, "dolphin": 3733, "cucumber": 3734, "arched": 3735, "searching": 3736, "shadowy": 3737, "vessels": 3738, "windy": 3739, "pepperonis": 3740, "microphones": 3741, "wireless": 3742, "ballpark": 3743, "mix": 3744, "drawings": 3745, "puzzle": 3746, "chop": 3747, "teriyaki": 3748, "balanced": 3749, "stereo": 3750, "windowed": 3751, "supermarket": 3752, "ona": 3753, "native": 3754, "barbecue": 3755, "graffitti": 3756, "adjust": 3757, "done": 3758, "super": 3759, "smothered": 3760, "countries": 3761, "filed": 3762, "bee": 3763, "shabby": 3764, "toilette": 3765, "lens": 3766, "whose": 3767, "mates": 3768, "dugout": 3769, "50": 3770, "magazines": 3771, "leaps": 3772, "mason": 3773, "orchid": 3774, "burrito": 3775, "brass": 3776, "milking": 3777, "mixing": 3778, "examines": 3779, "necessary": 3780, "foal": 3781, "peacefully": 3782, "names": 3783, "blackboard": 3784, "sectional": 3785, "marshy": 3786, "pies": 3787, "cracker": 3788, "differently": 3789, "linens": 3790, "slippers": 3791, "turf": 3792, "lookign": 3793, "sure": 3794, "safe": 3795, "residence": 3796, "solar": 3797, "downs": 3798, "intersecting": 3799, "corridor": 3800, "hygiene": 3801, "because": 3802, "concerned": 3803, "hybrid": 3804, "snap": 3805, "depot": 3806, "calculator": 3807, "bites": 3808, "lost": 3809, "monk": 3810, "wade": 3811, "circus": 3812, "triangular": 3813, "disorganized": 3814, "pointy": 3815, "rollers": 3816, "manhole": 3817, "projector": 3818, "domed": 3819, "thomas": 3820, "ethnic": 3821, "steal": 3822, "collect": 3823, "windsurfers": 3824, "juicing": 3825, "mickey": 3826, "volley": 3827, "photography": 3828, "central": 3829, "yogurt": 3830, "ships": 3831, "fee": 3832, "scrambled": 3833, "jackson": 3834, "merry": 3835, "hors": 3836, "canal": 3837, "pc": 3838, "obese": 3839, "owl": 3840, "broth": 3841, "shepherd": 3842, "neglected": 3843, "varying": 3844, "calves": 3845, "standup": 3846, "driven": 3847, "printed": 3848, "jockeys": 3849, "dries": 3850, "attendant": 3851, "changing": 3852, "auditorium": 3853, "leopard": 3854, "hoping": 3855, "grouped": 3856, "anime": 3857, "packing": 3858, "involved": 3859, "camouflaged": 3860, "hammer": 3861, "rights": 3862, "manual": 3863, "plush": 3864, "sunflowers": 3865, "herded": 3866, "mario": 3867, "automatic": 3868, "compute": 3869, "dug": 3870, "find": 3871, "memorial": 3872, "plaque": 3873, "volleyball": 3874, "cooler": 3875, "amazon": 3876, "dc": 3877, "feast": 3878, "unripe": 3879, "booze": 3880, "visit": 3881, "accents": 3882, "darkened": 3883, "pushed": 3884, "contest": 3885, "clothesline": 3886, "juicer": 3887, "typical": 3888, "spanning": 3889, "storefronts": 3890, "leashes": 3891, "florets": 3892, "ladybug": 3893, "mosaic": 3894, "pancake": 3895, "filtering": 3896, "source": 3897, "fourth": 3898, "trashcan": 3899, "mouths": 3900, "headset": 3901, "moss": 3902, "kathy": 3903, "directs": 3904, "googly": 3905, "chuck": 3906, "rounding": 3907, "outhouse": 3908, "rushes": 3909, "ovens": 3910, "drapes": 3911, "propellor": 3912, "mit": 3913, "chases": 3914, "bronze": 3915, "passanger": 3916, "ceilings": 3917, "mascot": 3918, "lama": 3919, "battery": 3920, "petals": 3921, "unknown": 3922}