alakazamworld/doom-dungeon-hg

Model Card for DOOM Dungeon HG (doom_v55_hg)

The EDM diffusion UNet backbone and the world-model-as-denoiser frame are DIAMOND's (Alonso et al., NeurIPS 2024), unchanged. DOOM Dungeon HG changes two things, and the rest of this card follows from them: the conditioning interface (a rendered 12-channel state tensor injected every frame) and the training recipe (two-phase Self Forcing). It is the world model the Dream Robot Gauntlet robot controllers were evolved inside.

A 69.8M-parameter action-conditioned diffusion world model of a DOOM-style dungeon, exported to ONNX for interactive serving. It renders 64x64 first-person gameplay one frame at a time, conditioned on the last 4 frames, the last 4 discrete actions, and (unlike its DIAMOND base) a 12-channel rendered state tensor built fresh every frame from exact game state: raycast depth, top-down walls and pose, entity layout, and per-ray surface identity (wall/floor/ceiling texture ids and heights). The model never has to infer where it is from its own pixels, and at the interactive 1-step setting each new frame costs exactly one denoiser forward pass.

Generated frames (left, labeled "THE DREAM") next to the true map and pose trail (right) at t = 0/16/32/48/64/79 of one driven rollout. The first-person pixels are entirely model-generated; the map is the exact game state the conditioning channels are built from.

Model Details

Model Description

DOOM Dungeon HG is a DIAMOND-family EDM denoiser trained on a DOOM Health Gathering ("HG") dungeon domain: stone corridors, nukage, medkits, and enemy sprites. Version 5.5 of the family added the 12-channel conditioning layout: 7 base channels (depth disparity, top-down walls and entities, FOV entity columns, lifecycle) plus 5 per-ray surface channels (wall texture, floor height, floor texture, ceiling height, ceiling texture). That layout lets a host edit geometry and materials live and have the dream follow.

The published checkpoint is the Self-Forcing lineage of that model (checkpoint directory v6sf_phase2_ep26 in the archive repo): Self Forcing per arXiv:2506.08009 trains the denoiser on its own autoregressive rollouts (stochastic gradient truncation, per-step EDM loss on the discrete inference schedule), followed by a phase-2 fine-tune; per the training configs, phase 2 of this program adds a video-level adversarial (R3GAN-style) critic to break L2 regression-to-mean blur. A serving-relevant consequence of the Self-Forcing recipe: conditioning noise sigma_cond is fixed at 0.5 at inference (raised from 0.005) to force reliance on the conditioning channels and damp autoregressive momentum. Inference must use 0.5 to match training.

• Developed by: Alakazam (alakazam.gg)
• Model type: Action-conditioned diffusion world model (interactive video generation), EDM-preconditioned UNet denoiser, ONNX export
• License: CC BY-NC-SA 4.0 (consistent with the Dream Robot Gauntlet release cards, which inherit their license from these checkpoints)
• Architecture family: DIAMOND (Alonso et al., 2024), v5.5 configuration with Self-Forcing fine-tune

Model Sources

• Canonical serving copy: this repository (alakazamworld/doom-dungeon-hg)
• Checkpoint archive (byte-identical mirror + siblings): Karajan42/doom-dungeon-55, path v6sf_phase2_ep26/
• Reference environment (python, byte-identical to the file used in the robot campaign): Karajan42/dream-robot-gauntlet-policies env/doom_env_v55.py
• Robot campaign data + lab notebook: Karajan42/dream-robot-gauntlet-data
• Earlier family member (v5.3, with its own card): Karajan42/diamond-doom-hg-v5.3

Checkpoint inventory

Files in this repository:

File	Size (bytes)	What it is
denoiser.onnx	279,689,999	The world model. fp32, ONNX opset 17, exported by PyTorch 2.9.0. 69,773,002 parameters. Static batch dimension of 1.
init_state.json	886,772	Warm-start state: T=4, C=3, H=64, W=64, a 12x64x64 obs_buffer (4 RGB frames, values in [-1, 1]) and a length-4 act_buffer.
model_meta.json	639	Authoritative serving metadata: action count, conditioning channel names, sampler settings. Hosts should read sampler settings from here, not hardcode them.
media/	~5.5 MB	Card media (contact sheet, rollout GIFs, robot figure). Not needed for serving.

Checksums (SHA-256):

denoiser.onnx    62417a400befccde771317c4e8c9788ade237b6ba8f8840dccb8301af4b3cc37
init_state.json  066162c3e96f49232c59ea35a6455854c5674acbf80c55190ee9dfaa49370500
model_meta.json  2fa60c8d00bc0a2c00ec8b2c87b1c1efbfab9ec87e200b190f46cafd3a78e941

denoiser.onnx is byte-identical (same SHA-256) to v6sf_phase2_ep26/denoiser.onnx in the doom-dungeon-55 archive, which additionally holds: a fp16 denoiser_wonnx.onnx export of this checkpoint (currently non-functional under strict-fp16 WebGPU validation; the production browser player uses the fp32 file), a root-level denoiser.onnx with identical metadata but different bytes (relationship to this checkpoint unverified; treat this repository as canonical), a phase-1 sibling (v6sf/), a torch training checkpoint (v6_6_pm1_ep150/agent.pt, different sub-lineage), and an optional fp16 display upscaler (v6up/temporal_sr.onnx, [N, 30, 64, 64] -> [N, 3, 256, 256]).

Architecture: DIAMOND, with two deviations

This is not a new generative architecture, and this card does not claim one: the backbone is DIAMOND's, unchanged. Where the model deviates from the paper is the conditioning interface and the training recipe; the serving contract below rests on those two deviations.

Shared with DIAMOND

The world-model-as-denoiser frame and the EDM-preconditioned UNet backbone come directly from DIAMOND (Alonso et al., 2024, arXiv:2405.12399): next-frame prediction as denoising, AR(4) frame conditioning stacked channel-wise, discrete-action embeddings, and GameNGen-style noise augmentation over the context frames.

Backbone spec, verified against the exported graph and the training config:

Component	Value
Backbone	DIAMOND EDM-preconditioned UNet (denoiser.inner_model.unet)
Parameters	69,773,002 (fp32)
UNet	depths [2, 2, 2, 2], channels [128, 256, 256, 512], self-attention at the deepest scale only
Conditioning embed dim	256
Frame conditioning	AR(4): last 4 RGB frames, stacked channel-wise into a 12x64x64 tensor
Action conditioning	last 4 discrete action ids (embedding), current action last
Noise augmentation	GameNGen-style buckets, 10 buckets; bucket 5 at inference
Training-side EDM constants	sigma_data 0.5, sigma_offset_noise 0.3, noised previous obs

The exported graph contains no token-routing modules: this is the plain v5.5 conditioning pathway, not the later token-routed v6.6 variant. One family-side addition rides on the stock UNet: entity-bias attention (guided attention toward enemy/item FOV columns, enabled since v5.4.3).

Deviation 1: a state-grounded conditioning interface

DIAMOND's denoiser sees past frames and action ids; everything else about the world is implicit, inferred from pixels. That is why classic world-model rollouts can drift: the model must keep hallucinating a world consistent with its own outputs. This model additionally injects a rendered 12-channel spatial state tensor every frame, built by the host from exact game state (a 64-ray camera-plane raycast plus top-down maps at the current pose). That turns the denoiser into a state-grounded neural renderer: geometry cannot drift, because geometry is an input. Pose, occupancy, entity layout, surface heights, and surface identity are supplied fresh every step; the model's learned job is to render that state, not to remember it.

The verified per-channel semantics (from doom_env_v55.py, the reference implementation the robot campaign ran) are in the serving-contract table below. Grouped: channels 0/1/8/10 are exact geometry and pose, channels 2-6 are entity state, and channels 7/9/11 are per-ray surface identity: integer texture ids for the wall each ray hits and the floor/ceiling of the last open cell along it.

Because surface identity is a conditioning input, appearance is an inference-time parameter: swap the texture ids in the host's maps and the model re-skins walls, floors, and ceilings on the next frame, with no retraining. The production client's id vocabulary names eight materials per surface class (walls: stone, metal, marble, wood, tech, flesh, rocky, zimmer; floors: tile, dirt, nukage, lava, water, grass, concrete, slime; ceilings: default, sky, wood, tech-lights, concrete, dirt, marble, metal; id -1 = no surface). The ids and the live-edit path are code-verified; how faithfully each individual id renders depends on its training coverage, which is not enumerable from the shipped artifacts.

Deviation 2: a two-phase Self Forcing recipe

The second deviation post-dates DIAMOND: Self Forcing (arXiv:2506.08009) trains the denoiser on its own autoregressive rollouts instead of teacher-forced ground truth. Phase 1 runs stochastic gradient truncation with a per-step EDM loss on the discrete inference schedule; phase 2 (per the training configs) adds a video-level adversarial critic to break L2 regression-to-mean blur. This checkpoint is phase-2 epoch 26.

Two serving consequences follow. Single-step sampling becomes a stable operating point (the sigma ladder [5.0, 0.0] collapses to exactly one model forward per frame), and conditioning noise sigma_cond is fixed at 0.5, raised from 0.005 to force reliance on the state channels and damp autoregressive momentum during rollouts; inference must use 0.5 to match training. The measured 145.1 ms/frame laptop-CPU figure below rides on this recipe.

Why it matters

These two deviations (geometry as an input, rollouts as the training distribution) are what made a 70M model stable enough to be used as a training environment: the ground-truth-free dream-eyes policies were evolved entirely inside it and transferred to rigid-body physics (policies, data).

The serving contract

Everything in this section is verified directly against denoiser.onnx (graph introspection), model_meta.json, and the reference environment doom_env_v55.py.

ONNX graph I/O

One call = one denoising evaluation. All shapes are static; batch is fixed at 1.

Input	Shape	Dtype	Contents
noisy_next_obs	[1, 3, 64, 64]	float32	Current noisy estimate of the next frame (see sampler below)
sigma	[1]	float32	Current sigma of the sampler step
sigma_cond	[1]	float32	Conditioning-noise level. Always 0.5 for this checkpoint
obs	[1, 12, 64, 64]	float32	Last 4 RGB frames in [-1, 1], stacked channel-wise, oldest first
act	[1, 4]	int64	Last 4 action ids, oldest first; act[0, 3] is the action being applied this step
minimap	[1, 12, 64, 64]	float32	Conditioning channels for the post-action pose (below)
noise_aug_bucket	[1]	int64	Noise-augmentation bucket. Always 5 at inference

Output	Shape	Dtype	Contents
denoised	[1, 3, 64, 64]	float32	Denoised next frame, clip to [-1, 1] before use

Sampler

EDM/Karras schedule from model_meta.json: rho=7, sigma_min=0.002, sigma_max=5.0, first-order Euler, s_churn=0, s_noise=1, s_cond=0.5, default_noise_aug_bucket=5.

The sampler warm-starts from the previous frame, not from pure noise:

• Initialize x = prev_frame + N(0, sigma_max^2).
• Euler steps over the sigma ladder; after each model call, clip the model output to [-1, 1] before forming the Euler slope.
• 1-step (interactive setting): sigma ladder [5.0, 0.0]. The Euler update collapses algebraically to frame = clip(denoised, -1, 1), so each frame costs one model forward. This is the setting the robot campaign and the interactive clients run.
• 3-step (quality default in model_meta.json): sigma ladder [5.0, 0.283082, 0.002, 0.0], three model forwards per frame.

Per-frame stochasticity enters only through the warm-start noise draw.

Action space

5 discrete actions (model_meta.json: num_actions=5). Pose integration is host-side, dead-reckoned, in a 1856x1856-unit arena over a 128x128 wall grid:

Id	Name	Host-side pose effect
0	NOOP	none
1	FORWARD	+13.15 units along yaw; blocked (no slide) if the target cell is a wall or a floor step > 41 units up
2	TURN_LEFT	yaw +14.06 degrees
3	TURN_RIGHT	yaw -14.06 degrees
4	ATTACK	no pose change; the model renders the attack

Minimap conditioning (12 x 64 x 64)

Channel names are from model_meta.json; constructions from doom_env_v55.py / the export script. "Per-ray" channels come from a 64-ray camera-plane raycast (90-degree FOV, max depth 3000 units) at the post-action pose; each is a 64-vector tiled identically across all 64 rows (K=1 single-sample layout).

Ch	Name	Verified encoding (doom_env_v55.py)
0	disparity	per-ray log-depth: log(clamp(d,1,3000)+1)/log(3001)*2-1, tiled; exact geometry
1	walls_td	top-down occupancy, 128 grid downsampled 2x: wall +1, open -1; player marker stamped on top (pose cell + a 4-cell heading ray, value +1); exact geometry + pose
2	items_td	top-down item positions (+ the same player marker); marker only when the arena is empty
3	items_fov	item columns in view; zeros when no items
4	enemies_td	top-down enemies on a -1 background (+ player marker)
5	enemies_fov	enemy columns in view; zeros when no enemies
6	lifecycle	entity lifecycle channel; zeros in the reference gym
7	wall_tex	per-ray texture id of the wall cell the ray hit: clamp(t,-1,8)/8*2-1; -1 = no surface (out-of-bounds hit); tiled; surface identity
8	floor_h	per-ray floor height at the last open cell along the ray, /256, clipped to [-1, 1]; tiled
9	floor_tex	per-ray floor texture id at the last open cell (same encoding as ch 7); tiled; surface identity
10	ceil_h	per-ray ceiling height at the last open cell, /256; tiled
11	ceil_tex	per-ray ceiling texture id at the last open cell; tiled; surface identity

The reference gym runs an empty flat arena (channels 2/3/5/6 zeroed except markers, floor 0, ceiling 128, wall texture id 0); the browser runtime feeds live items/enemies through the same channels. Because geometry and materials are conditioning inputs, live map edits are supported: mutate the wall grid or texture maps, rebuild the channels, and the dream follows.

Frame format

Frames are 3x64x64 float32 RGB (CHW) in [-1, 1]. Display mapping: (x + 1) / 2. The obs ring advances by dropping the oldest 3 channels and appending the new frame.

Per-step order of operations

1. Apply the action to the host-side pose (collision check against the wall grid).
2. Raycast at the new pose; build the 12-channel minimap.
3. Roll the action ring; append the current action at the end.
4. Run the sampler (1 or 3 denoiser calls) with the old obs ring.
5. Clip the result to [-1, 1]; this is the new frame.
6. Roll the obs ring; append the new frame.

Driving the model

A live driven rollout. Left: frames generated by this checkpoint, one denoiser forward per frame. Right: the true wall map and pose trail from which the conditioning is built each step.

Minimal inference loop, adapted faithfully from the reference environment (env/doom_env_v55.py in the policies repo, byte-identical to the campaign file):

import json, numpy as np, onnxruntime as ort

sess = ort.InferenceSession("denoiser.onnx")
init = json.load(open("init_state.json"))
obs = np.array(init["obs_buffer"], np.float32).reshape(12, 64, 64)  # 4 frames
act = np.array(init["act_buffer"], np.int64)                        # 4 ids
rng = np.random.default_rng(0)
SIGMAS = [5.0, 0.0]                       # 1-step; use [5.0, 0.283082, 0.002, 0.0] for 3-step

def denoise(x, sigma, minimap):
    return sess.run(None, {
        "noisy_next_obs": x[None], "sigma": np.array([sigma], np.float32),
        "sigma_cond": np.array([0.5], np.float32),
        "obs": obs[None], "act": act[None], "minimap": minimap[None],
        "noise_aug_bucket": np.array([5], np.int64)})[0][0]

def step(action_id, minimap):             # minimap: (12,64,64) for the post-action pose
    global obs, act
    act = np.roll(act, -1); act[-1] = action_id
    x = obs[-3:] + rng.normal(0, SIGMAS[0], (3, 64, 64)).astype(np.float32)
    for i in range(len(SIGMAS) - 1):
        den = np.clip(denoise(x, SIGMAS[i], minimap), -1, 1)
        x = x + (x - den) / SIGMAS[i] * (SIGMAS[i + 1] - SIGMAS[i])
    frame = np.clip(x, -1, 1)
    obs = np.concatenate([obs[3:], frame], 0)
    return frame                          # (3,64,64) in [-1,1]

The pose/raycast/minimap builder is ~150 lines of dependency-free numpy in the reference environment; a TypeScript implementation (v55Runner.ts / channelBuilder12.ts / heightRaycaster.ts) runs the same model in-browser through onnxruntime-web (WebGPU, wasm fallback).

Performance and footprint

Fact	Value	Status
Checkpoint size	279,689,999 bytes, fp32	verified
Denoiser forwards per frame	1 (interactive) / 3 (quality)	verified from code
Batch	static 1 (one session per loaded graph; re-export to change it)	verified from graph
Full step, Apple M1 Pro, CPU EP, onnxruntime 1.27.0, 1-step	145.1 ms/frame (6.9 fps), including the python-side raycast + channel build	measured (this card, 2026-07)
GPU throughput	not measured for this export	unverified; no numbers published
Browser	runs via onnxruntime-web WebGPU with wasm fallback (fp32 file)	verified from runtime code

The model has no fixed internal clock: one action advances the world one frame, and the playable frame rate is the inference rate. The training capture rate is not recorded in the shipped metadata.

Uses

Direct Use

An interactive dungeon simulator to play, probe, and train against: a host that implements the contract above gets a controllable first-person world with editable geometry and materials at one denoiser call per frame. It has served as a full training substrate: policies evolved entirely inside it transfer to an independent physics simulator (below).

Out-of-Scope Use

Anything commercial (license is non-commercial, share-alike). The model simulates one DOOM-style dungeon domain at 64x64; it is not a general video generator, not a physics oracle, and the deathmatch sibling (doom-engine-arena) is not contract-compatible (6-channel minimap, no noise-aug input, different arena constants).

The robot that trained inside this model

This checkpoint is the "gauntlet" of the Dream Robot Gauntlet release: spiking (LIF) differential-drive controllers were evolved with CMA-ES entirely inside this world model, then verified in Webots, a rigid-body physics simulator that shares no code with the dream. One regime went further: the controller's only sensing was proximity values read off the model's own generated pixels by a fitted linear probe ("dream-eyes", r_L=0.785 / r_R=0.869 held-out vs the oracle raycaster for this checkpoint).

Dream-eyes: the controller sees only these generated pixels (bars = its left/right proximity percept), and drives the dream with them.

Top: the generated dream (t = 5/40/75 of the dream-eyes rollout). Bottom: the Webots physics verification of the same champion (t ~ 8s/31s/54s of its own episode 0). Two different episodes with matched moments, not one drive filmed twice.

• Evolved controllers + probes + the exact environment: Karajan42/dream-robot-gauntlet-policies
• Episode logs, physics-verification runs, lab notebook: Karajan42/dream-robot-gauntlet-data

Training Details

What is verifiable from the training configurations in the DIAMOND fork: the v5.5 base was trained with the DIAMOND recipe on a DOOM Health Gathering + enemies domain (64px, 5 actions), on a dataset with K=1 per-ray conditioning (data/v5_5); the Self-Forcing program then ran phase 1 (rollout training, per-step EDM loss, stochastic gradient truncation) and phase 2 (adds an adversarial critic), from which this checkpoint is epoch 26. Exact dataset episode counts, step budgets, and the specific phase-2 run configuration for this epoch are not published here because they were not verifiable from the shipped artifacts; the v5.3 family card documents its own (earlier) data mix.

Limitations and unverified claims

• Long rollouts drift; the Self-Forcing fine-tune exists precisely to reduce autoregressive drift, but the dream is a plausible continuation, not exact state. Ground-truth pose/collision lives host-side by design.
• 64x64 native resolution, no audio.
• GPU serving throughput: not measured for this export (flagged above).
• Training data specifics for v5.5/v6sf: not restated here (flagged above).
• The root-level denoiser.onnx in the archive repo shares this metadata but not these bytes; its exact epoch is unverified. This repository is the canonical, documented copy.
• License is stated as CC BY-NC-SA 4.0 for consistency with the published Dream Robot Gauntlet cards, which declare their license inherited from these checkpoints.

Citation

Cite DIAMOND and Self Forcing, and link this repository.

@inproceedings{alonso2024diamond,
  title  = {Diffusion for World Modeling: Visual Details Matter in Atari},
  author = {Alonso, Eloi and Jelley, Adam and Micheli, Vincent and Kanervisto, Anssi and Storkey, Amos and Pearce, Tim and Fleuret, Fran{\c{c}}ois},
  booktitle = {Advances in Neural Information Processing Systems},
  year   = {2024},
  note   = {arXiv:2405.12399}
}

@article{huang2025selfforcing,
  title  = {Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion},
  author = {Huang, Xun and others},
  year   = {2025},
  note   = {arXiv:2506.08009}
}

Model Card Authors

Alakazam (alakazam.gg)