---
license: creativeml-openrail-m
base_model: sd2-community/stable-diffusion-2-1
tags:
  - stable-diffusion
  - lora
  - text-to-image
  - diffusers
  - peft
library_name: peft
---

# SD 2.1 Diorama LoRA — Diorama Machine

A LoRA adapter fine-tuned on Stable Diffusion 2.1 to generate miniature, isometric diorama-style architectural interiors. Built as part of an independent deep learning project exploring diffusion model fine-tuning and evaluation methodology.

**Live demo:** [Diorama Machine Space](https://huggingface.co/spaces/Sathya77/Miniature-Diorama-Style-Image-Generator)

---

## Why LoRA

Full fine-tuning of SD 2.1's UNet (869M parameters) wasn't feasible on the available 8GB GPU. LoRA freezes the base model and trains small low-rank matrices (`A`, `B`) injected alongside the attention projection layers (`to_q`, `to_k`, `to_v`, `to_out.0`) — the layers where text conditioning meets image features:

```
W_effective = W + (alpha / r) * (A @ B)
```

This project used `r=32, alpha=64`, bringing trainable parameters down to ~6.6M — 0.76% of the full model — which is what made training on diorama-style interior renders practical on consumer hardware in the first place, without needing to touch or store gradients for the frozen 869M-parameter base.

---

## Model Details

| | |
|---|---|
| **Base model** | [`sd2-community/stable-diffusion-2-1`](https://huggingface.co/sd2-community/stable-diffusion-2-1) |
| **Method** | LoRA (Low-Rank Adaptation) via [PEFT](https://github.com/huggingface/peft) |
| **Rank / Alpha** | r=32, alpha=64 |
| **Target modules** | `to_q`, `to_k`, `to_v`, `to_out.0` (UNet cross- and self-attention) |
| **Trainable params** | 6,639,616 (0.76% of the full 869M-parameter UNet) |
| **Training data** | ~2,000 diorama-style architectural interior renders |
| **Precision** | fp16 mixed precision, gradient clipping (max_norm=1.0) |
| **Optimizer** | 8-bit AdamW, cosine LR schedule with warmup |
| **Epochs** | 12 |

---

## Implementation — Dataset to Result

**1. Dataset collection**
~2,000 diorama-style architectural interior renders were sourced for training — isometric, miniature-scale interior scenes (bedrooms, living rooms, kitchens, bathrooms).

**2. Caption inspection and cleaning**
Raw captions contained a systematic corruption pattern: duplicated scene descriptions followed by scraped platform attribution text (e.g. *"a living room with a red couch a living room with a red couch ... on Behance"*), affecting the large majority of entries. A cleaning function detected the repeated-phrase pattern programmatically — comparing progressively shorter prefixes of each caption against the text immediately following them — and kept only a single, clean copy of the actual scene description, discarding the trailing attribution text.

**3. Trigger word / caption structure**
Cleaned captions were used as-is, without an artificial style trigger word prepended — the model learns the diorama aesthetic directly from the paired image/caption signal rather than from a keyword shortcut. At inference, no special trigger word is required in the prompt; plain scene descriptions (e.g. *"a bedroom with a bed and a mirror"*) are sufficient.

**4. Target objective — v-prediction**
SD 2.1 is trained with a **v-prediction** objective rather than epsilon (noise) prediction, unlike SD 1.5. The training loop computes loss against `noise_scheduler.get_velocity(latents, noise, timesteps)`, matching the base model's actual prediction target.

**5. Training loop**
Implemented from scratch in PyTorch: VAE-encoded latents, per-sample random timestep sampling, v-prediction noise-velocity targets, LoRA-only backpropagation, gradient clipping (max_norm=1.0), 8-bit AdamW, and a cosine learning rate schedule with warmup, run for 12 epochs with checkpointing every 3 epochs.

**6. Result**
The final checkpoint (epoch 12) was evaluated and deployed as the adapter published here.

---

## Usage

```python
from diffusers import StableDiffusionPipeline
from peft import PeftModel
import torch

pipe = StableDiffusionPipeline.from_pretrained(
    "sd2-community/stable-diffusion-2-1",
    torch_dtype=torch.float16,
).to("cuda")
pipe.unet = PeftModel.from_pretrained(pipe.unet, "Sathya77/sd21-diorama-lora")

img = pipe(
    "a room with a bed and a table",
    negative_prompt="blurry, distorted lines, melted architecture, sloppy, deformed, noise, messy details",
    num_inference_steps=50,
    guidance_scale=7.5,
).images[0]
img.save("output.png")
```