Agent guidelines

AGENT.md

@@ -0,0 +1,169 @@
+# AGENT.md
+
+This file provides guidance to AI coding agents when working with code in this repository.
+
+## Project Overview
+
+This is a Gradio Space that implements "Next Scene" cinematic image generation using Qwen-Image-Edit-2509 with LoRA fine-tuning. The application generates visually progressive image sequences with natural cinematic transitions from frame to frame, optimized for fast 4-step inference.
+
+**Key Model Components:**
+- Base model: `Qwen/Qwen-Image-Edit-2509` (image editing diffusion model)
+- Accelerated transformer: `linoyts/Qwen-Image-Edit-Rapid-AIO` (4-step optimized variant)
+- LoRA adapter: `lovis93/next-scene-qwen-image-lora-2509` (cinematic progression fine-tune)
+- Text encoder: `Qwen2.5-VL-72B-Instruct` (via Hugging Face InferenceClient for prompt enhancement)
+
+## Running the Application
+
+**Start the Gradio interface:**
+```bash
+python app.py
+```
+
+**Install dependencies:**
+```bash
+pip install -r requirements.txt
+```
+
+The app requires GPU access. It uses the `@spaces.GPU` decorator for Hugging Face Spaces zero-GPU allocation.
+
+## Architecture
+
+### Pipeline Flow
+
+1. **Input Processing** (`app.py:infer`):
+   - Accepts input images via Gradio Gallery (filepath-based)
+   - Optional prompt rewriting using `Qwen2.5-VL-72B-Instruct` API
+   - Automatic "Next Scene" prompt generation from images
+
+2. **Image Generation** (`qwenimage/pipeline_qwenimage_edit_plus.py`):
+   - Custom pipeline extending `DiffusionPipeline`
+   - Encodes images using VAE at 1024x1024 for latents
+   - Encodes conditioning images at 384x384 for text encoder
+   - Packs latents into 2x2 patches (latent dims must be divisible by 2)
+   - Uses `FlowMatchEulerDiscreteScheduler` for denoising
+
+3. **Optimization** (`optimization.py`):
+   - Ahead-of-time (AOT) compilation using `torch.export` and `spaces.aoti_compile`
+   - Dynamic shapes for variable sequence lengths
+   - Custom inductor configs for performance (max_autotune, cudagraphs)
+   - FlashAttention 3 integration via `QwenDoubleStreamAttnProcessorFA3`
+
+4. **Output Handling**:
+   - Saves outputs to `outputs/` directory with unique timestamps
+   - Maintains 20-image history gallery
+   - Optional video generation via `multimodalart/wan-2-2-first-last-frame` Space
+
+### Custom QwenImage Components
+
+**Location:** `qwenimage/` package
+
+- `pipeline_qwenimage_edit_plus.py` - Main diffusion pipeline with LoRA support
+- `transformer_qwenimage.py` - Custom transformer model with cache management
+- `qwen_fa3_processor.py` - FlashAttention 3 attention processor
+
+**Key architectural features:**
+- Latent packing/unpacking for 2x2 patch processing
+- Multi-image conditioning support
+- True CFG (classifier-free guidance) with separate pos/neg paths
+- Dual-stream attention with rotary embeddings
+- Cache contexts for conditional/unconditional forward passes
+
+### Prompt Engineering
+
+**Two-stage prompt system:**
+
+1. **Edit Instruction Rewriter** (`SYSTEM_PROMPT`):
+   - Normalizes user prompts into professional editing instructions
+   - Handles text replacement (requires quotes), object manipulation, style transfer
+   - Used when `rewrite_prompt=True` checkbox is enabled
+
+2. **Next Scene Generator** (`NEXT_SCENE_SYSTEM_PROMPT`):
+   - Automatically suggests cinematic camera movements
+   - Focus on visual progression (dolly, pan, zoom, framing changes)
+   - Auto-triggers when input images change
+
+Both use `Qwen2.5-VL-72B-Instruct` via Hugging Face InferenceClient with Nebius provider. Requires `HF_TOKEN` environment variable.
+
+## Important Implementation Details
+
+### Image Dimension Handling
+
+Images are automatically resized based on `calculate_dimensions()` function:
+- VAE images: resized to maintain 1024×1024 area (1,048,576 pixels)
+- Condition images: resized to maintain 384×384 area (147,456 pixels)
+- Output dimensions must be divisible by 16 (vae_scale_factor × 2)
+- Height/width default to `None` which auto-calculates from input aspect ratio
+
+### LoRA Integration
+
+The pipeline fuses the "next-scene" LoRA adapter at initialization:
+```python
+pipe.load_lora_weights("lovis93/next-scene-qwen-image-lora-2509", ...)
+pipe.set_adapters(["next-scene"], adapter_weights=[1.])
+pipe.fuse_lora(adapter_names=["next-scene"], lora_scale=1.)
+pipe.unload_lora_weights()
+```
+
+After fusion, the adapter weights are merged into the base model and cannot be unfused.
+
+### Video Generation Integration
+
+The `turn_into_video()` function:
+- Connects to external Gradio Space `multimodalart/wan-2-2-first-last-frame`
+- Requires first input image and last output image
+- Uses the original prompt (or "smooth cinematic transition" fallback)
+- Returns video path for display
+
+### Gradio Gallery Format
+
+Input/output galleries use `type="filepath"` (string paths) rather than PIL Image tuples. Helper functions handle format compatibility for legacy tuple support.
+
+## Environment Variables
+
+- `HF_TOKEN` - Required for Qwen2.5-VL API access (prompt rewriting/generation)
+
+## File Outputs
+
+Generated images are saved to `outputs/` directory with format:
+```
+output_{seed}_{index}_{timestamp_ms}.png
+```
+
+## Local Development and API Testing
+
+The `custom/` directory is fully gitignored and used for local development files. Specifically, it contains:
+
+- **API client scripts** - For testing the Gradio Space remotely via API after deployment to Hugging Face
+- **`API_GUIDE.txt`** - Auto-generated Gradio API documentation showing endpoint signatures and example usage
+- **Local testing environments** - Virtual environments or test data that shouldn't be committed
+
+**API Integration Pattern:**
+Once the Space is deployed to Hugging Face, you can interact with it programmatically using `gradio_client`:
+
+```python
+from gradio_client import Client, handle_file
+
+client = Client("Sneak-Moose/Qwen-Image-Edit-next-scene")
+result = client.predict(
+    images=[],
+    prompt="Next Scene: Camera dollies forward...",
+    seed=42,
+    randomize_seed=False,
+    true_guidance_scale=1.0,
+    num_inference_steps=4,
+    height=1024,
+    width=1024,
+    rewrite_prompt=False,
+    api_name="/infer"
+)
+```
+
+The `custom/API_GUIDE.txt` contains full documentation of all available endpoints including `/infer`, `/turn_into_video`, `/suggest_next_scene_prompt`, and utility functions.
+
+## Development Notes
+
+- The model loads on startup and applies AOT compilation during first inference
+- Compilation uses dynamic shapes to support variable text/image sequence lengths
+- The transformer uses custom cache contexts ("cond"/"uncond") to optimize CFG passes
+- True CFG applies norm-based rescaling: `comb_pred * (cond_norm / noise_norm)`
+- FlashAttention 3 processor must be set before compilation

CLAUDE.md

@@ -0,0 +1,169 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+This is a Gradio Space that implements "Next Scene" cinematic image generation using Qwen-Image-Edit-2509 with LoRA fine-tuning. The application generates visually progressive image sequences with natural cinematic transitions from frame to frame, optimized for fast 4-step inference.
+
+**Key Model Components:**
+- Base model: `Qwen/Qwen-Image-Edit-2509` (image editing diffusion model)
+- Accelerated transformer: `linoyts/Qwen-Image-Edit-Rapid-AIO` (4-step optimized variant)
+- LoRA adapter: `lovis93/next-scene-qwen-image-lora-2509` (cinematic progression fine-tune)
+- Text encoder: `Qwen2.5-VL-72B-Instruct` (via Hugging Face InferenceClient for prompt enhancement)
+
+## Running the Application
+
+**Start the Gradio interface:**
+```bash
+python app.py
+```
+
+**Install dependencies:**
+```bash
+pip install -r requirements.txt
+```
+
+The app requires GPU access. It uses the `@spaces.GPU` decorator for Hugging Face Spaces zero-GPU allocation.
+
+## Architecture
+
+### Pipeline Flow
+
+1. **Input Processing** (`app.py:infer`):
+   - Accepts input images via Gradio Gallery (filepath-based)
+   - Optional prompt rewriting using `Qwen2.5-VL-72B-Instruct` API
+   - Automatic "Next Scene" prompt generation from images
+
+2. **Image Generation** (`qwenimage/pipeline_qwenimage_edit_plus.py`):
+   - Custom pipeline extending `DiffusionPipeline`
+   - Encodes images using VAE at 1024x1024 for latents
+   - Encodes conditioning images at 384x384 for text encoder
+   - Packs latents into 2x2 patches (latent dims must be divisible by 2)
+   - Uses `FlowMatchEulerDiscreteScheduler` for denoising
+
+3. **Optimization** (`optimization.py`):
+   - Ahead-of-time (AOT) compilation using `torch.export` and `spaces.aoti_compile`
+   - Dynamic shapes for variable sequence lengths
+   - Custom inductor configs for performance (max_autotune, cudagraphs)
+   - FlashAttention 3 integration via `QwenDoubleStreamAttnProcessorFA3`
+
+4. **Output Handling**:
+   - Saves outputs to `outputs/` directory with unique timestamps
+   - Maintains 20-image history gallery
+   - Optional video generation via `multimodalart/wan-2-2-first-last-frame` Space
+
+### Custom QwenImage Components
+
+**Location:** `qwenimage/` package
+
+- `pipeline_qwenimage_edit_plus.py` - Main diffusion pipeline with LoRA support
+- `transformer_qwenimage.py` - Custom transformer model with cache management
+- `qwen_fa3_processor.py` - FlashAttention 3 attention processor
+
+**Key architectural features:**
+- Latent packing/unpacking for 2x2 patch processing
+- Multi-image conditioning support
+- True CFG (classifier-free guidance) with separate pos/neg paths
+- Dual-stream attention with rotary embeddings
+- Cache contexts for conditional/unconditional forward passes
+
+### Prompt Engineering
+
+**Two-stage prompt system:**
+
+1. **Edit Instruction Rewriter** (`SYSTEM_PROMPT`):
+   - Normalizes user prompts into professional editing instructions
+   - Handles text replacement (requires quotes), object manipulation, style transfer
+   - Used when `rewrite_prompt=True` checkbox is enabled
+
+2. **Next Scene Generator** (`NEXT_SCENE_SYSTEM_PROMPT`):
+   - Automatically suggests cinematic camera movements
+   - Focus on visual progression (dolly, pan, zoom, framing changes)
+   - Auto-triggers when input images change
+
+Both use `Qwen2.5-VL-72B-Instruct` via Hugging Face InferenceClient with Nebius provider. Requires `HF_TOKEN` environment variable.
+
+## Important Implementation Details
+
+### Image Dimension Handling
+
+Images are automatically resized based on `calculate_dimensions()` function:
+- VAE images: resized to maintain 1024×1024 area (1,048,576 pixels)
+- Condition images: resized to maintain 384×384 area (147,456 pixels)
+- Output dimensions must be divisible by 16 (vae_scale_factor × 2)
+- Height/width default to `None` which auto-calculates from input aspect ratio
+
+### LoRA Integration
+
+The pipeline fuses the "next-scene" LoRA adapter at initialization:
+```python
+pipe.load_lora_weights("lovis93/next-scene-qwen-image-lora-2509", ...)
+pipe.set_adapters(["next-scene"], adapter_weights=[1.])
+pipe.fuse_lora(adapter_names=["next-scene"], lora_scale=1.)
+pipe.unload_lora_weights()
+```
+
+After fusion, the adapter weights are merged into the base model and cannot be unfused.
+
+### Video Generation Integration
+
+The `turn_into_video()` function:
+- Connects to external Gradio Space `multimodalart/wan-2-2-first-last-frame`
+- Requires first input image and last output image
+- Uses the original prompt (or "smooth cinematic transition" fallback)
+- Returns video path for display
+
+### Gradio Gallery Format
+
+Input/output galleries use `type="filepath"` (string paths) rather than PIL Image tuples. Helper functions handle format compatibility for legacy tuple support.
+
+## Environment Variables
+
+- `HF_TOKEN` - Required for Qwen2.5-VL API access (prompt rewriting/generation)
+
+## File Outputs
+
+Generated images are saved to `outputs/` directory with format:
+```
+output_{seed}_{index}_{timestamp_ms}.png
+```
+
+## Local Development and API Testing
+
+The `custom/` directory is fully gitignored and used for local development files. Specifically, it contains:
+
+- **API client scripts** - For testing the Gradio Space remotely via API after deployment to Hugging Face
+- **`API_GUIDE.txt`** - Auto-generated Gradio API documentation showing endpoint signatures and example usage
+- **Local testing environments** - Virtual environments or test data that shouldn't be committed
+
+**API Integration Pattern:**
+Once the Space is deployed to Hugging Face, you can interact with it programmatically using `gradio_client`:
+
+```python
+from gradio_client import Client, handle_file
+
+client = Client("Sneak-Moose/Qwen-Image-Edit-next-scene")
+result = client.predict(
+    images=[],
+    prompt="Next Scene: Camera dollies forward...",
+    seed=42,
+    randomize_seed=False,
+    true_guidance_scale=1.0,
+    num_inference_steps=4,
+    height=1024,
+    width=1024,
+    rewrite_prompt=False,
+    api_name="/infer"
+)
+```
+
+The `custom/API_GUIDE.txt` contains full documentation of all available endpoints including `/infer`, `/turn_into_video`, `/suggest_next_scene_prompt`, and utility functions.
+
+## Development Notes
+
+- The model loads on startup and applies AOT compilation during first inference
+- Compilation uses dynamic shapes to support variable text/image sequence lengths
+- The transformer uses custom cache contexts ("cond"/"uncond") to optimize CFG passes
+- True CFG applies norm-based rescaling: `comb_pred * (cond_norm / noise_norm)`
+- FlashAttention 3 processor must be set before compilation