Remove obsolete workspace code folder

code/README.md

@@ -1,169 +0,0 @@
-# FLUX.1-dev LoRA Training Workspace
-
-This repo is a minimal local workspace for preparing a custom image dataset, launching a `FLUX.1-dev` LoRA run, and testing the resulting adapter.
-
-It uses a tracked local copy of Hugging Face's `diffusers` Flux LoRA trainer, patched for lower memory use, while keeping the project-specific logic here in this repo:
-
-- dataset normalization into a local `imagefolder` dataset
-- prompt and caption templating
-- reproducible training settings in `configs/flux_lora.toml`
-- a simple inference command for post-train validation
-
-## Project layout
-
-- [configs/flux_lora.toml](/home/zireael/lora/configs/flux_lora.toml)
-- [src/lora/cli.py](/home/zireael/lora/src/lora/cli.py)
-- [scripts/train_dreambooth_lora_flux_lowmem.py](/home/zireael/lora/scripts/train_dreambooth_lora_flux_lowmem.py)
-- [dataset](/home/zireael/lora/dataset/)
-
-## Why this setup
-
-Your dataset has two kinds of examples:
-
-- numbered source/output pairs like `1.jpg -> 1-output.png`
-- standalone keyword images that do not have matching source inputs
-
-`prepare-dataset` now preserves that split. Paired records include both the target image and its matching source image, while standalone keyword images are prepared as target-only samples.
-
-## 1. Prerequisites
-
-You need:
-
-- Python 3.11+
-- a CUDA-capable GPU
-- a Hugging Face account
-- accepted access to `black-forest-labs/FLUX.1-dev`
-
-Install and bootstrap:
-
-```bash
-uv sync
-uv pip install -e .
-source .venv/bin/activate
-lora install
-```
-
-Then authenticate and configure Accelerate:
-
-```bash
-hf auth login
-accelerate config default
-```
-
-## 2. Prepare the dataset
-
-This will:
-
-- convert all supported images in `dataset/` to RGB PNGs
-- write target images into `training_data/flux_aura_style/train/`
-- write paired source inputs into `training_data/flux_aura_style/conditioning/`
-- create `training_data/flux_aura_style/train/metadata.jsonl`
-
-Run:
-
-```bash
-lora prepare-dataset
-```
-
-The prompt template now defaults to:
-
-```text
-A strong colorful light is illuminating {subject}'s silhouette. Medium shot. The image is rendered in a smooth gradient of luminous light, giving a radiant and ethereal appearance. Subtle contour lighting highlights delicate outlines, adding depth and a refined, high-end cinematic glow
-```
-
-For numbered pairs, `{subject}` comes from `paired_subject` in [configs/flux_lora.toml](/home/zireael/lora/configs/flux_lora.toml). For standalone images, `{subject}` is derived from the filename stem.
-
-## 3. Share or fetch the dataset
-
-Uploading the prepared dataset to Hugging Face is recommended if other users should reproduce the run. Upload `training_data/flux_aura_style`, not the raw `dataset/` folder, unless you intentionally want to publish the original source images too.
-
-Before publishing, make sure every image is safe to redistribute and choose `--private` if the dataset should only be available to collaborators.
-
-```bash
-hf auth login
-hf repo create ezhoureal/flux-aura-style --type dataset --private
-hf upload ezhoureal/flux-aura-style training_data/flux_aura_style .
-```
-
-Other users can fetch it with:
-
-```bash
-hf download ezhoureal/flux-aura-style \
-  --repo-type dataset \
-  --local-dir training_data/flux_aura_style
-```
-
-Then they can train without running `lora prepare-dataset`:
-
-```bash
-uv sync
-uv run lora train
-```
-
-The prepared dataset must keep this layout:
-
-```text
-training_data/flux_aura_style/
-  train/
-    metadata.jsonl
-    pair-001.png
-    label-001.png
-  conditioning/
-    pair-001.png
-```
-
-## 4. Train the LoRA
-
-Start training with:
-
-```bash
-lora train
-```
-
-The launcher will:
-
-- run the tracked `scripts/train_dreambooth_lora_flux_lowmem.py` trainer
-- precompute all caption embeddings with CLIP/T5 before loading the FLUX transformer onto the GPU
-- free CLIP/T5 before transformer LoRA training starts
-
-The default config is intentionally conservative for a small dataset:
-
-- rank `8`
-- resolution `512`
-- `adamw` with learning rate `1e-4`
-- train batch size `2` with gradient accumulation `2`
-- dataloader workers `4`
-- in-training validation off, so the trainer does not reload the full inference pipeline while training
-
-If your GPU does not fit that profile, fall back in this order:
-
-- enable gradient checkpointing
-- reduce `train_batch_size` to `1`
-- increase `gradient_accumulation_steps` to keep the same effective batch size
-
-## 5. Test inference
-
-Once training finishes:
-
-```bash
-lora infer
-```
-
-Or pass a custom prompt:
-
-```bash
-lora infer "a side-profile portrait of a man in the style of zrlprfl, pastel haze, cinematic silhouette"
-```
-
-The sample image is written to:
-
-```text
-samples/flux-lora-test.png
-```
-
-## 6. Notes on Flux training
-
-- `FLUX.1-dev` is gated on Hugging Face, so you must accept the model terms before downloads work.
-- The official DreamBooth Flux trainer is text-to-image. It trains from the target image and `prompt` column; paired source images are preserved in `conditioning/` and referenced by metadata for future image-conditioned workflows, but this trainer does not consume them.
-- Flux LoRA training is memory-heavy. This workspace defaults to 512px, rank 8, gradient checkpointing, latent caching, and cached prompt embeddings to fit a 32 GB GPU more reliably.
-- If your GPU still runs out of memory, reduce `resolution`, reduce `rank`, or increase `gradient_accumulation_steps`.

code/configs/flux_lora.toml

@@ -1,46 +0,0 @@
-[dataset]
-source_dir = "dataset"
-prepared_dir = "training_data/flux_aura_style"
-
-[prompts]
-paired_subject = "man"
-instance_prompt = "A strong colorful light is illuminating man's silhouette. Medium shot. The image is rendered in a smooth gradient of luminous light, giving a radiant and ethereal appearance. Subtle contour lighting highlights delicate outlines, adding depth and a refined, high-end cinematic glow"
-prompt_template = "A strong colorful light is illuminating {subject}'s silhouette. Medium shot. The image is rendered in a smooth gradient of luminous light, giving a radiant and ethereal appearance. Subtle contour lighting highlights delicate outlines, adding depth and a refined, high-end cinematic glow"
-
-[training]
-model_name = "black-forest-labs/FLUX.1-dev"
-diffusers_ref = "a1c7df48013b1911231f34769ada4865a3acae20"
-training_script = "scripts/train_dreambooth_lora_flux_lowmem.py"
-output_dir = "outputs"
-mixed_precision = "bf16"
-resolution = 512
-train_batch_size = 2
-gradient_accumulation_steps = 2
-gradient_checkpointing = true
-optimizer = "adamw"
-learning_rate = 1e-4
-lr_scheduler = "constant"
-lr_warmup_steps = 0
-max_train_steps = 1000
-rank = 8
-lora_alpha = 8
-validation_prompt = ""
-validation_epochs = 100
-num_validation_images = 0
-seed = 42
-report_to = "tensorboard"
-repeats = 10
-max_sequence_length = 256
-cache_latents = true
-dataloader_num_workers = 4
-use_8bit_adam = false
-push_to_hub = false
-
-[inference]
-weight_name = "pytorch_lora_weights_v1.safetensors"
-prompt = "A strong colorful light is illuminating man's silhouette. Medium shot. The image is rendered in a smooth gradient of luminous light (pink, peach, lavendar), giving a radiant and ethereal appearance. Subtle contour lighting highlights delicate outlines, adding depth and a refined, high-end cinematic glow"
-output_path = "samples/flux-lora-test.png"
-height = 1024
-width = 1024
-guidance_scale = 3.5
-num_inference_steps = 28

code/main.py

@@ -1,10 +0,0 @@
-from pathlib import Path
-import sys
-
-sys.path.insert(0, str(Path(__file__).resolve().parent / "src"))
-
-from lora.cli import main
-
-
-if __name__ == "__main__":
-    raise SystemExit(main())

code/pyproject.toml

@@ -1,62 +0,0 @@
-[project]
-name = "lora"
-version = "0.1.0"
-description = "Local tooling for preparing and training a FLUX.1-dev LoRA."
-readme = "README.md"
-requires-python = ">=3.11"
-dependencies = [
-    "accelerate>=0.31.0",
-    "datasets>=3.0.0",
-    "diffusers @ git+https://github.com/huggingface/diffusers.git",
-    "fal-client>=0.7.0",
-    "ftfy",
-    "hf_transfer",
-    "huggingface-hub>=0.31.0",
-    "jinja2",
-    "peft>=0.11.1",
-    "pillow>=10.4.0",
-    "torch==2.11.0+cu128",
-    "torchvision==0.26.0+cu128",
-    "transformers>=4.41.2",
-    "sentencepiece",
-    "tensorboard",
-    "bitsandbytes>=0.49.2",
-    "protobuf>=7.35.0",
-    "requests>=2.34.2",
-]
-
-[dependency-groups]
-dev = [
-    "pyright>=1.1.400",
-    "pytest>=8.0.0",
-    "ruff>=0.11.0",
-]
-
-[project.scripts]
-lora = "lora.cli:main"
-
-[build-system]
-requires = ["setuptools>=68"]
-build-backend = "setuptools.build_meta"
-
-[[tool.uv.index]]
-url = "https://pypi.tuna.tsinghua.edu.cn/simple"
-default = true
-
-[[tool.uv.index]]
-name = "pytorch-cu128"
-url = "https://download.pytorch.org/whl/cu128"
-explicit = true
-
-[tool.uv.sources]
-torch = { index = "pytorch-cu128" }
-torchvision = { index = "pytorch-cu128" }
-
-[tool.setuptools]
-package-dir = {"" = "src"}
-
-[tool.setuptools.packages.find]
-where = ["src"]
-
-[tool.ruff]
-line-length = 100

code/scripts/generate_realistic_label_pairs.py

@@ -1,324 +0,0 @@
-#!/usr/bin/env python3
-from __future__ import annotations
-
-import argparse
-import base64
-import json
-import os
-import re
-import shutil
-import sys
-import time
-import urllib.error
-import urllib.request
-from dataclasses import dataclass
-from pathlib import Path
-from typing import Any
-
-
-REPO_ROOT = Path(__file__).resolve().parents[1]
-DEFAULT_TRAIN_DIR = REPO_ROOT / "training_data" / "flux_aura_style" / "train"
-DEFAULT_CONDITIONING_DIR = REPO_ROOT / "training_data" / "flux_aura_style" / "conditioning"
-DEFAULT_API_URL = "https://ark.cn-beijing.volces.com/api/v3/images/generations"
-DEFAULT_GENERATION_PROMPT = (
-    "Use the reference image only for composition, subject identity, pose, silhouette, and camera "
-    "framing. Generate a realistic natural photo of the same subject before the aura lighting style "
-    "was applied. Remove colorful glow, gradients, ethereal haze, rim-light effects, painterly "
-    "stylization, and text. Keep the result clean, plausible, detailed, and photorealistic."
-)
-
-
-@dataclass(frozen=True)
-class LabelRecord:
-    label_path: Path
-    prompt: str
-
-
-def parse_args() -> argparse.Namespace:
-    parser = argparse.ArgumentParser(
-        description=(
-            "Generate realistic conditioning images for label-*.png targets and append them as "
-            "new paired training records."
-        )
-    )
-    parser.add_argument(
-        "--api-key",
-        default=os.environ.get("ARK_API_KEY"),
-        help="Volcengine Ark API key. Defaults to ARK_API_KEY.",
-    )
-    parser.add_argument(
-        "--model",
-        default=os.environ.get("ARK_IMAGE_MODEL", "doubao-seedream-5-0-260128"),
-        help="Ark image generation model or endpoint ID. Defaults to ARK_IMAGE_MODEL.",
-    )
-    parser.add_argument("--api-url", default=DEFAULT_API_URL, help="Image generation API URL.")
-    parser.add_argument(
-        "--train-dir",
-        type=Path,
-        default=DEFAULT_TRAIN_DIR,
-        help="Directory containing label-*.png and metadata.jsonl.",
-    )
-    parser.add_argument(
-        "--conditioning-dir",
-        type=Path,
-        default=DEFAULT_CONDITIONING_DIR,
-        help="Directory where generated realistic pair conditioning images are written.",
-    )
-    parser.add_argument(
-        "--prompt",
-        default=DEFAULT_GENERATION_PROMPT,
-        help="Prompt used to generate the realistic conditioning image from each label reference.",
-    )
-    parser.add_argument(
-        "--size",
-        default="2048x2048",
-        help="Requested output size. Seedream 5.0 lite accepts values such as 2048x2048.",
-    )
-    parser.add_argument(
-        "--limit",
-        type=int,
-        default=None,
-        help="Maximum number of labels to process.",
-    )
-    parser.add_argument(
-        "--start-label",
-        type=int,
-        default=None,
-        help="First label number to process, inclusive.",
-    )
-    parser.add_argument(
-        "--end-label",
-        type=int,
-        default=None,
-        help="Last label number to process, inclusive.",
-    )
-    parser.add_argument(
-        "--request-timeout",
-        type=int,
-        default=300,
-        help="Per-request timeout in seconds.",
-    )
-    parser.add_argument(
-        "--sleep",
-        type=float,
-        default=0.0,
-        help="Seconds to sleep between successful API calls.",
-    )
-    parser.add_argument(
-        "--retries",
-        type=int,
-        default=2,
-        help="Number of retries after a failed API call.",
-    )
-    parser.add_argument(
-        "--dry-run",
-        action="store_true",
-        help="Print the planned label to pair mapping without calling the API or writing files.",
-    )
-    return parser.parse_args()
-
-
-def label_number(path: Path) -> int:
-    match = re.fullmatch(r"label-(\d+)\.png", path.name)
-    if not match:
-        raise ValueError(f"Not a label file: {path}")
-    return int(match.group(1))
-
-
-def pair_number(path: Path) -> int:
-    match = re.fullmatch(r"pair-(\d+)\.png", path.name)
-    if not match:
-        raise ValueError(f"Not a pair file: {path}")
-    return int(match.group(1))
-
-
-def read_metadata(metadata_path: Path) -> list[dict[str, Any]]:
-    with metadata_path.open("r", encoding="utf-8") as fh:
-        return [json.loads(line) for line in fh if line.strip()]
-
-
-def collect_labels(train_dir: Path, metadata_rows: list[dict[str, Any]]) -> list[LabelRecord]:
-    prompts_by_name = {
-        row["file_name"]: row.get("prompt", "")
-        for row in metadata_rows
-        if row.get("kind") == "label" and isinstance(row.get("file_name"), str)
-    }
-    records = []
-    for label_path in sorted(train_dir.glob("label-*.png"), key=label_number):
-        prompt = prompts_by_name.get(label_path.name)
-        if prompt is None:
-            print(f"Skipping {label_path.name}: no label metadata row found.", file=sys.stderr)
-            continue
-        records.append(LabelRecord(label_path=label_path, prompt=prompt))
-    return records
-
-
-def filter_labels(
-    records: list[LabelRecord],
-    start_label: int | None,
-    end_label: int | None,
-    limit: int | None,
-) -> list[LabelRecord]:
-    filtered = []
-    for record in records:
-        number = label_number(record.label_path)
-        if start_label is not None and number < start_label:
-            continue
-        if end_label is not None and number > end_label:
-            continue
-        filtered.append(record)
-    if limit is not None:
-        filtered = filtered[:limit]
-    return filtered
-
-
-def next_pair_numbers(train_dir: Path, count: int) -> list[int]:
-    existing = [pair_number(path) for path in train_dir.glob("pair-*.png")]
-    start = max(existing, default=0) + 1
-    return list(range(start, start + count))
-
-
-def image_as_data_url(path: Path) -> str:
-    encoded = base64.b64encode(path.read_bytes()).decode("ascii")
-    return f"data:image/png;base64,{encoded}"
-
-
-def strip_data_url_prefix(value: str) -> str:
-    if "," in value and value.lower().startswith("data:"):
-        return value.split(",", 1)[1]
-    return value
-
-
-def ark_generate_image(args: argparse.Namespace, label_path: Path) -> bytes:
-    payload = {
-        "model": args.model,
-        "prompt": args.prompt,
-        "image": image_as_data_url(label_path),
-        "size": args.size,
-        "sequential_image_generation": "disabled",
-        "response_format": "b64_json",
-        "output_format": "png",
-        "watermark": False,
-    }
-    body = json.dumps(payload).encode("utf-8")
-    request = urllib.request.Request(
-        args.api_url,
-        data=body,
-        headers={
-            "Authorization": f"Bearer {args.api_key}",
-            "Content-Type": "application/json",
-        },
-        method="POST",
-    )
-
-    last_error: Exception | None = None
-    for attempt in range(args.retries + 1):
-        try:
-            with urllib.request.urlopen(request, timeout=args.request_timeout) as response:
-                result = json.loads(response.read().decode("utf-8"))
-            data = result.get("data")
-            if not data:
-                raise RuntimeError(f"API response did not contain data: {result}")
-            b64_json = data[0].get("b64_json")
-            if not b64_json:
-                raise RuntimeError(f"API response did not contain b64_json: {result}")
-            return base64.b64decode(strip_data_url_prefix(b64_json))
-        except urllib.error.HTTPError as exc:
-            detail = exc.read().decode("utf-8", errors="replace")
-            last_error = RuntimeError(f"HTTP {exc.code}: {detail}")
-        except (urllib.error.URLError, TimeoutError, RuntimeError, json.JSONDecodeError) as exc:
-            last_error = exc
-
-        if attempt < args.retries:
-            time.sleep(2**attempt)
-
-    assert last_error is not None
-    raise last_error
-
-
-def append_metadata(metadata_path: Path, row: dict[str, Any]) -> None:
-    with metadata_path.open("a", encoding="utf-8") as fh:
-        fh.write(json.dumps(row, ensure_ascii=False) + "\n")
-
-
-def main() -> int:
-    args = parse_args()
-    train_dir = args.train_dir.resolve()
-    conditioning_dir = args.conditioning_dir.resolve()
-    metadata_path = train_dir / "metadata.jsonl"
-
-    if not train_dir.exists():
-        print(f"Train directory does not exist: {train_dir}", file=sys.stderr)
-        return 1
-    if not metadata_path.exists():
-        print(f"Metadata file does not exist: {metadata_path}", file=sys.stderr)
-        return 1
-    if not args.dry_run and not args.api_key:
-        print("Missing API key. Set ARK_API_KEY or pass --api-key.", file=sys.stderr)
-        return 1
-
-    rows = read_metadata(metadata_path)
-    labels = filter_labels(
-        collect_labels(train_dir, rows),
-        start_label=args.start_label,
-        end_label=args.end_label,
-        limit=args.limit,
-    )
-    if not labels:
-        print("No label images selected.", file=sys.stderr)
-        return 1
-
-    pair_numbers = next_pair_numbers(train_dir, len(labels))
-    plan = list(zip(labels, pair_numbers, strict=True))
-
-    print(
-        json.dumps(
-            {
-                "selected_labels": len(labels),
-                "first_pair": f"pair-{pair_numbers[0]:03d}.png",
-                "last_pair": f"pair-{pair_numbers[-1]:03d}.png",
-                "model": args.model,
-                "size": args.size,
-                "dry_run": args.dry_run,
-            },
-            indent=2,
-        )
-    )
-
-    if args.dry_run:
-        for label, number in plan:
-            print(f"{label.label_path.name} -> pair-{number:03d}.png")
-        return 0
-
-    conditioning_dir.mkdir(parents=True, exist_ok=True)
-    for index, (label, number) in enumerate(plan, start=1):
-        pair_name = f"pair-{number:03d}.png"
-        pair_target = train_dir / pair_name
-        conditioning_target = conditioning_dir / pair_name
-        if pair_target.exists() or conditioning_target.exists():
-            print(f"Refusing to overwrite existing files for {pair_name}", file=sys.stderr)
-            return 1
-
-        print(f"[{index}/{len(plan)}] Generating realistic conditioning for {label.label_path.name}")
-        image_bytes = ark_generate_image(args, label.label_path)
-        conditioning_target.write_bytes(image_bytes)
-        shutil.copy2(label.label_path, pair_target)
-        append_metadata(
-            metadata_path,
-            {
-                "file_name": pair_name,
-                "prompt": label.prompt,
-                "kind": "paired",
-                "conditioning_path": f"../conditioning/{pair_name}",
-                "source_label": label.label_path.name,
-            },
-        )
-        if args.sleep:
-            time.sleep(args.sleep)
-
-    print(f"Generated {len(plan)} realistic label pairs.")
-    return 0
-
-
-if __name__ == "__main__":
-    raise SystemExit(main())

code/scripts/run_flux2_edit_lora_fal.py

@@ -1,432 +0,0 @@
-#!/usr/bin/env python3
-from __future__ import annotations
-
-import argparse
-import json
-import os
-import sys
-import time
-import urllib.parse
-import urllib.request
-from pathlib import Path
-from typing import Any
-
-
-REPO_ROOT = Path(__file__).resolve().parents[1]
-DEFAULT_TRAINING_OUTPUT_DIR = REPO_ROOT / "outputs" / "fal_flux2_edit_lora"
-DEFAULT_TRAINING_RESULT = DEFAULT_TRAINING_OUTPUT_DIR / "fal-training-result.json"
-DEFAULT_OUTPUT_DIR = REPO_ROOT / "outputs" / "fal_flux2_edit_inference"
-DEFAULT_ENDPOINT = "fal-ai/flux-2/lora/edit"
-DEFAULT_PROMPT = (
-    "Transform this photorealistic image into the trained radiant aura style: smooth colorful "
-    "gradients, ethereal haze, subtle contour lighting, and a refined cinematic glow. Preserve the "
-    "subject identity, composition, pose, silhouette, camera framing, and important details."
-)
-
-
-def parse_args() -> argparse.Namespace:
-    parser = argparse.ArgumentParser(
-        description=(
-            "Apply the trained FLUX.2 edit LoRA on fal to a photorealistic input image and download "
-            "the stylized result."
-        )
-    )
-    parser.add_argument(
-        "input_image",
-        type=str,
-        help="Photorealistic input image path or URL.",
-    )
-    parser.add_argument(
-        "--prompt",
-        default=DEFAULT_PROMPT,
-        help="Edit instruction sent to FLUX.2 LoRA Edit.",
-    )
-    parser.add_argument(
-        "--lora",
-        default=None,
-        help=(
-            "LoRA weights URL, Hugging Face repo ID, or local safetensors path. If omitted, the "
-            "script reads --training-result or uses a local safetensors file from the training "
-            "output directory."
-        ),
-    )
-    parser.add_argument(
-        "--training-result",
-        type=Path,
-        default=DEFAULT_TRAINING_RESULT,
-        help="fal training result JSON written by scripts/train_flux2_edit_lora_fal.py.",
-    )
-    parser.add_argument(
-        "--lora-scale",
-        type=float,
-        default=1.0,
-        help="LoRA strength passed as the LoRAInput scale.",
-    )
-    parser.add_argument(
-        "--output-dir",
-        type=Path,
-        default=DEFAULT_OUTPUT_DIR,
-        help="Directory for downloaded images and the inference result JSON.",
-    )
-    parser.add_argument(
-        "--output-path",
-        type=Path,
-        default=None,
-        help="Optional path for the first downloaded output image.",
-    )
-    parser.add_argument(
-        "--endpoint",
-        default=DEFAULT_ENDPOINT,
-        help="fal endpoint id.",
-    )
-    parser.add_argument(
-        "--guidance-scale",
-        type=float,
-        default=2.5,
-        help="Prompt adherence. fal default is 2.5.",
-    )
-    parser.add_argument(
-        "--num-inference-steps",
-        type=int,
-        default=28,
-        help="Number of inference steps. fal accepts 4 to 50.",
-    )
-    parser.add_argument(
-        "--image-size",
-        default=None,
-        help='Optional output size as WIDTHxHEIGHT, for example "1024x1024". If omitted, fal chooses.',
-    )
-    parser.add_argument(
-        "--num-images",
-        type=int,
-        default=1,
-        help="Number of images to generate. fal accepts 1 to 4.",
-    )
-    parser.add_argument(
-        "--seed",
-        type=int,
-        default=None,
-        help="Optional seed for reproducible generations.",
-    )
-    parser.add_argument(
-        "--acceleration",
-        choices=("none", "regular", "high"),
-        default="regular",
-        help="fal acceleration level.",
-    )
-    parser.add_argument(
-        "--output-format",
-        choices=("jpeg", "png", "webp"),
-        default="png",
-        help="Output image format.",
-    )
-    parser.add_argument(
-        "--enable-prompt-expansion",
-        action="store_true",
-        help="Ask fal to expand the prompt before generation.",
-    )
-    parser.add_argument(
-        "--disable-safety-checker",
-        action="store_true",
-        help="Disable fal safety checker.",
-    )
-    parser.add_argument(
-        "--dry-run",
-        action="store_true",
-        help="Print the resolved request arguments without calling fal.",
-    )
-    return parser.parse_args()
-
-
-def is_url(value: str) -> bool:
-    parsed = urllib.parse.urlparse(value)
-    return parsed.scheme in {"http", "https"}
-
-
-def parse_image_size(value: str | None) -> dict[str, int] | None:
-    if value is None:
-        return None
-    try:
-        width_text, height_text = value.lower().split("x", 1)
-        width = int(width_text)
-        height = int(height_text)
-    except ValueError as exc:
-        raise ValueError('--image-size must look like "1024x1024".') from exc
-
-    if not 512 <= width <= 2048 or not 512 <= height <= 2048:
-        raise ValueError("--image-size width and height must be between 512 and 2048 pixels.")
-    return {"width": width, "height": height}
-
-
-def on_queue_update(update: object) -> None:
-    try:
-        import fal_client
-    except ImportError:
-        return
-
-    if isinstance(update, fal_client.InProgress) and update.logs:
-        for log in update.logs:
-            message = log.get("message")
-            if message:
-                print(message, flush=True)
-
-
-def load_training_result(path: Path) -> dict[str, Any] | None:
-    if not path.exists():
-        return None
-    data = json.loads(path.read_text(encoding="utf-8"))
-    result = data.get("result", data)
-    if not isinstance(result, dict):
-        raise ValueError(f"Training result JSON has no object result: {path}")
-    return result
-
-
-def lora_from_training_result(path: Path) -> str | None:
-    result = load_training_result(path)
-    if result is None:
-        return None
-
-    for key in ("diffusers_lora_file", "lora_file", "lora"):
-        value = result.get(key)
-        if isinstance(value, dict) and isinstance(value.get("url"), str):
-            return value["url"]
-        if isinstance(value, str):
-            return value
-
-    loras = result.get("loras")
-    if isinstance(loras, list):
-        for value in loras:
-            if isinstance(value, dict) and isinstance(value.get("url"), str):
-                return value["url"]
-            if isinstance(value, str):
-                return value
-    return None
-
-
-def latest_local_lora(training_output_dir: Path) -> Path | None:
-    candidates = sorted(
-        training_output_dir.glob("*.safetensors"),
-        key=lambda path: path.stat().st_mtime,
-        reverse=True,
-    )
-    return candidates[0] if candidates else None
-
-
-def resolve_lora(args: argparse.Namespace) -> str:
-    if args.lora:
-        return args.lora
-
-    lora = lora_from_training_result(args.training_result.resolve())
-    if lora:
-        return lora
-
-    local_lora = latest_local_lora(args.training_result.resolve().parent)
-    if local_lora is not None:
-        return str(local_lora)
-
-    raise FileNotFoundError(
-        "Could not find LoRA weights. Pass --lora, or run training without --no-download so "
-        f"{args.training_result} contains a diffusers_lora_file URL."
-    )
-
-
-def upload_input_image(value: str, *, dry_run: bool) -> str:
-    if is_url(value):
-        return value
-
-    path = Path(value).expanduser().resolve()
-    if not path.exists():
-        raise FileNotFoundError(f"Input image does not exist: {path}")
-    if dry_run:
-        return str(path)
-
-    import fal_client
-
-    uploaded_url = fal_client.upload_file(path)
-    print(f"Uploaded {path.name}: {uploaded_url}", flush=True)
-    return uploaded_url
-
-
-def upload_lora_if_local(value: str, *, dry_run: bool) -> str:
-    if is_url(value):
-        return value
-
-    path = Path(value).expanduser().resolve()
-    if not path.exists():
-        # The API also accepts Hugging Face repo IDs, which look like "owner/repo".
-        return value
-    if dry_run:
-        return str(path)
-
-    import fal_client
-
-    uploaded_url = fal_client.upload_file(path)
-    print(f"Uploaded {path.name}: {uploaded_url}", flush=True)
-    return uploaded_url
-
-
-def output_file_name(image: dict[str, Any], index: int, output_format: str) -> str:
-    file_name = image.get("file_name")
-    if isinstance(file_name, str) and file_name:
-        return file_name
-
-    url = image.get("url")
-    if isinstance(url, str):
-        parsed_name = Path(urllib.parse.urlparse(url).path).name
-        if parsed_name:
-            return parsed_name
-
-    suffix = "jpg" if output_format == "jpeg" else output_format
-    return f"stylized-{index + 1:02d}.{suffix}"
-
-
-def download_images(
-    images: list[dict[str, Any]],
-    output_dir: Path,
-    output_path: Path | None,
-    output_format: str,
-) -> list[Path]:
-    output_dir.mkdir(parents=True, exist_ok=True)
-    downloaded = []
-
-    for index, image in enumerate(images):
-        url = image.get("url")
-        if not isinstance(url, str) or not url:
-            continue
-
-        if index == 0 and output_path is not None:
-            destination = output_path.expanduser().resolve()
-            destination.parent.mkdir(parents=True, exist_ok=True)
-        else:
-            destination = output_dir / output_file_name(image, index, output_format)
-
-        urllib.request.urlretrieve(url, destination)
-        downloaded.append(destination)
-
-    return downloaded
-
-
-def build_arguments(
-    args: argparse.Namespace,
-    image_url: str,
-    lora_url: str,
-) -> dict[str, Any]:
-    request_arguments: dict[str, Any] = {
-        "prompt": args.prompt,
-        "guidance_scale": args.guidance_scale,
-        "num_inference_steps": args.num_inference_steps,
-        "num_images": args.num_images,
-        "acceleration": args.acceleration,
-        "enable_prompt_expansion": args.enable_prompt_expansion,
-        "enable_safety_checker": not args.disable_safety_checker,
-        "output_format": args.output_format,
-        "image_urls": [image_url],
-        "loras": [{"path": lora_url, "scale": args.lora_scale}],
-    }
-
-    image_size = parse_image_size(args.image_size)
-    if image_size is not None:
-        request_arguments["image_size"] = image_size
-    if args.seed is not None:
-        request_arguments["seed"] = args.seed
-
-    return request_arguments
-
-
-def run_with_fal(endpoint: str, request_arguments: dict[str, Any]) -> tuple[dict[str, Any], str | None]:
-    import fal_client
-
-    result = fal_client.subscribe(
-        endpoint,
-        arguments=request_arguments,
-        with_logs=True,
-        on_queue_update=on_queue_update,
-    )
-    if hasattr(result, "data"):
-        return dict(result.data), getattr(result, "request_id", None)
-    return dict(result), None
-
-
-def main() -> int:
-    args = parse_args()
-
-    if args.num_images < 1 or args.num_images > 4:
-        print("--num-images must be between 1 and 4.", file=sys.stderr)
-        return 1
-    if args.num_inference_steps < 4 or args.num_inference_steps > 50:
-        print("--num-inference-steps must be between 4 and 50.", file=sys.stderr)
-        return 1
-    if args.guidance_scale < 0 or args.guidance_scale > 20:
-        print("--guidance-scale must be between 0 and 20.", file=sys.stderr)
-        return 1
-    if not args.dry_run and not os.environ.get("FAL_KEY"):
-        print("Missing fal key. Set FAL_KEY before running inference.", file=sys.stderr)
-        return 1
-
-    try:
-        input_image_url = upload_input_image(args.input_image, dry_run=args.dry_run)
-        lora_url = upload_lora_if_local(resolve_lora(args), dry_run=args.dry_run)
-        request_arguments = build_arguments(args, input_image_url, lora_url)
-    except (FileNotFoundError, ValueError) as exc:
-        print(str(exc), file=sys.stderr)
-        return 1
-
-    print(
-        json.dumps(
-            {
-                "endpoint": args.endpoint,
-                "input_image": input_image_url,
-                "lora": lora_url,
-                "lora_scale": args.lora_scale,
-                "output_dir": str(args.output_dir.resolve()),
-                "dry_run": args.dry_run,
-            },
-            indent=2,
-        )
-    )
-
-    if args.dry_run:
-        print(json.dumps(request_arguments, indent=2))
-        return 0
-
-    output_dir = args.output_dir.resolve()
-    output_dir.mkdir(parents=True, exist_ok=True)
-    started_at = time.time()
-    result, request_id = run_with_fal(args.endpoint, request_arguments)
-    elapsed = time.time() - started_at
-
-    images = result.get("images", [])
-    if not isinstance(images, list):
-        print("fal response did not contain an images list.", file=sys.stderr)
-        return 1
-
-    downloaded = download_images(
-        [image for image in images if isinstance(image, dict)],
-        output_dir,
-        args.output_path,
-        args.output_format,
-    )
-    result_path = output_dir / "fal-inference-result.json"
-    result_path.write_text(
-        json.dumps(
-            {
-                "request_id": request_id,
-                "elapsed_seconds": elapsed,
-                "arguments": request_arguments,
-                "result": result,
-                "downloaded_images": [str(path) for path in downloaded],
-            },
-            indent=2,
-        ),
-        encoding="utf-8",
-    )
-
-    print(f"Wrote result JSON: {result_path}")
-    for path in downloaded:
-        print(f"Downloaded image: {path}")
-
-    return 0
-
-
-if __name__ == "__main__":
-    raise SystemExit(main())

code/scripts/run_flux2_edit_lora_local.py

@@ -1,479 +0,0 @@
-#!/usr/bin/env python3
-from __future__ import annotations
-
-import argparse
-import importlib.util
-import json
-import re
-import sys
-import time
-from pathlib import Path
-from typing import Any
-
-import torch
-from safetensors.torch import load_file, save_file
-
-
-REPO_ROOT = Path(__file__).resolve().parents[1]
-DEFAULT_LORA = REPO_ROOT / "fal_flux2_edit_lora" / "pytorch_lora_weights.safetensors"
-DEFAULT_OUTPUT_DIR = REPO_ROOT / "outputs" / "local_flux2_edit_inference"
-DEFAULT_MODEL = "diffusers/FLUX.2-dev-bnb-4bit"
-DEFAULT_PROMPT = (
-    "Transform this photorealistic image into the trained radiant aura style: smooth colorful "
-    "gradients, ethereal haze, subtle contour lighting, and a refined cinematic glow. Preserve the "
-    "subject identity, composition, pose, silhouette, camera framing, and important details."
-)
-SUPPORTED_IMAGE_SUFFIXES = {".avif", ".bmp", ".jpeg", ".jpg", ".png", ".webp"}
-
-
-def parse_args() -> argparse.Namespace:
-    parser = argparse.ArgumentParser(
-        description="Run local FLUX.2 image editing with the fal-trained LoRA, or validate it offline."
-    )
-    parser.add_argument(
-        "input_dir",
-        nargs="?",
-        type=Path,
-        help="Directory containing photorealistic input images.",
-    )
-    parser.add_argument("--prompt", default=DEFAULT_PROMPT, help="Edit prompt.")
-    parser.add_argument("--lora", type=Path, default=DEFAULT_LORA, help="Input LoRA safetensors file.")
-    parser.add_argument(
-        "--converted-lora",
-        type=Path,
-        default=None,
-        help="Optional path for a converted diffusers-format LoRA safetensors file.",
-    )
-    parser.add_argument(
-        "--model",
-        default=DEFAULT_MODEL,
-        help="Local path or Hugging Face model id. Defaults to the 4-bit FLUX.2-dev diffusers repo.",
-    )
-    parser.add_argument("--output-dir", type=Path, default=DEFAULT_OUTPUT_DIR)
-    parser.add_argument("--output-path", type=Path, default=None)
-    parser.add_argument(
-        "--batch-size",
-        type=int,
-        default=1,
-        help=(
-            "Number of input images to edit per pipeline call. Keep this low on <24GB VRAM; "
-            "try 2 first, then increase if memory allows."
-        ),
-    )
-    parser.add_argument("--height", type=int, default=1024)
-    parser.add_argument("--width", type=int, default=1024)
-    parser.add_argument("--num-inference-steps", type=int, default=28)
-    parser.add_argument("--guidance-scale", type=float, default=2.5)
-    parser.add_argument("--lora-scale", type=float, default=1.0)
-    parser.add_argument("--seed", type=int, default=None)
-    parser.add_argument(
-        "--torch-dtype",
-        choices=("auto", "float32", "float16", "bfloat16"),
-        default="bfloat16",
-        help="Pipeline dtype. Use bfloat16 on modern NVIDIA GPUs.",
-    )
-    parser.add_argument(
-        "--device",
-        default=None,
-        help="Torch device. Defaults to cuda if available, otherwise cpu.",
-    )
-    parser.add_argument(
-        "--device-map",
-        default=None,
-        help='Optional diffusers/accelerate device map, for example "balanced".',
-    )
-    parser.add_argument(
-        "--local-files-only",
-        action="store_true",
-        help="Do not download model files from Hugging Face.",
-    )
-    parser.add_argument(
-        "--check-only",
-        action="store_true",
-        help="Validate/convert LoRA against the default Flux2Transformer2DModel shape without loading the base model.",
-    )
-    return parser.parse_args()
-
-
-def dtype_from_arg(value: str) -> torch.dtype | str:
-    if value == "auto":
-        return "auto"
-    return {
-        "float32": torch.float32,
-        "float16": torch.float16,
-        "bfloat16": torch.bfloat16,
-    }[value]
-
-
-def require_module(import_name: str, install_name: str | None = None) -> None:
-    if importlib.util.find_spec(import_name) is None:
-        package = install_name or import_name
-        raise RuntimeError(f"Missing required package `{package}`. Install it with `uv add {package}`.")
-
-
-def uses_4bit_model(model: str) -> bool:
-    return "bnb-4bit" in model.lower() or "4bit" in model.lower()
-
-
-def preflight_environment(args: argparse.Namespace) -> None:
-    if args.check_only:
-        return
-
-    require_module("google.protobuf", "protobuf")
-
-    device_name = args.device or ("cuda" if torch.cuda.is_available() else "cpu")
-    if uses_4bit_model(args.model):
-        require_module("bitsandbytes")
-        if device_name == "cpu" or not torch.cuda.is_available():
-            raise RuntimeError(
-                "The 4-bit FLUX.2 model needs a CUDA GPU with bitsandbytes. "
-                "This environment does not expose CUDA to PyTorch."
-            )
-def convert_fal_key(key: str, tensor: torch.Tensor) -> dict[str, torch.Tensor]:
-    prefix = "base_model.model."
-    if not key.startswith(prefix):
-        return {key: tensor}
-
-    body = key.removeprefix(prefix)
-    suffix = ".lora_A.weight" if body.endswith(".lora_A.weight") else ".lora_B.weight"
-    base = body.removesuffix(suffix)
-
-    simple_map = {
-        "img_in": "x_embedder",
-        "txt_in": "context_embedder",
-        "time_in.in_layer": "time_guidance_embed.timestep_embedder.linear_1",
-        "time_in.out_layer": "time_guidance_embed.timestep_embedder.linear_2",
-        "guidance_in.in_layer": "time_guidance_embed.guidance_embedder.linear_1",
-        "guidance_in.out_layer": "time_guidance_embed.guidance_embedder.linear_2",
-        "double_stream_modulation_img.lin": "double_stream_modulation_img.linear",
-        "double_stream_modulation_txt.lin": "double_stream_modulation_txt.linear",
-        "single_stream_modulation.lin": "single_stream_modulation.linear",
-        "final_layer.linear": "proj_out",
-    }
-    if base in simple_map:
-        return {f"transformer.{simple_map[base]}{suffix}": tensor}
-
-    double_match = re.fullmatch(r"double_blocks\.(\d+)\.(img_attn|txt_attn)\.(qkv|proj)", base)
-    if double_match:
-        block, stream, layer = double_match.groups()
-        stem = f"transformer.transformer_blocks.{block}.attn"
-        if layer == "proj":
-            target = "to_out.0" if stream == "img_attn" else "to_add_out"
-            return {f"{stem}.{target}{suffix}": tensor}
-
-        targets = (
-            ("to_q", "to_k", "to_v")
-            if stream == "img_attn"
-            else ("add_q_proj", "add_k_proj", "add_v_proj")
-        )
-        if suffix == ".lora_A.weight":
-            return {f"{stem}.{target}{suffix}": tensor.clone() for target in targets}
-
-        chunks = tensor.chunk(3, dim=0)
-        return {f"{stem}.{target}{suffix}": chunk.contiguous() for target, chunk in zip(targets, chunks)}
-
-    single_match = re.fullmatch(r"single_blocks\.(\d+)\.(linear1|linear2)", base)
-    if single_match:
-        block, layer = single_match.groups()
-        target = "to_qkv_mlp_proj" if layer == "linear1" else "to_out"
-        return {f"transformer.single_transformer_blocks.{block}.attn.{target}{suffix}": tensor}
-
-    raise ValueError(f"Unsupported fal LoRA key: {key}")
-
-
-def convert_fal_lora_to_diffusers(input_path: Path, output_path: Path) -> dict[str, Any]:
-    state = load_file(input_path)
-    converted: dict[str, torch.Tensor] = {}
-    for key, tensor in state.items():
-        for new_key, new_tensor in convert_fal_key(key, tensor).items():
-            if new_key in converted:
-                raise ValueError(f"Duplicate converted LoRA key: {new_key}")
-            converted[new_key] = new_tensor
-
-    output_path.parent.mkdir(parents=True, exist_ok=True)
-    save_file(converted, output_path, metadata={"format": "pt"})
-    return {
-        "input_keys": len(state),
-        "converted_keys": len(converted),
-        "input_bytes": input_path.stat().st_size,
-        "converted_bytes": output_path.stat().st_size,
-    }
-
-
-def expected_linear_shapes() -> dict[str, tuple[int, ...]]:
-    from accelerate import init_empty_weights
-    from diffusers import Flux2Transformer2DModel
-
-    with init_empty_weights():
-        model = Flux2Transformer2DModel()
-    return {
-        f"transformer.{name}": tuple(module.weight.shape)
-        for name, module in model.named_modules()
-        if module.__class__.__name__ == "Linear"
-    }
-
-
-def validate_converted_lora(path: Path) -> dict[str, Any]:
-    state = load_file(path)
-    shapes = expected_linear_shapes()
-    missing_targets = []
-    bad_shapes = []
-    ranks = set()
-
-    for key, tensor in state.items():
-        if key.endswith(".lora_A.weight"):
-            target = key.removesuffix(".lora_A.weight")
-            ranks.add(tensor.shape[0])
-            expected = shapes.get(target)
-            if expected is None:
-                missing_targets.append(target)
-            elif tuple(tensor.shape[1:]) != (expected[1],):
-                bad_shapes.append((key, tuple(tensor.shape), expected))
-        elif key.endswith(".lora_B.weight"):
-            target = key.removesuffix(".lora_B.weight")
-            ranks.add(tensor.shape[1])
-            expected = shapes.get(target)
-            if expected is None:
-                missing_targets.append(target)
-            elif tuple(tensor.shape[:1]) != (expected[0],):
-                bad_shapes.append((key, tuple(tensor.shape), expected))
-        else:
-            missing_targets.append(key)
-
-    return {
-        "keys": len(state),
-        "target_modules": len({key.rsplit(".lora_", 1)[0] for key in state}),
-        "ranks": sorted(ranks),
-        "missing_targets": sorted(set(missing_targets)),
-        "bad_shapes": bad_shapes,
-        "valid": not missing_targets and not bad_shapes,
-    }
-
-
-def load_flux2_pipeline(args: argparse.Namespace, dtype: torch.dtype | str, device_name: str):
-    from diffusers import Flux2Pipeline
-
-    if uses_4bit_model(args.model) and device_name.startswith("cuda"):
-        from diffusers import AutoModel
-        from transformers import Mistral3ForConditionalGeneration
-
-        print("Loading 4-bit FLUX.2 with local text encoder on CPU and model CPU offload.", flush=True)
-        text_encoder = Mistral3ForConditionalGeneration.from_pretrained(
-            args.model,
-            subfolder="text_encoder",
-            torch_dtype=dtype,
-            device_map="cpu",
-            local_files_only=args.local_files_only,
-        )
-        transformer = AutoModel.from_pretrained(
-            args.model,
-            subfolder="transformer",
-            torch_dtype=dtype,
-            device_map="cpu",
-            local_files_only=args.local_files_only,
-        )
-        pipe = Flux2Pipeline.from_pretrained(
-            args.model,
-            text_encoder=text_encoder,
-            transformer=transformer,
-            torch_dtype=dtype,
-            local_files_only=args.local_files_only,
-        )
-        pipe.enable_model_cpu_offload()
-        return pipe
-
-    load_kwargs: dict[str, Any] = {
-        "torch_dtype": dtype,
-        "local_files_only": args.local_files_only,
-    }
-    if args.device_map is not None:
-        load_kwargs["device_map"] = args.device_map
-    elif device_name.startswith("cuda"):
-        load_kwargs["device_map"] = device_name
-
-    pipe = Flux2Pipeline.from_pretrained(args.model, **load_kwargs)
-    if "device_map" not in load_kwargs:
-        pipe.to(device_name)
-    return pipe
-
-
-def batched(values: list[Path], batch_size: int) -> list[list[Path]]:
-    return [values[index : index + batch_size] for index in range(0, len(values), batch_size)]
-
-
-def discover_input_images(input_dir: Path) -> list[Path]:
-    return sorted(
-        (
-            path
-            for path in input_dir.iterdir()
-            if path.is_file() and path.suffix.lower() in SUPPORTED_IMAGE_SUFFIXES
-        ),
-        key=lambda path: path.name.lower(),
-    )
-
-
-def output_paths_for_inputs(args: argparse.Namespace, input_images: list[Path]) -> list[Path]:
-    if args.output_path is None:
-        return [
-            args.output_dir / f"{input_path.stem}-flux2-local-stylized.png"
-            for input_path in input_images
-        ]
-
-    if args.output_path.suffix:
-        stem = args.output_path.with_suffix("")
-        suffix = args.output_path.suffix
-        return [
-            stem.with_name(f"{stem.name}-{index:04d}{suffix}")
-            for index, _input_path in enumerate(input_images, start=1)
-        ]
-
-    return [
-        args.output_path / f"{input_path.stem}-flux2-local-stylized.png"
-        for input_path in input_images
-    ]
-
-
-def generators_for_batch(
-    seed: int | None,
-    device_name: str,
-    *,
-    start_index: int,
-    batch_size: int,
-) -> torch.Generator | list[torch.Generator] | None:
-    if seed is None:
-        return None
-    if batch_size == 1:
-        return torch.Generator(device=device_name).manual_seed(seed + start_index)
-    return [
-        torch.Generator(device=device_name).manual_seed(seed + start_index + index)
-        for index in range(batch_size)
-    ]
-
-
-def run_inference(args: argparse.Namespace, lora_path: Path) -> list[Path]:
-    from diffusers.utils import load_image
-
-    device_name = args.device or ("cuda" if torch.cuda.is_available() else "cpu")
-    dtype = dtype_from_arg(args.torch_dtype)
-    input_paths = discover_input_images(args.input_dir.expanduser().resolve())
-    output_paths = output_paths_for_inputs(args, input_paths)
-
-    print(
-        f"Found {len(input_paths)} input image(s). Processing in batches of {args.batch_size}.",
-        flush=True,
-    )
-    print("Using text encoder mode: local", flush=True)
-    pipe = load_flux2_pipeline(args, dtype, device_name)
-    pipe.load_lora_weights(str(lora_path), adapter_name="aura")
-    pipe.set_adapters(["aura"], adapter_weights=[args.lora_scale])
-
-    for start_index, batch_paths in enumerate(batched(input_paths, args.batch_size)):
-        batch_offset = start_index * args.batch_size
-        input_images = [load_image(str(input_path)) for input_path in batch_paths]
-        image_arg: Any = input_images[0] if len(input_images) == 1 else input_images
-        prompt_arg: Any = args.prompt if len(input_images) == 1 else [args.prompt] * len(batch_paths)
-        call_kwargs: dict[str, Any] = {
-            "image": image_arg,
-            "height": args.height,
-            "width": args.width,
-            "num_inference_steps": args.num_inference_steps,
-            "guidance_scale": args.guidance_scale,
-            "generator": generators_for_batch(
-                args.seed,
-                device_name,
-                start_index=batch_offset,
-                batch_size=len(batch_paths),
-            ),
-            "prompt": prompt_arg,
-        }
-
-        images = pipe(**call_kwargs).images
-        if len(images) != len(batch_paths):
-            raise RuntimeError(f"Expected {len(batch_paths)} outputs from pipeline, received {len(images)}.")
-
-        for image, output_path in zip(images, output_paths[batch_offset : batch_offset + len(images)]):
-            output_path.parent.mkdir(parents=True, exist_ok=True)
-            image.save(output_path)
-
-    return output_paths
-
-
-def main() -> int:
-    args = parse_args()
-    lora_path = args.lora.expanduser().resolve()
-    if not lora_path.exists():
-        print(f"LoRA file does not exist: {lora_path}", file=sys.stderr)
-        return 1
-    if not args.check_only:
-        if args.input_dir is None:
-            print("input_dir is required unless --check-only is set.", file=sys.stderr)
-            return 1
-        if args.batch_size < 1:
-            print("--batch-size must be at least 1.", file=sys.stderr)
-            return 1
-        input_dir = args.input_dir.expanduser().resolve()
-        if not input_dir.exists():
-            print(f"Input directory does not exist: {input_dir}", file=sys.stderr)
-            return 1
-        if not input_dir.is_dir():
-            print(f"Input path is not a directory: {input_dir}", file=sys.stderr)
-            return 1
-        input_images = discover_input_images(input_dir)
-        if not input_images:
-            print(
-                f"No supported images found in {input_dir}. "
-                f"Supported extensions: {', '.join(sorted(SUPPORTED_IMAGE_SUFFIXES))}.",
-                file=sys.stderr,
-            )
-            return 1
-
-    output_dir = args.output_dir.expanduser().resolve()
-    output_dir.mkdir(parents=True, exist_ok=True)
-    converted_path = (
-        args.converted_lora.expanduser().resolve()
-        if args.converted_lora
-        else output_dir / "pytorch_lora_weights.diffusers.safetensors"
-    )
-
-    try:
-        conversion = convert_fal_lora_to_diffusers(lora_path, converted_path)
-        validation = validate_converted_lora(converted_path)
-    except Exception as exc:
-        print(str(exc), file=sys.stderr)
-        return 1
-
-    report: dict[str, Any] = {
-        "model": args.model,
-        "text_encoder_mode": "local",
-        "lora": str(lora_path),
-        "converted_lora": str(converted_path),
-        "conversion": conversion,
-        "validation": validation,
-    }
-    print(json.dumps(report, indent=2, default=str))
-
-    if not validation["valid"]:
-        print("Converted LoRA did not validate against Flux2Transformer2DModel.", file=sys.stderr)
-        return 1
-    if args.check_only:
-        return 0
-    try:
-        preflight_environment(args)
-    except RuntimeError as exc:
-        print(str(exc), file=sys.stderr)
-        return 1
-
-    started_at = time.time()
-    try:
-        image_paths = run_inference(args, converted_path)
-    except Exception as exc:
-        print(f"Local inference failed after {time.time() - started_at:.1f}s: {exc}", file=sys.stderr)
-        return 1
-
-    for image_path in image_paths:
-        print(f"Saved local FLUX.2 edit output: {image_path}")
-    return 0
-
-
-if __name__ == "__main__":
-    raise SystemExit(main())

code/scripts/train_dreambooth_lora_flux_lowmem.py

@@ -1,8 +0,0 @@
-#!/usr/bin/env python
-"""Compatibility wrapper for the packaged FLUX low-memory trainer."""
-
-from lora.trainers.flux_lowmem import main, parse_args
-
-
-if __name__ == "__main__":
-    main(parse_args())

code/scripts/train_flux2_edit_lora_fal.py

@@ -1,326 +0,0 @@
-#!/usr/bin/env python3
-from __future__ import annotations
-
-import argparse
-import json
-import os
-import re
-import sys
-import tempfile
-import time
-import urllib.parse
-import urllib.request
-import zipfile
-from pathlib import Path
-from typing import Any
-
-
-REPO_ROOT = Path(__file__).resolve().parents[1]
-DEFAULT_TRAIN_DIR = REPO_ROOT / "training_data" / "flux_aura_style" / "train"
-DEFAULT_OUTPUT_DIR = REPO_ROOT / "outputs" / "fal_flux2_edit_lora"
-DEFAULT_ENDPOINT = "fal-ai/flux-2-trainer-v2/edit"
-DEFAULT_CAPTION = (
-    "Apply a radiant aura lighting style with smooth colorful gradients, ethereal haze, subtle "
-    "contour lighting, and a refined cinematic glow while preserving the subject and composition."
-)
-
-
-def parse_args() -> argparse.Namespace:
-    parser = argparse.ArgumentParser(
-        description="Package paired image-edit data and submit a FLUX.2 edit LoRA training job to fal."
-    )
-    parser.add_argument(
-        "--train-dir",
-        type=Path,
-        default=DEFAULT_TRAIN_DIR,
-        help="Prepared train directory containing metadata.jsonl and target images.",
-    )
-    parser.add_argument(
-        "--output-dir",
-        type=Path,
-        default=DEFAULT_OUTPUT_DIR,
-        help="Directory for the packaged zip, result JSON, and downloaded LoRA files.",
-    )
-    parser.add_argument(
-        "--zip-path",
-        type=Path,
-        default=None,
-        help="Optional explicit path for the generated training zip.",
-    )
-    parser.add_argument(
-        "--default-caption",
-        default=DEFAULT_CAPTION,
-        help="Fallback edit instruction passed to fal if any pair has no prompt text file.",
-    )
-    parser.add_argument(
-        "--steps",
-        type=int,
-        default=1000,
-        help="Training steps. fal accepts 100 to 10000 in increments of 100.",
-    )
-    parser.add_argument(
-        "--learning-rate",
-        type=float,
-        default=0.00005,
-        help="LoRA learning rate.",
-    )
-    parser.add_argument(
-        "--output-lora-format",
-        choices=("fal", "comfy"),
-        default="fal",
-        help="Output weight naming format.",
-    )
-    parser.add_argument(
-        "--endpoint",
-        default=DEFAULT_ENDPOINT,
-        help="fal endpoint id.",
-    )
-    parser.add_argument(
-        "--limit",
-        type=int,
-        default=None,
-        help="Maximum number of paired examples to package.",
-    )
-    parser.add_argument(
-        "--start-pair",
-        type=int,
-        default=None,
-        help="First pair number to include, inclusive.",
-    )
-    parser.add_argument(
-        "--end-pair",
-        type=int,
-        default=None,
-        help="Last pair number to include, inclusive.",
-    )
-    parser.add_argument(
-        "--no-download",
-        action="store_true",
-        help="Do not download result files after training completes.",
-    )
-    parser.add_argument(
-        "--package-only",
-        action="store_true",
-        help="Only create the zip; do not upload or start training.",
-    )
-    parser.add_argument(
-        "--dry-run",
-        action="store_true",
-        help="Print selected pairs without creating a zip or calling fal.",
-    )
-    return parser.parse_args()
-
-
-def pair_number(path: str | Path) -> int:
-    match = re.fullmatch(r"pair-(\d+)\.[^.]+", Path(path).name)
-    if not match:
-        raise ValueError(f"Not a pair file name: {path}")
-    return int(match.group(1))
-
-
-def read_metadata(metadata_path: Path) -> list[dict[str, Any]]:
-    with metadata_path.open("r", encoding="utf-8") as fh:
-        return [json.loads(line) for line in fh if line.strip()]
-
-
-def selected_pair_rows(args: argparse.Namespace) -> list[dict[str, Any]]:
-    metadata_path = args.train_dir / "metadata.jsonl"
-    if not metadata_path.exists():
-        raise FileNotFoundError(f"Metadata file does not exist: {metadata_path}")
-
-    rows = []
-    for row in read_metadata(metadata_path):
-        if row.get("kind") != "paired":
-            continue
-        file_name = row.get("file_name")
-        conditioning_path = row.get("conditioning_path")
-        if not isinstance(file_name, str) or not isinstance(conditioning_path, str):
-            continue
-        number = pair_number(file_name)
-        if args.start_pair is not None and number < args.start_pair:
-            continue
-        if args.end_pair is not None and number > args.end_pair:
-            continue
-        rows.append(row)
-
-    rows.sort(key=lambda row: pair_number(row["file_name"]))
-    if args.limit is not None:
-        rows = rows[: args.limit]
-    return rows
-
-
-def resolve_conditioning_path(train_dir: Path, value: str) -> Path:
-    path = Path(value)
-    if path.is_absolute():
-        return path
-    return (train_dir / path).resolve()
-
-
-def validate_rows(train_dir: Path, rows: list[dict[str, Any]]) -> None:
-    for row in rows:
-        target_path = train_dir / row["file_name"]
-        conditioning_path = resolve_conditioning_path(train_dir, row["conditioning_path"])
-        if not target_path.exists():
-            raise FileNotFoundError(f"Missing target image: {target_path}")
-        if not conditioning_path.exists():
-            raise FileNotFoundError(f"Missing conditioning image: {conditioning_path}")
-
-
-def build_zip(train_dir: Path, rows: list[dict[str, Any]], zip_path: Path) -> None:
-    zip_path.parent.mkdir(parents=True, exist_ok=True)
-    with tempfile.NamedTemporaryFile(
-        prefix=f".{zip_path.name}.",
-        suffix=".tmp",
-        dir=zip_path.parent,
-        delete=False,
-    ) as tmp:
-        tmp_path = Path(tmp.name)
-
-    try:
-        with zipfile.ZipFile(tmp_path, "w", compression=zipfile.ZIP_DEFLATED) as archive:
-            for row in rows:
-                target_path = train_dir / row["file_name"]
-                conditioning_path = resolve_conditioning_path(train_dir, row["conditioning_path"])
-                stem = Path(row["file_name"]).stem
-
-                archive.write(conditioning_path, f"{stem}_start{conditioning_path.suffix.lower()}")
-                archive.write(target_path, f"{stem}_end{target_path.suffix.lower()}")
-
-                prompt = row.get("prompt")
-                if isinstance(prompt, str) and prompt.strip():
-                    archive.writestr(f"{stem}.txt", prompt.strip())
-
-        tmp_path.replace(zip_path)
-    except Exception:
-        tmp_path.unlink(missing_ok=True)
-        raise
-
-
-def on_queue_update(update: object) -> None:
-    try:
-        import fal_client
-    except ImportError:
-        return
-
-    if isinstance(update, fal_client.InProgress) and update.logs:
-        for log in update.logs:
-            message = log.get("message")
-            if message:
-                print(message, flush=True)
-
-
-def train_with_fal(args: argparse.Namespace, zip_path: Path) -> tuple[dict[str, Any], str | None]:
-    try:
-        import fal_client
-    except ImportError as exc:
-        raise RuntimeError(
-            "fal-client is not installed. Run `uv sync` after this script was added, or install "
-            "it with `uv pip install fal-client`."
-        ) from exc
-
-    image_data_url = fal_client.upload_file(str(zip_path)) # type: ignore
-    print(f"Uploaded dataset zip: {image_data_url}", flush=True)
-
-    result = fal_client.subscribe(
-        args.endpoint,
-        arguments={
-            "image_data_url": image_data_url,
-            "steps": args.steps,
-            "learning_rate": args.learning_rate,
-            "default_caption": args.default_caption,
-            "output_lora_format": args.output_lora_format,
-        },
-        with_logs=True,
-        on_queue_update=on_queue_update,
-    )
-    if hasattr(result, "data"):
-        return dict(result.data), getattr(result, "request_id", None)
-    return dict(result), None
-
-
-def download_file(file_info: dict[str, Any], output_dir: Path) -> Path | None:
-    url = file_info.get("url")
-    if not isinstance(url, str) or not url:
-        return None
-
-    file_name = file_info.get("file_name")
-    if not isinstance(file_name, str) or not file_name:
-        file_name = Path(urllib.parse.urlparse(url).path).name or "downloaded-file"
-
-    output_path = output_dir / file_name
-    urllib.request.urlretrieve(url, output_path)
-    return output_path
-
-
-def main() -> int:
-    args = parse_args()
-    train_dir = args.train_dir.resolve()
-    output_dir = args.output_dir.resolve()
-    zip_path = (args.zip_path or (output_dir / "flux2-edit-lora-pairs.zip")).resolve()
-
-    if args.steps < 100 or args.steps > 10000 or args.steps % 100 != 0:
-        print("--steps must be between 100 and 10000, in increments of 100.", file=sys.stderr)
-        return 1
-    if not args.package_only and not args.dry_run and not os.environ.get("FAL_KEY"):
-        print("Missing fal key. Set FAL_KEY before submitting training.", file=sys.stderr)
-        return 1
-
-    rows = selected_pair_rows(args)
-    if not rows:
-        print("No paired examples selected.", file=sys.stderr)
-        return 1
-    validate_rows(train_dir, rows)
-
-    print(
-        json.dumps(
-            {
-                "pairs": len(rows),
-                "first_pair": rows[0]["file_name"],
-                "last_pair": rows[-1]["file_name"],
-                "zip_path": str(zip_path),
-                "endpoint": args.endpoint,
-                "steps": args.steps,
-                "learning_rate": args.learning_rate,
-                "package_only": args.package_only,
-                "dry_run": args.dry_run,
-            },
-            indent=2,
-        )
-    )
-
-    if args.dry_run:
-        for row in rows:
-            print(f"{row['conditioning_path']} -> {row['file_name']}")
-        return 0
-
-    build_zip(train_dir, rows, zip_path)
-    print(f"Wrote dataset zip: {zip_path} ({zip_path.stat().st_size:,} bytes)")
-
-    if args.package_only:
-        return 0
-
-    output_dir.mkdir(parents=True, exist_ok=True)
-    started_at = time.time()
-    result, request_id = train_with_fal(args, zip_path)
-    elapsed = time.time() - started_at
-
-    result_path = output_dir / "fal-training-result.json"
-    result_path.write_text(
-        json.dumps({"request_id": request_id, "elapsed_seconds": elapsed, "result": result}, indent=2),
-        encoding="utf-8",
-    )
-    print(f"Wrote result JSON: {result_path}")
-
-    if not args.no_download:
-        for key in ("diffusers_lora_file", "config_file"):
-            value = result.get(key)
-            if isinstance(value, dict):
-                downloaded = download_file(value, output_dir)
-                if downloaded is not None:
-                    print(f"Downloaded {key}: {downloaded}")
-
-    return 0
-
-
-if __name__ == "__main__":
-    raise SystemExit(main())

code/src/lora/__init__.py

@@ -1,3 +0,0 @@
-__all__ = ["__version__"]
-
-__version__ = "0.1.0"

code/src/lora/cli.py

@@ -1,442 +0,0 @@
-from __future__ import annotations
-
-import argparse
-import json
-import os
-import re
-import shutil
-import subprocess
-import sys
-import textwrap
-import tomllib
-from dataclasses import dataclass
-from importlib.util import find_spec
-from pathlib import Path
-from typing import Any, Protocol, cast
-
-
-REPO_ROOT = Path(__file__).resolve().parents[2]
-DEFAULT_CONFIG = REPO_ROOT / "configs" / "flux_lora.toml"
-
-
-class CommandFunc(Protocol):
-    def __call__(self, args: argparse.Namespace) -> int: ...
-
-
-class PipelineResult(Protocol):
-    images: list[Any]
-
-
-@dataclass
-class PreparedItem:
-    image_source: Path
-    output_name: str
-    prompt: str
-    conditioning_source: Path | None = None
-    kind: str = "label"
-
-
-def load_config(path: Path) -> dict[str, Any]:
-    with path.open("rb") as fh:
-        return tomllib.load(fh)
-
-
-def parse_bool(value: bool) -> bool:
-    return bool(value)
-
-
-def normalize_subject(value: str) -> str:
-    return value.lower().replace("-", " ").replace("_", " ").strip()
-
-
-def build_prompt(subject: str, prompt_cfg: dict[str, Any]) -> str:
-    return prompt_cfg["prompt_template"].format(subject=subject)
-
-
-def convert_image(source: Path, target: Path) -> None:
-    from PIL import Image, ImageOps
-    from PIL.Image import Image as PILImage
-
-    with Image.open(source) as image:
-        transposed = cast(PILImage, ImageOps.exif_transpose(image))
-        rgb_image = transposed.convert("RGB")
-        rgb_image.save(target, format="PNG", optimize=True)
-
-
-def prepare_dataset(args: argparse.Namespace) -> int:
-    if find_spec("PIL") is None:
-        print("Pillow is required. Install project dependencies first.", file=sys.stderr)
-        return 1
-
-    config = load_config(Path(args.config))
-    dataset_cfg = config["dataset"]
-    prompt_cfg = config["prompts"]
-
-    source_dir = REPO_ROOT / dataset_cfg["source_dir"]
-    output_dir = REPO_ROOT / dataset_cfg["prepared_dir"]
-    image_dir = output_dir / "train"
-    conditioning_dir = output_dir / "conditioning"
-    metadata_path = image_dir / "metadata.jsonl"
-
-    if output_dir.exists():
-        shutil.rmtree(output_dir)
-
-    output_dir.mkdir(parents=True, exist_ok=True)
-    image_dir.mkdir(parents=True, exist_ok=True)
-    conditioning_dir.mkdir(parents=True, exist_ok=True)
-
-    supported = {".jpg", ".jpeg", ".png", ".webp", ".avif"}
-    files = sorted(
-        path for path in source_dir.iterdir() if path.is_file() and path.suffix.lower() in supported
-    )
-    if not files:
-        print(f"No supported images found in {source_dir}", file=sys.stderr)
-        return 1
-
-    source_pattern = re.compile(r"^(?P<index>\d+)$")
-    output_pattern = re.compile(r"^(?P<index>\d+)-output$")
-    pairs: dict[str, dict[str, Path]] = {}
-    label_images: list[Path] = []
-
-    for path in files:
-        stem = path.stem
-        source_match = source_pattern.match(stem)
-        output_match = output_pattern.match(stem)
-        if output_match:
-            pairs.setdefault(output_match.group("index"), {})["output"] = path
-            continue
-        if source_match:
-            pairs.setdefault(source_match.group("index"), {})["source"] = path
-            continue
-        label_images.append(path)
-
-    items: list[PreparedItem] = []
-    pair_subject = prompt_cfg["paired_subject"]
-    for pair_index in sorted(pairs, key=lambda value: int(value)):
-        pair = pairs[pair_index]
-        source = pair.get("source")
-        output = pair.get("output")
-        if source is None or output is None:
-            missing = "source" if source is None else "output"
-            print(f"Skipping pair {pair_index}: missing {missing} image.", file=sys.stderr)
-            continue
-
-        items.append(
-            PreparedItem(
-                image_source=output,
-                conditioning_source=source,
-                output_name=f"pair-{int(pair_index):03d}.png",
-                prompt=build_prompt(pair_subject, prompt_cfg),
-                kind="paired",
-            )
-        )
-
-    for label_index, image in enumerate(sorted(label_images), start=1):
-        items.append(
-            PreparedItem(
-                image_source=image,
-                output_name=f"label-{label_index:03d}.png",
-                prompt=build_prompt(normalize_subject(image.stem), prompt_cfg),
-                kind="label",
-            )
-        )
-
-    for item in items:
-        target = image_dir / item.output_name
-        convert_image(item.image_source, target)
-        if item.conditioning_source is not None:
-            convert_image(item.conditioning_source, conditioning_dir / item.output_name)
-
-    with metadata_path.open("w", encoding="utf-8") as fh:
-        for item in items:
-            row = {
-                "file_name": item.output_name,
-                "prompt": item.prompt,
-                "kind": item.kind,
-            }
-            if item.conditioning_source is not None:
-                row["conditioning_path"] = f"../conditioning/{item.output_name}"
-            fh.write(json.dumps(row) + "\n")
-
-    summary = {
-        "prepared_count": len(items),
-        "paired_count": sum(1 for item in items if item.kind == "paired"),
-        "label_count": sum(1 for item in items if item.kind == "label"),
-        "prepared_dir": str(output_dir),
-    }
-    print(json.dumps(summary, indent=2))
-    return 0
-
-
-def run_checked(cmd: list[str], cwd: Path | None = None) -> None:
-    subprocess.run(cmd, cwd=cwd, check=True)
-
-
-def missing_modules(module_names: list[str]) -> list[str]:
-    return [module_name for module_name in module_names if find_spec(module_name) is None]
-
-
-def ensure_diffusers_checkout(cache_dir: Path, ref: str) -> Path:
-    repo_dir = cache_dir / "diffusers"
-    if not repo_dir.exists():
-        run_checked(["git", "clone", "https://github.com/huggingface/diffusers.git", str(repo_dir)])
-    run_checked(["git", "fetch", "origin"], cwd=repo_dir)
-    run_checked(["git", "checkout", ref], cwd=repo_dir)
-    return repo_dir
-
-
-def append_flag(command: list[str], name: str, value: Any) -> None:
-    if value is None:
-        return
-    if isinstance(value, bool):
-        if value:
-            command.append(name)
-        return
-    command.extend([name, str(value)])
-
-
-def train(args: argparse.Namespace) -> int:
-    config = load_config(Path(args.config))
-    train_cfg = config["training"]
-    prompt_cfg = config["prompts"]
-
-    missing = missing_modules(
-        [
-            "accelerate",
-            "datasets",
-            "diffusers",
-            "ftfy",
-            "hf_transfer",
-            "peft",
-            "sentencepiece",
-            "tensorboard",
-            "torch",
-            "torchvision",
-            "transformers",
-        ]
-    )
-    if missing:
-        print(
-            "Training dependencies are missing: "
-            + ", ".join(missing)
-            + ". Run `uv sync` or `lora install` before `lora train`.",
-            file=sys.stderr,
-        )
-        return 1
-
-    prepared_dir = REPO_ROOT / config["dataset"]["prepared_dir"]
-    if not prepared_dir.exists():
-        print(
-            "Prepared dataset is missing. Run `lora prepare-dataset` first.",
-            file=sys.stderr,
-        )
-        return 1
-
-    command = [
-        "accelerate",
-        "launch",
-        "-m",
-        train_cfg.get("training_module", "lora.trainers.flux_lowmem"),
-        "--pretrained_model_name_or_path",
-        train_cfg["model_name"],
-        "--dataset_name",
-        str(prepared_dir),
-        "--caption_column",
-        "prompt",
-        "--instance_prompt",
-        prompt_cfg["instance_prompt"],
-        "--output_dir",
-        str(REPO_ROOT / train_cfg["output_dir"]),
-    ]
-
-    flags = {
-        "--mixed_precision": train_cfg["mixed_precision"],
-        "--resolution": train_cfg["resolution"],
-        "--train_batch_size": train_cfg["train_batch_size"],
-        "--gradient_accumulation_steps": train_cfg["gradient_accumulation_steps"],
-        "--optimizer": train_cfg["optimizer"],
-        "--learning_rate": train_cfg["learning_rate"],
-        "--lr_scheduler": train_cfg["lr_scheduler"],
-        "--lr_warmup_steps": train_cfg["lr_warmup_steps"],
-        "--max_train_steps": train_cfg["max_train_steps"],
-        "--rank": train_cfg["rank"],
-        "--lora_alpha": train_cfg["lora_alpha"],
-        "--validation_prompt": train_cfg.get("validation_prompt") or None,
-        "--validation_epochs": train_cfg["validation_epochs"],
-        "--num_validation_images": train_cfg["num_validation_images"],
-        "--seed": train_cfg["seed"],
-        "--report_to": train_cfg["report_to"],
-        "--repeats": train_cfg["repeats"],
-        "--max_sequence_length": train_cfg["max_sequence_length"],
-        "--dataloader_num_workers": train_cfg.get("dataloader_num_workers"),
-    }
-    for flag_name, value in flags.items():
-        append_flag(command, flag_name, value)
-
-    if parse_bool(train_cfg.get("gradient_checkpointing", False)):
-        command.append("--gradient_checkpointing")
-    if parse_bool(train_cfg.get("cache_latents", False)):
-        command.append("--cache_latents")
-    if parse_bool(train_cfg.get("use_8bit_adam", False)):
-        command.append("--use_8bit_adam")
-    if parse_bool(train_cfg.get("push_to_hub", False)):
-        command.append("--push_to_hub")
-
-    env = os.environ.copy()
-    env.setdefault("HF_HUB_ENABLE_HF_TRANSFER", "1")
-
-    print("Launching training command:\n")
-    print(" ".join(command))
-    print()
-    subprocess.run(command, cwd=REPO_ROOT, env=env, check=True)
-    return 0
-
-
-def infer(args: argparse.Namespace) -> int:
-    try:
-        import diffusers  # pyright: ignore[reportMissingImports]
-        import torch
-    except ImportError:
-        print(
-            "Inference requires diffusers and torch installed in the active environment.",
-            file=sys.stderr,
-        )
-        return 1
-
-    config = load_config(Path(args.config))
-    infer_cfg = config["inference"]
-    train_cfg = config["training"]
-
-    lora_dir = REPO_ROOT / train_cfg["output_dir"]
-    weight_name = infer_cfg["weight_name"]
-    prompt = args.prompt or infer_cfg["prompt"]
-    output_path = REPO_ROOT / infer_cfg["output_path"]
-    output_path.parent.mkdir(parents=True, exist_ok=True)
-
-    pipeline_cls = cast(Any, getattr(diffusers, "DiffusionPipeline"))
-    pipe = cast(
-        Any,
-        pipeline_cls.from_pretrained(
-            train_cfg["model_name"],
-            torch_dtype=getattr(torch, "bfloat16"),
-        ),
-    )
-    pipe.enable_model_cpu_offload()
-    pipe.load_lora_weights(str(lora_dir), weight_name=weight_name)
-
-    result = cast(
-        PipelineResult,
-        pipe(
-            prompt=prompt,
-            height=infer_cfg["height"],
-            width=infer_cfg["width"],
-            guidance_scale=infer_cfg["guidance_scale"],
-            num_inference_steps=infer_cfg["num_inference_steps"],
-            max_sequence_length=train_cfg["max_sequence_length"],
-        ),
-    )
-    image = result.images[0]
-    image.save(output_path)
-    print(json.dumps({"prompt": prompt, "output_path": str(output_path)}, indent=2))
-    return 0
-
-
-def install(args: argparse.Namespace) -> int:
-    config = load_config(Path(args.config))
-    train_cfg = config["training"]
-    cache_dir = REPO_ROOT / ".cache"
-    cache_dir.mkdir(exist_ok=True)
-    diffusers_dir = ensure_diffusers_checkout(cache_dir, train_cfg["diffusers_ref"])
-
-    steps = [
-        [sys.executable, "-m", "pip", "install", "-e", "."],
-        [sys.executable, "-m", "pip", "install", "-e", str(diffusers_dir)],
-        [
-            sys.executable,
-            "-m",
-            "pip",
-            "install",
-            "-r",
-            str(diffusers_dir / "examples" / "dreambooth" / "requirements_flux.txt"),
-        ],
-    ]
-    for step in steps:
-        run_checked(step, cwd=REPO_ROOT)
-
-    note = textwrap.dedent(
-        """
-        Environment bootstrap complete.
-        Next steps:
-        1. Accept the gated model terms for black-forest-labs/FLUX.1-dev on Hugging Face.
-        2. Run `hf auth login`.
-        3. Run `accelerate config default`.
-        4. Run `lora prepare-dataset`.
-        5. Run `lora train`.
-        """
-    ).strip()
-    print(note)
-    return 0
-
-
-def clean(args: argparse.Namespace) -> int:
-    config = load_config(Path(args.config))
-    paths = [
-        REPO_ROOT / config["dataset"]["prepared_dir"],
-        REPO_ROOT / config["training"]["output_dir"],
-        REPO_ROOT / config["inference"]["output_path"],
-    ]
-    for path in paths:
-        if path.is_dir():
-            shutil.rmtree(path)
-        elif path.exists():
-            path.unlink()
-    print("Removed generated dataset, output weights, and sample image.")
-    return 0
-
-
-def build_parser() -> argparse.ArgumentParser:
-    parser = argparse.ArgumentParser(description="FLUX.1-dev LoRA workspace utilities.")
-    parser.add_argument(
-        "--config",
-        default=str(DEFAULT_CONFIG),
-        help="Path to the TOML config file.",
-    )
-
-    subparsers = parser.add_subparsers(dest="command", required=True)
-
-    prepare_parser = subparsers.add_parser(
-        "prepare-dataset", help="Normalize images into a local HF dataset."
-    )
-    prepare_parser.set_defaults(func=prepare_dataset)
-
-    install_parser = subparsers.add_parser(
-        "install", help="Install local and upstream training dependencies."
-    )
-    install_parser.set_defaults(func=install)
-
-    train_parser = subparsers.add_parser(
-        "train", help="Launch the official diffusers FLUX LoRA trainer."
-    )
-    train_parser.set_defaults(func=train)
-
-    infer_parser = subparsers.add_parser(
-        "infer", help="Run a quick inference pass with the trained LoRA."
-    )
-    infer_parser.add_argument("prompt", nargs="?", help="Optional prompt override.")
-    infer_parser.set_defaults(func=infer)
-
-    clean_parser = subparsers.add_parser("clean", help="Remove generated artifacts.")
-    clean_parser.set_defaults(func=clean)
-
-    return parser
-
-
-def main(argv: list[str] | None = None) -> int:
-    parser = build_parser()
-    args = parser.parse_args(argv)
-    try:
-        func = cast(CommandFunc, args.func)
-        return func(args)
-    except subprocess.CalledProcessError as exc:
-        print(f"Command failed with exit code {exc.returncode}: {exc.cmd}", file=sys.stderr)
-        return exc.returncode or 1

code/src/lora/trainers/__init__.py

@@ -1 +0,0 @@
-"""Training entrypoints for the lora package."""

code/src/lora/trainers/flux_lowmem.py

@@ -1,2059 +0,0 @@
-#!/usr/bin/env python
-# coding=utf-8
-# Copyright 2025 The HuggingFace Inc. team. All rights reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# pyright: reportAttributeAccessIssue=false, reportPrivateImportUsage=false, reportMissingImports=false, reportArgumentType=false, reportPossiblyUnboundVariable=false, reportOptionalSubscript=false, reportUndefinedVariable=false, reportCallIssue=false, reportAssignmentType=false, reportOptionalMemberAccess=false
-
-# /// script
-# dependencies = [
-#     "diffusers @ git+https://github.com/huggingface/diffusers.git",
-#     "torch>=2.0.0",
-#     "accelerate>=0.31.0",
-#     "transformers>=4.41.2",
-#     "ftfy",
-#     "tensorboard",
-#     "Jinja2",
-#     "peft>=0.11.1",
-#     "sentencepiece",
-#     "torchvision",
-#     "datasets",
-#     "bitsandbytes",
-#     "prodigyopt",
-# ]
-# ///
-
-from __future__ import annotations
-
-import argparse
-import copy
-import itertools
-import logging
-import math
-import os
-import random
-import shutil
-import warnings
-from contextlib import nullcontext
-from pathlib import Path
-from typing import TYPE_CHECKING
-
-import numpy as np
-import torch
-import transformers
-from accelerate import Accelerator
-from accelerate.logging import get_logger
-from accelerate.utils import DistributedDataParallelKwargs, ProjectConfiguration, set_seed
-from huggingface_hub import create_repo, upload_folder
-from huggingface_hub.utils import insecure_hashlib
-from peft import LoraConfig, set_peft_model_state_dict
-from peft.utils import get_peft_model_state_dict
-from PIL import Image
-from PIL.ImageOps import exif_transpose
-from torch.utils.data import Dataset
-from torchvision import transforms
-from torchvision.transforms.functional import crop
-from tqdm.auto import tqdm
-from transformers import CLIPTokenizer, PretrainedConfig, T5TokenizerFast
-
-import diffusers
-from diffusers import (
-    AutoencoderKL,
-    FlowMatchEulerDiscreteScheduler,
-    FluxPipeline,
-    FluxTransformer2DModel,
-)
-from diffusers.models.autoencoders.vae import DiagonalGaussianDistribution
-from diffusers.optimization import get_scheduler
-from diffusers.training_utils import (
-    _collate_lora_metadata,
-    _set_state_dict_into_text_encoder,
-    cast_training_params,
-    compute_density_for_timestep_sampling,
-    compute_loss_weighting_for_sd3,
-    free_memory,
-)
-from diffusers.utils import (
-    check_min_version,
-    convert_unet_state_dict_to_peft,
-    is_wandb_available,
-)
-from diffusers.utils.hub_utils import load_or_create_model_card, populate_model_card
-from diffusers.utils.import_utils import is_torch_npu_available
-from diffusers.utils.torch_utils import is_compiled_module
-
-if TYPE_CHECKING:
-    from collections.abc import Sequence
-
-    from PIL.Image import Image as PILImage
-
-
-if is_wandb_available():
-    import wandb
-
-# Will error if the minimal version of diffusers is not installed. Remove at your own risks.
-check_min_version("0.39.0.dev0")
-
-logger = get_logger(__name__)
-
-
-def save_model_card(
-    repo_id: str,
-    images: list[PILImage] | None = None,
-    base_model: str | None = None,
-    train_text_encoder: bool = False,
-    instance_prompt: str | None = None,
-    validation_prompt: str | None = None,
-    repo_folder: str | os.PathLike[str] | None = None,
-) -> None:
-    if repo_folder is None:
-        raise ValueError("repo_folder must be provided when saving a model card.")
-
-    repo_folder_path = os.fspath(repo_folder)
-    widget_dict: list[dict[str, object]] = []
-    if images is not None:
-        for i, image in enumerate(images):
-            image.save(os.path.join(repo_folder_path, f"image_{i}.png"))
-            widget_dict.append(
-                {"text": validation_prompt if validation_prompt else " ", "output": {"url": f"image_{i}.png"}}
-            )
-
-    model_description = f"""
-# Flux DreamBooth LoRA - {repo_id}
-
-<Gallery />
-
-## Model description
-
-These are {repo_id} DreamBooth LoRA weights for {base_model}.
-
-The weights were trained using [DreamBooth](https://dreambooth.github.io/) with the [Flux diffusers trainer](https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/README_flux.md).
-
-Was LoRA for the text encoder enabled? {train_text_encoder}.
-
-## Trigger words
-
-You should use `{instance_prompt}` to trigger the image generation.
-
-## Download model
-
-[Download the *.safetensors LoRA]({repo_id}/tree/main) in the Files & versions tab.
-
-## Use it with the [🧨 diffusers library](https://github.com/huggingface/diffusers)
-
-```py
-from diffusers import AutoPipelineForText2Image
-import torch
-pipeline = AutoPipelineForText2Image.from_pretrained("black-forest-labs/FLUX.1-dev", torch_dtype=torch.bfloat16).to('cuda')
-pipeline.load_lora_weights('{repo_id}', weight_name='pytorch_lora_weights.safetensors')
-image = pipeline('{validation_prompt if validation_prompt else instance_prompt}').images[0]
-```
-
-For more details, including weighting, merging and fusing LoRAs, check the [documentation on loading LoRAs in diffusers](https://huggingface.co/docs/diffusers/main/en/using-diffusers/loading_adapters)
-
-## License
-
-Please adhere to the licensing terms as described [here](https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/LICENSE.md).
-"""
-    model_card = load_or_create_model_card(
-        repo_id_or_path=repo_id,
-        from_training=True,
-        license="other",
-        base_model=base_model,
-        prompt=instance_prompt,
-        model_description=model_description,
-        widget=widget_dict,
-    )
-    tags = [
-        "text-to-image",
-        "diffusers-training",
-        "diffusers",
-        "lora",
-        "flux",
-        "flux-diffusers",
-        "template:sd-lora",
-    ]
-
-    model_card = populate_model_card(model_card, tags=tags)
-    model_card.save(os.path.join(repo_folder_path, "README.md"))
-
-
-def load_text_encoders(
-    args: argparse.Namespace,
-    class_one: type,
-    class_two: type,
-    torch_dtype: torch.dtype | None = None,
-):
-    text_encoder_one = class_one.from_pretrained(
-        args.pretrained_model_name_or_path,
-        subfolder="text_encoder",
-        revision=args.revision,
-        variant=args.variant,
-        torch_dtype=torch_dtype,
-    )
-    text_encoder_two = class_two.from_pretrained(
-        args.pretrained_model_name_or_path,
-        subfolder="text_encoder_2",
-        revision=args.revision,
-        variant=args.variant,
-        torch_dtype=torch_dtype,
-    )
-    return text_encoder_one, text_encoder_two
-
-
-def log_validation(
-    pipeline,
-    args,
-    accelerator,
-    pipeline_args,
-    epoch,
-    torch_dtype,
-    is_final_validation=False,
-):
-    logger.info(
-        f"Running validation... \n Generating {args.num_validation_images} images with prompt:"
-        f" {args.validation_prompt}."
-    )
-    pipeline = pipeline.to(accelerator.device, dtype=torch_dtype)
-    pipeline.set_progress_bar_config(disable=True)
-
-    # run inference
-    generator = torch.Generator(device=accelerator.device).manual_seed(args.seed) if args.seed is not None else None
-    autocast_ctx = torch.autocast(accelerator.device.type) if not is_final_validation else nullcontext()
-
-    # pre-calculate  prompt embeds, pooled prompt embeds, text ids because t5 does not support autocast
-    with torch.no_grad():
-        prompt_embeds, pooled_prompt_embeds, text_ids = pipeline.encode_prompt(
-            pipeline_args["prompt"], prompt_2=pipeline_args["prompt"]
-        )
-    images = []
-    for _ in range(args.num_validation_images):
-        with autocast_ctx:
-            image = pipeline(
-                prompt_embeds=prompt_embeds, pooled_prompt_embeds=pooled_prompt_embeds, generator=generator
-            ).images[0]
-            images.append(image)
-
-    for tracker in accelerator.trackers:
-        phase_name = "test" if is_final_validation else "validation"
-        if tracker.name == "tensorboard":
-            np_images = np.stack([np.asarray(img) for img in images])
-            tracker.writer.add_images(phase_name, np_images, epoch, dataformats="NHWC")
-        if tracker.name == "wandb":
-            tracker.log(
-                {
-                    phase_name: [
-                        wandb.Image(image, caption=f"{i}: {args.validation_prompt}") for i, image in enumerate(images)
-                    ]
-                }
-            )
-
-    del pipeline
-    free_memory()
-
-    return images
-
-
-def import_model_class_from_model_name_or_path(
-    pretrained_model_name_or_path: str, revision: str, subfolder: str = "text_encoder"
-):
-    text_encoder_config = PretrainedConfig.from_pretrained(
-        pretrained_model_name_or_path, subfolder=subfolder, revision=revision
-    )
-    model_class = text_encoder_config.architectures[0]
-    if model_class == "CLIPTextModel":
-        from transformers import CLIPTextModel
-
-        return CLIPTextModel
-    elif model_class == "T5EncoderModel":
-        from transformers import T5EncoderModel
-
-        return T5EncoderModel
-    else:
-        raise ValueError(f"{model_class} is not supported.")
-
-
-def parse_args(input_args: Sequence[str] | None = None) -> argparse.Namespace:
-    parser = argparse.ArgumentParser(description="Simple example of a training script.")
-    parser.add_argument(
-        "--pretrained_model_name_or_path",
-        type=str,
-        default=None,
-        required=True,
-        help="Path to pretrained model or model identifier from huggingface.co/models.",
-    )
-    parser.add_argument(
-        "--revision",
-        type=str,
-        default=None,
-        required=False,
-        help="Revision of pretrained model identifier from huggingface.co/models.",
-    )
-    parser.add_argument(
-        "--variant",
-        type=str,
-        default=None,
-        help="Variant of the model files of the pretrained model identifier from huggingface.co/models, 'e.g.' fp16",
-    )
-    parser.add_argument(
-        "--dataset_name",
-        type=str,
-        default=None,
-        help=(
-            "The name of the Dataset (from the HuggingFace hub) containing the training data of instance images (could be your own, possibly private,"
-            " dataset). It can also be a path pointing to a local copy of a dataset in your filesystem,"
-            " or to a folder containing files that 🤗 Datasets can understand."
-        ),
-    )
-    parser.add_argument(
-        "--dataset_config_name",
-        type=str,
-        default=None,
-        help="The config of the Dataset, leave as None if there's only one config.",
-    )
-    parser.add_argument(
-        "--instance_data_dir",
-        type=str,
-        default=None,
-        help=("A folder containing the training data. "),
-    )
-
-    parser.add_argument(
-        "--cache_dir",
-        type=str,
-        default=None,
-        help="The directory where the downloaded models and datasets will be stored.",
-    )
-
-    parser.add_argument(
-        "--image_column",
-        type=str,
-        default="image",
-        help="The column of the dataset containing the target image. By "
-        "default, the standard Image Dataset maps out 'file_name' "
-        "to 'image'.",
-    )
-    parser.add_argument(
-        "--caption_column",
-        type=str,
-        default=None,
-        help="The column of the dataset containing the instance prompt for each image",
-    )
-
-    parser.add_argument("--repeats", type=int, default=1, help="How many times to repeat the training data.")
-
-    parser.add_argument(
-        "--class_data_dir",
-        type=str,
-        default=None,
-        required=False,
-        help="A folder containing the training data of class images.",
-    )
-    parser.add_argument(
-        "--instance_prompt",
-        type=str,
-        default=None,
-        required=True,
-        help="The prompt with identifier specifying the instance, e.g. 'photo of a TOK dog', 'in the style of TOK'",
-    )
-    parser.add_argument(
-        "--class_prompt",
-        type=str,
-        default=None,
-        help="The prompt to specify images in the same class as provided instance images.",
-    )
-    parser.add_argument(
-        "--max_sequence_length",
-        type=int,
-        default=512,
-        help="Maximum sequence length to use with with the T5 text encoder",
-    )
-    parser.add_argument(
-        "--validation_prompt",
-        type=str,
-        default=None,
-        help="A prompt that is used during validation to verify that the model is learning.",
-    )
-    parser.add_argument(
-        "--num_validation_images",
-        type=int,
-        default=4,
-        help="Number of images that should be generated during validation with `validation_prompt`.",
-    )
-    parser.add_argument(
-        "--validation_epochs",
-        type=int,
-        default=50,
-        help=(
-            "Run dreambooth validation every X epochs. Dreambooth validation consists of running the prompt"
-            " `args.validation_prompt` multiple times: `args.num_validation_images`."
-        ),
-    )
-    parser.add_argument(
-        "--rank",
-        type=int,
-        default=4,
-        help=("The dimension of the LoRA update matrices."),
-    )
-    parser.add_argument(
-        "--lora_alpha",
-        type=int,
-        default=4,
-        help="LoRA alpha to be used for additional scaling.",
-    )
-    parser.add_argument("--lora_dropout", type=float, default=0.0, help="Dropout probability for LoRA layers")
-
-    parser.add_argument(
-        "--with_prior_preservation",
-        default=False,
-        action="store_true",
-        help="Flag to add prior preservation loss.",
-    )
-    parser.add_argument("--prior_loss_weight", type=float, default=1.0, help="The weight of prior preservation loss.")
-    parser.add_argument(
-        "--num_class_images",
-        type=int,
-        default=100,
-        help=(
-            "Minimal class images for prior preservation loss. If there are not enough images already present in"
-            " class_data_dir, additional images will be sampled with class_prompt."
-        ),
-    )
-    parser.add_argument(
-        "--output_dir",
-        type=str,
-        default="flux-dreambooth-lora",
-        help="The output directory where the model predictions and checkpoints will be written.",
-    )
-    parser.add_argument("--seed", type=int, default=None, help="A seed for reproducible training.")
-    parser.add_argument(
-        "--resolution",
-        type=int,
-        default=512,
-        help=(
-            "The resolution for input images, all the images in the train/validation dataset will be resized to this"
-            " resolution"
-        ),
-    )
-    parser.add_argument(
-        "--center_crop",
-        default=False,
-        action="store_true",
-        help=(
-            "Whether to center crop the input images to the resolution. If not set, the images will be randomly"
-            " cropped. The images will be resized to the resolution first before cropping."
-        ),
-    )
-    parser.add_argument(
-        "--random_flip",
-        action="store_true",
-        help="whether to randomly flip images horizontally",
-    )
-    parser.add_argument(
-        "--train_text_encoder",
-        action="store_true",
-        help="Whether to train the text encoder. If set, the text encoder should be float32 precision.",
-    )
-    parser.add_argument(
-        "--train_batch_size", type=int, default=4, help="Batch size (per device) for the training dataloader."
-    )
-    parser.add_argument(
-        "--sample_batch_size", type=int, default=4, help="Batch size (per device) for sampling images."
-    )
-    parser.add_argument("--num_train_epochs", type=int, default=1)
-    parser.add_argument(
-        "--max_train_steps",
-        type=int,
-        default=None,
-        help="Total number of training steps to perform.  If provided, overrides num_train_epochs.",
-    )
-    parser.add_argument(
-        "--checkpointing_steps",
-        type=int,
-        default=500,
-        help=(
-            "Save a checkpoint of the training state every X updates. These checkpoints can be used both as final"
-            " checkpoints in case they are better than the last checkpoint, and are also suitable for resuming"
-            " training using `--resume_from_checkpoint`."
-        ),
-    )
-    parser.add_argument(
-        "--checkpoints_total_limit",
-        type=int,
-        default=None,
-        help=("Max number of checkpoints to store."),
-    )
-    parser.add_argument(
-        "--resume_from_checkpoint",
-        type=str,
-        default=None,
-        help=(
-            "Whether training should be resumed from a previous checkpoint. Use a path saved by"
-            ' `--checkpointing_steps`, or `"latest"` to automatically select the last available checkpoint.'
-        ),
-    )
-    parser.add_argument(
-        "--gradient_accumulation_steps",
-        type=int,
-        default=1,
-        help="Number of updates steps to accumulate before performing a backward/update pass.",
-    )
-    parser.add_argument(
-        "--gradient_checkpointing",
-        action="store_true",
-        help="Whether or not to use gradient checkpointing to save memory at the expense of slower backward pass.",
-    )
-    parser.add_argument(
-        "--learning_rate",
-        type=float,
-        default=1e-4,
-        help="Initial learning rate (after the potential warmup period) to use.",
-    )
-
-    parser.add_argument(
-        "--guidance_scale",
-        type=float,
-        default=3.5,
-        help="the FLUX.1 dev variant is a guidance distilled model",
-    )
-
-    parser.add_argument(
-        "--text_encoder_lr",
-        type=float,
-        default=5e-6,
-        help="Text encoder learning rate to use.",
-    )
-    parser.add_argument(
-        "--scale_lr",
-        action="store_true",
-        default=False,
-        help="Scale the learning rate by the number of GPUs, gradient accumulation steps, and batch size.",
-    )
-    parser.add_argument(
-        "--lr_scheduler",
-        type=str,
-        default="constant",
-        help=(
-            'The scheduler type to use. Choose between ["linear", "cosine", "cosine_with_restarts", "polynomial",'
-            ' "constant", "constant_with_warmup"]'
-        ),
-    )
-    parser.add_argument(
-        "--lr_warmup_steps", type=int, default=500, help="Number of steps for the warmup in the lr scheduler."
-    )
-    parser.add_argument(
-        "--lr_num_cycles",
-        type=int,
-        default=1,
-        help="Number of hard resets of the lr in cosine_with_restarts scheduler.",
-    )
-    parser.add_argument("--lr_power", type=float, default=1.0, help="Power factor of the polynomial scheduler.")
-    parser.add_argument(
-        "--dataloader_num_workers",
-        type=int,
-        default=0,
-        help=(
-            "Number of subprocesses to use for data loading. 0 means that the data will be loaded in the main process."
-        ),
-    )
-    parser.add_argument(
-        "--weighting_scheme",
-        type=str,
-        default="none",
-        choices=["sigma_sqrt", "logit_normal", "mode", "cosmap", "none"],
-        help=('We default to the "none" weighting scheme for uniform sampling and uniform loss'),
-    )
-    parser.add_argument(
-        "--logit_mean", type=float, default=0.0, help="mean to use when using the `'logit_normal'` weighting scheme."
-    )
-    parser.add_argument(
-        "--logit_std", type=float, default=1.0, help="std to use when using the `'logit_normal'` weighting scheme."
-    )
-    parser.add_argument(
-        "--mode_scale",
-        type=float,
-        default=1.29,
-        help="Scale of mode weighting scheme. Only effective when using the `'mode'` as the `weighting_scheme`.",
-    )
-    parser.add_argument(
-        "--optimizer",
-        type=str,
-        default="AdamW",
-        help=('The optimizer type to use. Choose between ["AdamW", "prodigy"]'),
-    )
-
-    parser.add_argument(
-        "--use_8bit_adam",
-        action="store_true",
-        help="Whether or not to use 8-bit Adam from bitsandbytes. Ignored if optimizer is not set to AdamW",
-    )
-
-    parser.add_argument(
-        "--adam_beta1", type=float, default=0.9, help="The beta1 parameter for the Adam and Prodigy optimizers."
-    )
-    parser.add_argument(
-        "--adam_beta2", type=float, default=0.999, help="The beta2 parameter for the Adam and Prodigy optimizers."
-    )
-    parser.add_argument(
-        "--prodigy_beta3",
-        type=float,
-        default=None,
-        help="coefficients for computing the Prodigy stepsize using running averages. If set to None, "
-        "uses the value of square root of beta2. Ignored if optimizer is adamW",
-    )
-    parser.add_argument("--prodigy_decouple", type=bool, default=True, help="Use AdamW style decoupled weight decay")
-    parser.add_argument("--adam_weight_decay", type=float, default=1e-04, help="Weight decay to use for unet params")
-    parser.add_argument(
-        "--adam_weight_decay_text_encoder", type=float, default=1e-03, help="Weight decay to use for text_encoder"
-    )
-
-    parser.add_argument(
-        "--lora_layers",
-        type=str,
-        default=None,
-        help=(
-            'The transformer modules to apply LoRA training on. Please specify the layers in a comma separated. E.g. - "to_k,to_q,to_v,to_out.0" will result in lora training of attention layers only'
-        ),
-    )
-
-    parser.add_argument(
-        "--adam_epsilon",
-        type=float,
-        default=1e-08,
-        help="Epsilon value for the Adam optimizer and Prodigy optimizers.",
-    )
-
-    parser.add_argument(
-        "--prodigy_use_bias_correction",
-        type=bool,
-        default=True,
-        help="Turn on Adam's bias correction. True by default. Ignored if optimizer is adamW",
-    )
-    parser.add_argument(
-        "--prodigy_safeguard_warmup",
-        type=bool,
-        default=True,
-        help="Remove lr from the denominator of D estimate to avoid issues during warm-up stage. True by default. "
-        "Ignored if optimizer is adamW",
-    )
-    parser.add_argument("--max_grad_norm", default=1.0, type=float, help="Max gradient norm.")
-    parser.add_argument("--push_to_hub", action="store_true", help="Whether or not to push the model to the Hub.")
-    parser.add_argument("--hub_token", type=str, default=None, help="The token to use to push to the Model Hub.")
-    parser.add_argument(
-        "--hub_model_id",
-        type=str,
-        default=None,
-        help="The name of the repository to keep in sync with the local `output_dir`.",
-    )
-    parser.add_argument(
-        "--logging_dir",
-        type=str,
-        default="logs",
-        help=(
-            "[TensorBoard](https://www.tensorflow.org/tensorboard) log directory. Will default to"
-            " *output_dir/runs/**CURRENT_DATETIME_HOSTNAME***."
-        ),
-    )
-    parser.add_argument(
-        "--allow_tf32",
-        action="store_true",
-        help=(
-            "Whether or not to allow TF32 on Ampere GPUs. Can be used to speed up training. For more information, see"
-            " https://pytorch.org/docs/stable/notes/cuda.html#tensorfloat-32-tf32-on-ampere-devices"
-        ),
-    )
-    parser.add_argument(
-        "--cache_latents",
-        action="store_true",
-        default=False,
-        help="Cache the VAE latents",
-    )
-    parser.add_argument(
-        "--report_to",
-        type=str,
-        default="tensorboard",
-        help=(
-            'The integration to report the results and logs to. Supported platforms are `"tensorboard"`'
-            ' (default), `"wandb"` and `"comet_ml"`. Use `"all"` to report to all integrations.'
-        ),
-    )
-    parser.add_argument(
-        "--mixed_precision",
-        type=str,
-        default=None,
-        choices=["no", "fp16", "bf16"],
-        help=(
-            "Whether to use mixed precision. Choose between fp16 and bf16 (bfloat16). Bf16 requires PyTorch >="
-            " 1.10.and an Nvidia Ampere GPU.  Default to the value of accelerate config of the current system or the"
-            " flag passed with the `accelerate.launch` command. Use this argument to override the accelerate config."
-        ),
-    )
-    parser.add_argument(
-        "--upcast_before_saving",
-        action="store_true",
-        default=False,
-        help=(
-            "Whether to upcast the trained transformer layers to float32 before saving (at the end of training). "
-            "Defaults to precision dtype used for training to save memory"
-        ),
-    )
-    parser.add_argument(
-        "--prior_generation_precision",
-        type=str,
-        default=None,
-        choices=["no", "fp32", "fp16", "bf16"],
-        help=(
-            "Choose prior generation precision between fp32, fp16 and bf16 (bfloat16). Bf16 requires PyTorch >="
-            " 1.10.and an Nvidia Ampere GPU.  Default to  fp16 if a GPU is available else fp32."
-        ),
-    )
-    parser.add_argument("--local_rank", type=int, default=-1, help="For distributed training: local_rank")
-    parser.add_argument("--enable_npu_flash_attention", action="store_true", help="Enabla Flash Attention for NPU")
-
-    if input_args is not None:
-        args = parser.parse_args(input_args)
-    else:
-        args = parser.parse_args()
-
-    if args.dataset_name is None and args.instance_data_dir is None:
-        raise ValueError("Specify either `--dataset_name` or `--instance_data_dir`")
-
-    if args.dataset_name is not None and args.instance_data_dir is not None:
-        raise ValueError("Specify only one of `--dataset_name` or `--instance_data_dir`")
-
-    env_local_rank = int(os.environ.get("LOCAL_RANK", -1))
-    if env_local_rank != -1 and env_local_rank != args.local_rank:
-        args.local_rank = env_local_rank
-
-    if args.with_prior_preservation:
-        if args.class_data_dir is None:
-            raise ValueError("You must specify a data directory for class images.")
-        if args.class_prompt is None:
-            raise ValueError("You must specify prompt for class images.")
-    else:
-        # logger is not available yet
-        if args.class_data_dir is not None:
-            warnings.warn("You need not use --class_data_dir without --with_prior_preservation.")
-        if args.class_prompt is not None:
-            warnings.warn("You need not use --class_prompt without --with_prior_preservation.")
-
-    return args
-
-
-class DreamBoothDataset(Dataset):
-    """
-    A dataset to prepare the instance and class images with the prompts for fine-tuning the model.
-    It pre-processes the images.
-    """
-
-    def __init__(
-        self,
-        args: argparse.Namespace,
-        instance_data_root: str | os.PathLike[str] | None,
-        instance_prompt: str,
-        class_prompt: str | None,
-        class_data_root: str | os.PathLike[str] | None = None,
-        class_num: int | None = None,
-        size: int = 1024,
-        repeats: int = 1,
-        center_crop: bool = False,
-    ) -> None:
-        self.args = args
-        self.size = size
-        self.center_crop = center_crop
-
-        self.instance_prompt = instance_prompt
-        self.custom_instance_prompts = None
-        self.class_prompt = class_prompt
-
-        # if --dataset_name is provided or a metadata jsonl file is provided in the local --instance_data directory,
-        # we load the training data using load_dataset
-        if self.args.dataset_name is not None:
-            try:
-                from datasets import load_dataset
-            except ImportError:
-                raise ImportError(
-                    "You are trying to load your data using the datasets library. If you wish to train using custom "
-                    "captions please install the datasets library: `pip install datasets`. If you wish to load a "
-                    "local folder containing images only, specify --instance_data_dir instead."
-                )
-            # Downloading and loading a dataset from the hub.
-            # See more about loading custom images at
-            # https://huggingface.co/docs/datasets/v2.0.0/en/dataset_script
-            dataset = load_dataset(
-                self.args.dataset_name,
-                self.args.dataset_config_name,
-                cache_dir=self.args.cache_dir,
-            )
-            # Preprocessing the datasets.
-            column_names = dataset["train"].column_names
-
-            # 6. Get the column names for input/target.
-            if self.args.image_column is None:
-                image_column = column_names[0]
-                logger.info(f"image column defaulting to {image_column}")
-            else:
-                image_column = self.args.image_column
-                if image_column not in column_names:
-                    raise ValueError(
-                        f"`--image_column` value '{self.args.image_column}' not found in dataset columns. Dataset columns are: {', '.join(column_names)}"
-                    )
-            instance_images = dataset["train"][image_column]
-
-            if self.args.caption_column is None:
-                logger.info(
-                    "No caption column provided, defaulting to instance_prompt for all images. If your dataset "
-                    "contains captions/prompts for the images, make sure to specify the "
-                    "column as --caption_column"
-                )
-                self.custom_instance_prompts = None
-            else:
-                if self.args.caption_column not in column_names:
-                    raise ValueError(
-                        f"`--caption_column` value '{self.args.caption_column}' not found in dataset columns. Dataset columns are: {', '.join(column_names)}"
-                    )
-                custom_instance_prompts = dataset["train"][self.args.caption_column]
-                # create final list of captions according to --repeats
-                self.custom_instance_prompts = []
-                for caption in custom_instance_prompts:
-                    self.custom_instance_prompts.extend(itertools.repeat(caption, repeats))
-        else:
-            self.instance_data_root = Path(instance_data_root)
-            if not self.instance_data_root.exists():
-                raise ValueError("Instance images root doesn't exists.")
-
-            instance_images = [Image.open(path) for path in list(Path(instance_data_root).iterdir())]
-            self.custom_instance_prompts = None
-
-        self.instance_images = []
-        for img in instance_images:
-            self.instance_images.extend(itertools.repeat(img, repeats))
-
-        self.pixel_values = []
-        train_resize = transforms.Resize(size, interpolation=transforms.InterpolationMode.BILINEAR)
-        train_crop = transforms.CenterCrop(size) if center_crop else transforms.RandomCrop(size)
-        train_flip = transforms.RandomHorizontalFlip(p=1.0)
-        train_transforms = transforms.Compose(
-            [
-                transforms.ToTensor(),
-                transforms.Normalize([0.5], [0.5]),
-            ]
-        )
-        for image in self.instance_images:
-            image = exif_transpose(image)
-            if not image.mode == "RGB":
-                image = image.convert("RGB")
-            image = train_resize(image)
-            if self.args.random_flip and random.random() < 0.5:
-                # flip
-                image = train_flip(image)
-            if self.args.center_crop:
-                y1 = max(0, int(round((image.height - self.args.resolution) / 2.0)))
-                x1 = max(0, int(round((image.width - self.args.resolution) / 2.0)))
-                image = train_crop(image)
-            else:
-                y1, x1, h, w = train_crop.get_params(image, (self.args.resolution, self.args.resolution))
-                image = crop(image, y1, x1, h, w)
-            image = train_transforms(image)
-            self.pixel_values.append(image)
-
-        self.num_instance_images = len(self.instance_images)
-        self._length = self.num_instance_images
-
-        if class_data_root is not None:
-            self.class_data_root = Path(class_data_root)
-            self.class_data_root.mkdir(parents=True, exist_ok=True)
-            self.class_images_path = list(self.class_data_root.iterdir())
-            if class_num is not None:
-                self.num_class_images = min(len(self.class_images_path), class_num)
-            else:
-                self.num_class_images = len(self.class_images_path)
-            self._length = max(self.num_class_images, self.num_instance_images)
-        else:
-            self.class_data_root = None
-
-        self.image_transforms = transforms.Compose(
-            [
-                transforms.Resize(size, interpolation=transforms.InterpolationMode.BILINEAR),
-                transforms.CenterCrop(size) if center_crop else transforms.RandomCrop(size),
-                transforms.ToTensor(),
-                transforms.Normalize([0.5], [0.5]),
-            ]
-        )
-
-    def __len__(self):
-        return self._length
-
-    def __getitem__(self, index):
-        example = {}
-        instance_index = index % self.num_instance_images
-        instance_image = self.pixel_values[instance_index]
-        example["instance_images"] = instance_image
-        example["instance_index"] = instance_index
-
-        if self.custom_instance_prompts:
-            caption = self.custom_instance_prompts[instance_index]
-            if caption:
-                example["instance_prompt"] = caption
-            else:
-                example["instance_prompt"] = self.instance_prompt
-
-        else:  # custom prompts were provided, but length does not match size of image dataset
-            example["instance_prompt"] = self.instance_prompt
-
-        if self.class_data_root:
-            class_image = Image.open(self.class_images_path[index % self.num_class_images])
-            class_image = exif_transpose(class_image)
-
-            if not class_image.mode == "RGB":
-                class_image = class_image.convert("RGB")
-            example["class_images"] = self.image_transforms(class_image)
-            example["class_prompt"] = self.class_prompt
-
-        return example
-
-
-def collate_fn(examples, with_prior_preservation=False):
-    pixel_values = [example["instance_images"] for example in examples]
-    prompts = [example["instance_prompt"] for example in examples]
-    instance_indices = [example["instance_index"] for example in examples]
-
-    # Concat class and instance examples for prior preservation.
-    # We do this to avoid doing two forward passes.
-    if with_prior_preservation:
-        pixel_values += [example["class_images"] for example in examples]
-        prompts += [example["class_prompt"] for example in examples]
-
-    pixel_values = torch.stack(pixel_values)
-    pixel_values = pixel_values.to(memory_format=torch.contiguous_format).float()
-
-    batch = {"pixel_values": pixel_values, "prompts": prompts, "instance_indices": instance_indices}
-    return batch
-
-
-def collate_instance_latents(examples):
-    pixel_values = torch.stack([example["instance_images"] for example in examples])
-    pixel_values = pixel_values.to(memory_format=torch.contiguous_format).float()
-    instance_indices = [example["instance_index"] for example in examples]
-    return {"pixel_values": pixel_values, "instance_indices": instance_indices}
-
-
-class PromptDataset(Dataset):
-    "A simple dataset to prepare the prompts to generate class images on multiple GPUs."
-
-    def __init__(self, prompt, num_samples):
-        self.prompt = prompt
-        self.num_samples = num_samples
-
-    def __len__(self):
-        return self.num_samples
-
-    def __getitem__(self, index):
-        example = {}
-        example["prompt"] = self.prompt
-        example["index"] = index
-        return example
-
-
-def tokenize_prompt(tokenizer, prompt, max_sequence_length):
-    text_inputs = tokenizer(
-        prompt,
-        padding="max_length",
-        max_length=max_sequence_length,
-        truncation=True,
-        return_length=False,
-        return_overflowing_tokens=False,
-        return_tensors="pt",
-    )
-    text_input_ids = text_inputs.input_ids
-    return text_input_ids
-
-
-def _encode_prompt_with_t5(
-    text_encoder,
-    tokenizer,
-    max_sequence_length=512,
-    prompt=None,
-    num_images_per_prompt=1,
-    device=None,
-    text_input_ids=None,
-):
-    prompt = [prompt] if isinstance(prompt, str) else prompt
-    batch_size = len(prompt)
-
-    if tokenizer is not None:
-        text_inputs = tokenizer(
-            prompt,
-            padding="max_length",
-            max_length=max_sequence_length,
-            truncation=True,
-            return_length=False,
-            return_overflowing_tokens=False,
-            return_tensors="pt",
-        )
-        text_input_ids = text_inputs.input_ids
-    else:
-        if text_input_ids is None:
-            raise ValueError("text_input_ids must be provided when the tokenizer is not specified")
-
-    prompt_embeds = text_encoder(text_input_ids.to(device))[0]
-
-    if hasattr(text_encoder, "module"):
-        dtype = text_encoder.module.dtype
-    else:
-        dtype = text_encoder.dtype
-    prompt_embeds = prompt_embeds.to(dtype=dtype, device=device)
-
-    _, seq_len, _ = prompt_embeds.shape
-
-    # duplicate text embeddings and attention mask for each generation per prompt, using mps friendly method
-    prompt_embeds = prompt_embeds.repeat(1, num_images_per_prompt, 1)
-    prompt_embeds = prompt_embeds.view(batch_size * num_images_per_prompt, seq_len, -1)
-
-    return prompt_embeds
-
-
-def _encode_prompt_with_clip(
-    text_encoder,
-    tokenizer,
-    prompt: str,
-    device=None,
-    text_input_ids=None,
-    num_images_per_prompt: int = 1,
-):
-    prompt = [prompt] if isinstance(prompt, str) else prompt
-    batch_size = len(prompt)
-
-    if tokenizer is not None:
-        text_inputs = tokenizer(
-            prompt,
-            padding="max_length",
-            max_length=77,
-            truncation=True,
-            return_overflowing_tokens=False,
-            return_length=False,
-            return_tensors="pt",
-        )
-
-        text_input_ids = text_inputs.input_ids
-    else:
-        if text_input_ids is None:
-            raise ValueError("text_input_ids must be provided when the tokenizer is not specified")
-
-    prompt_embeds = text_encoder(text_input_ids.to(device), output_hidden_states=False)
-
-    if hasattr(text_encoder, "module"):
-        dtype = text_encoder.module.dtype
-    else:
-        dtype = text_encoder.dtype
-    # Use pooled output of CLIPTextModel
-    prompt_embeds = prompt_embeds.pooler_output
-    prompt_embeds = prompt_embeds.to(dtype=dtype, device=device)
-
-    # duplicate text embeddings for each generation per prompt, using mps friendly method
-    prompt_embeds = prompt_embeds.repeat(1, num_images_per_prompt, 1)
-    prompt_embeds = prompt_embeds.view(batch_size * num_images_per_prompt, -1)
-
-    return prompt_embeds
-
-
-def encode_prompt(
-    text_encoders,
-    tokenizers,
-    prompt: str,
-    max_sequence_length,
-    device=None,
-    num_images_per_prompt: int = 1,
-    text_input_ids_list=None,
-):
-    prompt = [prompt] if isinstance(prompt, str) else prompt
-
-    if hasattr(text_encoders[0], "module"):
-        dtype = text_encoders[0].module.dtype
-    else:
-        dtype = text_encoders[0].dtype
-
-    pooled_prompt_embeds = _encode_prompt_with_clip(
-        text_encoder=text_encoders[0],
-        tokenizer=tokenizers[0],
-        prompt=prompt,
-        device=device if device is not None else text_encoders[0].device,
-        num_images_per_prompt=num_images_per_prompt,
-        text_input_ids=text_input_ids_list[0] if text_input_ids_list else None,
-    )
-
-    prompt_embeds = _encode_prompt_with_t5(
-        text_encoder=text_encoders[1],
-        tokenizer=tokenizers[1],
-        max_sequence_length=max_sequence_length,
-        prompt=prompt,
-        num_images_per_prompt=num_images_per_prompt,
-        device=device if device is not None else text_encoders[1].device,
-        text_input_ids=text_input_ids_list[1] if text_input_ids_list else None,
-    )
-
-    text_ids = torch.zeros(prompt_embeds.shape[1], 3).to(device=device, dtype=dtype)
-
-    return prompt_embeds, pooled_prompt_embeds, text_ids
-
-
-def get_cached_text_embeddings(
-    prompt_embed_cache: dict[str, tuple[torch.Tensor, torch.Tensor, torch.Tensor]],
-    prompts: Sequence[str],
-    device: torch.device,
-) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
-    cached = [prompt_embed_cache[prompt] for prompt in prompts]
-    prompt_embeds = torch.cat([entry[0] for entry in cached], dim=0).to(device)
-    pooled_prompt_embeds = torch.cat([entry[1] for entry in cached], dim=0).to(device)
-    text_ids = cached[0][2].to(device)
-    return prompt_embeds, pooled_prompt_embeds, text_ids
-
-
-def build_latents_cache(
-    train_dataset: Dataset,
-    vae: AutoencoderKL,
-    batch_size: int,
-    num_workers: int,
-    device: torch.device,
-    dtype: torch.dtype,
-) -> list[torch.Tensor]:
-    latents_cache: list[torch.Tensor | None] = [None] * len(train_dataset)
-    latent_cache_loader = torch.utils.data.DataLoader(
-        train_dataset,
-        batch_size=batch_size,
-        shuffle=False,
-        collate_fn=collate_instance_latents,
-        num_workers=num_workers,
-    )
-    for batch in tqdm(latent_cache_loader, desc="Caching latents"):
-        with torch.no_grad():
-            pixel_values = batch["pixel_values"].to(device, non_blocking=True, dtype=dtype)
-            batch_latents = vae.encode(pixel_values).latent_dist
-            for latent_index, cached_latent in zip(batch["instance_indices"], batch_latents.parameters):
-                latents_cache[latent_index] = cached_latent.detach().cpu()
-
-    if any(cached_latent is None for cached_latent in latents_cache):
-        raise RuntimeError("Latent cache build was incomplete.")
-
-    return [cached_latent for cached_latent in latents_cache if cached_latent is not None]
-
-
-def get_cached_latent_dist(
-    latents_cache: Sequence[torch.Tensor],
-    instance_indices: Sequence[int],
-    device: torch.device,
-    dtype: torch.dtype,
-) -> DiagonalGaussianDistribution:
-    cached_latent_params = torch.stack([latents_cache[index] for index in instance_indices]).to(
-        device=device,
-        dtype=dtype,
-        non_blocking=True,
-    )
-    return DiagonalGaussianDistribution(cached_latent_params)
-
-
-def main(args: argparse.Namespace) -> None:
-    if args.report_to == "wandb" and args.hub_token is not None:
-        raise ValueError(
-            "You cannot use both --report_to=wandb and --hub_token due to a security risk of exposing your token."
-            " Please use `hf auth login` to authenticate with the Hub."
-        )
-
-    if torch.backends.mps.is_available() and args.mixed_precision == "bf16":
-        # due to pytorch#99272, MPS does not yet support bfloat16.
-        raise ValueError(
-            "Mixed precision training with bfloat16 is not supported on MPS. Please use fp16 (recommended) or fp32 instead."
-        )
-
-    logging_dir = Path(args.output_dir, args.logging_dir)
-
-    accelerator_project_config = ProjectConfiguration(project_dir=args.output_dir, logging_dir=logging_dir)
-    kwargs = DistributedDataParallelKwargs(find_unused_parameters=True)
-    accelerator = Accelerator(
-        gradient_accumulation_steps=args.gradient_accumulation_steps,
-        mixed_precision=args.mixed_precision,
-        log_with=args.report_to,
-        project_config=accelerator_project_config,
-        kwargs_handlers=[kwargs],
-    )
-
-    # Disable AMP for MPS.
-    if torch.backends.mps.is_available():
-        accelerator.native_amp = False
-
-    if args.report_to == "wandb":
-        if not is_wandb_available():
-            raise ImportError("Make sure to install wandb if you want to use it for logging during training.")
-
-    # Make one log on every process with the configuration for debugging.
-    logging.basicConfig(
-        format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",
-        datefmt="%m/%d/%Y %H:%M:%S",
-        level=logging.INFO,
-    )
-    logger.info(accelerator.state, main_process_only=False)
-    if accelerator.is_local_main_process:
-        transformers.utils.logging.set_verbosity_warning()
-        diffusers.utils.logging.set_verbosity_info()
-    else:
-        transformers.utils.logging.set_verbosity_error()
-        diffusers.utils.logging.set_verbosity_error()
-
-    # If passed along, set the training seed now.
-    if args.seed is not None:
-        set_seed(args.seed)
-
-    # Generate class images if prior preservation is enabled.
-    if args.with_prior_preservation:
-        class_images_dir = Path(args.class_data_dir)
-        if not class_images_dir.exists():
-            class_images_dir.mkdir(parents=True)
-        cur_class_images = len(list(class_images_dir.iterdir()))
-
-        if cur_class_images < args.num_class_images:
-            has_supported_fp16_accelerator = torch.cuda.is_available() or torch.backends.mps.is_available()
-            torch_dtype = torch.float16 if has_supported_fp16_accelerator else torch.float32
-            if args.prior_generation_precision == "fp32":
-                torch_dtype = torch.float32
-            elif args.prior_generation_precision == "fp16":
-                torch_dtype = torch.float16
-            elif args.prior_generation_precision == "bf16":
-                torch_dtype = torch.bfloat16
-
-            pipeline = FluxPipeline.from_pretrained(
-                args.pretrained_model_name_or_path,
-                torch_dtype=torch_dtype,
-                revision=args.revision,
-                variant=args.variant,
-            )
-            pipeline.set_progress_bar_config(disable=True)
-
-            num_new_images = args.num_class_images - cur_class_images
-            logger.info(f"Number of class images to sample: {num_new_images}.")
-
-            sample_dataset = PromptDataset(args.class_prompt, num_new_images)
-            sample_dataloader = torch.utils.data.DataLoader(sample_dataset, batch_size=args.sample_batch_size)
-
-            sample_dataloader = accelerator.prepare(sample_dataloader)
-            pipeline.to(accelerator.device)
-
-            for example in tqdm(
-                sample_dataloader, desc="Generating class images", disable=not accelerator.is_local_main_process
-            ):
-                with torch.autocast(device_type=accelerator.device.type, dtype=torch_dtype):
-                    images = pipeline(prompt=example["prompt"]).images
-
-                for i, image in enumerate(images):
-                    hash_image = insecure_hashlib.sha1(image.tobytes()).hexdigest()
-                    image_filename = class_images_dir / f"{example['index'][i] + cur_class_images}-{hash_image}.jpg"
-                    image.save(image_filename)
-
-            del pipeline
-            free_memory()
-
-    # Handle the repository creation
-    if accelerator.is_main_process:
-        if args.output_dir is not None:
-            os.makedirs(args.output_dir, exist_ok=True)
-
-        if args.push_to_hub:
-            repo_id = create_repo(
-                repo_id=args.hub_model_id or Path(args.output_dir).name,
-                exist_ok=True,
-            ).repo_id
-
-    # Load the tokenizers
-    tokenizer_one = CLIPTokenizer.from_pretrained(
-        args.pretrained_model_name_or_path,
-        subfolder="tokenizer",
-        revision=args.revision,
-    )
-    tokenizer_two = T5TokenizerFast.from_pretrained(
-        args.pretrained_model_name_or_path,
-        subfolder="tokenizer_2",
-        revision=args.revision,
-    )
-
-    # import correct text encoder classes
-    text_encoder_cls_one = import_model_class_from_model_name_or_path(
-        args.pretrained_model_name_or_path, args.revision
-    )
-    text_encoder_cls_two = import_model_class_from_model_name_or_path(
-        args.pretrained_model_name_or_path, args.revision, subfolder="text_encoder_2"
-    )
-
-    # For mixed precision training we cast all non-trainable weights (vae, text_encoder and transformer) to half-precision
-    # as these weights are only used for inference, keeping weights in full precision is not required.
-    weight_dtype = torch.float32
-    if accelerator.mixed_precision == "fp16":
-        weight_dtype = torch.float16
-    elif accelerator.mixed_precision == "bf16":
-        weight_dtype = torch.bfloat16
-
-    if torch.backends.mps.is_available() and weight_dtype == torch.bfloat16:
-        # due to pytorch#99272, MPS does not yet support bfloat16.
-        raise ValueError(
-            "Mixed precision training with bfloat16 is not supported on MPS. Please use fp16 (recommended) or fp32 instead."
-        )
-
-    # Load scheduler and models
-    noise_scheduler = FlowMatchEulerDiscreteScheduler.from_pretrained(
-        args.pretrained_model_name_or_path, subfolder="scheduler"
-    )
-    noise_scheduler_copy = copy.deepcopy(noise_scheduler)
-    text_encoder_one, text_encoder_two = load_text_encoders(
-        args, text_encoder_cls_one, text_encoder_cls_two, weight_dtype
-    )
-    text_encoder_one.requires_grad_(False)
-    text_encoder_two.requires_grad_(False)
-
-    if args.train_text_encoder:
-        raise ValueError(
-            "This low-memory trainer does not support --train_text_encoder. "
-            "LoRA training is restricted to the transformer so CLIP/T5 can be freed before transformer training."
-        )
-
-    # Dataset and DataLoaders creation. This is intentionally before transformer/VAE device placement
-    # so all prompt embeddings can be cached and the text encoders freed first.
-    train_dataset = DreamBoothDataset(
-        args=args,
-        instance_data_root=args.instance_data_dir,
-        instance_prompt=args.instance_prompt,
-        class_prompt=args.class_prompt,
-        class_data_root=args.class_data_dir if args.with_prior_preservation else None,
-        class_num=args.num_class_images,
-        size=args.resolution,
-        repeats=args.repeats,
-        center_crop=args.center_crop,
-    )
-
-    train_dataloader = torch.utils.data.DataLoader(
-        train_dataset,
-        batch_size=args.train_batch_size,
-        shuffle=True,
-        collate_fn=lambda examples: collate_fn(examples, args.with_prior_preservation),
-        num_workers=args.dataloader_num_workers,
-    )
-
-    text_encoder_one.to(accelerator.device, dtype=weight_dtype)
-    text_encoder_two.to(accelerator.device, dtype=weight_dtype)
-
-    tokenizers = [tokenizer_one, tokenizer_two]
-    text_encoders = [text_encoder_one, text_encoder_two]
-
-    def compute_text_embeddings(prompt, text_encoders, tokenizers):
-        with torch.no_grad():
-            prompt_embeds, pooled_prompt_embeds, text_ids = encode_prompt(
-                text_encoders, tokenizers, prompt, args.max_sequence_length
-            )
-            prompt_embeds = prompt_embeds.to(accelerator.device)
-            pooled_prompt_embeds = pooled_prompt_embeds.to(accelerator.device)
-            text_ids = text_ids.to(accelerator.device)
-        return prompt_embeds, pooled_prompt_embeds, text_ids
-
-    prompt_embed_cache = {}
-    prompts_to_cache = {args.instance_prompt}
-    if train_dataset.custom_instance_prompts:
-        prompts_to_cache.update(prompt for prompt in train_dataset.custom_instance_prompts if prompt)
-    if args.with_prior_preservation:
-        prompts_to_cache.add(args.class_prompt)
-
-    for prompt in sorted(prompts_to_cache):
-        prompt_embed_cache[prompt] = tuple(tensor.cpu() for tensor in compute_text_embeddings(prompt, text_encoders, tokenizers))
-
-    del text_encoder_one, text_encoder_two, tokenizer_one, tokenizer_two, text_encoders, tokenizers
-    text_encoder_one = None
-    text_encoder_two = None
-    free_memory()
-
-    vae = AutoencoderKL.from_pretrained(
-        args.pretrained_model_name_or_path,
-        subfolder="vae",
-        revision=args.revision,
-        variant=args.variant,
-        torch_dtype=weight_dtype,
-    )
-    transformer = FluxTransformer2DModel.from_pretrained(
-        args.pretrained_model_name_or_path,
-        subfolder="transformer",
-        revision=args.revision,
-        variant=args.variant,
-        torch_dtype=weight_dtype,
-    )
-
-    # We only train the additional adapter LoRA layers on the denoising transformer.
-    transformer.requires_grad_(False)
-    vae.requires_grad_(False)
-
-    if args.enable_npu_flash_attention:
-        if is_torch_npu_available():
-            logger.info("npu flash attention enabled.")
-            transformer.set_attention_backend("_native_npu")
-        else:
-            raise ValueError("npu flash attention requires torch_npu extensions and is supported only on npu device ")
-
-    vae.to(accelerator.device, dtype=weight_dtype)
-    transformer.to(accelerator.device, dtype=weight_dtype)
-
-    if args.gradient_checkpointing:
-        transformer.enable_gradient_checkpointing()
-        if args.train_text_encoder:
-            text_encoder_one.gradient_checkpointing_enable()
-
-    if args.lora_layers is not None:
-        target_modules = [layer.strip() for layer in args.lora_layers.split(",")]
-    else:
-        target_modules = [
-            "attn.to_k",
-            "attn.to_q",
-            "attn.to_v",
-            "attn.to_out.0",
-            "attn.add_k_proj",
-            "attn.add_q_proj",
-            "attn.add_v_proj",
-            "attn.to_add_out",
-            "ff.net.0.proj",
-            "ff.net.2",
-            "ff_context.net.0.proj",
-            "ff_context.net.2",
-        ]
-
-    # now we will add new LoRA weights the transformer layers
-    transformer_lora_config = LoraConfig(
-        r=args.rank,
-        lora_alpha=args.lora_alpha,
-        lora_dropout=args.lora_dropout,
-        init_lora_weights="gaussian",
-        target_modules=target_modules,
-    )
-    transformer.add_adapter(transformer_lora_config)
-    if args.train_text_encoder:
-        text_lora_config = LoraConfig(
-            r=args.rank,
-            lora_alpha=args.lora_alpha,
-            lora_dropout=args.lora_dropout,
-            init_lora_weights="gaussian",
-            target_modules=["q_proj", "k_proj", "v_proj", "out_proj"],
-        )
-        text_encoder_one.add_adapter(text_lora_config)
-
-    def unwrap_model(model):
-        model = accelerator.unwrap_model(model)
-        model = model._orig_mod if is_compiled_module(model) else model
-        return model
-
-    # create custom saving & loading hooks so that `accelerator.save_state(...)` serializes in a nice format
-    def save_model_hook(models, weights, output_dir):
-        if accelerator.is_main_process:
-            transformer_lora_layers_to_save = None
-            text_encoder_one_lora_layers_to_save = None
-            modules_to_save = {}
-            for model in models:
-                if isinstance(model, type(unwrap_model(transformer))):
-                    transformer_lora_layers_to_save = get_peft_model_state_dict(model)
-                    modules_to_save["transformer"] = model
-                elif isinstance(model, type(unwrap_model(text_encoder_one))):
-                    text_encoder_one_lora_layers_to_save = get_peft_model_state_dict(model)
-                    modules_to_save["text_encoder"] = model
-                else:
-                    raise ValueError(f"unexpected save model: {model.__class__}")
-
-                # make sure to pop weight so that corresponding model is not saved again
-                weights.pop()
-
-            FluxPipeline.save_lora_weights(
-                output_dir,
-                transformer_lora_layers=transformer_lora_layers_to_save,
-                text_encoder_lora_layers=text_encoder_one_lora_layers_to_save,
-                **_collate_lora_metadata(modules_to_save),
-            )
-
-    def load_model_hook(models, input_dir):
-        transformer_ = None
-        text_encoder_one_ = None
-
-        while len(models) > 0:
-            model = models.pop()
-
-            if isinstance(model, type(unwrap_model(transformer))):
-                transformer_ = model
-            elif isinstance(model, type(unwrap_model(text_encoder_one))):
-                text_encoder_one_ = model
-            else:
-                raise ValueError(f"unexpected save model: {model.__class__}")
-
-        lora_state_dict = FluxPipeline.lora_state_dict(input_dir)
-
-        transformer_state_dict = {
-            f"{k.replace('transformer.', '')}": v for k, v in lora_state_dict.items() if k.startswith("transformer.")
-        }
-        transformer_state_dict = convert_unet_state_dict_to_peft(transformer_state_dict)
-        incompatible_keys = set_peft_model_state_dict(transformer_, transformer_state_dict, adapter_name="default")
-        if incompatible_keys is not None:
-            # check only for unexpected keys
-            unexpected_keys = getattr(incompatible_keys, "unexpected_keys", None)
-            if unexpected_keys:
-                logger.warning(
-                    f"Loading adapter weights from state_dict led to unexpected keys not found in the model: "
-                    f" {unexpected_keys}. "
-                )
-        if args.train_text_encoder:
-            # Do we need to call `scale_lora_layers()` here?
-            _set_state_dict_into_text_encoder(lora_state_dict, prefix="text_encoder.", text_encoder=text_encoder_one_)
-
-        # Make sure the trainable params are in float32. This is again needed since the base models
-        # are in `weight_dtype`. More details:
-        # https://github.com/huggingface/diffusers/pull/6514#discussion_r1449796804
-        if args.mixed_precision == "fp16":
-            models = [transformer_]
-            if args.train_text_encoder:
-                models.extend([text_encoder_one_])
-            # only upcast trainable parameters (LoRA) into fp32
-            cast_training_params(models)
-
-    accelerator.register_save_state_pre_hook(save_model_hook)
-    accelerator.register_load_state_pre_hook(load_model_hook)
-
-    # Enable TF32 for faster training on Ampere GPUs,
-    # cf https://pytorch.org/docs/stable/notes/cuda.html#tensorfloat-32-tf32-on-ampere-devices
-    if args.allow_tf32 and torch.cuda.is_available():
-        torch.backends.cuda.matmul.allow_tf32 = True
-
-    if args.scale_lr:
-        args.learning_rate = (
-            args.learning_rate * args.gradient_accumulation_steps * args.train_batch_size * accelerator.num_processes
-        )
-
-    # Make sure the trainable params are in float32.
-    if args.mixed_precision == "fp16":
-        models = [transformer]
-        if args.train_text_encoder:
-            models.extend([text_encoder_one])
-        # only upcast trainable parameters (LoRA) into fp32
-        cast_training_params(models, dtype=torch.float32)
-
-    transformer_lora_parameters = list(filter(lambda p: p.requires_grad, transformer.parameters()))
-    if args.train_text_encoder:
-        text_lora_parameters_one = list(filter(lambda p: p.requires_grad, text_encoder_one.parameters()))
-
-    # Optimization parameters
-    transformer_parameters_with_lr = {"params": transformer_lora_parameters, "lr": args.learning_rate}
-    if args.train_text_encoder:
-        # different learning rate for text encoder and unet
-        text_parameters_one_with_lr = {
-            "params": text_lora_parameters_one,
-            "weight_decay": args.adam_weight_decay_text_encoder,
-            "lr": args.text_encoder_lr if args.text_encoder_lr else args.learning_rate,
-        }
-        params_to_optimize = [transformer_parameters_with_lr, text_parameters_one_with_lr]
-    else:
-        params_to_optimize = [transformer_parameters_with_lr]
-
-    # Optimizer creation
-    if not (args.optimizer.lower() == "prodigy" or args.optimizer.lower() == "adamw"):
-        logger.warning(
-            f"Unsupported choice of optimizer: {args.optimizer}.Supported optimizers include [adamW, prodigy]."
-            "Defaulting to adamW"
-        )
-        args.optimizer = "adamw"
-
-    if args.use_8bit_adam and not args.optimizer.lower() == "adamw":
-        logger.warning(
-            f"use_8bit_adam is ignored when optimizer is not set to 'AdamW'. Optimizer was "
-            f"set to {args.optimizer.lower()}"
-        )
-
-    if args.optimizer.lower() == "adamw":
-        if args.use_8bit_adam:
-            try:
-                import bitsandbytes as bnb
-            except ImportError:
-                raise ImportError(
-                    "To use 8-bit Adam, please install the bitsandbytes library: `pip install bitsandbytes`."
-                )
-
-            optimizer_class = bnb.optim.AdamW8bit
-        else:
-            optimizer_class = torch.optim.AdamW
-
-        optimizer = optimizer_class(
-            params_to_optimize,
-            betas=(args.adam_beta1, args.adam_beta2),
-            weight_decay=args.adam_weight_decay,
-            eps=args.adam_epsilon,
-        )
-
-    if args.optimizer.lower() == "prodigy":
-        try:
-            import prodigyopt
-        except ImportError:
-            raise ImportError("To use Prodigy, please install the prodigyopt library: `pip install prodigyopt`")
-
-        optimizer_class = prodigyopt.Prodigy
-
-        if args.learning_rate <= 0.1:
-            logger.warning(
-                "Learning rate is too low. When using prodigy, it's generally better to set learning rate around 1.0"
-            )
-        if args.train_text_encoder and args.text_encoder_lr:
-            logger.warning(
-                f"Learning rates were provided both for the transformer and the text encoder- e.g. text_encoder_lr:"
-                f" {args.text_encoder_lr} and learning_rate: {args.learning_rate}. "
-                f"When using prodigy only learning_rate is used as the initial learning rate."
-            )
-            # changes the learning rate of text_encoder_parameters_one to be
-            # --learning_rate
-            params_to_optimize[1]["lr"] = args.learning_rate
-
-        optimizer = optimizer_class(
-            params_to_optimize,
-            betas=(args.adam_beta1, args.adam_beta2),
-            beta3=args.prodigy_beta3,
-            weight_decay=args.adam_weight_decay,
-            eps=args.adam_epsilon,
-            decouple=args.prodigy_decouple,
-            use_bias_correction=args.prodigy_use_bias_correction,
-            safeguard_warmup=args.prodigy_safeguard_warmup,
-        )
-
-    vae_config_shift_factor = vae.config.shift_factor
-    vae_config_scaling_factor = vae.config.scaling_factor
-    vae_config_block_out_channels = vae.config.block_out_channels
-    if args.cache_latents:
-        latents_cache = build_latents_cache(
-            train_dataset=train_dataset,
-            vae=vae,
-            batch_size=args.train_batch_size,
-            num_workers=args.dataloader_num_workers,
-            device=accelerator.device,
-            dtype=weight_dtype,
-        )
-
-        if not args.validation_prompt or args.num_validation_images <= 0:
-            del vae
-            free_memory()
-
-    # Scheduler and math around the number of training steps.
-    # Check the PR https://github.com/huggingface/diffusers/pull/8312 for detailed explanation.
-    num_warmup_steps_for_scheduler = args.lr_warmup_steps * accelerator.num_processes
-    if args.max_train_steps is None:
-        len_train_dataloader_after_sharding = math.ceil(len(train_dataloader) / accelerator.num_processes)
-        num_update_steps_per_epoch = math.ceil(len_train_dataloader_after_sharding / args.gradient_accumulation_steps)
-        num_training_steps_for_scheduler = (
-            args.num_train_epochs * accelerator.num_processes * num_update_steps_per_epoch
-        )
-    else:
-        num_training_steps_for_scheduler = args.max_train_steps * accelerator.num_processes
-
-    lr_scheduler = get_scheduler(
-        args.lr_scheduler,
-        optimizer=optimizer,
-        num_warmup_steps=num_warmup_steps_for_scheduler,
-        num_training_steps=num_training_steps_for_scheduler,
-        num_cycles=args.lr_num_cycles,
-        power=args.lr_power,
-    )
-
-    # Prepare everything with our `accelerator`.
-    if args.train_text_encoder:
-        (
-            transformer,
-            text_encoder_one,
-            optimizer,
-            train_dataloader,
-            lr_scheduler,
-        ) = accelerator.prepare(
-            transformer,
-            text_encoder_one,
-            optimizer,
-            train_dataloader,
-            lr_scheduler,
-        )
-    else:
-        transformer, optimizer, train_dataloader, lr_scheduler = accelerator.prepare(
-            transformer, optimizer, train_dataloader, lr_scheduler
-        )
-
-    # We need to recalculate our total training steps as the size of the training dataloader may have changed.
-    num_update_steps_per_epoch = math.ceil(len(train_dataloader) / args.gradient_accumulation_steps)
-    if args.max_train_steps is None:
-        args.max_train_steps = args.num_train_epochs * num_update_steps_per_epoch
-        if num_training_steps_for_scheduler != args.max_train_steps:
-            logger.warning(
-                f"The length of the 'train_dataloader' after 'accelerator.prepare' ({len(train_dataloader)}) does not match "
-                f"the expected length ({len_train_dataloader_after_sharding}) when the learning rate scheduler was created. "
-                f"This inconsistency may result in the learning rate scheduler not functioning properly."
-            )
-    # Afterwards we recalculate our number of training epochs
-    args.num_train_epochs = math.ceil(args.max_train_steps / num_update_steps_per_epoch)
-
-    # We need to initialize the trackers we use, and also store our configuration.
-    # The trackers initializes automatically on the main process.
-    if accelerator.is_main_process:
-        tracker_name = "dreambooth-flux-dev-lora"
-        accelerator.init_trackers(tracker_name, config=vars(args))
-
-    # Train!
-    total_batch_size = args.train_batch_size * accelerator.num_processes * args.gradient_accumulation_steps
-
-    logger.info("***** Running training *****")
-    logger.info(f"  Num examples = {len(train_dataset)}")
-    logger.info(f"  Num batches each epoch = {len(train_dataloader)}")
-    logger.info(f"  Num Epochs = {args.num_train_epochs}")
-    logger.info(f"  Instantaneous batch size per device = {args.train_batch_size}")
-    logger.info(f"  Total train batch size (w. parallel, distributed & accumulation) = {total_batch_size}")
-    logger.info(f"  Gradient Accumulation steps = {args.gradient_accumulation_steps}")
-    logger.info(f"  Total optimization steps = {args.max_train_steps}")
-    global_step = 0
-    first_epoch = 0
-
-    # Potentially load in the weights and states from a previous save
-    if args.resume_from_checkpoint:
-        if args.resume_from_checkpoint != "latest":
-            path = os.path.basename(args.resume_from_checkpoint)
-        else:
-            # Get the mos recent checkpoint
-            dirs = os.listdir(args.output_dir)
-            dirs = [d for d in dirs if d.startswith("checkpoint")]
-            dirs = sorted(dirs, key=lambda x: int(x.split("-")[1]))
-            path = dirs[-1] if len(dirs) > 0 else None
-
-        if path is None:
-            accelerator.print(
-                f"Checkpoint '{args.resume_from_checkpoint}' does not exist. Starting a new training run."
-            )
-            args.resume_from_checkpoint = None
-            initial_global_step = 0
-        else:
-            accelerator.print(f"Resuming from checkpoint {path}")
-            accelerator.load_state(os.path.join(args.output_dir, path))
-            global_step = int(path.split("-")[1])
-
-            initial_global_step = global_step
-            first_epoch = global_step // num_update_steps_per_epoch
-
-    else:
-        initial_global_step = 0
-
-    progress_bar = tqdm(
-        range(0, args.max_train_steps),
-        initial=initial_global_step,
-        desc="Steps",
-        # Only show the progress bar once on each machine.
-        disable=not accelerator.is_local_main_process,
-    )
-
-    def get_sigmas(timesteps, n_dim=4, dtype=torch.float32):
-        sigmas = noise_scheduler_copy.sigmas.to(device=accelerator.device, dtype=dtype)
-        schedule_timesteps = noise_scheduler_copy.timesteps.to(accelerator.device)
-        timesteps = timesteps.to(accelerator.device)
-        step_indices = [(schedule_timesteps == t).nonzero().item() for t in timesteps]
-
-        sigma = sigmas[step_indices].flatten()
-        while len(sigma.shape) < n_dim:
-            sigma = sigma.unsqueeze(-1)
-        return sigma
-
-    for epoch in range(first_epoch, args.num_train_epochs):
-        transformer.train()
-        if args.train_text_encoder:
-            text_encoder_one.train()
-            # set top parameter requires_grad = True for gradient checkpointing works
-            _te_one = unwrap_model(text_encoder_one)
-            (_te_one.text_model if hasattr(_te_one, "text_model") else _te_one).embeddings.requires_grad_(True)
-
-        for step, batch in enumerate(train_dataloader):
-            models_to_accumulate = [transformer]
-            if args.train_text_encoder:
-                models_to_accumulate.extend([text_encoder_one])
-            with accelerator.accumulate(models_to_accumulate):
-                prompts = batch["prompts"]
-
-                prompt_embeds, pooled_prompt_embeds, text_ids = get_cached_text_embeddings(
-                    prompt_embed_cache,
-                    prompts,
-                    accelerator.device,
-                )
-
-                # Convert images to latent space
-                if args.cache_latents:
-                    model_input = get_cached_latent_dist(
-                        latents_cache,
-                        batch["instance_indices"],
-                        accelerator.device,
-                        weight_dtype,
-                    ).sample()
-                else:
-                    pixel_values = batch["pixel_values"].to(dtype=vae.dtype)
-                    model_input = vae.encode(pixel_values).latent_dist.sample()
-                model_input = (model_input - vae_config_shift_factor) * vae_config_scaling_factor
-                model_input = model_input.to(dtype=weight_dtype)
-
-                vae_scale_factor = 2 ** (len(vae_config_block_out_channels) - 1)
-
-                latent_image_ids = FluxPipeline._prepare_latent_image_ids(
-                    model_input.shape[0],
-                    model_input.shape[2] // 2,
-                    model_input.shape[3] // 2,
-                    accelerator.device,
-                    weight_dtype,
-                )
-                # Sample noise that we'll add to the latents
-                noise = torch.randn_like(model_input)
-                bsz = model_input.shape[0]
-
-                # Sample a random timestep for each image
-                # for weighting schemes where we sample timesteps non-uniformly
-                u = compute_density_for_timestep_sampling(
-                    weighting_scheme=args.weighting_scheme,
-                    batch_size=bsz,
-                    logit_mean=args.logit_mean,
-                    logit_std=args.logit_std,
-                    mode_scale=args.mode_scale,
-                )
-                indices = (u * noise_scheduler_copy.config.num_train_timesteps).long()
-                timesteps = noise_scheduler_copy.timesteps[indices].to(device=model_input.device)
-
-                # Add noise according to flow matching.
-                # zt = (1 - texp) * x + texp * z1
-                sigmas = get_sigmas(timesteps, n_dim=model_input.ndim, dtype=model_input.dtype)
-                noisy_model_input = (1.0 - sigmas) * model_input + sigmas * noise
-
-                packed_noisy_model_input = FluxPipeline._pack_latents(
-                    noisy_model_input,
-                    batch_size=model_input.shape[0],
-                    num_channels_latents=model_input.shape[1],
-                    height=model_input.shape[2],
-                    width=model_input.shape[3],
-                )
-
-                # handle guidance
-                if unwrap_model(transformer).config.guidance_embeds:
-                    guidance = torch.tensor([args.guidance_scale], device=accelerator.device)
-                    guidance = guidance.expand(model_input.shape[0])
-                else:
-                    guidance = None
-
-                # Predict the noise residual
-                model_pred = transformer(
-                    hidden_states=packed_noisy_model_input,
-                    # YiYi notes: divide it by 1000 for now because we scale it by 1000 in the transformer model (we should not keep it but I want to keep the inputs same for the model for testing)
-                    timestep=timesteps / 1000,
-                    guidance=guidance,
-                    pooled_projections=pooled_prompt_embeds,
-                    encoder_hidden_states=prompt_embeds,
-                    txt_ids=text_ids,
-                    img_ids=latent_image_ids,
-                    return_dict=False,
-                )[0]
-                model_pred = FluxPipeline._unpack_latents(
-                    model_pred,
-                    height=model_input.shape[2] * vae_scale_factor,
-                    width=model_input.shape[3] * vae_scale_factor,
-                    vae_scale_factor=vae_scale_factor,
-                )
-
-                # these weighting schemes use a uniform timestep sampling
-                # and instead post-weight the loss
-                weighting = compute_loss_weighting_for_sd3(weighting_scheme=args.weighting_scheme, sigmas=sigmas)
-
-                # flow matching loss
-                target = noise - model_input
-
-                if args.with_prior_preservation:
-                    # Chunk the noise and model_pred into two parts and compute the loss on each part separately.
-                    model_pred, model_pred_prior = torch.chunk(model_pred, 2, dim=0)
-                    target, target_prior = torch.chunk(target, 2, dim=0)
-                    weighting, weighting_prior = torch.chunk(weighting, 2, dim=0)
-
-                    # Compute prior loss
-                    prior_loss = torch.mean(
-                        (weighting_prior.float() * (model_pred_prior.float() - target_prior.float()) ** 2).reshape(
-                            target_prior.shape[0], -1
-                        ),
-                        1,
-                    )
-                    prior_loss = prior_loss.mean()
-
-                # Compute regular loss.
-                loss = torch.mean(
-                    (weighting.float() * (model_pred.float() - target.float()) ** 2).reshape(target.shape[0], -1),
-                    1,
-                )
-                loss = loss.mean()
-
-                if args.with_prior_preservation:
-                    # Add the prior loss to the instance loss.
-                    loss = loss + args.prior_loss_weight * prior_loss
-
-                accelerator.backward(loss)
-                if accelerator.sync_gradients:
-                    params_to_clip = (
-                        itertools.chain(transformer.parameters(), text_encoder_one.parameters())
-                        if args.train_text_encoder
-                        else transformer.parameters()
-                    )
-                    accelerator.clip_grad_norm_(params_to_clip, args.max_grad_norm)
-
-                optimizer.step()
-                lr_scheduler.step()
-                optimizer.zero_grad()
-
-            # Checks if the accelerator has performed an optimization step behind the scenes
-            if accelerator.sync_gradients:
-                progress_bar.update(1)
-                global_step += 1
-
-                if accelerator.is_main_process:
-                    if global_step % args.checkpointing_steps == 0:
-                        # _before_ saving state, check if this save would set us over the `checkpoints_total_limit`
-                        if args.checkpoints_total_limit is not None:
-                            checkpoints = os.listdir(args.output_dir)
-                            checkpoints = [d for d in checkpoints if d.startswith("checkpoint")]
-                            checkpoints = sorted(checkpoints, key=lambda x: int(x.split("-")[1]))
-
-                            # before we save the new checkpoint, we need to have at _most_ `checkpoints_total_limit - 1` checkpoints
-                            if len(checkpoints) >= args.checkpoints_total_limit:
-                                num_to_remove = len(checkpoints) - args.checkpoints_total_limit + 1
-                                removing_checkpoints = checkpoints[0:num_to_remove]
-
-                                logger.info(
-                                    f"{len(checkpoints)} checkpoints already exist, removing {len(removing_checkpoints)} checkpoints"
-                                )
-                                logger.info(f"removing checkpoints: {', '.join(removing_checkpoints)}")
-
-                                for removing_checkpoint in removing_checkpoints:
-                                    removing_checkpoint = os.path.join(args.output_dir, removing_checkpoint)
-                                    shutil.rmtree(removing_checkpoint)
-
-                        save_path = os.path.join(args.output_dir, f"checkpoint-{global_step}")
-                        accelerator.save_state(save_path)
-                        logger.info(f"Saved state to {save_path}")
-
-            logs = {"loss": loss.detach().item(), "lr": lr_scheduler.get_last_lr()[0]}
-            progress_bar.set_postfix(**logs)
-            accelerator.log(logs, step=global_step)
-
-            if global_step >= args.max_train_steps:
-                break
-
-        if accelerator.is_main_process:
-            if args.validation_prompt and args.num_validation_images > 0 and epoch % args.validation_epochs == 0:
-                # create pipeline
-                if not args.train_text_encoder:
-                    text_encoder_one, text_encoder_two = load_text_encoders(
-                        args, text_encoder_cls_one, text_encoder_cls_two, weight_dtype
-                    )
-                pipeline = FluxPipeline.from_pretrained(
-                    args.pretrained_model_name_or_path,
-                    vae=vae,
-                    text_encoder=unwrap_model(text_encoder_one),
-                    text_encoder_2=unwrap_model(text_encoder_two),
-                    transformer=unwrap_model(transformer),
-                    revision=args.revision,
-                    variant=args.variant,
-                    torch_dtype=weight_dtype,
-                )
-                pipeline_args = {"prompt": args.validation_prompt}
-                images = log_validation(
-                    pipeline=pipeline,
-                    args=args,
-                    accelerator=accelerator,
-                    pipeline_args=pipeline_args,
-                    epoch=epoch,
-                    torch_dtype=weight_dtype,
-                )
-                if not args.train_text_encoder:
-                    del text_encoder_one, text_encoder_two
-                    free_memory()
-
-                images = None
-                del pipeline
-
-    # Save the lora layers
-    accelerator.wait_for_everyone()
-    if accelerator.is_main_process:
-        modules_to_save = {}
-        transformer = unwrap_model(transformer)
-        if args.upcast_before_saving:
-            transformer.to(torch.float32)
-        else:
-            transformer = transformer.to(weight_dtype)
-        transformer_lora_layers = get_peft_model_state_dict(transformer)
-        modules_to_save["transformer"] = transformer
-
-        if args.train_text_encoder:
-            text_encoder_one = unwrap_model(text_encoder_one)
-            text_encoder_lora_layers = get_peft_model_state_dict(text_encoder_one.to(torch.float32))
-            modules_to_save["text_encoder"] = text_encoder_one
-        else:
-            text_encoder_lora_layers = None
-
-        FluxPipeline.save_lora_weights(
-            save_directory=args.output_dir,
-            transformer_lora_layers=transformer_lora_layers,
-            text_encoder_lora_layers=text_encoder_lora_layers,
-            **_collate_lora_metadata(modules_to_save),
-        )
-
-        images = []
-        if args.validation_prompt and args.num_validation_images > 0:
-            # Final inference
-            # Load previous pipeline only when validation output is requested.
-            pipeline = FluxPipeline.from_pretrained(
-                args.pretrained_model_name_or_path,
-                revision=args.revision,
-                variant=args.variant,
-                torch_dtype=weight_dtype,
-            )
-            # load attention processors
-            pipeline.load_lora_weights(args.output_dir)
-
-            # run inference
-            pipeline_args = {"prompt": args.validation_prompt}
-            images = log_validation(
-                pipeline=pipeline,
-                args=args,
-                accelerator=accelerator,
-                pipeline_args=pipeline_args,
-                epoch=epoch,
-                is_final_validation=True,
-                torch_dtype=weight_dtype,
-            )
-            del pipeline
-
-        if args.push_to_hub:
-            save_model_card(
-                repo_id,
-                images=images,
-                base_model=args.pretrained_model_name_or_path,
-                train_text_encoder=args.train_text_encoder,
-                instance_prompt=args.instance_prompt,
-                validation_prompt=args.validation_prompt,
-                repo_folder=args.output_dir,
-            )
-            upload_folder(
-                repo_id=repo_id,
-                folder_path=args.output_dir,
-                commit_message="End of training",
-                ignore_patterns=["step_*", "epoch_*"],
-            )
-
-        images = None
-
-    accelerator.end_training()
-
-
-if __name__ == "__main__":
-    main(parse_args())

code/tests/test_flux_lowmem.py

@@ -1,67 +0,0 @@
-from __future__ import annotations
-
-# pyright: reportPrivateImportUsage=false
-
-import sys
-import unittest
-from pathlib import Path
-
-import torch
-from torch import device, equal, ones, randn, zeros
-
-sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src"))
-
-from lora.trainers.flux_lowmem import get_cached_latent_dist, get_cached_text_embeddings
-
-
-class GetCachedTextEmbeddingsTests(unittest.TestCase):
-    def test_reuses_single_text_ids_tensor_for_batched_prompts(self) -> None:
-        seq_len = 5
-        prompt_embed_cache = {
-            "a": (
-                randn(1, seq_len, 4),
-                randn(1, 6),
-                zeros(seq_len, 3),
-            ),
-            "b": (
-                randn(1, seq_len, 4),
-                randn(1, 6),
-                ones(seq_len, 3),
-            ),
-        }
-
-        prompt_embeds, pooled_prompt_embeds, text_ids = get_cached_text_embeddings(
-            prompt_embed_cache,
-            ["a", "b"],
-            device("cpu"),
-        )
-
-        self.assertEqual(prompt_embeds.shape, (2, seq_len, 4))
-        self.assertEqual(pooled_prompt_embeds.shape, (2, 6))
-        self.assertEqual(text_ids.shape, (seq_len, 3))
-        self.assertTrue(equal(text_ids, prompt_embed_cache["a"][2]))
-
-
-class GetCachedLatentDistTests(unittest.TestCase):
-    def test_rebuilds_batched_distribution_from_instance_indices(self) -> None:
-        latents_cache = [
-            randn(32, 64, 64),
-            randn(32, 64, 64),
-            randn(32, 64, 64),
-        ]
-
-        latent_dist = get_cached_latent_dist(
-            latents_cache,
-            [2, 0],
-            device("cpu"),
-            torch.float32,
-        )
-
-        self.assertEqual(latent_dist.parameters.shape, (2, 32, 64, 64))
-        self.assertTrue(equal(latent_dist.parameters[0], latents_cache[2]))
-        self.assertTrue(equal(latent_dist.parameters[1], latents_cache[0]))
-        self.assertEqual(latent_dist.sample().shape, (2, 16, 64, 64))
-
-
-if __name__ == "__main__":
-    unittest.main()