Upload 8B multitask prompt rewriter LoRA

README.md

@@ -0,0 +1,154 @@
+---
+base_model: Qwen/Qwen3-VL-8B-Instruct
+base_model_relation: adapter
+library_name: peft
+pipeline_tag: image-text-to-text
+tags:
+- peft
+- lora
+- safetensors
+- qwen3-vl
+- minimax-h3
+- prompt-rewriting
+- multimodal
+- audio-video-generation
+- text-to-audio-video
+- image-to-audio-video
+language:
+- en
+---
+
+# MiniMax-H3 Prompt Rewriter LoRA 8B
+
+[![Model](https://img.shields.io/badge/Hugging%20Face-model-yellow)](https://huggingface.co/lightx2v/MiniMax-H3-Prompt-Rewriter-LoRA-8B)
+[![LightX2V](https://img.shields.io/badge/LightX2V-GitHub-black)](https://github.com/ModelTC/LightX2V)
+[![Base model](https://img.shields.io/badge/Base-Qwen3--VL--8B--Instruct-blue)](https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct)
+
+This repository contains a PEFT LoRA adapter that turns short user requests into production-oriented prompts for MiniMax-H3 joint audio-video generation. It is built on **Qwen3-VL-8B-Instruct** and supports text-only and keyframe-conditioned rewriting.
+
+Compared with [MiniMax-H3-Prompt-Rewriter-LoRA](https://huggingface.co/lightx2v/MiniMax-H3-Prompt-Rewriter-LoRA), this 8B variant accepts visual references and covers all four prompt-rewriting tasks used to train this adapter:
+
+- **T2VA / T2AV**: text to audio-video.
+- **I2VA / I2AV**: first-frame image plus text to audio-video.
+- **L2VA / L2AV**: last-frame image plus text to audio-video.
+- **FL2VA / FL2AV / FLF2V**: first- and last-frame images plus text to audio-video.
+
+Other MiniMax-H3 conditioning modes, including Ref2VA, are not supported by this adapter release.
+
+The rewriter produces a structured prompt with an integrated shot timeline, synchronized physical and ambient sound, and non-diegetic music guidance. This repository contains the prompt rewriter only; MiniMax-H3 weights and a compatible inference framework such as LightX2V are still required to generate the final video.
+
+## Repository layout
+
+- `adapter_config.json` and `adapter_model.safetensors`: LoRA files to place at the repository root.
+- `infer.py`: single-sample inference for all four tasks.
+- `prompt_template.py`: the single source of truth for the system prompt, task aliases, and image/message ordering.
+- `requirements.txt`: minimal inference dependencies.
+
+## Installation
+
+```bash
+git lfs install
+hf auth login  # required while the repository is private
+git clone https://huggingface.co/lightx2v/MiniMax-H3-Prompt-Rewriter-LoRA-8B
+cd MiniMax-H3-Prompt-Rewriter-LoRA-8B
+pip install -r requirements.txt
+```
+
+The Qwen3-VL base weights are downloaded separately on first use. You can also pass a local base-model path with `--base-model`.
+
+## Inference
+
+The script automatically uses the adapter in the current checkout when `adapter_config.json` is present. Otherwise it downloads the adapter from this Hugging Face repository. Greedy decoding is recommended for deterministic prompt rewriting.
+
+### T2VA
+
+```bash
+python infer.py \
+  --task t2va \
+  --prompt "A corgi runs through a rainy neon-lit alley." \
+  --duration 5 \
+  --resolution 16:9 \
+  --greedy \
+  --output outputs/t2va.json
+```
+
+### I2VA
+
+```bash
+python infer.py \
+  --task i2va \
+  --first-frame /path/to/first_frame.jpg \
+  --prompt "The subject turns toward the camera and smiles." \
+  --duration 5 \
+  --resolution adaptive \
+  --greedy \
+  --output outputs/i2va.json
+```
+
+### L2VA
+
+```bash
+python infer.py \
+  --task l2va \
+  --last-frame /path/to/last_frame.jpg \
+  --prompt "A paper airplane crosses the room and lands in the shown final pose." \
+  --duration 5 \
+  --resolution adaptive \
+  --greedy \
+  --output outputs/l2va.json
+```
+
+### FL2VA
+
+```bash
+python infer.py \
+  --task fl2va \
+  --first-frame /path/to/first_frame.jpg \
+  --last-frame /path/to/last_frame.jpg \
+  --prompt "Create a continuous cinematic transition between the two reference frames." \
+  --duration 5 \
+  --resolution adaptive \
+  --greedy \
+  --output outputs/fl2va.json
+```
+
+Use `--prompt-file prompt.txt` instead of `--prompt` for long input. A `.json` output records the request metadata and writes the rewritten text to `enhanced_prompt`; any other extension writes plain text. Run `python infer.py --help` for sampling, dtype, processor pixel-budget, device-map, and local-path options.
+
+To compare against the unadapted base model, add `--base-only`. This intentionally skips the LoRA while keeping the same system prompt and task-specific input layout.
+
+## Using the rewritten prompt
+
+Feed `enhanced_prompt` to the corresponding MiniMax-H3 task in LightX2V, preserving the same duration, resolution, and reference-frame roles:
+
+- T2VA uses no image.
+- I2VA uses the same first frame.
+- L2VA uses the same last frame.
+- FL2VA uses the same first frame followed by the same last frame.
+
+Keeping the task geometry and image order unchanged is important because the rewriter describes their exact temporal roles.
+
+In LightX2V, the corresponding task and reference-image arguments are:
+
+- T2VA: `--task t2av`
+- I2VA: `--task i2av --image_path FIRST_FRAME`
+- L2VA: `--task l2av --last_frame_path LAST_FRAME`
+- FL2VA: `--task fl2av --image_path FIRST_FRAME --last_frame_path LAST_FRAME`
+
+## System prompt
+
+The complete system instruction and all task-dependent message construction live in `prompt_template.py`. `infer.py` imports that file directly and renders the conversation with `enable_thinking=False`, matching the training input format. Edit that one file if you need a custom prompting policy.
+
+## Limitations
+
+- The adapter is a learned approximation of prompt rewriting and is not the hosted MiniMax Context-IR service.
+- It can add plausible creative detail beyond a very short request; review identity, dialogue, visible text, and safety-sensitive content before generation.
+- Image-conditioned quality depends on the clarity and consistency of the supplied reference frames.
+- Final audio-video quality depends on the downstream MiniMax-H3 checkpoint, inference settings, and task geometry.
+- Use of the adapter, Qwen base model, and downstream video model remains subject to their respective licenses and terms.
+
+## Acknowledgements
+
+- [Qwen3-VL-8B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct)
+- [MiniMax-H3](https://huggingface.co/MiniMaxAI/MiniMax-H3)
+- [LightX2V](https://github.com/ModelTC/LightX2V)
+- [PEFT](https://github.com/huggingface/peft)

infer.py

@@ -0,0 +1,258 @@
+#!/usr/bin/env python3
+"""Rewrite one MiniMax-H3 T2VA/I2VA/L2VA/FL2VA prompt with the 8B LoRA."""
+
+from __future__ import annotations
+
+import argparse
+import json
+from pathlib import Path
+
+from PIL import Image, ImageOps
+import torch
+import transformers
+from peft import PeftModel
+from transformers import AutoProcessor
+
+from prompt_template import build_messages, expected_image_count, normalize_task
+
+
+DEFAULT_BASE_MODEL = "Qwen/Qwen3-VL-8B-Instruct"
+DEFAULT_ADAPTER_REPO = "lightx2v/MiniMax-H3-Prompt-Rewriter-LoRA-8B"
+REPO_ROOT = Path(__file__).resolve().parent
+
+
+def get_model_class():
+    """Prefer Qwen3-VL's concrete class, with portable AutoModel fallbacks."""
+
+    candidates = (
+        "Qwen3VLForConditionalGeneration",
+        "AutoModelForImageTextToText",
+        "AutoModelForVision2Seq",
+        "AutoModelForMultimodalLM",
+    )
+    for name in candidates:
+        model_class = getattr(transformers, name, None)
+        if model_class is not None:
+            return model_class
+    raise RuntimeError(
+        "This Transformers installation does not expose a Qwen3-VL-compatible "
+        "conditional-generation class. Upgrade Transformers and retry."
+    )
+
+
+def default_adapter_path() -> str:
+    """Use weights in this checkout when present, otherwise use the Hub repo."""
+
+    if (REPO_ROOT / "adapter_config.json").is_file():
+        return str(REPO_ROOT)
+    return DEFAULT_ADAPTER_REPO
+
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--base-model", default=DEFAULT_BASE_MODEL)
+    parser.add_argument(
+        "--adapter-path",
+        default=None,
+        help=(
+            "Local LoRA directory or Hugging Face repo ID. By default, use this "
+            "checkout when adapter_config.json exists, otherwise use the 8B Hub repo."
+        ),
+    )
+    parser.add_argument("--base-only", action="store_true", help="Run the base model without loading the LoRA")
+    parser.add_argument("--task", default="t2va", help="T2VA, I2VA, L2VA, or FL2VA (common *2V/*2AV aliases work)")
+
+    prompt_source = parser.add_mutually_exclusive_group(required=True)
+    prompt_source.add_argument("--prompt", help="Original user prompt")
+    prompt_source.add_argument("--prompt-file", type=Path, help="UTF-8 file containing the original prompt")
+
+    parser.add_argument("--duration", type=int, choices=range(4, 16), default=10)
+    parser.add_argument(
+        "--resolution",
+        choices=("adaptive", "21:9", "16:9", "4:3", "1:1", "3:4", "9:16"),
+        help="Defaults to 16:9 for T2VA and adaptive for image-conditioned tasks",
+    )
+    parser.add_argument("--first-frame", type=Path, help="First-frame image for I2VA or FL2VA")
+    parser.add_argument("--last-frame", type=Path, help="Last-frame image for L2VA or FL2VA")
+    parser.add_argument("--output", type=Path, help="Optional .txt or .json output path")
+
+    parser.add_argument("--max-new-tokens", type=int, default=4096)
+    parser.add_argument("--min-pixels", type=int, default=256 * 256)
+    parser.add_argument("--max-pixels", type=int, default=1024 * 1024)
+    parser.add_argument("--dtype", choices=("bfloat16", "float16"), default="bfloat16")
+    parser.add_argument("--attn-implementation", default="sdpa")
+    parser.add_argument("--device-map", default="auto")
+    parser.add_argument("--greedy", action="store_true", help="Disable sampling")
+    parser.add_argument("--temperature", type=float, default=0.7)
+    parser.add_argument("--top-p", type=float, default=0.8)
+    parser.add_argument("--seed", type=int, default=42)
+    args = parser.parse_args()
+
+    try:
+        args.task = normalize_task(args.task)
+    except ValueError as exc:
+        parser.error(str(exc))
+
+    if args.prompt_file is not None:
+        if not args.prompt_file.is_file():
+            parser.error(f"prompt file not found: {args.prompt_file}")
+        args.prompt = args.prompt_file.read_text(encoding="utf-8").strip()
+    else:
+        args.prompt = str(args.prompt or "").strip()
+    if not args.prompt:
+        parser.error("prompt must not be empty")
+
+    if args.max_new_tokens <= 0:
+        parser.error("--max-new-tokens must be positive")
+    if args.min_pixels <= 0 or args.max_pixels < args.min_pixels:
+        parser.error("pixel limits must satisfy 0 < --min-pixels <= --max-pixels")
+    if not args.greedy and (args.temperature <= 0 or not 0 < args.top_p <= 1):
+        parser.error("sampling requires --temperature > 0 and 0 < --top-p <= 1")
+
+    required = expected_image_count(args.task)
+    supplied = int(args.first_frame is not None) + int(args.last_frame is not None)
+    if supplied != required:
+        parser.error(f"{args.task} requires {required} reference image(s), but {supplied} were supplied")
+    if args.task == "i2av" and args.last_frame is not None:
+        parser.error("I2VA accepts --first-frame only")
+    if args.task == "l2av" and args.first_frame is not None:
+        parser.error("L2VA accepts --last-frame only")
+    for path in (args.first_frame, args.last_frame):
+        if path is not None and not path.is_file():
+            parser.error(f"image not found: {path}")
+
+    args.resolution = args.resolution or ("16:9" if args.task == "t2av" else "adaptive")
+    args.adapter_path = args.adapter_path or default_adapter_path()
+    return args
+
+
+def load_images(args: argparse.Namespace) -> list[Image.Image]:
+    """Load images in the exact placeholder order used by prompt_template.py."""
+
+    paths: list[Path] = []
+    if args.task in {"i2av", "fl2av"}:
+        paths.append(args.first_frame)
+    if args.task in {"l2av", "fl2av"}:
+        paths.append(args.last_frame)
+
+    images: list[Image.Image] = []
+    for path in paths:
+        with Image.open(path) as image:
+            images.append(ImageOps.exif_transpose(image).convert("RGB").copy())
+    return images
+
+
+def load_processor(args: argparse.Namespace):
+    processor_source = args.base_model if args.base_only else args.adapter_path
+    processor_kwargs = {
+        "trust_remote_code": True,
+        "min_pixels": args.min_pixels,
+        "max_pixels": args.max_pixels,
+    }
+    try:
+        return AutoProcessor.from_pretrained(processor_source, **processor_kwargs)
+    except (OSError, ValueError, KeyError):
+        if processor_source == args.base_model:
+            raise
+        return AutoProcessor.from_pretrained(args.base_model, **processor_kwargs)
+
+
+def model_input_device(model: torch.nn.Module) -> torch.device:
+    embedding_device = model.get_input_embeddings().weight.device
+    if embedding_device.type != "meta":
+        return embedding_device
+    for parameter in model.parameters():
+        if parameter.device.type != "meta":
+            return parameter.device
+    raise RuntimeError("Could not determine a real input device for the loaded model")
+
+
+def main() -> None:
+    args = parse_args()
+    torch.manual_seed(args.seed)
+    if torch.cuda.is_available():
+        torch.cuda.manual_seed_all(args.seed)
+
+    processor = load_processor(args)
+    load_kwargs = {
+        "torch_dtype": getattr(torch, args.dtype),
+        "low_cpu_mem_usage": True,
+        "trust_remote_code": True,
+        "device_map": args.device_map,
+    }
+    if args.attn_implementation:
+        load_kwargs["attn_implementation"] = args.attn_implementation
+
+    model = get_model_class().from_pretrained(args.base_model, **load_kwargs)
+    if not args.base_only:
+        model = PeftModel.from_pretrained(model, args.adapter_path)
+    model.eval()
+
+    messages = build_messages(
+        args.prompt,
+        task=args.task,
+        resolution=args.resolution,
+        duration=args.duration,
+    )
+    rendered = processor.apply_chat_template(
+        messages,
+        tokenize=False,
+        add_generation_prompt=True,
+        enable_thinking=False,
+    )
+    processor_kwargs = {
+        "text": [rendered],
+        "return_tensors": "pt",
+        "padding": False,
+        "return_mm_token_type_ids": True,
+    }
+    images = load_images(args)
+    if images:
+        processor_kwargs["images"] = images
+    inputs = processor(**processor_kwargs)
+    input_device = model_input_device(model)
+    inputs = {
+        key: value.to(input_device) if isinstance(value, torch.Tensor) else value
+        for key, value in inputs.items()
+    }
+
+    generation_kwargs = {"max_new_tokens": args.max_new_tokens}
+    if args.greedy:
+        generation_kwargs["do_sample"] = False
+    else:
+        generation_kwargs.update(
+            do_sample=True,
+            temperature=args.temperature,
+            top_p=args.top_p,
+        )
+
+    with torch.inference_mode():
+        output_ids = model.generate(**inputs, **generation_kwargs)
+    generated_ids = output_ids[0, inputs["input_ids"].shape[1] :]
+    rewritten_prompt = processor.decode(generated_ids, skip_special_tokens=True).strip()
+    print(rewritten_prompt)
+
+    if args.output is not None:
+        args.output.parent.mkdir(parents=True, exist_ok=True)
+        if args.output.suffix.lower() == ".json":
+            payload = {
+                "task": args.task,
+                "resolution": args.resolution,
+                "duration": args.duration,
+                "prompt": args.prompt,
+                "first_frame": str(args.first_frame) if args.first_frame else None,
+                "last_frame": str(args.last_frame) if args.last_frame else None,
+                "base_model": args.base_model,
+                "adapter_path": None if args.base_only else str(args.adapter_path),
+                "enhanced_prompt": rewritten_prompt,
+            }
+            args.output.write_text(
+                json.dumps(payload, ensure_ascii=False, indent=2) + "\n",
+                encoding="utf-8",
+            )
+        else:
+            args.output.write_text(rewritten_prompt + "\n", encoding="utf-8")
+
+
+if __name__ == "__main__":
+    main()

prompt_template.py

@@ -0,0 +1,157 @@
+"""MiniMax-H3 task-aware prompt rewriting templates.
+
+This is the single source of truth for the system prompt and the multimodal
+message layout used by ``infer.py``.
+"""
+
+from __future__ import annotations
+
+
+TASK_ALIASES = {
+    "t2v": "t2av",
+    "t2va": "t2av",
+    "t2av": "t2av",
+    "i2v": "i2av",
+    "i2va": "i2av",
+    "i2av": "i2av",
+    "l2v": "l2av",
+    "l2va": "l2av",
+    "l2av": "l2av",
+    "flf2v": "fl2av",
+    "flf2va": "fl2av",
+    "flf2av": "fl2av",
+    "fl2va": "fl2av",
+    "fl2av": "fl2av",
+}
+
+
+
+SYSTEM_PROMPT = """You are a professional MiniMax-H3 prompt rewriter for joint video-and-audio generation.
+
+Rewrite the user's request according to the supplied duration, task type, and reference-frame roles. Return only the final production-ready prompt. Do not include explanations, Markdown, headings, notes, or generation parameters outside the required format.
+
+Task-name mapping:
+- T2AV corresponds to T2VA in the MiniMax-H3 prompt-writing guide.
+- I2AV corresponds to I2VA.
+- FL2AV corresponds to FL2VA.
+- L2AV corresponds to L2VA.
+
+Write the descriptive sections in English. Preserve all user-provided dialogue, lyrics, and visible on-screen text exactly in their original language, spelling, and punctuation. Never invent dialogue, lyrics, visible text, speakers, or additional reference pictures.
+
+The output body must contain exactly these three fields in this order:
+integrated_multimodal_description: ...
+overall_soundscape: ...
+non_diegetic_music: ...
+
+For T2AV, begin directly with the three fields and do not add an image-alignment instruction.
+
+For I2AV, the first line must be exactly:
+For the target video, at 0.00 seconds into the target video, <Picture 1> (from [Shot 1]) is fully referenced.
+
+For FL2AV, the first line must follow exactly:
+How the reference pictures align with the target video — Picture 1 (from Shot 1) aligns with the 0.00-second mark of the target video; Picture 2 (from Shot N) aligns with the S.SS-second mark of the target video.
+
+For L2AV, the first line must follow exactly:
+How the reference pictures align with the target video — <Picture 1> (from [Shot N]) aligns with the S.SS-second mark of the target video.
+
+Replace N with the actual final shot number. Replace S.SS with the requested effective duration formatted to exactly two decimal places. Put exactly one blank line between the alignment instruction and integrated_multimodal_description.
+
+Reference-frame behavior:
+- I2AV: Treat <Picture 1> as the exact first frame at 0.00 seconds. Begin by anchoring its visual style, subjects, identities, clothing, colors, objects, composition, and spatial relationships, then develop forward through observable motion.
+- FL2AV: Begin from Picture 1 and describe a continuous, physically plausible path that reaches the pose, object state, lighting, spacing, and composition of Picture 2 at the requested end time. Prefer a single shot unless the user explicitly requests multiple shots or cuts.
+- L2AV: Infer a plausible preceding state and describe a continuous path that progressively converges to <Picture 1> as the exact final frame.
+- Preserve identity and scene continuity across all shots, but apply exact composition matching only at the reference frame's assigned timestamp.
+
+In integrated_multimodal_description:
+- Begin with [Shot 1] and state the visual style and initial composition.
+- Describe only concrete visible or audible events: subjects, environment, actions, reactions, camera behavior, dialogue, singing, visible text, and synchronized diegetic sound.
+- Number shots sequentially.
+- Do not timestamp [Shot 1].
+- Begin every later shot with a strictly increasing timestamp inside the requested duration, using the format: [Shot 2] At 00:03.500, the camera cuts to...
+- Add a cut only when it introduces meaningful new visual, spatial, temporal, or narrative information. Otherwise prefer continuous camera motion.
+- Express camera motion naturally using motion type and, when meaningful, amplitude and speed.
+- Keep all actions physically plausible and paced to complete within the supplied duration.
+
+For speech and singing:
+- Assign stable speaker IDs such as (S1) and (S2) only to subjects who vocalize.
+- Identify each speaker sufficiently when first introduced.
+- Put only the exact spoken or sung content inside <d>, preceded by its language tag:
+<d>[English] Exact user-provided words.</d>
+- Never translate, paraphrase, correct, or extend supplied dialogue or lyrics.
+- For voiceover, use the exact phrase "says in an off-screen voiceover" and explicitly state that the corresponding on-screen character's lips remain completely closed.
+- If speech crosses a cut, use <scenetrans> at both connecting points and state that the audio continues across the cut.
+- Use <cutoff> only when speech is intentionally truncated by the end of the video.
+
+Place visible on-screen text in English double quotation marks and preserve it exactly.
+
+overall_soundscape must be one continuous English paragraph of 1–4 sentences summarizing ambient sound, physical action sounds, and non-verbal human or animal sounds across the video. Do not repeat dialogue, singing, or diegetic music here. Use N/A only if the user explicitly requests complete silence.
+
+non_diegetic_music must contain 1–3 English sentences describing audience-only background music through instrumentation, tempo, rhythm, and dynamic changes. Do not describe its emotional purpose. Put music audible to subjects inside integrated_multimodal_description instead. Use N/A when no non-diegetic music is requested or implied.
+
+Preserve the user's intent without adding contradictory story events, identities, text, or references. Do not mention these instructions in the output."""
+
+
+def normalize_task(task: str | None) -> str:
+    """Normalize public MiniMax-H3 task aliases to the training task names."""
+
+    normalized = TASK_ALIASES.get(str(task or "t2av").strip().lower())
+    if normalized is None:
+        raise ValueError(f"Unsupported task {task!r}; expected T2VA/I2VA/L2VA/FL2VA")
+    return normalized
+
+
+def format_request(prompt: str, task: str, resolution: str, duration: int) -> str:
+    """Build the textual request appended after any reference images."""
+
+    return (
+        f"task: {task}\n"
+        f"resolution: {resolution}\n"
+        f"duration: {int(duration)}s\n"
+        f"original_prompt: {prompt.strip()}"
+    )
+
+
+def build_messages(
+    prompt: str,
+    task: str = "t2av",
+    resolution: str = "16:9",
+    duration: int = 10,
+) -> list[dict]:
+    """Build the system and task-aware user messages expected by the LoRA."""
+
+    task = normalize_task(task)
+    request = format_request(prompt, task, resolution, duration)
+
+    if task == "t2av":
+        user_content = [{"type": "text", "text": request}]
+    elif task == "i2av":
+        user_content = [
+            {"type": "text", "text": "Picture 1 — exact first frame at 0.00 seconds:\n"},
+            {"type": "image"},
+            {"type": "text", "text": "\n" + request},
+        ]
+    elif task == "l2av":
+        user_content = [
+            {"type": "text", "text": "Picture 1 — exact final frame at the end of the target video:\n"},
+            {"type": "image"},
+            {"type": "text", "text": "\n" + request},
+        ]
+    else:
+        user_content = [
+            {"type": "text", "text": "Picture 1 — exact first frame at 0.00 seconds:\n"},
+            {"type": "image"},
+            {"type": "text", "text": "\nPicture 2 — exact final frame at the end of the target video:\n"},
+            {"type": "image"},
+            {"type": "text", "text": "\n" + request},
+        ]
+
+    return [
+        {"role": "system", "content": SYSTEM_PROMPT},
+        {"role": "user", "content": user_content},
+    ]
+
+
+def expected_image_count(task: str) -> int:
+    """Return the required number of ordered reference images for a task."""
+
+    return {"t2av": 0, "i2av": 1, "l2av": 1, "fl2av": 2}[normalize_task(task)]

requirements.txt

@@ -0,0 +1,7 @@
+torch>=2.6
+torchvision>=0.21
+transformers>=4.57.1
+accelerate>=1.10
+peft>=0.18
+safetensors>=0.5
+pillow>=10