README.md

---
base_model: stabilityai/stable-diffusion-xl-base-1.0
library_name: diffusers
license: openrail++
tags:
- text-to-image
- text-to-image
- diffusers-training
- diffusers
- lora
- template:sd-lora
- stable-diffusion-xl
- stable-diffusion-xl-diffusers
instance_prompt: a photo of sks chair
widget:
- text: A photo of sks chair in an apartment
  output:
    url: image_0.png
- text: A photo of sks chair in an apartment
  output:
    url: image_1.png
- text: A photo of sks chair in an apartment
  output:
    url: image_2.png
- text: A photo of sks chair in an apartment
  output:
    url: image_3.png
datasets:
- AdamLucek/green-chair
---

# SDXL LoRA DreamBooth - AdamLucek/sdxl-base-1.0-greenchair-dreambooth-lora

## Model description

These are LoRA DreamBooth weights for [stabilityai/stable-diffusion-xl-base-1.0](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0).

The weights were trained using [DreamBooth](https://dreambooth.github.io/) on the [AdamLucek/green-chair](https://huggingface.co/datasets/AdamLucek/green-chair) Dataset.

LoRA for the text encoder was enabled: **True**.

Special VAE used for training: [madebyollin/sdxl-vae-fp16-fix](https://huggingface.co/madebyollin/sdxl-vae-fp16-fix).

## Trigger words

You should use **a photo of sks chair** to trigger the image generation.

## Example Generations

**Reference Image**
<img src="https://cdn-uploads.huggingface.co/production/uploads/65ba68a15d2ef0a4b2c892b4/hvvyMxmnXK36wAEoEPCn2.jpeg" width="350">
**Generated Images**
<img src="https://cdn-uploads.huggingface.co/production/uploads/65ba68a15d2ef0a4b2c892b4/tTVfzzi4H09-H8w4dAyA3.jpeg" width="500">
*on a street in new york*, *in a desert*, *in a jungle*, *in the color blue*
## Download model

Weights for this model are available in Safetensors format.

## Intended uses & limitations

<div style="display: flex; align-items: center;">
  <img src="https://upload.wikimedia.org/wikipedia/commons/d/d0/Google_Colaboratory_SVG_Logo.svg" width="100">
  <a href="https://colab.research.google.com/drive/1v503hMrThIy87xozZBMPSDBc53Bukk_1?usp=sharing" style="margin-left: 10px;">COLAB Notebook Here</a>
</div>

#### How to use

```python
# Load Stable Diffusion XL Base 1.0
pipe = DiffusionPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0", 
    torch_dtype=torch.float16,
    variant="fp16", 
    use_safetensors=True
).to("cuda")

# Optional, enable cpu offloading
pipe.enable_model_cpu_offload()

# Load LoRA Weights
pipe.load_lora_weights("AdamLucek/sdxl-base-1.0-greenchair-dreambooth-lora")

# Generate an Image
image = pipe(
    prompt = "a photo of sks chair",
    num_inference_steps=50,
    height=1024,
    width=1024,
).images[0]

# Save the Image
image.save("green_chair.png")
```

#### Limitations and bias

**Note**: Limited tuning of hyperparameters

**Note**: See original Stable Diffusion XL Base 1.0 page for additional limitations and biases

## Training details

**Video Overview**
<a href="https://youtu.be/v89kB4OScOA">
  <img src="https://i.imgur.com/fW6hHu2.png" width="350">
</a>

Trained using [Dreambooth](https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/README_sdxl.md) via [Diffusers](https://huggingface.co/docs/diffusers/main/en/index) on a single A100

Training Script:
```
accelerate launch train_dreambooth_lora_sdxl.py \
  --pretrained_model_name_or_path="stabilityai/stable-diffusion-xl-base-1.0"  \
  --dataset_name="AdamLucek/green-chair" \
  --pretrained_vae_model_name_or_path="madebyollin/sdxl-vae-fp16-fix" \
  --output_dir="lora-trained-xl" \
  --train_text_encoder \
  --instance_prompt="a photo of sks chair" \
  --resolution=1024 \
  --train_batch_size=1 \
  --gradient_accumulation_steps=4 \
  --learning_rate=1e-4 \
  --lr_scheduler="constant" \
  --lr_warmup_steps=0 \
  --max_train_steps=500 \
  --validation_prompt="A photo of sks chair in an apartment" \
  --validation_epochs=25 \
  --seed="0" \
  --hub_model_id="sdxl-base-1.0-greenchair-dreambooth-lora" \
  --push_to_hub
```