---
license: apache-2.0
base_model:
- ByteDance-Seed/BAGEL-7B-MoT
base_model_relation: quantized
pipeline_tag: any-to-any
library_name: bagel
tags: 
  - fp8
  - quantized
  - bagel
  - mot 
---

Original model is https://huggingface.co/ByteDance-Seed/BAGEL-7B-MoT

ema-FP8.safetensors is float8_e4m3fn.

float8_e4m3fn weight of: https://huggingface.co/ByteDance-Seed/BAGEL-7B-MoT

## Benchmark  Spec: 24GB 4090 + 60GB RAM  
### Default setting, Timesteps 25 steps  
| Features | Speed (seconds) | GPU VRAM Usage | CPU RAM Usage |
|---------------------|------------------|----------------|----------------|
| 📝 Text to Image    | 128.90 s          | 16.18 GB       | 14.22 GB       |
| 🖌️ Image Edit        | 138.67 s          | 15.08 GB       | 14.21 GB       |
| 🖼️ Image Understanding | 102.68 s      | 15.08 GB       |13.66 GB        |

[Benchmark Images](https://huggingface.co/meimeilook/BAGEL-7B-MoT-FP8/tree/main/Benchmark)

## Support
### Runs with less than 12GB of GPU memory.  
### ram + vram = about 31GB 
####  * *12GB is much slower than 24GB due to CPU offload. It will be 1.5x much slower than 24GB* 


![4070-12GB.jpg](https://huggingface.co/meimeilook/BAGEL-7B-MoT-FP8/resolve/main/Assets/4070-12GB.jpg "4070-12GB.jpg")



## How to Install：
### new venv
1. git clone https://github.com/bytedance-seed/BAGEL.git
2. cd BAGEL
3. conda create -n bagel python=3.10 -y
4. conda activate bagel



### install
5. install pytorch 2.5.1   
CUDA 12.4  
pip install torch==2.5.1 torchvision==0.20.1 --index-url https://download.pytorch.org/whl/cu124  

6. pip install [flash_attn-2.7.0.post1+cu12torch2.5cxx11abiFALSE-cp310-cp310-linux_x86_64.whl](https://github.com/Dao-AILab/flash-attention/releases/download/v2.7.0.post1/flash_attn-2.7.0.post1+cu12torch2.5cxx11abiFALSE-cp310-cp310-linux_x86_64.whl)  
**more whl:  [https://github.com/Dao-AILab/flash-attention/releases](https://github.com/Dao-AILab/flash-attention/releases)  
 It needs to be the same as the Python version, PyTorch version, CUDA version, and flash_attn WHL.**

7. pip install -r requirements.txt  
(edit requirements.txt, without flash_attn==2.5.8, make it #flash_attn==2.5.8)  

8. pip install gradio  pynvml  (#pynvml for check vram stats.)  



## Models & Settings:
0. Download [huggingface.co/ByteDance-Seed/BAGEL-7B-MoT](https://huggingface.co/ByteDance-Seed/BAGEL-7B-MoT)(without ema.safetensors) & 
[ema-FP8.safetensors](https://huggingface.co/meimeilook/BAGEL-7B-MoT-FP8/blob/main/ema-FP8.safetensors)
and make it like this. 

```
folders
├── BAGEL
│   └── app-fp8.py
└── BAGEL-7B-MoT
    └── ema-FP8.safetensors
```
0. Open app-fp8.py via Notepad or VScode etc.  

1. Replace model_path to yours.

```
parser.add_argument("--model_path", type=str, default="/root/your_path/BAGEL-7B-MoT")
```

2. Edit your spec:

```
cpu_mem_for_offload = "16GiB"
gpu_mem_per_device = "24GiB"  #default：24GiB  you can set 16GB within 24GB with 4090,more slower.
```

3. Be more efficient

```
NUM_ADDITIONAL_LLM_LAYERS_TO_GPU = 5  
# (5 for 24gb VRAM, >5 for 32gb VRAM, have a try)
# The default is 10 layers in GPU, use it can be 15 layers in GPU with 4090.
```



## How to Use:
1. CD BAGEL
2. conda activate bagel
3. python app-fp8.py
4. Open [127.0.0.1:7860](https://127.0.0.1:7860)  


![demo.jpg](https://huggingface.co/meimeilook/BAGEL-7B-MoT-FP8/resolve/main/Assets/demo.jpg "demo.jpg")