Instructions to use ethzanalytics/blip2-flan-t5-xl-sharded with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ethzanalytics/blip2-flan-t5-xl-sharded with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="ethzanalytics/blip2-flan-t5-xl-sharded")

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("ethzanalytics/blip2-flan-t5-xl-sharded")
model = AutoModelForMultimodalLM.from_pretrained("ethzanalytics/blip2-flan-t5-xl-sharded")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ethzanalytics/blip2-flan-t5-xl-sharded with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ethzanalytics/blip2-flan-t5-xl-sharded"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ethzanalytics/blip2-flan-t5-xl-sharded",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/ethzanalytics/blip2-flan-t5-xl-sharded

SGLang

How to use ethzanalytics/blip2-flan-t5-xl-sharded with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ethzanalytics/blip2-flan-t5-xl-sharded" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ethzanalytics/blip2-flan-t5-xl-sharded",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ethzanalytics/blip2-flan-t5-xl-sharded" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ethzanalytics/blip2-flan-t5-xl-sharded",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use ethzanalytics/blip2-flan-t5-xl-sharded with Docker Model Runner:
```
docker model run hf.co/ethzanalytics/blip2-flan-t5-xl-sharded
```

Sharded BLIP-2 Model Card - flan-t5-xl

This is a sharded version of the blip2-flan-t5-xl which leverages Flan T5-xl for image-to-text tasks such as image captioning and visual question answering.

this model repo is sharded so it can be easily loaded on low-RAM Colab runtimes :)
Refer to the original model card for more details about the model description, intended uses, and limitations, as well as instructions for how to use the model on CPU and GPU in different precisions.

Usage

Refer to the original model card for details or see this blog post. Here is how you can use it on CPU:

Install

Requires the current main of transformers (at time of writing):

pip install accelerate git+https://github.com/huggingface/transformers.git -U -q

Use (this is for CPU, check out the original model card/blog for fp16 and int8 usage)

import requests
from PIL import Image
from transformers import BlipProcessor, Blip2ForConditionalGeneration

model_name = "ethzanalytics/blip2-flan-t5-xl-sharded"
processor = BlipProcessor.from_pretrained(model_name)
model = Blip2ForConditionalGeneration.from_pretrained(model_name)

img_url = 'https://storage.googleapis.com/sfr-vision-language-research/BLIP/demo.jpg' 
raw_image = Image.open(requests.get(img_url, stream=True).raw).convert('RGB')

question = "how many dogs are in the picture?"
inputs = processor(raw_image, question, return_tensors="pt")

out = model.generate(**inputs)
print(processor.decode(out[0], skip_special_tokens=True))

Downloads last month: 14

Safetensors

Model size

4B params

Tensor type

F32

Inference Providers NEW

Image-Text-to-Text

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

ethzanalytics
/

blip2-flan-t5-xl-sharded

Sharded BLIP-2 Model Card - flan-t5-xl

Usage

Space using ethzanalytics/blip2-flan-t5-xl-sharded 1