output = llm(
"Once upon a time,",
max_tokens=512,
echo=True
)
print(output)InsecureErasure/flan-t5-xxl-encoder-only-GGUF
Quantized GGUF version of Google FLAN T5-XXL (encoder only)
This repo contains additional GGUF quantizations of Google's FLAN-T5 XXL (encoder only) not available in silveroxides/flan-t5-xxl-encoder-only-GGUF, produced using llama.cpp.
These are aimed at users with limited hardware — requiring less memory and storage than higher quantization formats. Other repositories on HF include the decoder layers, resulting in a much bigger file. If you're using this for generating embeddings for Chroma or FLUX models (as I am), you only need the encoder layers.
Big kudos to silveroxides.
| Chroma Flash Q5_K_S |  |
 |
| Chroma Flash Q4_K_S |  |
 |
| flan-t5-xxl-TE-only Q4_K_M | flan-t5-xxl-TE-only Q3_K_M |
- Downloads last month
- 52
Hardware compatibility
Log In to add your hardware
2-bit
3-bit
4-bit
5-bit
6-bit
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support
Model tree for InsecureErasure/flan-t5-xxl-encoder-only-GGUF
Base model
silveroxides/flan-t5-xxl-encoder-only
# !pip install llama-cpp-python from llama_cpp import Llama llm = Llama.from_pretrained( repo_id="InsecureErasure/flan-t5-xxl-encoder-only-GGUF", filename="", )