Text Generation
Transformers
Safetensors
PyTorch
phi
gptq
quantized
causal-lm
phi-2
text-generation-inference
8-bit precision
🧠 Phi-2 GPTQ (Quantized)
This repository provides a 4-bit GPTQ quantized version of the Phi-2 model by Microsoft, optimized for efficient inference using gptqmodel.
📌 Model Details
- Base Model: Microsoft Phi-2
- Quantization: GPTQ (4-bit)
- Quantizer:
GPTQModel - Framework: PyTorch + HuggingFace Transformers
- Device Support: CUDA (GPU)
- License: Apache 2.0
🚀 Features
- ✅ Lightweight: 4-bit quantization significantly reduces memory usage
- ✅ Fast Inference: Ideal for deployment on consumer GPUs
- ✅ Compatible: Works with
transformers,optimum, andgptqmodel - ✅ CUDA-accelerated: Automatically uses GPU for speed
📚 Usage
This model is ready-to-use with the Hugging Face transformers library.
🧪 Intended Use
- Research and development
- Prototyping generative applications
- Fast inference environments with limited GPU memory
📖 References
- Microsoft Phi-2: https://huggingface.co/microsoft/phi-2
- GPTQModel: https://github.com/ModelCoud/GPTQModel
- Transformers: https://github.com/huggingface/transformers
⚖️ License
This model is distributed under the Apache License 2.0.
- Downloads last month
- 6
Model tree for STiFLeR7/Phi2-GPTQ
Base model
microsoft/phi-2
Install from pip and serve model
# Install vLLM from pip: pip install vllm# Start the vLLM server: vllm serve "STiFLeR7/Phi2-GPTQ"# Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "STiFLeR7/Phi2-GPTQ", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'