output = llm(
"Once upon a time,",
max_tokens=512,
echo=True
)
print(output)4bit GGUF quantization of TinyLlama-1.1B-intermediate-step-955k-token-2T
I used this script to generate the file using this command:
python make-ggml.py ~/ooba/models/TinyLlama_TinyLlama-1.1B-intermediate-step-955k-token-2T/ --model_type=llama --quants=Q4_K_M
The original model is so small that there is only one safetensors file named model.safetensors, so I had to change that filename to model-00001-of-00001.safetensors to make the script load the model properly.
- Downloads last month
- 16
Hardware compatibility
Log In to add your hardware
4-bit
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support
# !pip install llama-cpp-python from llama_cpp import Llama llm = Llama.from_pretrained( repo_id="SlimeQ/TinyLlama_TinyLlama-1.1B-intermediate-step-955k-token-2T-Q4_K_M-GGUF", filename="TinyLlama_TinyLlama-1.1B-intermediate-step-955k-token-2T.Q4_K_M.gguf", )