Hugging Face's logo

Aryanne
/
Sheared-LLaMA-2.7B-gguf

GGUF
Model card Files
xet
 Community
How to use from
llama.cpp
Install from brew 
brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf Aryanne/Sheared-LLaMA-2.7B-gguf:
# Run inference directly in the terminal:
llama-cli -hf Aryanne/Sheared-LLaMA-2.7B-gguf:
Install from WinGet (Windows) 
winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf Aryanne/Sheared-LLaMA-2.7B-gguf:
# Run inference directly in the terminal:
llama-cli -hf Aryanne/Sheared-LLaMA-2.7B-gguf:
Use pre-built binary 
# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Aryanne/Sheared-LLaMA-2.7B-gguf:
# Run inference directly in the terminal:
./llama-cli -hf Aryanne/Sheared-LLaMA-2.7B-gguf:
Build from source code 
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Aryanne/Sheared-LLaMA-2.7B-gguf:
# Run inference directly in the terminal:
./build/bin/llama-cli -hf Aryanne/Sheared-LLaMA-2.7B-gguf:
Use Docker 
docker model run hf.co/Aryanne/Sheared-LLaMA-2.7B-gguf:
Quick Links

Some GGUF v2 quantizations of the model princeton-nlp/Sheared-LLaMA-2.7B

Paper: https://arxiv.org/pdf/2310.06694.pdf
Code: https://github.com/princeton-nlp/LLM-Shearing
Models: Sheared-LLaMA-1.3B, Sheared-LLaMA-2.7B

Sheared-LLaMA-2.7B is a model pruned and further pre-trained from meta-llama/Llama-2-7b-hf. We dynamically load data from different domains in the RedPajama dataset. We use 0.4B tokens for pruning and 50B tokens for continued pre-training the pruned model.

  • Smaller-scale
  • Same vocabulary as LLaMA1 and LLaMA2
  • Derived with a budget of 50B tokens by utilizing existing strong LLMs

Downstream Tasks

We evaluate on an extensive set of downstream tasks including reasoning, reading comprehension, language modeling and knowledge intensive tasks. Our Sheared-LLaMA models outperform existing large language models.

Model # Pre-training Tokens Average Performance
LLaMA2-7B 2T 64.6

1.3B

Model # Pre-training Tokens Average Performance
OPT-1.3B 300B 48.2
Pythia-1.4B 300B 48.9
Sheared-LLaMA-1.3B 50B 51.0

3B

Model # Pre-training Tokens Average Performance
OPT-2.7B 300B 51.4
Pythia-2.8B 300B 52.5
INCITE-Base-3B 800B 54.7
Open-LLaMA-3B-v1 1T 55.1
Open-LLaMA-3B-v2 1T 55.7
Sheared-LLaMA-2.7B 50B 56.7
Downloads last month
29
GGUF
Model size
3B params
Architecture
llama
Hardware compatibility
Log In to add your hardware

2-bit

4-bit

5-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Paper for Aryanne/Sheared-LLaMA-2.7B-gguf