aniris/Qwen2.5-0.5B-Instruct-Q4_K_M-GGUF

This model was converted to GGUF format from Qwen/Qwen2.5-0.5B-Instruct using llama.cpp via the ggml.ai's GGUF-my-repo space. Refer to the original model card for more details on the model.
Qwen2.5 를 Q4_K_M로 양자화한 GGUF 모델입니다.

Model summary

• Base model: Qwen/Qwen2.5-0.5B-Instruct

• Format: GGUF

• Quantization: Q4_K_M (4-bit, grouped, mixed-precision)

• Architecture: Qwen2.5 (Transformer with RoPE, SwiGLU, RMSNorm, GQA) :contentReference[oaicite:0]{index=0}

  한국어 안내 (Korean quick note)

이 리포지토리는 Qwen/Qwen2.5-0.5B-Instruct 원본을 llama.cpp용 GGUF (Q4_K_M)로 양자화한 체크포인트입니다.

• 가벼운 로컬 챗봇, 코드 보조, 간단한 한/영 다국어 실험에 적합합니다.
• 정확한 사실성, 강한 추론, 안전성이 매우 중요한 용도에는 더 큰 모델이나 고정밀 버전을 권장합니다.

Acknowledgements

This repository only provides a quantized GGUF checkpoint. All training and evaluation of the base model were done by the Qwen team.

If you use this model, please cite:

@misc{qwen2.5,
  title  = {Qwen2.5: A Party of Foundation Models},
  url    = {https://qwenlm.github.io/blog/qwen2.5/},
  author = {Qwen Team},
  month  = {September},
  year   = {2024}
}

Q4_K_M는 “품질 vs 메모리” 밸런스가 좋은 4비트 가중치 양자화라서,

• 저사양 노트북 / 경량 서버 계열에서 실행할 수 있고,
• 간단한 챗봇, 코드 어시스턴트, 로컬 도구 연동 등에 적합합니다.

Use with llama.cpp

Install llama.cpp through brew (works on Mac and Linux)

brew install llama.cpp

Invoke the llama.cpp server or the CLI.

CLI:

llama-cli --hf-repo aniris/Qwen2.5-0.5B-Instruct-Q4_K_M-GGUF --hf-file qwen2.5-0.5b-instruct-q4_k_m.gguf -p "The meaning to life and the universe is"

Server:

llama-server --hf-repo aniris/Qwen2.5-0.5B-Instruct-Q4_K_M-GGUF --hf-file qwen2.5-0.5b-instruct-q4_k_m.gguf -c 2048

Note: You can also use this checkpoint directly through the usage steps listed in the Llama.cpp repo as well.

Step 1: Clone llama.cpp from GitHub.

git clone https://github.com/ggerganov/llama.cpp

Step 2: Move into the llama.cpp folder and build it with LLAMA_CURL=1 flag along with other hardware-specific flags (for ex: LLAMA_CUDA=1 for Nvidia GPUs on Linux).

cd llama.cpp && LLAMA_CURL=1 make

Step 3: Run inference through the main binary.

./llama-cli --hf-repo aniris/Qwen2.5-0.5B-Instruct-Q4_K_M-GGUF --hf-file qwen2.5-0.5b-instruct-q4_k_m.gguf -p "The meaning to life and the universe is"

or

./llama-server --hf-repo aniris/Qwen2.5-0.5B-Instruct-Q4_K_M-GGUF --hf-file qwen2.5-0.5b-instruct-q4_k_m.gguf -c 2048