Instructions to use rubra-ai/Mistral-7B-Instruct-v0.3-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use rubra-ai/Mistral-7B-Instruct-v0.3-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf rubra-ai/Mistral-7B-Instruct-v0.3-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf rubra-ai/Mistral-7B-Instruct-v0.3-GGUF:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf rubra-ai/Mistral-7B-Instruct-v0.3-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf rubra-ai/Mistral-7B-Instruct-v0.3-GGUF:Q4_K_M
Use pre-built binary
# Download pre-built binary from: # https://github.com/ggerganov/llama.cpp/releases # Start a local OpenAI-compatible server with a web UI: ./llama-server -hf rubra-ai/Mistral-7B-Instruct-v0.3-GGUF:Q4_K_M # Run inference directly in the terminal: ./llama-cli -hf rubra-ai/Mistral-7B-Instruct-v0.3-GGUF:Q4_K_M
Build from source code
git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-server llama-cli # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama-server -hf rubra-ai/Mistral-7B-Instruct-v0.3-GGUF:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama-cli -hf rubra-ai/Mistral-7B-Instruct-v0.3-GGUF:Q4_K_M
Use Docker
docker model run hf.co/rubra-ai/Mistral-7B-Instruct-v0.3-GGUF:Q4_K_M
- LM Studio
- Jan
- Ollama
How to use rubra-ai/Mistral-7B-Instruct-v0.3-GGUF with Ollama:
ollama run hf.co/rubra-ai/Mistral-7B-Instruct-v0.3-GGUF:Q4_K_M
- Unsloth Studio
How to use rubra-ai/Mistral-7B-Instruct-v0.3-GGUF with Unsloth Studio:
Install Unsloth Studio (macOS, Linux, WSL)
curl -fsSL https://unsloth.ai/install.sh | sh # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for rubra-ai/Mistral-7B-Instruct-v0.3-GGUF to start chatting
Install Unsloth Studio (Windows)
irm https://unsloth.ai/install.ps1 | iex # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for rubra-ai/Mistral-7B-Instruct-v0.3-GGUF to start chatting
Using HuggingFace Spaces for Unsloth
# No setup required # Open https://huggingface.co/spaces/unsloth/studio in your browser # Search for rubra-ai/Mistral-7B-Instruct-v0.3-GGUF to start chatting
- Docker Model Runner
How to use rubra-ai/Mistral-7B-Instruct-v0.3-GGUF with Docker Model Runner:
docker model run hf.co/rubra-ai/Mistral-7B-Instruct-v0.3-GGUF:Q4_K_M
- Lemonade
How to use rubra-ai/Mistral-7B-Instruct-v0.3-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull rubra-ai/Mistral-7B-Instruct-v0.3-GGUF:Q4_K_M
Run and chat with the model
lemonade run user.Mistral-7B-Instruct-v0.3-GGUF-Q4_K_M
List all available models
lemonade list
- Atomic Chat
Rubra Mistral 7B Instruct v0.3 GGUF
Original model: rubra-ai/Mistral-7B-Instruct-v0.3
Model Description
Mistral-7B-Instruct-v0.3 is the result of further post-training on the base model mistralai/Mistral-7B-Instruct-v0.3. This model is designed for high performance in various instruction-following tasks and complex interactions, including multi-turn function calling and detailed conversations.
Training Data
The model underwent additional training on a proprietary dataset encompassing diverse instruction-following, chat, and function calling data. This post-training process enhances the model's ability to integrate tools and manage complex interaction scenarios effectively.
How to use
Refer to https://docs.rubra.ai/inference/llamacpp for usage. Feel free to ask/open issues up in our Github repo: https://github.com/rubra-ai/rubra
Limitations and Bias
While the model performs well on a wide range of tasks, it may still produce biased or incorrect outputs. Users should exercise caution and critical judgment when using the model in sensitive or high-stakes applications. The model's outputs are influenced by the data it was trained on, which may contain inherent biases.
Ethical Considerations
Users should ensure that the deployment of this model adheres to ethical guidelines and consider the potential societal impact of the generated text. Misuse of the model for generating harmful or misleading content is strongly discouraged.
Acknowledgements
We would like to thank Mistral for the model.
Contact Information
For questions or comments about the model, please reach out to the rubra team.
Citation
If you use this work, please cite it as:
@misc {rubra_ai_2024,
author = { Sanjay Nadhavajhala and Yingbei Tong },
title = { Rubra-Mistral-7B-Instruct-v0.3 },
year = 2024,
url = { https://huggingface.co/rubra-ai/Mistral-7B-Instruct-v0.3 },
doi = { 10.57967/hf/2684 },
publisher = { Hugging Face }
}
- Downloads last month
- 202
Collection including rubra-ai/Mistral-7B-Instruct-v0.3-GGUF
Evaluation results
- 5-shot on MMLUself-reported59.120
- 0-shot on GPQAself-reported29.910
- 8-shot, CoT on GSM-8Kself-reported43.290
- 4-shot, CoT on MATHself-reported11.140
- GPT-4 as Judge on MT-benchself-reported7.690
docker model run hf.co/rubra-ai/Mistral-7B-Instruct-v0.3-GGUF: