Instructions to use trollek/Tyr-4B-v0.1-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- llama-cpp-python
How to use trollek/Tyr-4B-v0.1-GGUF with llama-cpp-python:
# !pip install llama-cpp-python from llama_cpp import Llama llm = Llama.from_pretrained( repo_id="trollek/Tyr-4B-v0.1-GGUF", filename="Tyr-4B-v0.1-q3_K_L.gguf", )
llm.create_chat_completion( messages = "No input example has been defined for this model task." )
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use trollek/Tyr-4B-v0.1-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf trollek/Tyr-4B-v0.1-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf trollek/Tyr-4B-v0.1-GGUF:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf trollek/Tyr-4B-v0.1-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf trollek/Tyr-4B-v0.1-GGUF:Q4_K_M
Use pre-built binary
# Download pre-built binary from: # https://github.com/ggerganov/llama.cpp/releases # Start a local OpenAI-compatible server with a web UI: ./llama-server -hf trollek/Tyr-4B-v0.1-GGUF:Q4_K_M # Run inference directly in the terminal: ./llama-cli -hf trollek/Tyr-4B-v0.1-GGUF:Q4_K_M
Build from source code
git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-server llama-cli # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama-server -hf trollek/Tyr-4B-v0.1-GGUF:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama-cli -hf trollek/Tyr-4B-v0.1-GGUF:Q4_K_M
Use Docker
docker model run hf.co/trollek/Tyr-4B-v0.1-GGUF:Q4_K_M
- LM Studio
- Jan
- Ollama
How to use trollek/Tyr-4B-v0.1-GGUF with Ollama:
ollama run hf.co/trollek/Tyr-4B-v0.1-GGUF:Q4_K_M
- Unsloth Studio
How to use trollek/Tyr-4B-v0.1-GGUF with Unsloth Studio:
Install Unsloth Studio (macOS, Linux, WSL)
curl -fsSL https://unsloth.ai/install.sh | sh # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for trollek/Tyr-4B-v0.1-GGUF to start chatting
Install Unsloth Studio (Windows)
irm https://unsloth.ai/install.ps1 | iex # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for trollek/Tyr-4B-v0.1-GGUF to start chatting
Using HuggingFace Spaces for Unsloth
# No setup required # Open https://huggingface.co/spaces/unsloth/studio in your browser # Search for trollek/Tyr-4B-v0.1-GGUF to start chatting
- Atomic Chat new
- Docker Model Runner
How to use trollek/Tyr-4B-v0.1-GGUF with Docker Model Runner:
docker model run hf.co/trollek/Tyr-4B-v0.1-GGUF:Q4_K_M
- Lemonade
How to use trollek/Tyr-4B-v0.1-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull trollek/Tyr-4B-v0.1-GGUF:Q4_K_M
Run and chat with the model
lemonade run user.Tyr-4B-v0.1-GGUF-Q4_K_M
List all available models
lemonade list
Tyr-4B-v0.1
Tyr was the Norse god that accidentally one of his hands. "It tasted like chicken" was the review from the biggest and baddest of good boys in Imaginationland, an avid hand enjoyer. The figurative THE GRIM went on to say that neither he nor his buddy Carl will stop eating hands.
Merging seems like the way to go when it comes to training language models on a budget, and NinjaMouse3 would be ridiculous. GL;HF
Model Description
This model was created by training LoRAs (rsLoRA with NEFTune noise alpha at 5) and DELLA merge them. It saves space and time this way and the result is good. The WhiteRabbitNeo datasets are the focus in this one along with coding.
Incedentally, it seems uncensored. It was trained using the ChatML template and can be used with or without a system prompt.
I am very grateful for all the different components that made this model possible. From H2O and their danube models, through Huggingface and Llama-Factory for making the fine-tuning easy, and to all the great dataset creators. Thank you!
Apache-2.0 + WhiteRabbitNeo Extended Version
Licence: Usage Restrictions
You agree not to use the Model or Derivatives of the Model:
- In any way that violates any applicable national or international law or regulation or infringes upon the lawful rights and interests of any third party;
- For military use in any way;
- For the purpose of exploiting, harming or attempting to exploit or harm minors in any way;
- To generate or disseminate verifiably false information and/or content with the purpose of harming others;
- To generate or disseminate inappropriate content subject to applicable regulatory requirements;
- To generate or disseminate personal identifiable information without due authorization or for unreasonable use;
- To defame, disparage or otherwise harass others;
- For fully automated decision making that adversely impacts an individual’s legal rights or otherwise creates or modifies a binding, enforceable obligation;
- For any use intended to or which has the effect of discriminating against or harming individuals or groups based on online or offline social behavior or known or predicted personal or personality characteristics;
- To exploit any of the vulnerabilities of a specific group of persons based on their age, social, physical or mental characteristics, in order to materially distort the behavior of a person pertaining to that group in a manner that causes or is likely to cause that person or another person physical or psychological harm;
- For any use intended to or which has the effect of discriminating against individuals or groups based on legally protected characteristics or categories.
- Downloads last month
- 19
3-bit
4-bit
5-bit
6-bit
8-bit
Model tree for trollek/Tyr-4B-v0.1-GGUF
Base model
h2oai/h2o-danube3-4b-base