Instructions to use tensorblock/llama3.2-typhoon2-t1-3b-research-preview-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use tensorblock/llama3.2-typhoon2-t1-3b-research-preview-GGUF with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="tensorblock/llama3.2-typhoon2-t1-3b-research-preview-GGUF") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("tensorblock/llama3.2-typhoon2-t1-3b-research-preview-GGUF", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use tensorblock/llama3.2-typhoon2-t1-3b-research-preview-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf tensorblock/llama3.2-typhoon2-t1-3b-research-preview-GGUF:Q2_K # Run inference directly in the terminal: llama cli -hf tensorblock/llama3.2-typhoon2-t1-3b-research-preview-GGUF:Q2_K
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf tensorblock/llama3.2-typhoon2-t1-3b-research-preview-GGUF:Q2_K # Run inference directly in the terminal: llama cli -hf tensorblock/llama3.2-typhoon2-t1-3b-research-preview-GGUF:Q2_K
Use pre-built binary
# Download pre-built binary from: # https://github.com/ggerganov/llama.cpp/releases # Start a local OpenAI-compatible server with a web UI: ./llama-server -hf tensorblock/llama3.2-typhoon2-t1-3b-research-preview-GGUF:Q2_K # Run inference directly in the terminal: ./llama-cli -hf tensorblock/llama3.2-typhoon2-t1-3b-research-preview-GGUF:Q2_K
Build from source code
git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-server llama-cli # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama-server -hf tensorblock/llama3.2-typhoon2-t1-3b-research-preview-GGUF:Q2_K # Run inference directly in the terminal: ./build/bin/llama-cli -hf tensorblock/llama3.2-typhoon2-t1-3b-research-preview-GGUF:Q2_K
Use Docker
docker model run hf.co/tensorblock/llama3.2-typhoon2-t1-3b-research-preview-GGUF:Q2_K
- LM Studio
- Jan
- vLLM
How to use tensorblock/llama3.2-typhoon2-t1-3b-research-preview-GGUF with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "tensorblock/llama3.2-typhoon2-t1-3b-research-preview-GGUF" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "tensorblock/llama3.2-typhoon2-t1-3b-research-preview-GGUF", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/tensorblock/llama3.2-typhoon2-t1-3b-research-preview-GGUF:Q2_K
- SGLang
How to use tensorblock/llama3.2-typhoon2-t1-3b-research-preview-GGUF with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "tensorblock/llama3.2-typhoon2-t1-3b-research-preview-GGUF" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "tensorblock/llama3.2-typhoon2-t1-3b-research-preview-GGUF", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "tensorblock/llama3.2-typhoon2-t1-3b-research-preview-GGUF" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "tensorblock/llama3.2-typhoon2-t1-3b-research-preview-GGUF", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Ollama
How to use tensorblock/llama3.2-typhoon2-t1-3b-research-preview-GGUF with Ollama:
ollama run hf.co/tensorblock/llama3.2-typhoon2-t1-3b-research-preview-GGUF:Q2_K
- Unsloth Studio
How to use tensorblock/llama3.2-typhoon2-t1-3b-research-preview-GGUF with Unsloth Studio:
Install Unsloth Studio (macOS, Linux, WSL)
curl -fsSL https://unsloth.ai/install.sh | sh # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for tensorblock/llama3.2-typhoon2-t1-3b-research-preview-GGUF to start chatting
Install Unsloth Studio (Windows)
irm https://unsloth.ai/install.ps1 | iex # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for tensorblock/llama3.2-typhoon2-t1-3b-research-preview-GGUF to start chatting
Using HuggingFace Spaces for Unsloth
# No setup required # Open https://huggingface.co/spaces/unsloth/studio in your browser # Search for tensorblock/llama3.2-typhoon2-t1-3b-research-preview-GGUF to start chatting
- Pi
How to use tensorblock/llama3.2-typhoon2-t1-3b-research-preview-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf tensorblock/llama3.2-typhoon2-t1-3b-research-preview-GGUF:Q2_K
Configure the model in Pi
# Install Pi: npm install -g @mariozechner/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "tensorblock/llama3.2-typhoon2-t1-3b-research-preview-GGUF:Q2_K" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Hermes Agent new
How to use tensorblock/llama3.2-typhoon2-t1-3b-research-preview-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf tensorblock/llama3.2-typhoon2-t1-3b-research-preview-GGUF:Q2_K
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default tensorblock/llama3.2-typhoon2-t1-3b-research-preview-GGUF:Q2_K
Run Hermes
hermes
- Atomic Chat new
- OpenClaw new
How to use tensorblock/llama3.2-typhoon2-t1-3b-research-preview-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf tensorblock/llama3.2-typhoon2-t1-3b-research-preview-GGUF:Q2_K
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "tensorblock/llama3.2-typhoon2-t1-3b-research-preview-GGUF:Q2_K" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
- Docker Model Runner
How to use tensorblock/llama3.2-typhoon2-t1-3b-research-preview-GGUF with Docker Model Runner:
docker model run hf.co/tensorblock/llama3.2-typhoon2-t1-3b-research-preview-GGUF:Q2_K
- Lemonade
How to use tensorblock/llama3.2-typhoon2-t1-3b-research-preview-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull tensorblock/llama3.2-typhoon2-t1-3b-research-preview-GGUF:Q2_K
Run and chat with the model
lemonade run user.llama3.2-typhoon2-t1-3b-research-preview-GGUF-Q2_K
List all available models
lemonade list
License Conflict: llama3.2 vs CC BY-NC 4.0
Hi, I’d like to report a potential license conflict in tensorblock/llama3.2-typhoon2-t1-3b-research-preview-GGUF. Based on the model card, this model is distributed under the LLaMA 3.2 Community License. However, the training dataset scb10x/typhoon-t1-3b-research-preview-data is published under the CC BY-NC 4.0 license.
This combination raises potential license compatibility concerns, as LLaMA 3.2 and CC BY-NC 4.0 impose different, and potentially conflicting, restrictions on model use, redistribution, and downstream licensing
⚠️ Key incompatibilities:
LLaMA 3.2 License:
• Prohibits relicensing or sublicensing under other licenses (e.g., CC BY-NC)
• Allows limited commercial use (MAU under threshold), subject to Meta’s Acceptable Use Policy
• Requires that derivative models retain the LLaMA name and be distributed under the LLaMA 3.2 License
CC BY-NC 4.0 License:
• Strictly prohibits any commercial use of the dataset and derivative works
• Requires attribution to the dataset authors
• May apply NonCommercial restrictions to downstream outputs (including trained models)
This could lead to uncertainty for downstream users regarding:
• Whether the model can be used for research or commercial applications
• Whether attribution to the dataset is required (currently not mentioned)
• Whether commercial usage restrictions under CC BY-NC are being fully inherited
While both licenses limit commercial use, they do so in different and incompatible ways:
CC BY-NC 4.0 flatly prohibits commercial use;
LLaMA 3.2 allows conditional commercial use under specific thresholds;
LLaMA 3.2 prohibits relicensing, meaning the model cannot legally inherit the CC BY-NC license, even if required.
This makes it legally unclear how the model should be used, and may result in a license violation of either or both upstream sources.
🔹 Suggestion:
To help clarify the licensing situation and ensure alignment with upstream terms, here are a few options to consider:
1. Clearly document in the model card or README that the model was trained on CC BY-NC 4.0–licensed data, and is thus subject to non-commercial use only.
2. Add attribution for the dataset (e.g., a "Data sources" section with license info and dataset link).
3. Clarify that even though the model is released under the LLaMA 3.2 License, users must also comply with the dataset’s non-commercial use requirement.
4. If commercial use is desired, consider retraining the model on datasets with more permissive licenses.
Hope this helps! Let me know if you have any questions or need more info.
Thanks for your attention!
Your reply would be much appreciated!