Instructions to use remiai3/gpt_oss_20b_GGUF_project_guide with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use remiai3/gpt_oss_20b_GGUF_project_guide with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="remiai3/gpt_oss_20b_GGUF_project_guide")

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("remiai3/gpt_oss_20b_GGUF_project_guide", dtype="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use remiai3/gpt_oss_20b_GGUF_project_guide with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "remiai3/gpt_oss_20b_GGUF_project_guide"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "remiai3/gpt_oss_20b_GGUF_project_guide",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/remiai3/gpt_oss_20b_GGUF_project_guide

SGLang

How to use remiai3/gpt_oss_20b_GGUF_project_guide with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "remiai3/gpt_oss_20b_GGUF_project_guide" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "remiai3/gpt_oss_20b_GGUF_project_guide",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "remiai3/gpt_oss_20b_GGUF_project_guide" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "remiai3/gpt_oss_20b_GGUF_project_guide",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use remiai3/gpt_oss_20b_GGUF_project_guide with Docker Model Runner:
```
docker model run hf.co/remiai3/gpt_oss_20b_GGUF_project_guide
```

gpt_oss_20b_GGUF_project_guide / download_model.py

remiai3

Upload 6 files

46e7744 verified 10 months ago

Raw

History Blame Contribute Delete

1.8 kB

	"""
	download_model.py
	- Paste your Hugging Face token into HUGGINGFACE_TOKEN below.
	- By default it will try to download from REPO_ID and only files matching PATTERN.
	- It prints the path of the downloaded .gguf file on success.
	"""

	import os
	import glob
	from huggingface_hub import login, snapshot_download

	# ---- EDIT: paste your token here (or set HUGGINGFACE_TOKEN env var) ----
	HUGGINGFACE_TOKEN = "hf_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
	# -----------------------------------------------------------------------

	# Replace with the repo that contains your GGUF files (change if needed)
	REPO_ID = "unsloth/gpt-oss-20b-GGUF"
	LOCAL_DIR = "models/oss_20b_gguf"
	# Pattern to fetch only the Q2_K_L weight file:
	PATTERN = "*Q2_K_L.gguf"

	if not HUGGINGFACE_TOKEN or HUGGINGFACE_TOKEN.startswith("PASTE_"):
	raise SystemExit("Please paste your Hugging Face token into HUGGINGFACE_TOKEN variable in this file.")

	print("Logging in to Hugging Face hub...")
	login(token=HUGGINGFACE_TOKEN)

	print(f"Downloading from repo: {REPO_ID} --> local dir: {LOCAL_DIR}")
	path = snapshot_download(
	repo_id=REPO_ID,
	local_dir=LOCAL_DIR,
	token=HUGGINGFACE_TOKEN,
	allow_patterns=[PATTERN],
	resume_download=True,
	)

	# find the downloaded .gguf file
	candidates = glob.glob(os.path.join(LOCAL_DIR, "*", ".gguf"), recursive=True)
	candidates = [c for c in candidates if "Q2_K_L" in os.path.basename(c)]
	if not candidates:
	raise SystemExit("Download finished but no Q2_K_L.gguf found in the target folder. Check REPO_ID or PATTERN.")
	gguf_path = os.path.abspath(candidates[0])

	print("Download complete.")
	print("GGUF model path:", gguf_path)
	print("\nSet MODEL_PATH in app.py to this path (or leave app.py to auto-detect 'models/*/.gguf').")