Bubba is a fine-tuned LLM based on OpenAI’s Chat GPT-5. This release packages the fine-tuned weights (or adapters) for practical, low-latency instruction following, summarization, reasoning, and light code generation. It is intended for local or self-hosted environments and RAG (Retrieval-Augmented Generation) stacks that require predictable, fast outputs.

Quantized, and fine-tuned GGUF based on OpenAI’s gpt-oss-20b
Format: GGUF (for llama.cpp and compatible runtimes) • Quantization: Q4_K_XL (4-bit, K-grouped, extra-low loss)
File: bubba-20b-Q4_K_XL.gguf

⚙️ Streamlit UI

🐍 Code Repository

🧠 Overview

• This repo provides a 4-bit K-quantized .gguf for fast local inference of a 20B-parameter model derived from OpenAI’s gpt-oss-20b (as reported by the uploader).
• Use cases: general chat/instruction following, coding help, knowledge Q&A (see Intended Use & Limitations).
• Works with: llama.cpp, llama-cpp-python, KoboldCPP, Text Generation WebUI, LM Studio, and other GGUF-compatible backends.
• Hardware guidance (rule of thumb): ~12–16 GB VRAM/RAM for comfortable batch-1 inference with Q4_K_XL; CPU-only works too (expect lower tokens/s).

---

Key Features

• Instruction-tuned derivative of gpt-oss-20b for concise, helpful responses.
• Optimized defaults for short to medium prompts; strong compatibility with RAG pipelines.
• Flexible distribution: full finetuned weights or lightweight LoRA/QLoRA adapters.
• Compatible with popular runtimes and libraries (Transformers, PEFT, vLLM, Text Generation Inference).

⚠️ Provenance & license: This quant is produced from a base model claimed to be OpenAI’s gpt-oss-20b. Please review and comply with the original model’s license/terms. The GGUF quantization inherits those terms. See the License section.

⚙️ Fine-Tuning Datasets

• The Stafford Act of 2019

• Incident Management Handbook

• OMB Circular A-123 Management's Responsibility For Risk Management & Internal Controls

• Federal Government Standards For Internal Controls

• CFR Title-2 Uniform Administrative Requirements Cost Principles & Audit

• The GPRA Modernization Act Of 2010

• Government Performance & Results Act

• Statements Of Federal Financial Accounting Concepts And Standards

• The Congressional Budget & Impoundment Control Act Of 1974

• The Anti-Deficiency Reform & Enforcement Act Of 2018

• The-Anti-Deficiency-Act

• Federal Acquisition Regulation

• FASAB Handbook of Federal Accounting Standards

• Statements Of Federal Financial Accounting Concepts & Standards

• US Standard General Ledger Accounts & Definitions

• Principles Of Federal Appropriations Law

• Government Auditing Standards

• DOD Regulation 7000-14 Financial Management

• OMB Circular A-11 Preparation Submission & Execution Of The Budget

• Inspector General Act Of 1978

• The Balanced Budget & Emergency Deficit Control Act Of 1985

• The Chief Financial Officers Act 1990

• The Fiscal Responsibility Act of 2023

• The Statutory Pay-As-You-Go Act of 2010

• CFR Title-31 Money & Finance

• Federal Manager's Financial Integrity Act of 1982

• Digital Accountability And Transparency Act of 2014

• The Budget Control Act 2011

• The Federal Trust Fund Accounting Guide

• OMB Circular A-11 Section-120 The Apportionment Process

• OMB Circular A-11 Section-82 Schedule-X

• OMB Circular A-11 Section-100 Sequestration

• The Economy Act of 1932

• The Chief Financial Officers Act of 1990

• CFR Title-2 Uniform Administrative Requirements For Cost Principles & Audit

• The Supplemental Appropriation Act of 1955

• Independent Offices Appropriation Act of 1953

• Program Management Improvement Accountability Act of 2016

Technical Specifications

Property	Value / Guidance
Base model	gpt-oss-20b (decoder-only Transformer)
Parameters	~20B (as per upstream)
Tokenizer	Use the upstream tokenizer associated with gpt-oss-20b
Context window	Determined by the upstream base; set accordingly in your runtime
Fine-tuning	Supervised Fine-Tuning (SFT); optional preference optimization (DPO/ORPO)
Precision	FP16/BF16 recommended; 4-bit (bnb) for single-GPU experimentation
Intended runtimes	Hugging Face Transformers, PEFT, vLLM, TGI (Text Generation Inference)

Note: Please adjust any specifics (context length, tokenizer name) to match the exact upstream build you use for gpt-oss-20b.

---

Files

File / Folder	Description
README.md	This model card
config.json / tokenizer files	Configuration and tokenizer artifacts (from upstream)
pytorch_model.safetensors	Full fine-tuned weights (if released as full model)
adapter_model.safetensors	LoRA/QLoRA adapters only (if released as adapters)
training_args.json (optional)	Minimal training configuration for reproducibility

Only one of “full weights” or “adapters” may be included depending on how you distribute Bubba.

---

📝 Intended Use & Limitations

Intended Use

• Instruction following, general dialogue
• Code assistance (reasoning, boilerplate, refactoring)
• Knowledge/Q&A within the model’s training cutoff

Out-of-Scope / Known Limitations

• Factuality: may produce inaccurate or outdated info
• Safety: can emit biased or unsafe text; apply your own filters/guardrails
• High-stakes decisions: not for medical, legal, financial, or safety-critical use

🎯 Quick Start

Examples: Using the Bubba LLM (Fine-tuned from gpt-oss-20b)

This guide shows several ways to run Bubba locally or on a server. Examples cover full weights, LoRA/QLoRA adapters, vLLM, and Text Generation Inference (TGI), plus prompt patterns and RAG.

---

🐍 Python (Transformers) — Full Weights

Install

pip install "transformers>=4.44.0" accelerate torch --upgrade

Load and generate

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_name = "your-namespace/Bubba-gpt-oss-20b-finetuned"
tok = AutoTokenizer.from_pretrained(model_name, use_fast=True)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

prompt = "In 5 bullet points, explain retrieval-augmented generation and when to use it."
inputs = tok(prompt, return_tensors="pt").to(model.device)

out = model.generate(
    **inputs,
    max_new_tokens=256,
    temperature=0.7,
    top_p=0.9
)
print(tok.decode(out[0], skip_special_tokens=True))

Notes

• device_map="auto" will place weights across available GPUs/CPU.
• Prefer BF16 if supported; otherwise FP16. For VRAM-constrained experiments, see 4-bit below.

---

🧩 Python (PEFT) — Adapters on Top of the Base

Install

pip install "transformers>=4.44.0" peft accelerate torch --upgrade

Load base + LoRA/QLoRA adapters

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base_name = "openai/gpt-oss-20b"   # replace with the exact upstream base you use
lora_name = "your-namespace/Bubba-gpt-oss-20b-finetuned"

tok = AutoTokenizer.from_pretrained(base_name, use_fast=True)
base = AutoModelForCausalLM.from_pretrained(
    base_name,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)
model = PeftModel.from_pretrained(base, lora_name)

prompt = "Draft a JSON spec with keys: goal, steps[], risks[], success_metric."
inputs = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=200, temperature=0.6, top_p=0.9)
print(tok.decode(out[0], skip_special_tokens=True))

---

💾 4-bit (bitsandbytes) — Memory-Efficient Loading

Install

pip install "transformers>=4.44.0" accelerate bitsandbytes --upgrade

Load with 4-bit quantization

    from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
    import torch

    bnb = BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_use_double_quant=True,
        bnb_4bit_quant_type="nf4",
        bnb_4bit_compute_dtype=torch.bfloat16
    )

    model_name = "your-namespace/Bubba-gpt-oss-20b-finetuned"
    tok = AutoTokenizer.from_pretrained(model_name, use_fast=True)
    model = AutoModelForCausalLM.from_pretrained(
        model_name,
        quantization_config=bnb,
        device_map="auto"
    )

    prompt = "Explain beam search vs. nucleus sampling in three short bullets."
    inputs = tok(prompt, return_tensors="pt").to("cuda" if torch.cuda.is_available() else "cpu")
    out = model.generate(**inputs, max_new_tokens=160, temperature=0.7, top_p=0.9)
    print(tok.decode(out[0], skip_special_tokens=True))

---

🚀 Serve with vLLM (OpenAI-compatible API)

Install and launch (example)

    pip install vllm
    python -m vllm.entrypoints.openai.api_server \
      --model your-namespace/Bubba-gpt-oss-20b-finetuned \
      --dtype bfloat16 --max-model-len 8192 \
      --port 8000

Call the endpoint (Python)

    import requests, json
    url = "http://localhost:8000/v1/chat/completions"
    headers = {"Content-Type": "application/json"}

    data = {
      "model": "your-namespace/Bubba-gpt-oss-20b-finetuned",
      "messages": [
        {"role": "system", "content": "You are concise and factual."},
        {"role": "user", "content": "Give a 4-step checklist for evaluating a RAG pipeline."}
      ],
      "temperature": 0.7,
      "max_tokens": 256,
      "stream": True
    }

    with requests.post(url, headers=headers, data=json.dumps(data), stream=True) as r:
        for line in r.iter_lines():
            if line and line.startswith(b"data: "):
                chunk = line[len(b"data: "):].decode("utf-8")
                if chunk == "[DONE]":
                    break
                print(chunk, flush=True)

---

📦 Serve with Text Generation Inference (TGI)

Run the server (Docker)

docker run --gpus all --shm-size 1g -p 8080:80 \
  -e MODEL_ID=your-namespace/Bubba-gpt-oss-20b-finetuned \
  ghcr.io/huggingface/text-generation-inference:latest

Call the server (HTTP)

curl http://localhost:8080/generate \
  -X POST -d '{
    "inputs": "Summarize pros/cons of hybrid search (BM25 + embeddings).",
    "parameters": {"max_new_tokens": 200, "temperature": 0.7, "top_p": 0.9}
  }' \
  -H "Content-Type: application/json"

---

🧠 Prompt Patterns

Direct instruction (concise)

You are a precise assistant. In 6 bullets, explain evaluation metrics for retrieval (Recall@k,
MRR, nDCG). Keep each bullet under 20 words.

Constrained JSON output

System: Output only valid JSON. No prose.
User: Produce {"goal":"", "steps":[""], "risks":[""], "metrics":[""]} for testing a QA bot.

Guarded answer

If the answer isn’t derivable from the context, say “I don’t know” and ask for the missing info.

Few-shot structure

Example:
Q: Map 3 tasks to suitable embedding dimensions.
A: 256: short titles; 768: support FAQs; 1024: multi-paragraph knowledge base.

---

📚 Basic RAG

# 1) Retrieve
chunks = retriever.search("compare vector DBs for legal discovery", k=5)

# 2) Build prompt
context = "\n".join([f"• {c.text} [{c.source}]" for c in chunks])
prompt = f"""
You are a helpful assistant. Use only the context to answer.
Context:
{context}

Question:
What selection criteria should teams use when picking a vector DB for scale and cost?
"""

# 3) Generate (Transformers / vLLM / TGI)
inputs = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=256, temperature=0.6, top_p=0.9)
print(tok.decode(out[0], skip_special_tokens=True))

📁 1. Document Ingestion

from langchain.document_loaders import TextLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter

loader = TextLoader("docs/corpus.txt")
documents = loader.load()

splitter = RecursiveCharacterTextSplitter(chunk_size=700, chunk_overlap=150)
docs = splitter.split_documents(documents)

---

🔍 2. Embedding & Vector Indexing

from langchain.embeddings import HuggingFaceEmbeddings
from langchain.vectorstores import FAISS

embedding = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")
vectorstore = FAISS.from_documents(docs, embedding)

---

🔄 3. Retrieval + Prompt Formatting

retriever = vectorstore.as_retriever(search_kwargs={"k": 5})
retrieved_docs = retriever.get_relevant_documents("What role does Bubba play in improving document QA?")

context = "\n\n".join([doc.page_content for doc in retrieved_docs])

prompt = f"""
You are Bubba, a reasoning-heavy assistant. Use only the context below to answer:

<context>
{context}
</context>

<question>
What role does Bubba play in improving document QA?
</question>
"""

---

🧠 4. LLM Inference with Bubba

./main -m Bubba.Q4_K_M.gguf -p "$prompt" -n 768 -t 16 -c 4096 --color

Bubba’s output will include a context-aware, citation-grounded response backed by the retrieved input.

---

📝 Notes

• Bubba (20B parameter model) may require more memory than smaller models like Bro or Leeroy.
• Use a higher -c value (context size) to accommodate longer prompts with more chunks.
• GPU acceleration is recommended for smooth generation if your hardware supports it.

---

⚙️ Parameter Tips

• Temperature: 0.6–0.9 (lower = more deterministic)
• Top-p: 0.8–0.95 (tune one knob at a time)
• Max new tokens: 128–384 for chat; longer for drafting
• Repetition penalty: 1.05–1.2 if loops appear
• Batch size: use padding_side="left" and dynamic padding for throughput
• Context length: set to your runtime’s max; compress context via selective retrieval

---

🛟 Troubleshooting

• CUDA OOM:
  Lower max_new_tokens; enable 4-bit; shard across GPUs; reduce context length.
• Slow throughput:
  Use vLLM/TGI with tensor/PP sharding; enable paged attention; pin to BF16.
• Messy JSON:
  Use a JSON-only system prompt; set temperature ≤0.6; add a JSON schema in the prompt.
• Domain shift:
  Consider small adapter tuning on your domain data; add retrieval grounding.

---

🔍 Minimal Batch Inference Example

prompts = [
  "List 5 key features of FAISS.",
  "Why would I choose pgvector over Milvus?"
]
inputs = tok(prompts, return_tensors="pt", padding=True).to(model.device)
out = model.generate(**inputs, max_new_tokens=160, temperature=0.7, top_p=0.9)
for i, seq in enumerate(out):
    print(f"--- Prompt {i+1} ---")
    print(tok.decode(seq, skip_special_tokens=True))

---

Inference Tips

• Prefer BF16 if available; otherwise FP16. For limited VRAM, try 4-bit (bitsandbytes) to explore.
• Start with max_new_tokens between 128–384 and temperature 0.6–0.9; tune top_p for stability.
• For RAG, constrain prompt length and adopt strict chunking/citation formatting for better grounding.

---

📘 WebUI

• Place the GGUF in text-generation-webui/models/bubba-20b-Q4_K_XL/
• Launch with the llama.cpp loader (or llama-cpp-python backend)
• Select the model in the UI, adjust context length, GPU layers, and sampling

🧩 KoboldCPP

./koboldcpp \
  -m bubba-20b-Q4_K_XL.gguf \
  --contextsize 4096 \
  --gpulayers 35 \
  --usecublas

⚡ LM Studio

1. Open LM Studio → Models → Local models → Add local model and select the .gguf.
2. In Chat, pick the model, set Context length (≤ base model max), and adjust GPU Layers.
3. For API use, enable Local Server and target the exposed endpoint with OpenAI-compatible clients.

❓ Prompting

This build is instruction-tuned (downstream behavior depends on your base). Common prompt patterns work:

Simple instruction

Write a concise summary of the benefits of grouped 4-bit quantization.

ChatML-like

<|system|>
You are a helpful, concise assistant.
<|user|>
Compare Q4_K_XL vs Q5_K_M in terms of quality and RAM.
<|assistant|>

Code task

Task: Write a Python function that computes perplexity given log-likelihoods.
Constraints: Include docstrings and type hints.

Tip: Keep prompts explicit and structured (roles, constraints, examples).
Suggested starting points: temperature 0.2–0.8, top_p 0.8–0.95, repeat_penalty 1.05–1.15.

• No special chat template is strictly required. Use clear instructions and keep prompts concise. For multi-turn workflows, persist conversation state externally or via your app’s memory/RAG layer.

Example system style

You are a concise, accurate assistant. Prefer step-by-step reasoning only when needed.
Cite assumptions and ask for missing constraints.

• Guro is a prompt library designed to supercharge AI agents and assistants with task-specific personas -ie, total randos.
• From academic writing to financial analysis, technical support, SEO, and beyond
• Guro provides precision-crafted prompt templates ready to drop into your LLM workflows.

⚙️ Performance & Memory Guidance (Rules of Thumb)

• RAM/VRAM for Q4_K_XL (20B): ~12–16 GB for batch-1 inference (varies by backend and offloading).
• Throughput: Highly dependent on CPU/GPU, backend, context length, and GPU offload.
  Start with -ngl as high as your VRAM allows, then tune threads/batch sizes.
• Context window: Do not exceed the base model’s maximum (quantization does not increase it).

💻 Files

• bubba-20b-Q4_K_XL.gguf — 4-bit K-quantized weights (XL variant)
• tokenizer.* — packed inside GGUF (no separate files needed)

Integrity: Verify your download (e.g., SHA256) if provided by the host/mirror.

⚙️ GGUF Format

1. Start from the base gpt-oss-20b weights (FP16/BF16).
2. Convert to GGUF with llama.cpp’s convert tooling (or equivalent for the base arch).
3. Quantize with llama.cpp quantize to Q4_K_XL.
4. Sanity-check perplexity/behavior, package with metadata.

Exact scripts/commits may vary by environment; please share your pipeline for full reproducibility if you fork this card.

🏁 Safety, Bias & Responsible Use

Large language models can generate plausible but incorrect or harmful content and may reflect societal biases. If you deploy this model:

• Add moderation/guardrails and domain-specific filters.
• Provide user disclaimers and feedback channels.
• Keep human-in-the-loop for consequential outputs.

🕒 License and Usage

This model package derives from Chat GPT-5 so you're responsible for ensuring your use complies with the upstream model license and any dataset terms. For commercial deployment, review OpenAI’s license and your organization’s compliance requirements.

• Bubba is published under the MIT General Public License v3

🧩 Attribution

If this quant helped you, consider citing like:

bubba-20b–Q4_K_XL.gguf (2025).
Quantized GGUF build derived from OpenAI’s gpt-oss-20b.
Retrieved from the Hugging Face Hub.

❓ FAQ

Does quantization change the context window or tokenizer?
No. Those are inherited from the base model; quantization only changes weight representation.

Why am I hitting out-of-memory?
Lower -ngl (fewer GPU layers), reduce context (-c), or switch to a smaller quant (e.g., Q3_K). Ensure no other large models occupy VRAM.

Best sampler settings?
Start with temp 0.7, top_p 0.9, repeat_penalty 1.1.
Lower temperature for coding/planning; raise for creative writing.

📝 Changelog

• v1.0 — Initial release of bubba-20b-Q4_K_XL.gguf.