Model Card: LoRA Adapter for GPT-OSS 20B BF16

Model Summary

This repository contains a PEFT LoRA adapter for the base model togethercomputer/gpt-oss-20b-bf16. The adapter is designed to specialize the base model for a target behavior, style, or domain while keeping the original foundation model weights intact. It is not a standalone model; it must be used together with the base model.

The adapter artifacts include:

A LoRA adapter checkpoint in Safetensors format
Tokenizer assets and a chat template
Training metadata captured during fine-tuning

Model Details

Model Description

Model type: Parameter-efficient fine-tuning adapter for a causal language model
Base model: togethercomputer/gpt-oss-20b-bf16
Architecture: GPT-OSS causal language model (GptOssForCausalLM)
Adapter type: LoRA
Task: Text generation, chat-style interaction, and instruction-following behavior consistent with the base model
Framework: PEFT + Transformers
License: Please follow the license of the base model. The adapter artifacts are derived from the base model and should be used under the same terms unless otherwise stated.

Intended Use

This adapter is intended for:

Research and experimentation with domain or style adaptation
Instruction-following and chat-style generation
Controlled adaptation of a large language model for specific workflows

Out-of-Scope Use

This adapter should not be used for:

High-stakes decision-making without human oversight
Medical, legal, financial, or safety-critical advice without verification
Automated content moderation or decision-making without domain review
Any use that violates the license or terms of the base model

Base Model Information

Base model name: togethercomputer/gpt-oss-20b-bf16
Base model family: GPT-OSS
Base model size: 20B-class foundation model
Base model capabilities: General-purpose text generation and reasoning inherited from the base model

Adapter Configuration

The adapter was trained as a LoRA adapter with the following configuration:

PEFT type: LoRA
Rank: 64
LoRA alpha: 256
LoRA dropout: 0.05
Bias: none
Target modules: q_proj, k_proj, o_proj, v_proj
Task type: CAUSAL_LM
Initialization: LoRA initialization enabled

Training Details

Training Procedure

The provided training metadata indicates that the adapter was fine-tuned for 5 epochs and 165 global steps.

Observed training behavior from the available logs:

Initial training loss: approximately 2.19
Final logged training loss: approximately 0.80
Evaluation loss at the recorded checkpoint: approximately 0.97

Training Hyperparameters

The following details are available from the supplied artifacts:

Epochs: 5
Global steps: 165
Evaluation interval: every 33 steps
Framework: PEFT/Transformers

Additional optimizer and batch-size settings were not included in the provided artifacts and should be documented if the training run is reproduced.

Evaluation and Limitations

Known Limitations

This adapter inherits the strengths and weaknesses of the base model. It may:

Generate plausible but incorrect information
Reflect social or cultural biases present in the base model data
Struggle with factual precision in specialized domains
Exhibit inconsistent behavior for edge cases or ambiguous prompts

Recommended Use

Users should:

Evaluate the adapter on their intended task before deployment
Test for factuality, safety, and stylistic consistency
Monitor outputs for hallucinations, harmful content, or policy violations
Keep human review in the loop for high-impact applications

Usage

A minimal example for loading the base model and adapter is shown below.

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

base_model_name = "togethercomputer/gpt-oss-20b-bf16"
adapter_path = "./"

tokenizer = AutoTokenizer.from_pretrained(base_model_name)
model = AutoModelForCausalLM.from_pretrained(
    base_model_name,
    torch_dtype="auto",
    device_map="auto",
)
model = PeftModel.from_pretrained(model, adapter_path)

prompt = "Write a concise summary of the benefits of parameter-efficient fine-tuning."
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=200)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Data and Safety Notes

The adapter should be treated as a behavior-modifying layer on top of the base model rather than as an independent model. Its quality depends heavily on the underlying base model, the adaptation data, and the evaluation process used by the adopter.

Authors and Contact

Author: Not specified in the provided artifacts
Contact: Add your preferred contact or organization details here

Citation

If you use this adapter in a publication or project, please cite the base model and the PEFT/Transformers libraries.

Model Card Version

Version: 1.0
Last updated: 2026-07-06