---
license: mit
base_model: EleutherAI/pythia-70m
library_name: transformers
pipeline_tag: text-generation
tags:
  - bwsk
  - combinator-analysis
  - transformer
  - reversible-backprop
  - convergence-training
datasets:
  - wikitext
metrics:
  - perplexity
model-index:
  - name: bwsk-pythia-70m
    results:
      - task:
          type: text-generation
          name: Fine-tune (Conventional)
        dataset:
          name: wikitext
          type: wikitext
        metrics:
          - name: perplexity
            type: perplexity
            value: 28.7759
            verified: false
      - task:
          type: text-generation
          name: Fine-tune (BWSK Analyzed)
        dataset:
          name: wikitext
          type: wikitext
        metrics:
          - name: perplexity
            type: perplexity
            value: 28.9248
            verified: false
      - task:
          type: text-generation
          name: Fine-tune (BWSK Reversible)
        dataset:
          name: wikitext
          type: wikitext
        metrics:
          - name: perplexity
            type: perplexity
            value: 28.9305
            verified: false
      - task:
          type: text-generation
          name: From Scratch (Conventional)
        dataset:
          name: wikitext
          type: wikitext
        metrics:
          - name: perplexity
            type: perplexity
            value: 201.5793
            verified: false
      - task:
          type: text-generation
          name: From Scratch (BWSK Analyzed)
        dataset:
          name: wikitext
          type: wikitext
        metrics:
          - name: perplexity
            type: perplexity
            value: 215.2512
            verified: false
      - task:
          type: text-generation
          name: From Scratch (BWSK Reversible)
        dataset:
          name: wikitext
          type: wikitext
        metrics:
          - name: perplexity
            type: perplexity
            value: 194.3137
            verified: false
---

# BWSK Pythia-70M

**Pythia-70M** (70M params) trained in **6 variants** (3 BWSK modes x 2 experiments) on WikiText-2 with full convergence training and early stopping.

This repo contains all model weights, configs, and training results in a single consolidated repository.

## What is BWSK?

BWSK is a framework that classifies every neural network operation as **S-type** (information-preserving, reversible, coordination-free) or **K-type** (information-erasing, synchronization point) using combinator logic. This classification enables reversible backpropagation through S-phases to save memory, and CALM-based parallelism analysis.

## Model Overview

| Property | Value |
|----------|-------|
| **Base Model** | [EleutherAI/pythia-70m](https://huggingface.co/EleutherAI/pythia-70m) |
| **Architecture** | Transformer (causal_lm) |
| **Parameters** | 70M |
| **Dataset** | WikiText-2 |
| **Eval Metric** | Perplexity |

## S/K Classification

| Type | Ratio |
|------|-------|
| **S-type** (information-preserving) | 67.8% |
| **K-type** (information-erasing) | 32.2% |

## Fine-tune Results

| Mode | Final Loss | Val Perplexity | Test Perplexity | Peak Memory | Time | Epochs |
|------|------------|----------|----------|----------|----------|----------|
| Conventional | 3.2062 | 29.58 | 28.78 | 4.1 GB | 1.3m | 4 |
| BWSK Analyzed | 3.4344 | 29.68 | 28.92 | 4.1 GB | 1.3m | 4 |
| BWSK Reversible | 3.2864 | 29.63 | 28.93 | 3.4 GB | 1.4m | 4 |

**Memory savings (reversible vs conventional):** 15.9%

## From Scratch Results

| Mode | Final Loss | Val Perplexity | Test Perplexity | Peak Memory | Time | Epochs |
|------|------------|----------|----------|----------|----------|----------|
| Conventional | 3.5755 | 195.11 | 201.58 | 4.1 GB | 3.5m | 6 |
| BWSK Analyzed | 3.4921 | 207.35 | 215.25 | 4.1 GB | 4.1m | 6 |
| BWSK Reversible | 3.2354 | 186.11 | 194.31 | 3.4 GB | 4.5m | 6 |

**Memory savings (reversible vs conventional):** 16.0%

## Repository Structure

```
├── README.md
├── results.json
├── finetune-conventional/
│   ├── model.safetensors
│   ├── config.json
│   └── training_results.json
├── finetune-bwsk-analyzed/
│   ├── model.safetensors
│   ├── config.json
│   └── training_results.json
├── finetune-bwsk-reversible/
│   ├── model.safetensors
│   ├── config.json
│   └── training_results.json
├── scratch-conventional/
│   ├── model.safetensors
│   ├── config.json
│   └── training_results.json
├── scratch-bwsk-analyzed/
│   ├── model.safetensors
│   ├── config.json
│   └── training_results.json
├── scratch-bwsk-reversible/
│   ├── model.safetensors
│   ├── config.json
│   └── training_results.json
```

## Usage

Load a specific variant:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load fine-tuned conventional variant
model = AutoModelForCausalLM.from_pretrained(
    "tzervas/bwsk-pythia-70m", subfolder="finetune-conventional"
)
tokenizer = AutoTokenizer.from_pretrained(
    "tzervas/bwsk-pythia-70m", subfolder="finetune-conventional"
)

# Load from-scratch BWSK reversible variant
model = AutoModelForCausalLM.from_pretrained(
    "tzervas/bwsk-pythia-70m", subfolder="scratch-bwsk-reversible"
)
```

## Training Configuration

| Setting | Value |
|---------|-------|
| **Optimizer** | AdamW |
| **LR (fine-tune)** | 5e-05 |
| **LR (from-scratch)** | 3e-04 |
| **LR Schedule** | Cosine with warmup |
| **Max Grad Norm** | 1.0 |
| **Mixed Precision** | AMP (float16) |
| **Early Stopping** | Patience 3 |
| **Batch Size** | 8 |
| **Sequence Length** | 512 |

## Links

- [GitHub Repository](https://github.com/tzervas/ai-s-combinator)
- [Whitepaper](https://github.com/tzervas/ai-s-combinator/blob/main/docs/WHITEPAPER.md)
- [Full Training Report](https://github.com/tzervas/ai-s-combinator/blob/main/docs/FULL_TRAINING_REPORT.md)

## Citation

```bibtex
@software{zervas2026bwsk,
  author = {Zervas, Tyler},
  title = {BWSK: Combinator-Typed Neural Network Analysis},
  year = {2026},
  url = {https://github.com/tzervas/ai-s-combinator},
}
```

## License

MIT