# SAE + Probes: pythia-7b on commonsense

## Model
- **Model**: EleutherAI/pythia-6.9b
- **Pooling**: max

## Dataset
- **Dataset**: hendrycks/ethics
- **Subset**: commonsense

## SAE Configuration
- **Expansion Factor**: 4
- **Sparsity Coefficient**: 0.0001

## Probe Configuration
- **Weight Decay (L2)**: 0
- **L1 Penalty**: 1e-05

## Results
- **Best Layer**: 28
- **Best Val Accuracy**: 0.6804
- **Best Test Accuracy**: 0.5354

## Layer Results

| Layer | Val Accuracy | Train Accuracy | SAE Sparsity |
|-------|--------------|----------------|---------------|
| 3 | 0.5155 | 0.7353 | 23.86% |
| 9 | 0.5619 | 0.7755 | 22.42% |
| 16 | 0.6701 | 0.8317 | 25.61% |
| 22 | 0.6443 | 0.8245 | 25.34% |
| 28 🏆 | 0.6804 | 0.8187 | 21.95% |