---
license: apache-2.0
base_model: BayesRL/Olmo3-IVON-SFT-7B
datasets:
  - BytedTsinghua-SIA/DAPO-Math-17k
language:
  - en
pipeline_tag: text-generation
library_name: transformers
tags:
  - rlvr
  - grpo
  - ivon
  - variational-learning
  - 3po
  - m3po
  - math
---

# Olmo3-M3PO-7B

📄 **Paper:** [Parameter Exploration for RLVR via Variational Learning](https://huggingface.co/papers/2608.09805) · [arXiv:2608.09805](https://arxiv.org/abs/2608.09805)

📦 **Code:** [insait-institute/c3po](https://github.com/insait-institute/c3po)

Olmo-3 7B fine-tuned with **M3PO**, from the paper **"Parameter Exploration for RLVR via Variational
Learning"**.

3PO is a family of *parameter-space* exploration strategies for Reinforcement Learning with Verifiable
Rewards (RLVR). Instead of relying only on action-space heuristics (temperature, clipping, entropy
bonuses), 3PO samples model weights from an approximate posterior learned with the variational optimizer
IVON; the amount of weight noise becomes an extra control lever for
exploration.

**M3PO** draws `M` Monte-Carlo weight perturbations from the IVON posterior per gradient step; rollouts
and advantages are computed per sample and the gradients are averaged. To keep compute roughly matched to
GRPO, the group size is reduced (`GROUP_SIZE = G/M`).

## Training

| | |
|---|---|
| Base / warm-start | [`BayesRL/Olmo3-IVON-SFT-7B`](https://huggingface.co/BayesRL/Olmo3-IVON-SFT-7B) |
| Foundation model | `allenai/Olmo-3-1025-7B` |
| Algorithm | M3PO (GRPO + IVON, `M` MC perturbations per step, equal-compute) |
| RL data | [DAPO-Math-17k](https://huggingface.co/datasets/BytedTsinghua-SIA/DAPO-Math-17k) |
| Optimizer | IVON, lr `1.0`, ESS (λ) `1e9` |
| Hardware | 8× NVIDIA H200 (144 GB) |

## Evaluation

Evaluated on AIME 2024–2026, MATH-500, AMC 2023, and Minerva. See the paper for full results.

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("BayesRL/Olmo3-M3PO-7B")
tok = AutoTokenizer.from_pretrained("BayesRL/Olmo3-M3PO-7B")
```

## Citation

```bibtex
@misc{venkatkrishna2026parameterexploration,
      title={Parameter Exploration for RLVR via Variational Learning}, 
      author={Vatsal Venkatkrishna and Nico Daheim and Iryna Gurevych},
      year={2026},
      eprint={2608.09805},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2608.09805}, 
}
```