---
library_name: transformers
pipeline_tag: text-generation
language:
  - en
base_model:
  - google/gemma-4-12B
datasets:
  - atenareply/asterion-training-corpus-gemma4
tags:
  - gemma4
  - continued-pretraining
  - domain-adaptation
license: apache-2.0
---

# Gemma-4-12B Asterion — Domain CPT

google/gemma-4-12B after full-parameter Continued Pre-Training on the Asterion Space Operations corpus (1.887B-token fused corpus; training stopped at the eval-loss plateau after ~229M tokens seen) plus reused Mars Express telemetry.

## Overview

- **Stage**: Continued Pre-Training (full fine-tune, CLM, 8-bit AdamW)
- **Lineage**: gemma-4-12B → **CPT** (this model)
- **Method**: Full-parameter CPT (LoRA demonstrably under-performs on knowledge injection [[loraless]](#citations)) with paged 8-bit AdamW — the optimizer quantization is what fits a 12B full-FT on ONE H200 (fp32 AdamW alone would be ~144 GB). 12% FineWeb-Edu replay [[replay]](#citations); NO fixed end of training: a stop-loss on the held-out eval_domain_loss plateau decides when to stop (1 epoch = data ceiling only [[datacon]](#citations)); bf16 mandatory (Gemma-family fp16 overflow).
- **Domain**: fictional — Asterion Space Operations (24-satellite constellation: alert triage, anomaly investigation, subsystem ops) + reused Mars Express telemetry.

## Training

| | |
|---|---|
| Corpus | asterion-training-corpus-gemma4 — 1.887B Gemma tokens post-dedup (85% Asterion / 3% telemetry / 12% replay), source pinned 7f0c3236 |
| Objective | CLM (next-token), full fine-tune, bf16, paged_adamw_8bit |
| LR / schedule | 1.5e-5 cosine re-warm, warmup 0.03 |
| End of training | stop-loss (eval plateau): closed at step 1,750/14,400 (12% of one epoch, ~229M tokens seen) — the last 3 evals improved ≤0.012 each |
| Seq / batch / HW | seq 4096, eff_batch 32, per_device=4, 1×H200 (~2,600 tok/s; the 262K-vocab fp32 logits tensor is the binding memory constraint) |

## Evaluation

| Metric | Value | Note |
|---|---|---|
| PPL Asterion held-out | **1.83** | base gemma-4-12B: 4.80 (-62%) |
| PPL Mars telemetry | **1.25** | base: 3.67 |
| PPL general (FineWeb-Edu) | **8.34** | base: 8.55 — replay works, no forgetting |
| eval_domain_loss | **0.614** | 1.176 at step 0 → 0.614 at step 1,750 |

_Eval is a held-out, deterministic verifiable harness (synthetic tasks); baseline = the pre-SFT ParamΔ model. See `noval-corp/scripts/eval_agentic.py`._

## Intended use & limitations

Domain-knowledge backbone for the Asterion round. Not instruction-tuned (see the `-instruct-paramdelta` / `-agentic` siblings).

**Limitations:**
- Fictional domain — the model 'knows' an invented satellite operator.
- No chat/tool behaviour on its own (base-style CPT checkpoint).
- Saw ~12% of the corpus (stop-loss end): unique per-document facts in the unseen 88% are not in the weights — by design, downstream tasks provide specifics via tool results in context. Held-out PPL (1.83) shows the domain itself generalized.

## Innovations tested

- **12B full-FT on one GPU** — paged 8-bit AdamW + gradient checkpointing + a logits-aware payload model (the 262,144-token vocab makes the fp32 logits tensor, 4.3 GB/seq @4096, the binding memory term — not activations).

## Citations

- CPT replay (anti-forgetting) — Ibrahim et al., 2024. arXiv:2403.08763
- Data-constrained scaling (≤4 epochs) — Muennighoff et al., 2023. arXiv:2305.16264
- LoRA learns less on CPT — Biderman et al., 2024. arXiv:2405.09673

---
_Card generated by `noval-corp/scripts/gen_model_cards.py` (standardized across the noval-corp model family)._