atenareply/gemma-4-12b-asterion

Gemma-4-12B Asterion — Domain CPT

google/gemma-4-12B after full-parameter Continued Pre-Training on the Asterion Space Operations corpus (1.887B-token fused corpus; training stopped at the eval-loss plateau after ~229M tokens seen) plus reused Mars Express telemetry.

Overview

• Stage: Continued Pre-Training (full fine-tune, CLM, 8-bit AdamW)
• Lineage: gemma-4-12B → CPT (this model)
• Method: Full-parameter CPT (LoRA demonstrably under-performs on knowledge injection [loraless]) with paged 8-bit AdamW — the optimizer quantization is what fits a 12B full-FT on ONE H200 (fp32 AdamW alone would be ~144 GB). 12% FineWeb-Edu replay [replay]; NO fixed end of training: a stop-loss on the held-out eval_domain_loss plateau decides when to stop (1 epoch = data ceiling only [datacon]); bf16 mandatory (Gemma-family fp16 overflow).
• Domain: fictional — Asterion Space Operations (24-satellite constellation: alert triage, anomaly investigation, subsystem ops) + reused Mars Express telemetry.

Training

Corpus	asterion-training-corpus-gemma4 — 1.887B Gemma tokens post-dedup (85% Asterion / 3% telemetry / 12% replay), source pinned 7f0c3236
Objective	CLM (next-token), full fine-tune, bf16, paged_adamw_8bit
LR / schedule	1.5e-5 cosine re-warm, warmup 0.03
End of training	stop-loss (eval plateau): closed at step 1,750/14,400 (12% of one epoch, ~229M tokens seen) — the last 3 evals improved ≤0.012 each
Seq / batch / HW	seq 4096, eff_batch 32, per_device=4, 1×H200 (~2,600 tok/s; the 262K-vocab fp32 logits tensor is the binding memory constraint)

Evaluation

Metric	Value	Note
PPL Asterion held-out	1.83	base gemma-4-12B: 4.80 (-62%)
PPL Mars telemetry	1.25	base: 3.67
PPL general (FineWeb-Edu)	8.34	base: 8.55 — replay works, no forgetting
eval_domain_loss	0.614	1.176 at step 0 → 0.614 at step 1,750

Eval is a held-out, deterministic verifiable harness (synthetic tasks); baseline = the pre-SFT ParamΔ model. See noval-corp/scripts/eval_agentic.py.

Intended use & limitations

Domain-knowledge backbone for the Asterion round. Not instruction-tuned (see the -instruct-paramdelta / -agentic siblings).

Limitations:

• Fictional domain — the model 'knows' an invented satellite operator.
• No chat/tool behaviour on its own (base-style CPT checkpoint).
• Saw ~12% of the corpus (stop-loss end): unique per-document facts in the unseen 88% are not in the weights — by design, downstream tasks provide specifics via tool results in context. Held-out PPL (1.83) shows the domain itself generalized.

Innovations tested

• 12B full-FT on one GPU — paged 8-bit AdamW + gradient checkpointing + a logits-aware payload model (the 262,144-token vocab makes the fp32 logits tensor, 4.3 GB/seq @4096, the binding memory term — not activations).

Citations

• CPT replay (anti-forgetting) — Ibrahim et al., 2024. arXiv:2403.08763
• Data-constrained scaling (≤4 epochs) — Muennighoff et al., 2023. arXiv:2305.16264
• LoRA learns less on CPT — Biderman et al., 2024. arXiv:2405.09673

---

Card generated by noval-corp/scripts/gen_model_cards.py (standardized across the noval-corp model family).