Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,145 +1,16 @@
----
-license: other
-language:
-  - ru
-  - en
-tags:
-  - causal-lm
-  - bilingual
-  - moe
-  - pytorch
-  - experimental
-  - nullxes
-  - veridian
-pipeline_tag: text-generation
-library_name: pytorch
----
+# veridian-beta
 
----
+Current checkpoint:
+- `step_0000050.pt`
 
-# VERIDIAN Ru MoE Beta
+Base:
+- VERIDIAN 2B `step_0004750.pt`
 
-### by NULLXES
+Pipeline:
+- continued pretraining from `step_0004750.pt`
+- quick clean SFT from `pretrain_stage2_5 step_0001000.pt`
 
-**VERIDIAN** is an experimental bilingual (RU/EN) language model developed by **NULLXES**.
-
-This checkpoint is the first public beta of the VERIDIAN line — a raw but intentional snapshot of the model’s early architecture, identity layer, and bilingual behavior.
-
-VERIDIAN is not designed as a generic chatbot. It is being built as a **foundational model for execution-oriented AI systems** — including digital employees, assistants, and real-time integrated agents.
-
----
-
-## 🧠 Core Idea
-
-VERIDIAN sits inside the broader **NULLXES AI infrastructure stack**, where models are expected not just to generate text, but to:
-
-* operate inside real systems
-* interact with tools and workflows
-* act as components of larger AI-driven architectures
-
-This beta reflects that direction at an early stage.
-
----
-
-## ⚙️ Current Capabilities
-
-* Bilingual baseline (Russian / English)
-* Early identity anchoring
-* Experimental sparse MoE architecture
-* Initial behavior shaping toward system integration
-
----
-
-## 🚧 Status
-
-* **Version:** beta-0
-* **Stage:** early experimental
-* **Languages:** RU / EN
-* **Format:** PyTorch → `safetensors`
-
----
-
-## ⚠️ Known Limitations
-
-* Tokenizer splits names, brands, and custom tokens inconsistently
-* Output quality is unstable across domains
-* Reasoning depth is limited
-* Instruction-following is still weak
-* Not optimized for long-context tasks
-* Hugging Face integration is minimal (artifact-focused)
-
----
-
-## 📦 Files
-
-* `veridian_beta_0.safetensors` — model weights
-* `veridian_beta_0_config.yaml` — original config
-* `config.json` — HF-compatible config
-* `tokenizer.json` — tokenizer
-* `tokenizer_config.json` — tokenizer metadata
-* `special_tokens_map.json` — special tokens
-* `generation_config.json` — generation defaults
-
----
-
-## 🚀 Intended Use
-
-* Research and experimentation
-* Internal system testing
-* Prototyping bilingual agents
-* Custom pipeline integration
-* Studying early-stage MoE behavior
-
----
-
-## ⛔ Not Recommended For
-
-* Production deployment
-* High-stakes or safety-critical decisions
-* Reliability-sensitive applications
-* Benchmark comparisons with mature models
-* Polished end-user chat experiences
-
----
-
-## 🧩 Positioning
-
-VERIDIAN is part of a shift from:
-
-> "models that answer"
-
-to:
-
-> "models that operate"
-
-It is being developed as an execution layer inside AI systems — not just an interface.
-
----
-
-## 🏗️ About NULLXES
-
-**NULLXES** builds next-generation AI infrastructure and digital employees designed to operate in real-world environments.
-
-VERIDIAN is a foundational model initiative within this ecosystem.
-
----
-
-## 🔮 Roadmap (high-level)
-
-* Tokenizer stability improvements
-* Stronger bilingual coherence
-* Instruction tuning and alignment layers
-* More consistent identity behavior
-* Expanded training datasets
-* Integration into real-time systems (voice / avatar / agents)
-* Progression toward multimodal capabilities
-
----
-
-## 💬 Release Note
-
-This is not a finished model.
-
-This is a **starting checkpoint** — raw, imperfect, but directional.
-
-VERIDIAN will evolve alongside the NULLXES ecosystem into a system-level AI component.
+Files:
+- `step_0000050.pt`
+- `tokenizer.json`
+- `configs/`

configs/sft_stage5_clean_512.yaml

@@ -0,0 +1,12 @@
+name: veridian-sft-stage5-clean-512
+streaming: false
+shuffle_buffer_size: 1024
+sequence_length: 512
+min_document_tokens: 0
+
+datasets:
+  - name: clean_stage5
+    path: data/stage5_clean/veridian_stage5_clean.jsonl
+    split: train
+    text_field: messages
+    weight: 1.0

configs/sft_stage6_from_cpt1000_512.yaml

@@ -0,0 +1,24 @@
+name: veridian-sft-stage6-from-cpt1000-512
+seed: 42
+device: cuda
+dtype: bfloat16
+global_batch_size_tokens: 16384
+micro_batch_size_sequences: 1
+gradient_accumulation_steps: 32
+num_train_steps: 100
+learning_rate: 2.0e-7
+min_learning_rate: 5.0e-8
+warmup_steps: 10
+weight_decay: 0.0
+adam_beta1: 0.9
+adam_beta2: 0.95
+adam_epsilon: 1.0e-8
+grad_clip_norm: 0.15
+log_every: 10
+eval_every: 0
+save_every: 50
+checkpoint_dir: checkpoints/sft_stage6_cpt1000
+bf16: true
+use_tf32: true
+activation_checkpointing: true
+compile_model: false

configs/veridian_2b_bpe131k.yaml

@@ -0,0 +1,27 @@
+name: veridian-2b-bpe131k
+vocab_size: 131072
+hidden_size: 2048
+intermediate_size: 5632
+num_hidden_layers: 20
+num_attention_heads: 16
+num_key_value_heads: 4
+num_local_experts: 4
+num_shared_experts: 1
+num_experts_per_tok: 2
+max_position_embeddings: 1024
+target_max_position_embeddings: 1024
+pad_token_id: 0
+bos_token_id: 1
+eos_token_id: 2
+attention_dropout: 0.0
+residual_dropout: 0.0
+rms_norm_eps: 1.0e-6
+rope_theta: 100000.0
+router_aux_loss_coef: 0.01
+router_z_loss_coef: 1.0e-4
+capacity_factor_train: 1.0
+capacity_factor_eval: 1.0
+initializer_range: 0.02
+tie_word_embeddings: true
+use_bias: false
+use_cache: true

step_0000050.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4dbdac8eeea99b330aa155c87daf6e58e03e7458ed1e43f2f744a99bd8f87144
+size 47264986666