Upload README.md with huggingface_hub

README.md

@@ -1,199 +1,277 @@
 ---
+base_model: LiquidAI/LFM2.5-2.6B
+datasets:
+- saidutta69/fable-5-premium
 library_name: transformers
-tags: []
+pipeline_tag: text-generation
+tags:
+- lfm2
+- full-parameter-fine-tuning
+- supervised-fine-tuning
+- assistant-only-loss
+- tool-use
+- coding-agent
+- conversational
+- heretic
+- uncensored
+- decensored
+- abliterated
+- reproducible
 ---
+# This is a decensored version of [AyoubChLin/lfm2.5-2.6b-fable5-coding-agent](https://huggingface.co/AyoubChLin/lfm2.5-2.6b-fable5-coding-agent), made using [Heretic](https://heretic-project.org) v1.4.0
 
-# Model Card for Model ID
+> [!TIP]
+> **This model is reproducible!**
+>
+> See the [README](reproduce/README.md) in the `reproduce` directory for more information.
 
-<!-- Provide a quick summary of what the model is/does. -->
+## Abliteration parameters
+
+| Parameter | Value |
+| :-------- | :---: |
+| **direction_index** | 17.56 |
+| **attn.o_proj.max_weight** | 1.42 |
+| **attn.o_proj.max_weight_position** | 21.61 |
+| **attn.o_proj.min_weight** | 1.41 |
+| **attn.o_proj.min_weight_distance** | 16.49 |
+| **mlp.down_proj.max_weight** | 1.35 |
+| **mlp.down_proj.max_weight_position** | 20.51 |
+| **mlp.down_proj.min_weight** | 1.33 |
+| **mlp.down_proj.min_weight_distance** | 14.17 |
 
+## Performance
 
+| Metric | This model | Original model ([AyoubChLin/lfm2.5-2.6b-fable5-coding-agent](https://huggingface.co/AyoubChLin/lfm2.5-2.6b-fable5-coding-agent)) |
+| :----- | :--------: | :---------------------------: |
+| **KL divergence** | 0.0140 | 0 *(by definition)* |
+| **Refusals** | 7/100 | 96/100 |
 
-## Model Details
+-----
 
-### Model Description
 
-<!-- Provide a longer summary of what this model is. -->
+# LFM2.5-2.6B Fable-5 Coding Agent
 
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
+`AyoubChLin/lfm2.5-2.6b-fable5-coding-agent` is a **full-parameter supervised fine-tune** of [`LiquidAI/LFM2.5-2.6B`](https://huggingface.co/LiquidAI/LFM2.5-2.6B) on [`saidutta69/fable-5-premium`](https://huggingface.co/datasets/saidutta69/fable-5-premium).
 
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
+The run optimized assistant responses in multi-turn conversations, including reasoning-style text and tool-call patterns. All **2,697,198,592 parameters** were trainable. This repository contains a complete BF16 model checkpoint—not a LoRA, QLoRA, PEFT adapter, or quantized-weight checkpoint. The 8-bit optimizer affected optimizer-state storage only.
 
-### Model Sources [optional]
+## Model details
 
-<!-- Provide the basic links for the model. -->
+| Field | Value |
+|---|---|
+| Base model | `LiquidAI/LFM2.5-2.6B` |
+| Architecture | Causal language model |
+| Fine-tuning method | Full-parameter supervised fine-tuning |
+| Parameters | 2,697,198,592 total; 100% trainable |
+| Training precision | BF16, with TF32 enabled |
+| Maximum sequence length used for SFT | 32,000 tokens |
+| Training objective | Assistant-only next-token loss |
+| Chat formatting | Base model's native chat template |
+| Tool-call preprocessing | JSON argument strings converted to mappings for the native LFM2.5 tool-call format |
+| Reasoning data | Preserved during training (`PRESERVE_THINKING=True`) |
 
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
+## Intended use
 
-## Uses
+This checkpoint is intended for research and evaluation involving:
 
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+- multi-turn assistant behavior;
+- code generation and explanation;
+- structured tool-call generation in a controlled agent harness; and
+- further evaluation or domain adaptation.
 
-### Direct Use
+It should not be treated as production-ready based on the evidence currently available. The recorded run did not measure code correctness, tool-call validity, factuality, security, safety, bias, multilingual performance, instruction following, or agent-task completion.
 
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+## Training data
 
-[More Information Needed]
+The run loaded the `openai_chat` Parquet files explicitly so each published split was included once. It used the **first 5,000 rows** of the training split and the complete validation and test splits.
 
-### Downstream Use [optional]
+Only assistant tokens contributed to the loss. System, user, tool-result, and padding tokens were masked with label `-100`; assistant tool calls remained supervised. No row was removed by the post-tokenization assistant-label check.
 
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+### Tokenized split statistics
 
-[More Information Needed]
+| Split | Rows | Mean tokens | P95 tokens | Rows truncated at 32,000 | Mean supervised assistant tokens |
+|---|---:|---:|---:|---:|---:|
+| Train | 5,000 | 23,167.7 | 32,000 | 2,609 (52.18%) | 6,335.1 |
+| Validation | 318 | 23,010.3 | 32,000 | 165 (51.89%) | 6,331.9 |
+| Test | 319 | 22,812.0 | 32,000 | 154 (48.28%) | 6,370.4 |
 
-### Out-of-Scope Use
+Before truncation, the 5,000 selected training rows had the following length distribution:
 
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+| Statistic | Tokens |
+|---|---:|
+| P50 | 33,351 |
+| P90 | 65,820 |
+| P95 | 76,467 |
+| P99 | 92,063 |
+| Maximum | 104,776 |
 
-[More Information Needed]
+Because more than half of the selected training rows exceeded the 32,000-token training cap, long conversations were frequently truncated.
 
-## Bias, Risks, and Limitations
+## Training procedure
 
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+| Hyperparameter | Recorded value |
+|---|---|
+| Epochs | 1 |
+| Micro-batch size | 2 |
+| Gradient accumulation | 4 |
+| Effective batch size | 8 sequences per optimizer step |
+| Evaluation batch size | 1 |
+| Learning rate | `2e-5` |
+| Weight decay | `0.1` |
+| Scheduler | Cosine |
+| Warm-up argument | `0.03` supplied to `warmup_steps` |
+| Optimizer | 8-bit AdamW (`adamw_bnb_8bit`) |
+| Gradient clipping | `1.0` |
+| Gradient checkpointing | Enabled, non-reentrant |
+| Seed / data seed | 42 / 42 |
+| Evaluation cadence | Every 100 optimizer steps |
+| Checkpoint strategy | Once per epoch, model weights only |
+| Hardware | 1× NVIDIA B200, 178.4 GiB VRAM |
+| Software observed | PyTorch 2.8.0+cu129; CUDA 12.9; Transformers 5.15.0 |
+
+Checkpoints were saved with `save_only_model=True`. They are suitable for evaluation or deployment, but they do not contain optimizer and scheduler states for an exact training resume.
+
+## Results
+
+| Split / metric | Value | Derived perplexity |
+|---|---:|---:|
+| Training loss | 0.1316 | 1.1406 |
+| Validation loss | 0.3445 | 1.4113 |
+| Held-out test loss | 0.3458 | 1.4131 |
 
-[More Information Needed]
-
-### Recommendations
-
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-
-## How to Get Started with the Model
-
-Use the code below to get started with the model.
-
-[More Information Needed]
-
-## Training Details
-
-### Training Data
-
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-
-[More Information Needed]
-
-### Training Procedure
-
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-
-#### Preprocessing [optional]
-
-[More Information Needed]
-
-
-#### Training Hyperparameters
-
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-
-#### Speeds, Sizes, Times [optional]
-
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-
-[More Information Needed]
-
-## Evaluation
-
-<!-- This section describes the evaluation protocols and provides the results. -->
-
-### Testing Data, Factors & Metrics
-
-#### Testing Data
-
-<!-- This should link to a Dataset Card if possible. -->
-
-[More Information Needed]
-
-#### Factors
-
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-
-[More Information Needed]
-
-#### Metrics
-
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-
-[More Information Needed]
-
-### Results
-
-[More Information Needed]
-
-#### Summary
-
-
-
-## Model Examination [optional]
-
-<!-- Relevant interpretability work for the model goes here -->
-
-[More Information Needed]
-
-## Environmental Impact
-
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-
-## Technical Specifications [optional]
-
-### Model Architecture and Objective
-
-[More Information Needed]
-
-### Compute Infrastructure
-
-[More Information Needed]
-
-#### Hardware
-
-[More Information Needed]
-
-#### Software
-
-[More Information Needed]
-
-## Citation [optional]
-
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-
-**BibTeX:**
-
-[More Information Needed]
-
-**APA:**
-
-[More Information Needed]
-
-## Glossary [optional]
-
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-
-[More Information Needed]
-
-## More Information [optional]
-
-[More Information Needed]
-
-## Model Card Authors [optional]
-
-[More Information Needed]
-
-## Model Card Contact
-
-[More Information Needed]
+Training completed in **15,553.2 seconds** (approximately **4 h 19 min 13 s**) at 0.321 samples/second and 0.040 optimizer steps/second. The run reported approximately `2.134e18` floating-point operations.
+
+Perplexity is calculated as `exp(loss)`. All losses cover only the assistant tokens selected by the masking procedure, so they are not directly comparable with full-sequence language-model losses. Training loss is averaged over the optimization trajectory, whereas validation and test losses were measured after training.
+
+The held-out test split was not used for optimization or periodic validation. No pre-fine-tuning baseline, external benchmark, confidence interval, or repeated-seed result was recorded. These results establish held-out assistant-token loss for this run; they do not by themselves demonstrate improvement over the base model or general coding-agent quality.
+
+## Qualitative observation
+
+For one interval-merging prompt, the checkpoint produced a structured plan and emitted a native `write(...)` tool call without an explicit tool schema in the prompt. The generation reached the configured `max_new_tokens=768` limit before completing the program, and the resulting code was not executed or scored.
+
+This is an illustration, not an evaluation. In deployment:
+
+1. Provide explicit tool definitions through the serving or agent layer.
+2. Parse, authorize, and validate every generated tool call before execution.
+3. Run generated code in a sandbox and verify it with independent tests.
+4. Do not expose preserved reasoning traces when the product requires private internal reasoning.
+
+## Inference with Transformers
+
+Install a recent Transformers release:
+
+```bash
+%pip install -q --upgrade \
+    "transformers==5.15.0" \
+    "accelerate>=1.10,<2" \
+    "safetensors>=0.6" \
+    "huggingface_hub>=0.34"
+```
+
+Then apply the checkpoint's native chat template:
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+MODEL_ID = "AyoubChLin/lfm2.5-2.6b-fable5-coding-agent"
+
+tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
+
+model = AutoModelForCausalLM.from_pretrained(
+    MODEL_ID,
+    dtype=torch.float16,          # Important: T4 uses FP16, not BF16
+    device_map={"": 0},           # Keep the complete model on GPU 0
+    low_cpu_mem_usage=True,
+    attn_implementation="sdpa",   # No external flash-attn installation
+)
+
+model.eval()
+
+parameter = next(model.parameters())
+print("Model device:", parameter.device)
+print("Model dtype:", parameter.dtype)
+
+assert parameter.device.type == "cuda"
+assert parameter.dtype == torch.float16
+
+messages = [
+    {
+        "role": "system",
+        "content": (
+            "You are a careful coding assistant. Return complete, executable "
+            "code and briefly explain how it was verified."
+        ),
+    },
+    {
+        "role": "user",
+        "content": (
+            "Write a Python function that merges overlapping integer intervals. "
+            "Include pytest tests."
+        ),
+    },
+]
+
+inputs = tokenizer.apply_chat_template(
+    messages,
+    tokenize=True,
+    add_generation_prompt=True,
+    return_tensors="pt",
+    return_dict=True,
+    truncation=True,
+    max_length=8192,              # Safe starting point for a T4
+).to("cuda:0")
+
+prompt_length = inputs["input_ids"].shape[1]
+
+with torch.inference_mode():
+    output = model.generate(
+        **inputs,
+        max_new_tokens=768,
+        do_sample=True,
+        temperature=0.1,
+        top_k=50,
+        repetition_penalty=1.1,
+        use_cache=True,
+        pad_token_id=tokenizer.pad_token_id,
+        eos_token_id=tokenizer.eos_token_id,
+    )
+
+generated_tokens = output[0, prompt_length:]
+
+print(
+    tokenizer.decode(
+        generated_tokens,
+        skip_special_tokens=False,
+    )
+)
+
+```
+
+`skip_special_tokens=False` preserves native reasoning and tool-call delimiters for inspection by a compatible parser. Do not send raw reasoning or unvalidated tool syntax directly to end users or executors.
+
+## Reproducibility notes
+
+- The source run used a single NVIDIA B200 with native BF16 support.
+- The model remained in BF16 and all parameters were updated; `adamw_bnb_8bit` reduced optimizer-state memory only.
+- OpenAI-style tool-call argument strings were normalized into mappings before the native chat template was applied.
+- `PRESERVE_THINKING=True` retained supplied thinking content.
+- The variable named `WARMUP_RATIO` was passed to `warmup_steps`, not `warmup_ratio`; this card reports the executed configuration rather than reinterpreting it.
+- The bitsandbytes runtime reported that no CUDA 12.9 binary was available and loaded its CUDA 12.8 build instead.
+- The environment reported Linux kernel 4.19.0, below the Trainer warning's recommended minimum of 5.5.0.
+
+## Limitations and responsible use
+
+- Generated code and tool calls may be incomplete, incorrect, unsafe, or incompatible with the target environment.
+- Reasoning-style text may be exposed because the training data preserved it.
+- The training set was a deterministic 5,000-row prefix rather than the complete published training split.
+- Heavy 32K truncation may weaken behavior that depends on information appearing late in long conversations.
+- Tool-call patterns were learned without complete tool schemas; applications must supply schemas and enforce permissions externally.
+- The checkpoint inherits limitations from the base model and the fine-tuning dataset.
+
+Review and comply with the licenses and terms of both the [base model](https://huggingface.co/LiquidAI/LFM2.5-2.6B) and the [training dataset](https://huggingface.co/datasets/saidutta69/fable-5-premium) before use or redistribution. This model card does not grant additional rights.
+
+## Acknowledgements
+
+- Base model: [LiquidAI/LFM2.5-2.6B](https://huggingface.co/LiquidAI/LFM2.5-2.6B)
+- Training dataset: [saidutta69/fable-5-premium](https://huggingface.co/datasets/saidutta69/fable-5-premium)