Upload 11 files

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+banner.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,3 +1,216 @@
 ---
 license: apache-2.0
+language:
+- en
+library_name: transformers
+pipeline_tag: text-generation
+base_model: BananaMind/BananaMind-2-Mini
+datasets:
+- HuggingFaceTB/smol-smoltalk
+tags:
+- causal-lm
+- chat
+- instruction-tuned
+- full-finetune
+- small-language-model
+- bananamind
+- bananamind2
+- bananamind2-mini
+- smoltalk
+- pytorch
+- safetensors
+- custom-code
+- trust-remote-code
 ---
+
+![BananaMind 2 Mini Chat](banner.png)
+
+# BananaMind-2-Mini-Chat
+
+BananaMind-2-Mini-Chat is the instruction-tuned version of [BananaMind-2-Mini](https://huggingface.co/BananaMind/BananaMind-2-Mini). It was fully fine-tuned on [HuggingFaceTB/smol-smoltalk](https://huggingface.co/datasets/HuggingFaceTB/smol-smoltalk) with loss applied only to assistant tokens and the assistant-ending EOS token.
+
+The model has **25,178,752 parameters**, a **4,096-token context window**, and a custom **8,192-token digit-aware byte-level BPE tokenizer**. It supports system prompts, multi-turn conversations, and KV-cached generation.
+
+## Model Details
+
+| Field | Value |
+|---|---:|
+| Parameters | 25,178,752 |
+| Base model | `BananaMind/BananaMind-2-Mini` |
+| Architecture | BananaMind2Mini decoder-only Transformer |
+| Layers | 14 |
+| Hidden size | 384 |
+| Intermediate size | 1,024 |
+| Attention heads | 6 |
+| KV heads | 2 |
+| Head dimension | 64 |
+| Attention | Grouped-query attention with QK norm |
+| MLP | SwiGLU |
+| Position embeddings | RoPE, theta 100,000 |
+| Vocabulary size | 8,192 |
+| Context length | 4,096 |
+| Embeddings | Tied input/output embeddings |
+| Generation cache | KV cache supported |
+| Weight format | safetensors |
+| Training type | Full-parameter supervised fine-tuning |
+
+## Benchmarks
+
+### BananaMind Instruct Bench 1.1
+
+Self-reported results from the official [BananaMind Instruct Bench 1.1](https://huggingface.co/datasets/BananaMind/BananaMind-Instruct-Bench-1.1) script. This is a 300-example, difficulty- and category-weighted instruction benchmark. Generation was deterministic with `repetition_penalty=1.1` and seed 42.
+
+| Model | Overall ELO | General | Multi-turn | System Prompts | Context Recall | Code |
+|---|---:|---:|---:|---:|---:|---:|
+| BananaMind-2-Medium-Chat | 787 | 534 | 872 | 666 | 1,085 | 1,291 |
+| **BananaMind-2-Mini-Chat** | **654** | **389** | **812** | **743** | **924** | **667** |
+| Supra 1.5 50M Instruct* | 647 | 520 | 804 | 590 | 860 | 780 |
+| BananaMind-2-Nano-Chat | 643 | 353 | 796 | 600 | 978 | 885 |
+| Supra 1.0 50M Instruct* | 520 | 531 | 405 | 590 | 794 | 667 |
+
+**Detailed BananaMind-2-Mini-Chat result**
+
+| Scope | Examples | Passed | Weighted Score | ELO |
+|---|---:|---:|---:|---:|
+| **Overall** | **300** | **40** | **9.64%** | **654** |
+| General | 120 | 6 | 3.42% | 389 |
+| Multi-turn | 75 | 19 | 16.84% | 812 |
+| System Prompts | 60 | 7 | 8.95% | 743 |
+| Context Recall | 30 | 8 | 17.89% | 924 |
+| Code | 15 | 0 | 0.00% | 667 |
+
+| Difficulty | Examples | Passed | Weighted Score | ELO |
+|---|---:|---:|---:|---:|
+| Easy | 100 | 33 | 34.62% | 821 |
+| Medium | 100 | 7 | 7.45% | 593 |
+| Hard | 100 | 0 | 0.00% | 296 |
+
+\* Supra 1.5 used the benchmark's Alpaca-style fallback, with conversation context supplied through its `### Input` field. Supra 1.0 used the same fallback and a copied checkpoint with 3.0x linear RoPE scaling, extending its context from 1,024 to 3,072 tokens without additional long-context training so it could fit the benchmark inputs.
+
+These scores are self-evaluated and may vary with the benchmark revision, Transformers version, dtype, hardware, and generation settings.
+
+## Instruction Tuning
+
+| Field | Value |
+|---|---:|
+| Dataset | `HuggingFaceTB/smol-smoltalk` |
+| Dataset split | `train` |
+| Dataset rows | 460,341 |
+| Epochs | 1 |
+| Final optimizer step | 3,193 |
+| Packed tokens processed | 470,782,108 |
+| Supervised assistant tokens | 360,967,001 |
+| Sequence length | 4,096 |
+| Micro batch | 12 |
+| Gradient accumulation | 3 |
+| Effective batch | 36 sequences |
+| Peak learning rate | 2e-4 |
+| Warmup | 100 steps |
+| LR schedule | Constant after warmup |
+| Optimizer | AdamW |
+| Betas | 0.9, 0.95 |
+| Weight decay | 0.1 |
+| Gradient clipping | 1.0 |
+| Seed | 1337 |
+
+System and user messages were retained as context but masked from the loss. All parameters were trainable; no LoRA modules or adapters were used.
+
+## Chat Template
+
+The tokenizer includes a Jinja template for `system`, `user`, and `assistant` messages:
+
+```text
+<BOS><|system|>
+{system message}
+<|user|>
+{user message}
+<|assistant|>
+{assistant response}<EOS>
+```
+
+The role markers use the existing tokenizer vocabulary. Fine-tuning did not add new tokens.
+
+## Usage
+
+This model uses custom architecture code and must be loaded with `trust_remote_code=True`.
+
+```bash
+pip install -U transformers safetensors torch
+```
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_id = "BananaMind/BananaMind-2-Mini-Chat"
+device = "cuda" if torch.cuda.is_available() else "cpu"
+dtype = torch.bfloat16 if device == "cuda" and torch.cuda.is_bf16_supported() else torch.float32
+
+tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
+model = AutoModelForCausalLM.from_pretrained(
+    model_id,
+    trust_remote_code=True,
+    torch_dtype=dtype,
+).to(device).eval()
+
+messages = [
+    {"role": "system", "content": "You are a concise and helpful assistant."},
+    {"role": "user", "content": "Write one Python line that prints 2 + 3."},
+]
+
+inputs = tokenizer.apply_chat_template(
+    messages,
+    add_generation_prompt=True,
+    return_tensors="pt",
+    return_dict=True,
+)
+inputs = {name: tensor.to(device) for name, tensor in inputs.items()}
+
+with torch.inference_mode():
+    output = model.generate(
+        **inputs,
+        max_new_tokens=192,
+        do_sample=False,
+        repetition_penalty=1.1,
+        eos_token_id=tokenizer.eos_token_id,
+        pad_token_id=tokenizer.eos_token_id,
+        use_cache=True,
+    )
+
+new_tokens = output[0, inputs["input_ids"].shape[1]:]
+print(tokenizer.decode(new_tokens, skip_special_tokens=True))
+```
+
+For multi-turn chat, append the generated assistant response and the next user message to `messages`, then render the chat template again.
+
+## Suggested Generation Settings
+
+- `do_sample=False`
+- `repetition_penalty=1.1`
+- `max_new_tokens=128` to `256`
+- `use_cache=True`
+
+Small models can hallucinate facts, fail arithmetic, produce invalid code, or enter repetitive continuations. Keep a finite generation limit and do not use the model for high-stakes decisions.
+
+## Repository Files
+
+| File | Description |
+|---|---|
+| `config.json` | Transformers model configuration |
+| `model.safetensors` | Fine-tuned model weights |
+| `tokenizer.json` | Custom 8,192-token tokenizer |
+| `tokenizer_config.json` | Tokenizer metadata |
+| `chat_template.jinja` | System/user/assistant chat template |
+| `generation_config.json` | Generation configuration |
+| `configuration_bananamind2mini.py` | Custom Transformers config class |
+| `modeling_bananamind2mini.py` | Custom Transformers model class |
+| `sft_metadata.json` | Fine-tuning provenance and token counts |
+| `banner.png` | Model-card banner |
+
+## Intended Use
+
+This model is intended for compact-model research, local chat experiments, educational demonstrations, and instruction-tuning comparisons.
+
+## License
+
+Apache 2.0

chat_template.jinja

@@ -0,0 +1,10 @@
+{{ bos_token }}{% for message in messages %}{% if message['role'] == 'assistant' %}<|assistant|>
+{{ message['content'] }}{{ eos_token }}
+{% elif message['role'] == 'system' %}<|system|>
+{{ message['content'] }}
+{% elif message['role'] == 'user' %}<|user|>
+{{ message['content'] }}
+{% else %}<|{{ message['role'] }}|>
+{{ message['content'] }}
+{% endif %}{% endfor %}{% if add_generation_prompt %}<|assistant|>
+{% endif %}

config.json

@@ -0,0 +1,30 @@
+{
+  "architectures": [
+    "BananaMind2MiniForCausalLM"
+  ],
+  "auto_map": {
+    "AutoConfig": "configuration_bananamind2mini.BananaMind2MiniConfig",
+    "AutoModelForCausalLM": "modeling_bananamind2mini.BananaMind2MiniForCausalLM"
+  },
+  "bos_token_id": 1,
+  "dtype": "float32",
+  "eos_token_id": 2,
+  "head_dim": 64,
+  "hidden_size": 384,
+  "intermediate_size": 1024,
+  "max_position_embeddings": 4096,
+  "model_type": "bananamind2_mini",
+  "num_attention_heads": 6,
+  "num_hidden_layers": 14,
+  "num_key_value_heads": 2,
+  "pad_token_id": 0,
+  "rms_norm_eps": 1e-06,
+  "rope_theta": 100000.0,
+  "tie_word_embeddings": true,
+  "transformers_version": "5.7.0",
+  "unk_token_id": 3,
+  "use_cache": true,
+  "vocab_size": 8192,
+  "z_loss_coeff": 0.0,
+  "_name_or_path": "BananaMind/BananaMind-2-Mini-Chat"
+}

configuration_bananamind2mini.py

@@ -0,0 +1,48 @@
+"""BananaMind 2 Mini configuration."""
+from transformers import PretrainedConfig
+
+
+class BananaMind2MiniConfig(PretrainedConfig):
+    model_type = "bananamind2_mini"
+
+    def __init__(
+        self,
+        vocab_size=8192,
+        hidden_size=384,
+        num_hidden_layers=14,
+        num_attention_heads=6,
+        num_key_value_heads=2,
+        head_dim=64,
+        intermediate_size=1024,
+        max_position_embeddings=4096,
+        rope_theta=100000.0,
+        rms_norm_eps=1e-6,
+        tie_word_embeddings=True,
+        use_cache=True,
+        z_loss_coeff=0.0,
+        bos_token_id=1,
+        eos_token_id=2,
+        pad_token_id=0,
+        unk_token_id=3,
+        **kwargs,
+    ):
+        self.vocab_size = vocab_size
+        self.hidden_size = hidden_size
+        self.num_hidden_layers = num_hidden_layers
+        self.num_attention_heads = num_attention_heads
+        self.num_key_value_heads = num_key_value_heads
+        self.head_dim = head_dim
+        self.intermediate_size = intermediate_size
+        self.max_position_embeddings = max_position_embeddings
+        self.rope_theta = rope_theta
+        self.rms_norm_eps = rms_norm_eps
+        self.use_cache = use_cache
+        self.z_loss_coeff = z_loss_coeff
+        super().__init__(
+            tie_word_embeddings=tie_word_embeddings,
+            bos_token_id=bos_token_id,
+            eos_token_id=eos_token_id,
+            pad_token_id=pad_token_id,
+            unk_token_id=unk_token_id,
+            **kwargs,
+        )

generation_config.json

@@ -0,0 +1,8 @@
+{
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "max_length": 4096,
+  "pad_token_id": 0,
+  "transformers_version": "5.7.0",
+  "use_cache": true
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:eb4ad342c45820929ac72e5b778b2a1d7f7329ee57e51d63eaa012dfbd9d61ae
+size 100731072

modeling_bananamind2mini.py

@@ -0,0 +1,310 @@
+"""BananaMind 2 Mini model implementation for Hugging Face Transformers."""
+import math
+from typing import Optional
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from transformers import PreTrainedModel
+from transformers.cache_utils import Cache, DynamicCache
+from transformers.generation.utils import GenerationMixin
+from transformers.modeling_outputs import CausalLMOutputWithPast
+
+from .configuration_bananamind2mini import BananaMind2MiniConfig
+
+
+class RMSNorm(nn.Module):
+    def __init__(self, dim, eps=1e-6):
+        super().__init__()
+        self.eps = eps
+        self.weight = nn.Parameter(torch.ones(dim))
+
+    def forward(self, x):
+        x_float = x.float()
+        rms = torch.rsqrt(x_float.pow(2).mean(-1, keepdim=True) + self.eps)
+        return (x_float * rms * self.weight.float()).type_as(x)
+
+
+def build_rope_inv_freq(head_dim, theta=100000.0):
+    return 1.0 / (theta ** (torch.arange(0, head_dim, 2, dtype=torch.float32) / head_dim))
+
+
+def precompute_freqs_cis(head_dim, seq_len, theta=100000.0):
+    freqs = build_rope_inv_freq(head_dim, theta)
+    t = torch.arange(seq_len, dtype=torch.float32)
+    freqs = torch.outer(t, freqs)
+    return torch.polar(torch.ones_like(freqs), freqs)
+
+
+def apply_rotary_emb(q, k, freqs_cis):
+    q_complex = torch.view_as_complex(q.float().reshape(*q.shape[:-1], -1, 2))
+    k_complex = torch.view_as_complex(k.float().reshape(*k.shape[:-1], -1, 2))
+    freqs_cis = freqs_cis.unsqueeze(0).unsqueeze(0)
+    q_out = torch.view_as_real(q_complex * freqs_cis).flatten(-2)
+    k_out = torch.view_as_real(k_complex * freqs_cis).flatten(-2)
+    return q_out.type_as(q), k_out.type_as(k)
+
+
+class BananaMind2MiniAttention(nn.Module):
+    def __init__(self, config, layer_idx):
+        super().__init__()
+        self.layer_idx = layer_idx
+        self.n_head = config.num_attention_heads
+        self.n_kv_heads = config.num_key_value_heads
+        self.head_dim = config.head_dim
+        self.n_rep = self.n_head // self.n_kv_heads
+
+        self.q_proj = nn.Linear(config.hidden_size, self.n_head * self.head_dim, bias=False)
+        self.k_proj = nn.Linear(config.hidden_size, self.n_kv_heads * self.head_dim, bias=False)
+        self.v_proj = nn.Linear(config.hidden_size, self.n_kv_heads * self.head_dim, bias=False)
+        self.o_proj = nn.Linear(self.n_head * self.head_dim, config.hidden_size, bias=False)
+        self.q_norm = RMSNorm(self.head_dim, eps=config.rms_norm_eps)
+        self.k_norm = RMSNorm(self.head_dim, eps=config.rms_norm_eps)
+
+    def forward(
+        self,
+        x,
+        freqs_cis,
+        attention_mask=None,
+        past_key_values=None,
+        use_cache=False,
+    ):
+        batch_size, seq_len, _ = x.shape
+        q = self.q_proj(x).view(
+            batch_size,
+            seq_len,
+            self.n_head,
+            self.head_dim,
+        ).transpose(1, 2)
+        k = self.k_proj(x).view(
+            batch_size,
+            seq_len,
+            self.n_kv_heads,
+            self.head_dim,
+        ).transpose(1, 2)
+        v = self.v_proj(x).view(
+            batch_size,
+            seq_len,
+            self.n_kv_heads,
+            self.head_dim,
+        ).transpose(1, 2)
+
+        q = self.q_norm(q)
+        k = self.k_norm(k)
+        q, k = apply_rotary_emb(q, k, freqs_cis)
+
+        past_length = 0
+        if use_cache and past_key_values is not None:
+            past_length = past_key_values.get_seq_length(self.layer_idx)
+            k, v = past_key_values.update(k, v, self.layer_idx)
+
+        kv_len = k.size(-2)
+        k = k.unsqueeze(2).expand(
+            batch_size,
+            self.n_kv_heads,
+            self.n_rep,
+            kv_len,
+            self.head_dim,
+        ).reshape(batch_size, self.n_head, kv_len, self.head_dim)
+        v = v.unsqueeze(2).expand(
+            batch_size,
+            self.n_kv_heads,
+            self.n_rep,
+            kv_len,
+            self.head_dim,
+        ).reshape(batch_size, self.n_head, kv_len, self.head_dim)
+
+        attn_mask = None
+        is_causal = past_length == 0 and attention_mask is None
+        if not is_causal:
+            query_positions = past_length + torch.arange(seq_len, device=x.device)
+            key_positions = torch.arange(kv_len, device=x.device)
+            causal = key_positions.unsqueeze(0) <= query_positions.unsqueeze(1)
+            attn_mask = causal[None, None, :, :]
+            if attention_mask is not None:
+                key_padding = attention_mask.to(torch.bool)
+                if key_padding.size(-1) < kv_len:
+                    cached_padding = torch.ones(
+                        key_padding.size(0),
+                        kv_len - key_padding.size(-1),
+                        dtype=torch.bool,
+                        device=key_padding.device,
+                    )
+                    key_padding = torch.cat((cached_padding, key_padding), dim=-1)
+                else:
+                    key_padding = key_padding[:, -kv_len:]
+                attn_mask = attn_mask & key_padding[:, None, None, :]
+            is_causal = False
+
+        y = F.scaled_dot_product_attention(
+            q,
+            k,
+            v,
+            attn_mask=attn_mask,
+            is_causal=is_causal,
+        )
+        y = y.transpose(1, 2).contiguous().view(
+            batch_size,
+            seq_len,
+            self.n_head * self.head_dim,
+        )
+        return self.o_proj(y)
+
+
+class BananaMind2MiniSwiGLUMLP(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        self.w_gate = nn.Linear(config.hidden_size, config.intermediate_size, bias=False)
+        self.w_up = nn.Linear(config.hidden_size, config.intermediate_size, bias=False)
+        self.w_down = nn.Linear(config.intermediate_size, config.hidden_size, bias=False)
+
+    def forward(self, x):
+        return self.w_down(F.silu(self.w_gate(x)) * self.w_up(x))
+
+
+class BananaMind2MiniBlock(nn.Module):
+    def __init__(self, config, layer_idx):
+        super().__init__()
+        self.ln_1 = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
+        self.attn = BananaMind2MiniAttention(config, layer_idx)
+        self.ln_2 = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
+        self.mlp = BananaMind2MiniSwiGLUMLP(config)
+
+    def forward(
+        self,
+        x,
+        freqs_cis,
+        attention_mask=None,
+        past_key_values=None,
+        use_cache=False,
+    ):
+        x = x + self.attn(
+            self.ln_1(x),
+            freqs_cis,
+            attention_mask=attention_mask,
+            past_key_values=past_key_values,
+            use_cache=use_cache,
+        )
+        return x + self.mlp(self.ln_2(x))
+
+
+class BananaMind2MiniPreTrainedModel(PreTrainedModel):
+    config_class = BananaMind2MiniConfig
+    base_model_prefix = "transformer"
+    supports_gradient_checkpointing = False
+
+    def _init_weights(self, module):
+        std = 0.02
+        if hasattr(module, "NANOGPT_SCALE_INIT"):
+            std *= 2 * self.config.num_hidden_layers ** -0.5
+        if isinstance(module, nn.Linear):
+            torch.nn.init.normal_(module.weight, mean=0.0, std=std)
+        elif isinstance(module, nn.Embedding):
+            torch.nn.init.normal_(module.weight, mean=0.0, std=0.02)
+
+
+class BananaMind2MiniForCausalLM(BananaMind2MiniPreTrainedModel, GenerationMixin):
+    _tied_weights_keys = {"lm_head.weight": "transformer.wte.weight"}
+
+    def __init__(self, config):
+        super().__init__(config)
+        self.config = config
+        self.transformer = nn.ModuleDict(
+            {
+                "wte": nn.Embedding(config.vocab_size, config.hidden_size),
+                "h": nn.ModuleList(
+                    [
+                        BananaMind2MiniBlock(config, layer_idx)
+                        for layer_idx in range(config.num_hidden_layers)
+                    ]
+                ),
+                "ln_f": RMSNorm(config.hidden_size, eps=config.rms_norm_eps),
+            }
+        )
+        self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
+        if config.tie_word_embeddings:
+            self.lm_head.weight = self.transformer["wte"].weight
+        self._embd_scale = math.sqrt(config.hidden_size)
+        self._freqs_cis_cache = None
+        self.post_init()
+
+    def get_input_embeddings(self):
+        return self.transformer["wte"]
+
+    def set_input_embeddings(self, value):
+        self.transformer["wte"] = value
+
+    def get_output_embeddings(self):
+        return self.lm_head
+
+    def set_output_embeddings(self, new_embeddings):
+        self.lm_head = new_embeddings
+
+    def _get_freqs_cis(self, seq_len, device):
+        cache = self._freqs_cis_cache
+        if cache is None or cache.device != device or cache.size(0) < seq_len:
+            # Keep the reusable RoPE table as a normal tensor even if its first
+            # construction happens inside torch.inference_mode().
+            with torch.inference_mode(False):
+                cache = precompute_freqs_cis(
+                    self.config.head_dim,
+                    seq_len,
+                    self.config.rope_theta,
+                ).to(device)
+            self._freqs_cis_cache = cache
+        return cache[:seq_len]
+
+    def forward(
+        self,
+        input_ids,
+        attention_mask=None,
+        labels=None,
+        past_key_values: Optional[Cache] = None,
+        use_cache=None,
+        **kwargs,
+    ):
+        _, seq_len = input_ids.shape
+        if use_cache is None:
+            use_cache = self.config.use_cache and labels is None
+        if use_cache and past_key_values is None:
+            past_key_values = DynamicCache(config=self.config)
+
+        past_length = past_key_values.get_seq_length() if use_cache else 0
+        total_length = past_length + seq_len
+        if total_length > self.config.max_position_embeddings:
+            raise ValueError(
+                f"Sequence length {total_length} exceeds the configured maximum "
+                f"of {self.config.max_position_embeddings}"
+            )
+
+        x = self.transformer["wte"](input_ids) * self._embd_scale
+        freqs_cis = self._get_freqs_cis(total_length, input_ids.device)[past_length:]
+
+        for block in self.transformer["h"]:
+            x = block(
+                x,
+                freqs_cis,
+                attention_mask=attention_mask,
+                past_key_values=past_key_values,
+                use_cache=use_cache,
+            )
+
+        x = self.transformer["ln_f"](x)
+        logits = self.lm_head(x)
+
+        loss = None
+        if labels is not None:
+            shift_logits = logits[..., :-1, :].contiguous()
+            shift_labels = labels[..., 1:].contiguous()
+            loss = F.cross_entropy(
+                shift_logits.float().reshape(-1, shift_logits.size(-1)),
+                shift_labels.reshape(-1),
+            )
+            if self.config.z_loss_coeff:
+                loss = loss + self.config.z_loss_coeff * logits.float().pow(2).mean()
+
+        return CausalLMOutputWithPast(
+            loss=loss,
+            logits=logits,
+            past_key_values=past_key_values if use_cache else None,
+        )

sft_metadata.json

@@ -0,0 +1,12 @@
+{
+  "base_model": "BananaMind-2-Mini",
+  "dataset": "HuggingFaceTB/smol-smoltalk",
+  "dataset_config": "default",
+  "dataset_split": "train",
+  "training_type": "full_parameter_supervised_finetuning",
+  "loss_mask": "assistant_content_and_eos_only",
+  "global_step": 3193,
+  "tokens_seen": 470782108,
+  "supervised_tokens_seen": 360967001,
+  "repo_id": "BananaMind/BananaMind-2-Mini-Chat"
+}

tokenizer_config.json

@@ -0,0 +1,11 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<|bos|>",
+  "eos_token": "<|eos|>",
+  "is_local": true,
+  "local_files_only": true,
+  "model_max_length": 4096,
+  "pad_token": "<|pad|>",
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": "<|unk|>"
+}