Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,107 @@
+---
+license: mit
+base_model:
+- meta-llama/Llama-4-Scout-17B-16E-Instruct
+library_name: transformers
+---
+
+# Llama-4-Scout-1.7B-0.4B-Instruct
+
+This is a tiny version of [meta-llama/Llama-4-Scout-17B-16E-Instruct](https://huggingface.co/meta-llama/Llama-4-Scout-17B-16E-Instruct) created for testing and development.
+
+## Model Details
+
+- **Base Model**: meta-llama/Llama-4-Scout-17B-16E-Instruct
+- **Architecture**: llama4 (multimodal vision-language with MoE)
+- **Total Parameters**: 1.72B
+- **Activated Parameters**: ~0.43B (1 expert activated per token out of 4)
+
+## Configuration Changes
+
+The following parameters were reduced from the original model:
+
+| Parameter | Original | Tiny |
+|-----------|----------|------|
+| **Text Model** | | |
+| num_hidden_layers | 48 | 8 |
+| num_local_experts | 16 | 4 |
+| num_experts_per_tok | 1 | 1 |
+| hidden_size | 5120 | 2048 |
+| intermediate_size | 8192 | 3072 |
+| intermediate_size_mlp | 16384 | 6144 |
+| num_attention_heads | 40 | 16 |
+| num_key_value_heads | 8 | 4 |
+| layer_types | 48 layers (chunked/full pattern) | 8 layers (maintains 3:1 pattern) |
+| **Vision Model** | | |
+| num_hidden_layers | 34 | 6 |
+| hidden_size | 1408 | 768 |
+| intermediate_size | 5632 | 3072 |
+| num_attention_heads | 16 | 12 |
+
+## Architecture Preservation
+
+The tiny model maintains the original Llama-4-Scout architecture patterns:
+- **MoE Structure**: Retained mixture-of-experts with shared expert
+- **Attention Pattern**: Maintains the chunked_attention/full_attention pattern (every 4th layer is full_attention)
+- **No-RoPE Layers**: Preserved the pattern where 3 out of every 4 layers use alternative position encoding
+
+## Checkpoint Structure
+
+The model is saved as a single safetensors file following the original checkpoint structure:
+- `language_model.model.layers.{X}.feed_forward.experts.*`
+- `language_model.model.layers.{X}.feed_forward.shared_expert.*`
+- `vision_model.model.layers.{X}.*`
+
+This structure is compatible with transformers' `Llama4ForConditionalGeneration`.
+
+## Usage
+
+```python
+from transformers import Llama4ForConditionalGeneration, AutoProcessor
+
+model = Llama4ForConditionalGeneration.from_pretrained(
+    "inference-optimization/Llama-4-Scout-1.7B-0.4B-Instruct",
+    device_map="auto"
+)
+processor = AutoProcessor.from_pretrained("inference-optimization/Llama-4-Scout-1.7B-0.4B-Instruct")
+
+# Text-only input
+text = "Hello, world!"
+inputs = processor.tokenizer(text, return_tensors="pt").to(model.device)
+outputs = model.generate(**inputs, max_new_tokens=20)
+print(processor.tokenizer.decode(outputs[0]))
+```
+
+## Creation Process
+
+This model was created using the llm-compressor `create-tiny-model` skill:
+
+1. **Config Modification**: Reduced layers, experts, and hidden dimensions while preserving architectural patterns
+2. **Weight Initialization**: Randomly initialized weights using the model's init_weights() method
+3. **Fine-tuning Attempt**: Attempted text-only fine-tuning on a small corpus (note: the multimodal architecture made standard text-only fine-tuning ineffective, but the model structure is valid)
+4. **Validation**: Verified model loads correctly and can perform inference
+
+## Notes
+
+**Important**: This is a tiny model with randomly initialized weights intended for **testing and development purposes only**. It is not trained and will not produce meaningful outputs. The vision tower is completely untrained.
+
+### Use Cases
+- Testing model loading and inference pipelines
+- Validating quantization and compression workflows
+- Debugging multimodal model handling
+- CI/CD pipeline testing with realistic model sizes
+- Memory profiling and optimization experiments
+
+### Limitations
+- Randomly initialized weights (not trained)
+- Will generate nonsensical outputs
+- Vision capabilities are non-functional
+- Not suitable for any production use or evaluation benchmarks
+
+## Technical Warnings
+
+When loading this model, you may see the warning:
+```
+[transformers] `rope_parameters`'s high_freq_factor field must be greater than low_freq_factor
+```
+This is a known issue with the Llama-4 config and can be safely ignored.

chat_template.jinja

@@ -0,0 +1,111 @@
+{{- bos_token }}
+{%- if custom_tools is defined and custom_tools%}
+    {%- set tools = custom_tools %}
+{%- endif %}
+{%- if tools is defined and tools %}
+    {%- set tool_definition = tool_definition ~ (tools | tojson(indent=4)) %}
+{%- else %}
+    {%- set tools = none %}
+{%- endif %}
+
+
+{#- This block extracts the system message, so we can slot it into the right place. #}
+{%- if messages[0]['role'] == 'system' %}
+    {%- set user_provided_system_message = true %}
+    {%- if messages[0]['content'] is string %}
+        {%- set system_message = messages[0]['content']|trim %}
+    {%- else %}
+        {%- set system_message = messages[0]['content'][0]['text']|trim %}
+    {%- endif %}
+    {%- set messages = messages[1:] %}
+{%- else %}
+    {%- if tools is not none  %}
+        {#- Since not system_message was provided by user, if tool is provided, system_message is now default tool system message #}
+        {#- This system message is from llama website:https://www.llama.com/docs/model-cards-and-prompt-formats/llama4/  #}
+        {%- set system_message = "You are a helpful assistant and an expert in function composition. You can answer general questions using your internal knowledge OR invoke functions when necessary. Follow these strict guidelines:\n\n1. FUNCTION CALLS:\n- ONLY use functions that are EXPLICITLY listed in the function list below\n- If NO functions are listed (empty function list []), respond ONLY with internal knowledge or \"I don't have access to [Unavailable service] information\"\n- If a function is not in the list, respond ONLY with internal knowledge or \"I don't have access to [Unavailable service] information\"\n- If ALL required parameters are present AND the query EXACTLY matches a listed function's purpose: output ONLY the function call(s)\n- Use exact format: [func_name1(param1=value1, param2=value2), func_name2(...)]\nExamples:\nCORRECT: [get_weather(location=\"Vancouver\"), calculate_route(start=\"Boston\", end=\"New York\")] <- Only if get_weather and calculate_route are in function list\nINCORRECT: get_weather(location=\"New York\")\nINCORRECT: Let me check the weather: [get_weather(location=\"New York\")]\nINCORRECT: [get_events(location=\"Singapore\")] <- If function not in list\n\n2. RESPONSE RULES:\n- For pure function requests matching a listed function: ONLY output the function call(s)\n- For knowledge questions: ONLY output text\n- For missing parameters: ONLY request the specific missing parameters\n- For unavailable services (not in function list): output ONLY with internal knowledge or \"I don't have access to [Unavailable service] information\". Do NOT execute a function call.\n- If the query asks for information beyond what a listed function provides: output ONLY with internal knowledge about your limitations\n- NEVER combine text and function calls in the same response\n- NEVER suggest alternative functions when the requested service is unavailable\n- NEVER create or invent new functions not listed below\n\n3. STRICT BOUNDARIES:\n- ONLY use functions from the list below - no exceptions\n- NEVER use a function as an alternative to unavailable information\n- NEVER call functions not present in the function list\n- NEVER add explanatory text to function calls\n- NEVER respond with empty brackets\n- Use proper Python/JSON syntax for function calls\n- Check the function list carefully before responding\n\n4. TOOL RESPONSE HANDLING:\n- When receiving tool responses: provide concise, natural language responses\n- Don't repeat tool response verbatim\n- Don't add supplementary information\n\nHere is a list of functions in JSON format that you can invoke:\n" %}
+    {%- else %}
+        {%- set system_message = "" %}
+    {%- endif %}
+{%- endif %}
+{#- Now writing the system message: use the user provided system message if user_provided_system_message, else default tool system message if tools presented #}
+{%- if system_message %}
+    {#- always use user provided system message to override default tool system message #}
+    {{- "<|header_start|>system<|header_end|>\n\n" }}
+    {{- system_message }}
+    {%- if user_provided_system_message and tools %}
+        {{- "\nHere is a list of functions in JSON format that you can invoke. Use exact format: [func_name1(param1=value1, param2=value2), func_name2(...)]\n" }}
+        {{- tool_definition -}}
+        {%- elif tool_definition %}
+        {{- tool_definition -}}
+    {%- endif %}
+    {{- "<|eot|>" }}
+{%- endif %}
+
+{#- Now deal with all other messages #}
+{%- for message in messages %}
+    {#- Base case: messages that are not from tool role and has empty tool_call list  #}
+    {%- if not (message.role == 'ipython' or message.role == 'tool' or ('tool_calls' in message and  message.tool_calls|length != 0 )) %}
+        {{- '<|header_start|>' + message['role'] + '<|header_end|>\n\n' }}
+        {%- if message['content'] is string %}
+            {{- message['content'] }}
+        {%- else %}
+            {%- for content in message['content'] %}
+                {%- if content['type'] == 'image' %}
+                    {{- '<|image|>' }}
+                {%- elif content['type'] == 'text' %}
+                    {{- content['text'] | trim }}
+                {%- endif %}
+            {%- endfor %}
+        {%- endif %}
+    {{- "<|eot|>" }}
+    {#- Tool case: messages has non-empty tool_call list, must from assistant #}
+    {%- elif 'tool_calls' in message %}
+        {#- assume tool_calls are always coming from assistant #}
+        {%- if message.role == 'assistant' %}
+            {{- '<|header_start|>assistant<|header_end|>\n\n' -}}
+        {%- if message['content'] is string %}
+            {{- message['content'] }}
+        {%- else %}
+            {%- for content in message['content'] %}
+                {%- if content['type'] == 'image' %}
+                    {{- '<|image|>' }}
+                {%- elif content['type'] == 'text' %}
+                    {{- content['text'] }}
+                {%- endif %}
+            {%- endfor %}
+        {%- endif %}
+            {{- "[" }}
+        {%- for tool_call in message.tool_calls %}
+            {%- if tool_call.function is defined %}
+                {%- set tool_call = tool_call.function %}
+            {%- endif %}
+                {{-  tool_call.name + '(' -}}
+            {%- for param in tool_call.arguments %}
+                {{- param + '="' -}}
+                {{- "%s" | format(tool_call.arguments[param]) -}}
+                {{- '"' -}}
+                {% if not loop.last %}, {% endif %}
+            {%- endfor %}
+            {{- ')' -}}
+            {% if not loop.last %}, {% endif %}
+        {%- endfor %}
+        {{- "]<|eot|>" }}
+{%- endif %}
+{#- Tool_response case: messages are from tool_response  #}
+    {%- elif message.role == "tool" or message.role == "ipython" %}
+        {{- "<|header_start|>ipython<|header_end|>\n\n" }}
+        {%- if message.content is string %}
+            {{-  message.content  | tojson }}
+        {%- else %}
+            {%- for content in message['content']  %}
+                {%- if content['type']  == 'text' %}
+                    {{-  content['text'] | tojson }}
+                {%- endif %}
+            {%- endfor %}
+        {%- endif %}
+        {{- "<|eot|>" }}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|header_start|>assistant<|header_end|>\n\n' }}
+{%- endif %}

checkpoint-500/config.json

@@ -0,0 +1,120 @@
+{
+  "architectures": [
+    "Llama4ForConditionalGeneration"
+  ],
+  "boi_token_index": 200080,
+  "dtype": "float32",
+  "eoi_token_index": 200081,
+  "image_token_index": 200092,
+  "model_type": "llama4",
+  "text_config": {
+    "_attn_implementation_autoset": true,
+    "attention_bias": false,
+    "attention_chunk_size": 8192,
+    "attention_dropout": 0.0,
+    "attn_scale": 0.1,
+    "attn_temperature_tuning": true,
+    "bos_token_id": 200000,
+    "dtype": "float32",
+    "eos_token_id": [
+      200001,
+      200007,
+      200008
+    ],
+    "floor_scale": 8192,
+    "for_llm_compressor": false,
+    "head_dim": 128,
+    "hidden_act": "silu",
+    "hidden_size": 2048,
+    "initializer_range": 0.02,
+    "interleave_moe_layer_step": 1,
+    "intermediate_size": 3072,
+    "intermediate_size_mlp": 6144,
+    "layer_types": [
+      "chunked_attention",
+      "chunked_attention",
+      "chunked_attention",
+      "full_attention",
+      "chunked_attention",
+      "chunked_attention",
+      "chunked_attention",
+      "full_attention"
+    ],
+    "max_position_embeddings": 10485760,
+    "model_type": "llama4_text",
+    "moe_layers": [
+      0,
+      1,
+      2,
+      3,
+      4,
+      5,
+      6,
+      7
+    ],
+    "no_rope_layer_interval": 4,
+    "no_rope_layers": [
+      1,
+      1,
+      1,
+      0,
+      1,
+      1,
+      1,
+      0
+    ],
+    "num_attention_heads": 16,
+    "num_experts_per_tok": 1,
+    "num_hidden_layers": 8,
+    "num_key_value_heads": 4,
+    "num_local_experts": 4,
+    "output_router_logits": false,
+    "pad_token_id": 200018,
+    "rms_norm_eps": 1e-05,
+    "rope_parameters": {
+      "factor": 16.0,
+      "high_freq_factor": 1.0,
+      "low_freq_factor": 1.0,
+      "original_max_position_embeddings": 8192,
+      "rope_theta": 500000.0,
+      "rope_type": "llama3"
+    },
+    "router_aux_loss_coef": 0.001,
+    "router_jitter_noise": 0.0,
+    "tie_word_embeddings": false,
+    "use_cache": true,
+    "use_qk_norm": true,
+    "vocab_size": 202048
+  },
+  "tie_word_embeddings": false,
+  "transformers_version": "5.10.1",
+  "use_cache": false,
+  "vision_config": {
+    "_attn_implementation_autoset": true,
+    "attention_dropout": 0.0,
+    "dtype": "float32",
+    "hidden_act": "gelu",
+    "hidden_size": 768,
+    "image_size": 336,
+    "initializer_range": 0.02,
+    "intermediate_size": 3072,
+    "model_type": "llama4_vision_model",
+    "multi_modal_projector_bias": false,
+    "norm_eps": 1e-05,
+    "num_attention_heads": 12,
+    "num_channels": 3,
+    "num_hidden_layers": 6,
+    "patch_size": 14,
+    "pixel_shuffle_ratio": 0.5,
+    "projector_dropout": 0.0,
+    "projector_input_dim": 2048,
+    "projector_output_dim": 2048,
+    "rope_parameters": {
+      "rope_theta": 10000,
+      "rope_type": "default"
+    },
+    "vision_feature_layer": -1,
+    "vision_feature_select_strategy": "default",
+    "vision_output_dim": 2048
+  }
+}

checkpoint-500/generation_config.json

@@ -0,0 +1,14 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 200000,
+  "eos_token_id": [
+    200001,
+    200007,
+    200008
+  ],
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "pad_token_id": 200018,
+  "transformers_version": "5.10.1",
+  "use_cache": true
+}

checkpoint-500/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:739252e404227a38b8f48330f3fa86169e7c162382cd134cb2743dda6b333ec3
+size 6898648848

checkpoint-500/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:757c8865bc6fb275bd2bd43e19c1fb1d8104fca25c210766a47538a941bfd0d7
+size 13332487139

checkpoint-500/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ad039bf1b460e4f46a4f0c9fde92bda8a17a5127d6c3692469420560b3b1c3ff
+size 14645

checkpoint-500/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3105ebe8471f9890c3eb1f20cc0f2520fa5fdb0128474bbc87e607b2ec7c53dc
+size 1465

checkpoint-500/trainer_state.json

@@ -0,0 +1,391 @@
+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 125.0,
+  "eval_steps": 500,
+  "global_step": 500,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.25,
+      "grad_norm": 0.0,
+      "learning_rate": 5e-05,
+      "loss": 12.21626091003418,
+      "step": 1
+    },
+    {
+      "epoch": 2.5,
+      "grad_norm": 0.0,
+      "learning_rate": 4.91e-05,
+      "loss": 12.21626196967231,
+      "step": 10
+    },
+    {
+      "epoch": 5.0,
+      "grad_norm": 0.0,
+      "learning_rate": 4.8100000000000004e-05,
+      "loss": 12.216262054443359,
+      "step": 20
+    },
+    {
+      "epoch": 7.5,
+      "grad_norm": 0.0,
+      "learning_rate": 4.71e-05,
+      "loss": 12.216262054443359,
+      "step": 30
+    },
+    {
+      "epoch": 10.0,
+      "grad_norm": 0.0,
+      "learning_rate": 4.61e-05,
+      "loss": 12.216262054443359,
+      "step": 40
+    },
+    {
+      "epoch": 12.5,
+      "grad_norm": 0.0,
+      "learning_rate": 4.5100000000000005e-05,
+      "loss": 12.216262054443359,
+      "step": 50
+    },
+    {
+      "epoch": 15.0,
+      "grad_norm": 0.0,
+      "learning_rate": 4.41e-05,
+      "loss": 12.216262054443359,
+      "step": 60
+    },
+    {
+      "epoch": 17.5,
+      "grad_norm": 0.0,
+      "learning_rate": 4.3100000000000004e-05,
+      "loss": 12.216262054443359,
+      "step": 70
+    },
+    {
+      "epoch": 20.0,
+      "grad_norm": 0.0,
+      "learning_rate": 4.21e-05,
+      "loss": 12.216262054443359,
+      "step": 80
+    },
+    {
+      "epoch": 22.5,
+      "grad_norm": 0.0,
+      "learning_rate": 4.11e-05,
+      "loss": 12.216262054443359,
+      "step": 90
+    },
+    {
+      "epoch": 25.0,
+      "grad_norm": 0.0,
+      "learning_rate": 4.0100000000000006e-05,
+      "loss": 12.216262054443359,
+      "step": 100
+    },
+    {
+      "epoch": 27.5,
+      "grad_norm": 0.0,
+      "learning_rate": 3.91e-05,
+      "loss": 12.216262054443359,
+      "step": 110
+    },
+    {
+      "epoch": 30.0,
+      "grad_norm": 0.0,
+      "learning_rate": 3.8100000000000005e-05,
+      "loss": 12.216262054443359,
+      "step": 120
+    },
+    {
+      "epoch": 32.5,
+      "grad_norm": 0.0,
+      "learning_rate": 3.71e-05,
+      "loss": 12.216262054443359,
+      "step": 130
+    },
+    {
+      "epoch": 35.0,
+      "grad_norm": 0.0,
+      "learning_rate": 3.61e-05,
+      "loss": 12.216262054443359,
+      "step": 140
+    },
+    {
+      "epoch": 37.5,
+      "grad_norm": 0.0,
+      "learning_rate": 3.51e-05,
+      "loss": 12.216262054443359,
+      "step": 150
+    },
+    {
+      "epoch": 40.0,
+      "grad_norm": 0.0,
+      "learning_rate": 3.41e-05,
+      "loss": 12.216262054443359,
+      "step": 160
+    },
+    {
+      "epoch": 42.5,
+      "grad_norm": 0.0,
+      "learning_rate": 3.3100000000000005e-05,
+      "loss": 12.216262054443359,
+      "step": 170
+    },
+    {
+      "epoch": 45.0,
+      "grad_norm": 0.0,
+      "learning_rate": 3.21e-05,
+      "loss": 12.216262054443359,
+      "step": 180
+    },
+    {
+      "epoch": 47.5,
+      "grad_norm": 0.0,
+      "learning_rate": 3.1100000000000004e-05,
+      "loss": 12.216262054443359,
+      "step": 190
+    },
+    {
+      "epoch": 50.0,
+      "grad_norm": 0.0,
+      "learning_rate": 3.01e-05,
+      "loss": 12.216262054443359,
+      "step": 200
+    },
+    {
+      "epoch": 52.5,
+      "grad_norm": 0.0,
+      "learning_rate": 2.91e-05,
+      "loss": 12.216262054443359,
+      "step": 210
+    },
+    {
+      "epoch": 55.0,
+      "grad_norm": 0.0,
+      "learning_rate": 2.8100000000000005e-05,
+      "loss": 12.216262054443359,
+      "step": 220
+    },
+    {
+      "epoch": 57.5,
+      "grad_norm": 0.0,
+      "learning_rate": 2.7100000000000005e-05,
+      "loss": 12.216262054443359,
+      "step": 230
+    },
+    {
+      "epoch": 60.0,
+      "grad_norm": 0.0,
+      "learning_rate": 2.61e-05,
+      "loss": 12.216262054443359,
+      "step": 240
+    },
+    {
+      "epoch": 62.5,
+      "grad_norm": 0.0,
+      "learning_rate": 2.51e-05,
+      "loss": 12.216262054443359,
+      "step": 250
+    },
+    {
+      "epoch": 65.0,
+      "grad_norm": 0.0,
+      "learning_rate": 2.41e-05,
+      "loss": 12.216262054443359,
+      "step": 260
+    },
+    {
+      "epoch": 67.5,
+      "grad_norm": 0.0,
+      "learning_rate": 2.3100000000000002e-05,
+      "loss": 12.216262054443359,
+      "step": 270
+    },
+    {
+      "epoch": 70.0,
+      "grad_norm": 0.0,
+      "learning_rate": 2.2100000000000002e-05,
+      "loss": 12.216262054443359,
+      "step": 280
+    },
+    {
+      "epoch": 72.5,
+      "grad_norm": 0.0,
+      "learning_rate": 2.11e-05,
+      "loss": 12.216262054443359,
+      "step": 290
+    },
+    {
+      "epoch": 75.0,
+      "grad_norm": 0.0,
+      "learning_rate": 2.01e-05,
+      "loss": 12.216262054443359,
+      "step": 300
+    },
+    {
+      "epoch": 77.5,
+      "grad_norm": 0.0,
+      "learning_rate": 1.91e-05,
+      "loss": 12.216262054443359,
+      "step": 310
+    },
+    {
+      "epoch": 80.0,
+      "grad_norm": 0.0,
+      "learning_rate": 1.81e-05,
+      "loss": 12.216262054443359,
+      "step": 320
+    },
+    {
+      "epoch": 82.5,
+      "grad_norm": 0.0,
+      "learning_rate": 1.7100000000000002e-05,
+      "loss": 12.216262054443359,
+      "step": 330
+    },
+    {
+      "epoch": 85.0,
+      "grad_norm": 0.0,
+      "learning_rate": 1.6100000000000002e-05,
+      "loss": 12.216262054443359,
+      "step": 340
+    },
+    {
+      "epoch": 87.5,
+      "grad_norm": 0.0,
+      "learning_rate": 1.51e-05,
+      "loss": 12.216262054443359,
+      "step": 350
+    },
+    {
+      "epoch": 90.0,
+      "grad_norm": 0.0,
+      "learning_rate": 1.4099999999999999e-05,
+      "loss": 12.216262054443359,
+      "step": 360
+    },
+    {
+      "epoch": 92.5,
+      "grad_norm": 0.0,
+      "learning_rate": 1.3100000000000002e-05,
+      "loss": 12.216262054443359,
+      "step": 370
+    },
+    {
+      "epoch": 95.0,
+      "grad_norm": 0.0,
+      "learning_rate": 1.2100000000000001e-05,
+      "loss": 12.216262054443359,
+      "step": 380
+    },
+    {
+      "epoch": 97.5,
+      "grad_norm": 0.0,
+      "learning_rate": 1.11e-05,
+      "loss": 12.216262054443359,
+      "step": 390
+    },
+    {
+      "epoch": 100.0,
+      "grad_norm": 0.0,
+      "learning_rate": 1.0100000000000002e-05,
+      "loss": 12.216262054443359,
+      "step": 400
+    },
+    {
+      "epoch": 102.5,
+      "grad_norm": 0.0,
+      "learning_rate": 9.100000000000001e-06,
+      "loss": 12.216262054443359,
+      "step": 410
+    },
+    {
+      "epoch": 105.0,
+      "grad_norm": 0.0,
+      "learning_rate": 8.1e-06,
+      "loss": 12.216262054443359,
+      "step": 420
+    },
+    {
+      "epoch": 107.5,
+      "grad_norm": 0.0,
+      "learning_rate": 7.1e-06,
+      "loss": 12.216262054443359,
+      "step": 430
+    },
+    {
+      "epoch": 110.0,
+      "grad_norm": 0.0,
+      "learning_rate": 6.1e-06,
+      "loss": 12.216262054443359,
+      "step": 440
+    },
+    {
+      "epoch": 112.5,
+      "grad_norm": 0.0,
+      "learning_rate": 5.1e-06,
+      "loss": 12.216262054443359,
+      "step": 450
+    },
+    {
+      "epoch": 115.0,
+      "grad_norm": 0.0,
+      "learning_rate": 4.1000000000000006e-06,
+      "loss": 12.216262054443359,
+      "step": 460
+    },
+    {
+      "epoch": 117.5,
+      "grad_norm": 0.0,
+      "learning_rate": 3.1e-06,
+      "loss": 12.216262054443359,
+      "step": 470
+    },
+    {
+      "epoch": 120.0,
+      "grad_norm": 0.0,
+      "learning_rate": 2.1000000000000002e-06,
+      "loss": 12.216262054443359,
+      "step": 480
+    },
+    {
+      "epoch": 122.5,
+      "grad_norm": 0.0,
+      "learning_rate": 1.1e-06,
+      "loss": 12.216262054443359,
+      "step": 490
+    },
+    {
+      "epoch": 125.0,
+      "grad_norm": 0.0,
+      "learning_rate": 1.0000000000000001e-07,
+      "loss": 12.216262054443359,
+      "step": 500
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 500,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 125,
+  "save_steps": 500,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 2013482975232000.0,
+  "train_batch_size": 1,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-500/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5bc989356c0d6b41f61495cf20b7417c186211fe618bb03360462df9b72cb0fb
+size 5201

config.json

@@ -0,0 +1,120 @@
+{
+  "architectures": [
+    "Llama4ForConditionalGeneration"
+  ],
+  "boi_token_index": 200080,
+  "dtype": "float32",
+  "eoi_token_index": 200081,
+  "image_token_index": 200092,
+  "model_type": "llama4",
+  "text_config": {
+    "_attn_implementation_autoset": true,
+    "attention_bias": false,
+    "attention_chunk_size": 8192,
+    "attention_dropout": 0.0,
+    "attn_scale": 0.1,
+    "attn_temperature_tuning": true,
+    "bos_token_id": 200000,
+    "dtype": "float32",
+    "eos_token_id": [
+      200001,
+      200007,
+      200008
+    ],
+    "floor_scale": 8192,
+    "for_llm_compressor": false,
+    "head_dim": 128,
+    "hidden_act": "silu",
+    "hidden_size": 2048,
+    "initializer_range": 0.02,
+    "interleave_moe_layer_step": 1,
+    "intermediate_size": 3072,
+    "intermediate_size_mlp": 6144,
+    "layer_types": [
+      "chunked_attention",
+      "chunked_attention",
+      "chunked_attention",
+      "full_attention",
+      "chunked_attention",
+      "chunked_attention",
+      "chunked_attention",
+      "full_attention"
+    ],
+    "max_position_embeddings": 10485760,
+    "model_type": "llama4_text",
+    "moe_layers": [
+      0,
+      1,
+      2,
+      3,
+      4,
+      5,
+      6,
+      7
+    ],
+    "no_rope_layer_interval": 4,
+    "no_rope_layers": [
+      1,
+      1,
+      1,
+      0,
+      1,
+      1,
+      1,
+      0
+    ],
+    "num_attention_heads": 16,
+    "num_experts_per_tok": 1,
+    "num_hidden_layers": 8,
+    "num_key_value_heads": 4,
+    "num_local_experts": 4,
+    "output_router_logits": false,
+    "pad_token_id": 200018,
+    "rms_norm_eps": 1e-05,
+    "rope_parameters": {
+      "factor": 16.0,
+      "high_freq_factor": 1.0,
+      "low_freq_factor": 1.0,
+      "original_max_position_embeddings": 8192,
+      "rope_theta": 500000.0,
+      "rope_type": "llama3"
+    },
+    "router_aux_loss_coef": 0.001,
+    "router_jitter_noise": 0.0,
+    "tie_word_embeddings": false,
+    "use_cache": true,
+    "use_qk_norm": true,
+    "vocab_size": 202048
+  },
+  "tie_word_embeddings": false,
+  "transformers_version": "5.10.1",
+  "use_cache": false,
+  "vision_config": {
+    "_attn_implementation_autoset": true,
+    "attention_dropout": 0.0,
+    "dtype": "float32",
+    "hidden_act": "gelu",
+    "hidden_size": 768,
+    "image_size": 336,
+    "initializer_range": 0.02,
+    "intermediate_size": 3072,
+    "model_type": "llama4_vision_model",
+    "multi_modal_projector_bias": false,
+    "norm_eps": 1e-05,
+    "num_attention_heads": 12,
+    "num_channels": 3,
+    "num_hidden_layers": 6,
+    "patch_size": 14,
+    "pixel_shuffle_ratio": 0.5,
+    "projector_dropout": 0.0,
+    "projector_input_dim": 2048,
+    "projector_output_dim": 2048,
+    "rope_parameters": {
+      "rope_theta": 10000,
+      "rope_type": "default"
+    },
+    "vision_feature_layer": -1,
+    "vision_feature_select_strategy": "default",
+    "vision_output_dim": 2048
+  }
+}

generation_config.json

@@ -0,0 +1,14 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 200000,
+  "eos_token_id": [
+    200001,
+    200007,
+    200008
+  ],
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "pad_token_id": 200018,
+  "transformers_version": "5.10.1",
+  "use_cache": true
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:739252e404227a38b8f48330f3fa86169e7c162382cd134cb2743dda6b333ec3
+size 6898648848

processor_config.json

@@ -0,0 +1,33 @@
+{
+  "fake_image_token": "<|image|>",
+  "image_processor": {
+    "data_format": "channels_first",
+    "default_to_square": true,
+    "do_convert_rgb": true,
+    "do_normalize": true,
+    "do_rescale": true,
+    "do_resize": true,
+    "image_mean": [
+      0.5,
+      0.5,
+      0.5
+    ],
+    "image_processor_type": "Llama4ImageProcessor",
+    "image_std": [
+      0.5,
+      0.5,
+      0.5
+    ],
+    "max_patches": 16,
+    "resample": 2,
+    "rescale_factor": 0.00392156862745098,
+    "resize_to_max_canvas": false,
+    "size": {
+      "height": 336,
+      "width": 336
+    }
+  },
+  "image_token": "<|image|>",
+  "patch_size": 14,
+  "processor_class": "Llama4Processor"
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f32e154d2e27a25934998e1500e6e6504ef0572ca67d28be95c29d923fd5e6b0
+size 27948865

tokenizer_config.json

@@ -0,0 +1,16 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<|begin_of_text|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|eot|>",
+  "is_local": true,
+  "local_files_only": false,
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 10485760,
+  "pad_token": "<|finetune_right_pad|>",
+  "processor_class": "Llama4Processor",
+  "tokenizer_class": "TokenizersBackend"
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5bc989356c0d6b41f61495cf20b7417c186211fe618bb03360462df9b72cb0fb
+size 5201