Upload folder using huggingface_hub

README.md

@@ -27,8 +27,8 @@ This model represents a **systematic exploration** of the combination of reasoni
 ## 🔬 Model Lineage & Methodology
 
 ### Parent Models
-- **Primary**: [DeepSeek-R1-Distill-Qwen-1.5B](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B) - A model trained via large-scale reinforcement learning, demonstrating advanced reasoning capabilities across various benchmarks.
-- **Secondary**: [Gensyn/Qwen2.5-1.5B-Instruct](https://huggingface.co/Gensyn/Qwen2.5-1.5B-Instruct) - A model designed for instruction-following tasks, showcasing proficiency in understanding and executing user prompts.
+- **Primary**: [DeepSeek-R1-Distill-Qwen-1.5B](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B) - A distilled model that demonstrates advanced reasoning capabilities through reinforcement learning, achieving competitive performance across various benchmarks.
+- **Secondary**: [Gensyn/Qwen2.5-1.5B-Instruct](https://huggingface.co/Gensyn/Qwen2.5-1.5B-Instruct) - A model designed for instruction-following tasks, providing a strong foundation for generating coherent and contextually relevant responses.
 
 ### Merge Configuration
 ```yaml
@@ -46,40 +46,40 @@ dtype: bfloat16
 ```
 
 ### Research Rationale
-The combination of DeepSeek-R1's reasoning capabilities with Qwen2.5's instruction-following abilities was hypothesized to create a model that excels in tasks requiring both logical reasoning and adherence to user instructions, thereby addressing the limitations observed in each individual model.
+The combination of DeepSeek-R1's reasoning capabilities with Qwen2.5's instruction-following abilities was hypothesized to create a model that excels in tasks requiring both logical reasoning and adherence to user prompts, thereby addressing limitations observed in individual models.
 
 ## 🎯 Intended Use & Research Applications
 
 ### Primary Research Use Cases
 - Complex problem-solving in mathematics and reasoning tasks
 - Instruction-following applications in educational tools
-- Benchmarking performance in natural language understanding and generation
+- Benchmarking against state-of-the-art models in reasoning and language understanding
 
 ### Production Considerations
-While this model shows promise in enhancing reasoning and instruction-following tasks, it is essential to consider the potential for overfitting to specific types of queries and the need for careful prompt engineering to achieve optimal results.
+While this model shows promise in enhancing reasoning and instruction-following tasks, it is essential to consider the potential for overfitting to specific types of queries and the need for careful prompt engineering to achieve optimal performance.
 
 ## 📊 Evaluation & Validation
 
 ### Research Metrics
-The model's performance was evaluated using a range of benchmarks, including MMLU, DROP, and LiveCodeBench, with results indicating improved performance in reasoning tasks compared to baseline models.
+Evaluation was conducted using a variety of benchmarks, including MMLU, DROP, and LiveCodeBench, with results indicating improved performance in reasoning tasks compared to baseline models.
 
 ### Known Capabilities
 - Enhanced reasoning patterns through the integration of reinforcement learning techniques
-- Improved instruction adherence in complex queries
+- Improved instruction adherence and contextual understanding in generated responses
 
 ### Performance Characteristics
-Quantitative results indicate that the model outperforms individual parent models in several benchmarks, particularly in tasks requiring both reasoning and instruction adherence.
+Quantitative results indicate that the model outperforms individual parent models in several key benchmarks, demonstrating the effectiveness of the merging strategy.
 
 ## ⚠️ Limitations & Research Boundaries
 
 ### Technical Limitations
-The model may exhibit limitations in generating coherent outputs for highly complex queries, particularly those that require extensive context or multi-step reasoning.
+The model may exhibit limitations in handling highly specialized queries or tasks that require domain-specific knowledge beyond the training data.
 
 ### Research Scope
-This research does not explore the full range of capabilities of either parent model but focuses specifically on the intersection of reasoning and instruction-following.
+This research focuses on the integration of reasoning and instruction-following capabilities and does not explore other potential model combinations or architectures.
 
 ### Ethical Considerations
-Potential biases inherent in the training data of the parent models may affect the outputs of the merged model. Responsible use guidelines should be followed to mitigate risks associated with biased outputs.
+Bias risks inherent in the training data of the parent models may persist in the merged model. Responsible use guidelines should be followed to mitigate potential misuse.
 
 ## 🔬 Research Framework
 

config.json

@@ -1,5 +1,4 @@
 {
-  "_name_or_path": "Gensyn/Qwen2.5-1.5B-Instruct",
   "architectures": [
     "Qwen2ForCausalLM"
   ],
@@ -10,6 +9,36 @@
   "hidden_size": 1536,
   "initializer_range": 0.02,
   "intermediate_size": 8960,
+  "layer_types": [
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention"
+  ],
   "max_position_embeddings": 32768,
   "max_window_layers": 21,
   "model_type": "qwen2",
@@ -17,11 +46,12 @@
   "num_hidden_layers": 28,
   "num_key_value_heads": 2,
   "rms_norm_eps": 1e-06,
+  "rope_scaling": null,
   "rope_theta": 1000000.0,
   "sliding_window": null,
   "tie_word_embeddings": true,
   "torch_dtype": "bfloat16",
-  "transformers_version": "4.44.0",
+  "transformers_version": "4.53.0",
   "use_cache": true,
   "use_sliding_window": false,
   "vocab_size": 151936

model-00001-of-00004.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8fbeffabb36c64e581959ebc641c0d09961cf38c8319ff1069fdd14b471cd569
+oid sha256:f65df8ba299179ed078158f700fd523d8dd179cb68de306a0da52ebb33496ff6
 size 988548664

model-00002-of-00004.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f64d563aa0f608e1312ea5efdd6a03e0d4b0587d3aa0bc553d8ca2be3f3e2af4
+oid sha256:b57bf8aba2b675dc1391a187b383343e8fbd7c44e50203d0c2215c3f5a3fdde3
 size 974516536

model-00003-of-00004.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:34a16b908c0a31d6f997ba5534af6c7be0dd4db3aaa9007fc36a1692cec0f184
+oid sha256:b2c1830719b56e9e836c9165b5175aff5e76225b17db98e6634a03623eb202b3
 size 991020472

model-00004-of-00004.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b8267a0db5ca06da83f5dc5c1a4da2543766734e33b29f47b6b620ee0722a7df
+oid sha256:d9ab763cc47a0fbf8c72e2f7ab5a7abb76c802df19131f44449612a538669210
 size 600128448