Upload folder using huggingface_hub

.trillim_cache.json

@@ -0,0 +1,11 @@
+{
+  "config_hash": "1eff32ddbdc3af6fa70f226808356ac33682ea50fc6d61411e9634ec11895676",
+  "config_h_hash": "1b3947afb731149bdf242207ff1a9abaf719bbeb3fdd787ecb1f81cc23493c7c",
+  "generate_h_hash": "82c7216b1f207e7e384eec66eb1b6406e7dcd42121cbb87534a611e5cdcc90dc",
+  "safetensor_mtimes": {
+    "model.safetensors": 1768459059.0255861
+  },
+  "output_dir": "/home/darksca/Vineet/Code/Startups/1bitLLM/DarkNet/include/model",
+  "adapter_state": null,
+  "version": 1
+}

README.md

@@ -1,80 +1,100 @@
 ---
-license: llama3
-tags:
-  - llama3
-  - ternary
-  - trillim
-  - cpu-inference
-base_model: HF1BitLLM/Llama3-8B-1.58-100B-tokens
-extra_gated_prompt: >-
-  This model is derived from Meta Llama 3. By accessing this model you agree to
-  the Meta Llama 3 Community License Agreement.
+library_name: transformers
+base_model:
+- meta-llama/Meta-Llama-3-8B-Instruct
 ---
 
-# Llama3-TRNQ
+# Model Card for Model ID
 
-Built with Meta Llama 3.
+### Llama3-8B-1.58 Models
 
-Ternary-quantized version of [HF1BitLLM/Llama3-8B-1.58-100B-tokens](https://huggingface.co/HF1BitLLM/Llama3-8B-1.58-100B-tokens), packaged for the [Trillim DarkNet](https://huggingface.co/Trillim) inference engine. The source model is a BitNet 1.58b quantization of [Meta-Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B), trained on the FineWeb-edu dataset.
+The **Llama3-8B-1.58** models are large language models fine-tuned on the **BitNet 1.58b architecture**, starting from the base model **Llama-3-8B-Instruct**.
+
+For a deeper dive into the methods and results, check out our [blog post](https://huggingface.co/blog/1_58_llm_extreme_quantization).
 
-This model runs entirely on CPU — no GPU required.
 
 ## Model Details
 
-| | |
-|---|---|
-| **Architecture** | Llama (LlamaForCausalLM) |
-| **Parameters** | ~8B |
-| **Hidden size** | 4096 |
-| **Layers** | 32 |
-| **Attention heads** | 32 (8 KV heads) |
-| **Context length** | 8192 |
-| **Quantization** | Ternary ({-1, 0, 1}) |
-| **Source model** | [HF1BitLLM/Llama3-8B-1.58-100B-tokens](https://huggingface.co/HF1BitLLM/Llama3-8B-1.58-100B-tokens) |
-| **License** | [Meta Llama 3 Community License](https://llama.meta.com/llama3/license/) |
+### Model Sources
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [Model](https://huggingface.co/HF1BitLLM/Llama3-8B-1.58-100B-tokens)
+- **Paper:** [The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits](https://arxiv.org/abs/2402.17764)
+
+
+## How to Get Started with the Model
 
-## Usage
+You can easily load and test our model in Transformers. Just follow the code below:
 
+Start by installing the transformers version with the correct configuration to load bitnet models
 ```bash
-pip install trillim
-trillim pull Trillim/Llama3-TRNQ
-trillim serve Trillim/Llama3-TRNQ
+pip install git+https://github.com/huggingface/transformers.git@refs/pull/33410/head
 ```
+And then load the model : 
+```python
 
-This starts an OpenAI-compatible API server at `http://127.0.0.1:8000`.
+model = AutoModelForCausalLM.from_pretrained("HF1BitLLM/Llama3-8B-1.58-100B-tokens", device_map="cuda", torch_dtype=torch.bfloat16)    
+tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct")
 
-For interactive CLI chat:
+input_text = "Daniel went back to the the the garden. Mary travelled to the kitchen. Sandra journeyed to the kitchen. Sandra went to the hallway. John went to the bedroom. Mary went back to the garden. Where is Mary?\nAnswer:"
 
-```bash
-trillim chat Trillim/Llama3-TRNQ
+input_ids = tokenizer.encode(input_text, return_tensors="pt").cuda()
+output = model.generate(input_ids, max_length=10, do_sample=False)
+generated_text = tokenizer.decode(output[0], skip_special_tokens=True)
+print(generated_text)
 ```
 
-## What's in this repo
+## Training Details
+
+### Training Data
+
+The model was trained on a subset of [FineWeb-edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu)
+
+### Training Process
 
-| File | Description |
-|---|---|
-| `qmodel.tensors` | Ternary-quantized weights in Trillim format |
-| `rope.cache` | Precomputed RoPE embeddings |
-| `config.json` | Model configuration |
-| `tokenizer.json` | Tokenizer |
-| `tokenizer_config.json` | Tokenizer configuration |
-| `trillim_config.json` | Trillim metadata |
+1. **Starting Point**
+   - Best-performing checkpoint from the 10 billion token runs with a linear lambda scheduler
 
-## License
+2. **Training Duration**
+   - Fine-tuned for an additional 45,000 steps
+   - Reached a total of 100 billion tokens
 
-This model is derived from Meta Llama 3 and is subject to the [Meta Llama 3 Community License Agreement](https://llama.meta.com/llama3/license/).
+3. **Dataset**
+   - FineWeb-edu dataset
 
-### Meta Llama 3 Community License Agreement
+4. **Batch Size**
+   - 2 million tokens per step
+   - Total per run: 45,000 steps * 2 million tokens = 90 billion tokens
+   - Combined with initial 10 billion tokens to reach 100 billion
 
-Meta Llama 3 is licensed under the Meta Llama 3 Community License, Copyright (c) Meta Platforms, Inc. All Rights Reserved.
+5. **Learning Rate Experiments**
+   - Tested various learning rates to find optimal setting, according the to experiments, the best performing peak lr is 1e-5
 
-**Acceptable Use Policy:** Your use of Llama 3 must comply with applicable laws and regulations and you must comply with Meta's [Acceptable Use Policy](https://llama.meta.com/llama3/use-policy/).
+6. **Performance**
+   - Close to Llama3 8B on some metrics
+   - Behind Llama3 8B in overall average performance
 
-**License terms:** The full license text is available at [https://llama.meta.com/llama3/license/](https://llama.meta.com/llama3/license/). Key terms include:
+7. **Evaluation**
+   - Metrics included perplexity, MMLU scores, and other standard benchmarks
 
-- You may use, reproduce, distribute, and create derivative works of Llama 3.
-- If you distribute or make Llama 3 available to a third party, you must provide a copy of this Agreement and include "Built with Meta Llama 3" in related materials.
-- You must not use Llama 3 to improve any other large language model (excluding Llama 3 derivatives).
-- If the monthly active users of your products or services exceed 700 million, you must request a license from Meta.
+These extended training runs on 100 billion tokens pushed the boundaries of highly quantized models, bringing performance closer to half-precision models like Llama3.
 
-For the complete license terms, please refer to the [official license](https://llama.meta.com/llama3/license/).
+
+## Evaluation
+
+The evaluation of the models is done on the nanotron checkpoints using LightEval : 
+
+![results](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/1.58llm_extreme_quantization/metrics_100B_table.png)
+
+
+
+## Citation
+
+```bash
+@misc{,
+      title={1.58-Bit LLM: A New Era of Extreme Quantization}, 
+      author={Mohamed Mekkouri and Marc Sun and Leandro von Werra and Thomas Wolf},
+      year={2024},
+}
+```

qmodel.tensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d1556b45b32e5c6f336ba4e90aefbf290fc2096fdae3d0155e57574549881502
-size 3321887504
+oid sha256:ace580c8ded34d25fc8ed35ea8fe38be66dd17eabc130a18373d085fbacfc3a2
+size 3846711056

trillim_config.json

@@ -1,8 +1,13 @@
 {
-    "trillim_version": "0.1.0",
-    "format_version": 1,
-    "quantization": "ternary",
-    "source_model": "HF1BitLLM/Llama3-8B-1.58-100B-tokens",
-    "architecture": "llama",
-    "platforms": ["x86_64", "aarch64"]
+  "trillim_version": "0.2.0",
+  "format_version": 2,
+  "type": "model",
+  "quantization": "ternary",
+  "source_model": "/fsx/mohamed/scripts_training/quantized_model_llama_8B_10",
+  "architecture": "llama",
+  "platforms": [
+    "x86_64",
+    "aarch64"
+  ],
+  "base_model_config_hash": "2a4a26b03d12d1fc60386f184537621d0d2a1fd455815999418389c66a906b91"
 }