Update README.md

README.md

@@ -9,31 +9,24 @@ language:
 # Model Card for Model ID
 
 <!-- Provide a quick summary of what the model is/does. -->
+An open-source LLaMa language model of 13b parameters fine-tuned to follow instructions in italian.
 
-This modelcard aims to be a base template for new models. It has been generated using [this raw template](https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/templates/modelcard_template.md?plain=1).
-
-## Model Details
 
 ### Model Description
 
-<!-- Provide a longer summary of what this model is. -->
+This model is an open-source LLM of 13b parameters based on [OpenLLaMA](https://github.com/openlm-research/open_llama), an open-source replica of Meta AI's LLaMA.
+The model was fine-tuned in order to follow instructions, as proposed in [Alpaca](https://github.com/tatsu-lab/stanford_alpaca), 
+but using [LoRA](https://arxiv.org/pdf/2106.09685.pdf) technique and a bigger dataset of instruction/answers in italian, [cosimoiaia/Loquace-102k](cosimoiaia/Loquace-102k).
 
+This repository contains the model merged with the LoRA adapters obtained in the fine-tuning procedure.
 
 
-- **Developed by:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
+- **Developed by:** Stefano Scotta (stefano.scotta@rai.it)
+- **Model type:** LLM fine-tuned to follow instructions
+- **Language(s) (NLP):** Italian
 - **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-
-### Model Sources [optional]
+- **Finetuned from model:** [openlm-research/open_llama_13b](openlm-research/open_llama_13b)
 
-<!-- Provide the basic links for the model. -->
-
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
 
 ## Uses
 
@@ -80,63 +73,40 @@ Use the code below to get started with the model.
 ### Training Data
 
 <!-- This should link to a Data Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+The model was fine-tinuned on [cosimoiaia/Loquace-102k](cosimoiaia/Loquace-102k), a dataset of 102k question/answer pairs in italian.
 
-[More Information Needed]
 
 ### Training Procedure 
 
 <!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-
-#### Preprocessing [optional]
-
-[More Information Needed]
+The fine-tuning procedure was done using [LoRA](https://arxiv.org/pdf/2106.09685.pdf) approach following closely what done for fine-tuning models like [Alpaca-LoRA](https://github.com/tloen/alpaca-lora).
 
 
 #### Training Hyperparameters
 
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+ **Training setting:**
+- train epochs=3,
 
-#### Speeds, Sizes, Times [optional]
+- learning_rate=3e-4,
 
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+- optimizer="adamw_hf"
 
-[More Information Needed]
+- mixed precision training: float16 
 
-## Evaluation
+ **LoRA configuration:**
+-  r= 8
+  
+- lora_alpha=16
+  
+- target_modules=["q_proj","v_proj"]
 
-<!-- This section describes the evaluation protocols and provides the results. -->
+- lora_dropout=0.05
+  
+- bias="none"
+  
+- task_type=TaskType.CAUSAL_LM
+  
 
-### Testing Data, Factors & Metrics
-
-#### Testing Data
-
-<!-- This should link to a Data Card if possible. -->
-
-[More Information Needed]
-
-#### Factors
-
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-
-[More Information Needed]
-
-#### Metrics
-
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-
-[More Information Needed]
-
-### Results
-
-[More Information Needed]
-
-#### Summary
-
-
-
-## Model Examination [optional]
-
-<!-- Relevant interpretability work for the model goes here -->
 
 [More Information Needed]
 
@@ -146,11 +116,10 @@ Use the code below to get started with the model.
 
 Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
 
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
+- **Hardware Type:** 1 NVIDIA A100/40Gb
+- **Hours used:** 68
+- **Cloud Provider:** Private Infrastructure
+- **Carbon Emitted:** 7.34 kg eq. CO2
 
 ## Technical Specifications [optional]
 
@@ -194,8 +163,8 @@ Carbon emissions can be estimated using the [Machine Learning Impact calculator]
 
 ## Model Card Authors [optional]
 
-[More Information Needed]
+Stefano Scotta (stefano.scotta@rai.it)
 
 ## Model Card Contact
 
-[More Information Needed]
+stefano.scotta@rai.it