---
license: mit
tags:
- unsloth
- trl
- sft
datasets:
- olafdil/French_MultiSpeaker_Diarization
language:
- fr
base_model:
- unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit
pipeline_tag: text-generation
---
# Fine-Tuned Model: Meta-Llama-3.1-8B-Instruct-bnb-4bit

This is a fine-tuned version of the [Meta-Llama-3.1-8B-Instruct-bnb-4bit](https://huggingface.co/unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit) model, adapted for French multi-speaker diarization tasks. Below, you'll find details about the fine-tuning process, dataset, and how to use this model.

---

## Model Details

- **Base Model**: Meta-Llama-3.1-8B-Instruct-bnb-4bit
- **Quantization**: 4-bit quantization for reduced memory usage
- **Purpose**: Fine-tuned for multi-speaker diarization in French.
- **Techniques**:
  - LoRA (Low-Rank Adaptation) for efficient fine-tuning.
  - Target modules: `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj`.
  - Rank: `16`
  - LoRA alpha: `16`
  - Gradient checkpointing: Enabled.

---

## Dataset

The model was fine-tuned on the `French_MultiSpeaker_Diarization` dataset, hosted on the Hugging Face Hub:

- **Dataset Name**: [French_MultiSpeaker_Diarization](https://huggingface.co/datasets/olafdil/French_MultiSpeaker_Diarization)
- **Split Used**: Train
- **Dataset Content**:
  - Multispeaker conversational data in French.
  - Includes labeled diarization information to improve diarization capabilities.

---

## Training Configuration

### Hyperparameters

- **Max Sequence Length**: `120,000`
- **LoRA Dropout**: `0`
- **Bias**: `none`
- **Use Gradient Checkpointing**: Enabled for efficiency.
- **Custom Prompting**: Chat templates applied for formatting prompts (e.g., `llama-3.1` template).

### Training Workflow

1. **Model Loading**:
   - Loaded the base model using `FastLanguageModel.from_pretrained()`.
   - Applied 4-bit quantization for memory efficiency.

2. **Dataset Preparation**:
   - The dataset was tokenized using a custom chat template from the `unsloth.chat_templates` library.
   - Prompts formatted with `apply_chat_template()` to suit the diarization task.

3. **Fine-Tuning**:
   - LoRA applied to specific layers for adaptation.
   - Gradient checkpointing enabled to reduce memory overhead during training.

---

## Usage

### Load the Model
You can load this model directly from Hugging Face:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "olafdil/FrDiarization-Llama-3.1-8B-4bit"
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="auto")
tokenizer = AutoTokenizer.from_pretrained(model_name)
```
### Inference Example
```python
template = """
I have an audio transcription where multiple speakers are involved in a conversation.
Your task is to distinguish the different speakers and diarize the text accordingly.
Each speaker's dialogue should be clearly labeled, such as 'Speaker 1:', 'Speaker 2:', etc.
Ensure that the labels remain consistent throughout the transcription and that the text is formatted neatly.
Here's the transcription:
"""
transciption = "Your input transcription here"
prompt = template + transcription

inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

## Dependencies

The following libraries were used:

- `transformers`
- `datasets`
- `unsloth`
- `torch`

To install the dependencies, you can use:

```bash
pip install transformers datasets torch unsloth
```

## Limitations

- The model has been fine-tuned specifically for French multi-speaker diarization tasks and may not generalize well to other tasks or languages.
- 4-bit quantization reduces memory usage but may slightly affect precision.

---

## Citation

If you use this model, please consider citing the base model and the dataset:

- **Base Model**: Meta-Llama-3.1-8B-Instruct-bnb-4bit
- **Dataset**: French MultiSpeaker Diarization