README.md

---
license: llama3.2
language:
- en
base_model:
- meta-llama/Llama-3.2-3B-Instruct
pipeline_tag: text-generation
tags:
- Speech Recognition
- ATC
- PEFT
- LoRA
- Unsloth
---

# ATC Communication Expert Model (LoRA Adapters)

A specialized set of LoRA adapters fine-tuned for improving and analyzing Air Traffic Control (ATC) communications, extracting relevant information from raw transcripts.

## Model Details

### Model Description

These adapters fine-tune the meta-llama/Llama-3.2-3B-Instruct model to specialize in processing Air Traffic Control communications. When applied to the base model, it can:

- Improve raw ATC transcripts with proper punctuation and formatting
- Identify communication intentions (pilot requests, ATC instructions, etc.)
- Extract key information such as flight numbers, altitudes, headings, and other numerical data
- Analyze speaker roles and communication patterns

The adapters were created using LoRA (Low-Rank Adaptation) with PEFT (Parameter-Efficient Fine-Tuning) techniques to efficiently adapt the Llama 3B model to this specialized domain.

- **Developed by:** [Sang-Buster](https://github.com/Sang-Buster)
- **Model type:** LoRA adapters for meta-llama/Llama-3.2-3B-Instruct
- **Language(s):** English, specialized for ATC terminology
- **License:** Same as the base model
- **Finetuned from model:** meta-llama/Llama-3.2-3B-Instruct

## Uses

### Direct Use

These adapters are intended for:
- Transcribing and formatting raw ATC communications
- Training ATC communication skills
- Analyzing ATC communication patterns
- Extracting structured data from ATC communications
- Educational purposes for those learning ATC communication protocols

### Downstream Use

The model can be integrated into:
- Air traffic management training systems
- Communication analysis tools
- ATC transcript post-processing pipelines
- Aviation safety monitoring systems
- Radio communication enhancement systems

### Out-of-Scope Use

This model is not suitable for:
- Real-time ATC operations or safety-critical decision-making
- Full language translation (it's specialized for ATC terminology only)
- General language processing outside the ATC domain
- Any application where model errors could impact flight safety

## Bias, Risks, and Limitations

- The model is specialized for ATC communications and may not perform well on general text
- It may have limitations with accents or non-standard ATC phraseology
- Performance depends on audio transcription quality for real-world applications
- Not intended for safety-critical applications without human verification
- May have biases based on the training data distribution

### Recommendations

- Always have human verification for safety-critical applications
- Use in conjunction with standard ATC protocols, not as a replacement
- Provide clear domain context for optimal performance
- Test thoroughly with diverse ATC communications before deployment
- Consider fine-tuning further on your specific ATC subdomain if needed

## How to Get Started with the Model

```python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load the base model
base_model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.2-3B-Instruct",
    torch_dtype="auto",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-3B-Instruct")

# Load the adapter
model = PeftModel.from_pretrained(base_model, "atc_llama")

# Process an ATC message
instruction = "As an ATC communication expert, improve this transcript and analyze its intentions and data."
message = "southwest five niner two turn left heading three four zero descend and maintain flight level two five zero"

prompt = f"<|begin_of_text|><|header_start|>user<|header_end|>\n\n{instruction}\n\nOriginal: {message}<|eot|><|header_start|>assistant<|header_end|>\n\n"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

# Generate improved transcript and analysis
outputs = model.generate(**inputs, max_new_tokens=512, do_sample=False)
response = tokenizer.decode(outputs[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print(response)
```

## Training Details

### Training Data

The model was trained on a dataset of ATC communications with:
- Original raw transcripts
- Properly punctuated and formatted versions
- Annotated intentions (PSC, PSR, PRP, PRQ, PRB, PAC, ASC, AGI, ACR, END)
- Extracted numerical data (altitudes, headings, flight numbers, etc.)
- Speaker and listener information

### Training Procedure

The model was fine-tuned using LoRA with the following approach:
- Parameter-efficient fine-tuning using PEFT
- LoRA applied to key attention layers (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj)
- Optimized with Unsloth for efficiency

#### Training Hyperparameters

- **Base model:** meta-llama/Llama-3.2-3B-Instruct
- **LoRA rank:** 16
- **LoRA alpha:** 16
- **Learning rate:** 0.0002
- **Batch size:** 4
- **Gradient accumulation steps:** 4
- **Epochs:** 3
- **Warmup ratio:** 0.03
- **Max sequence length:** 2048
- **Training regime:** BF16 mixed precision where available, FP16 otherwise
- **Optimizer:** AdamW 8-bit

## Evaluation

### Testing

The adapters should be tested on diverse ATC communications, including:
- Clearances and instructions
- Pilot requests and reports
- Emergency communications
- Different accents and speaking patterns

## Technical Specifications

### Model Architecture and Objective

- **Base architecture:** meta-llama/Llama-3.2-3B-Instruct
- **Fine-tuning method:** LoRA with PEFT
- **Optimization library:** Unsloth
- **Training objective:** Improving and analyzing ATC communications

### Compute Infrastructure

- **Framework versions:**
  - PEFT (compatible with the base model)
  - Unsloth (for efficient LoRA fine-tuning)
  - Transformers (compatible with the base model)
  - PyTorch (with BF16 support where available)