File size: 5,938 Bytes
64aae6f bba3d15 64aae6f | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 | ---
license: llama3.2
language:
- en
base_model:
- meta-llama/Llama-3.2-3B-Instruct
pipeline_tag: text-generation
tags:
- Speech Recognition
- ATC
- PEFT
- LoRA
- Unsloth
---
# ATC Communication Expert Model (LoRA Adapters)
A specialized set of LoRA adapters fine-tuned for improving and analyzing Air Traffic Control (ATC) communications, extracting relevant information from raw transcripts.
## Model Details
### Model Description
These adapters fine-tune the meta-llama/Llama-3.2-3B-Instruct model to specialize in processing Air Traffic Control communications. When applied to the base model, it can:
- Improve raw ATC transcripts with proper punctuation and formatting
- Identify communication intentions (pilot requests, ATC instructions, etc.)
- Extract key information such as flight numbers, altitudes, headings, and other numerical data
- Analyze speaker roles and communication patterns
The adapters were created using LoRA (Low-Rank Adaptation) with PEFT (Parameter-Efficient Fine-Tuning) techniques to efficiently adapt the Llama 3B model to this specialized domain.
- **Developed by:** [Sang-Buster](https://github.com/Sang-Buster)
- **Model type:** LoRA adapters for meta-llama/Llama-3.2-3B-Instruct
- **Language(s):** English, specialized for ATC terminology
- **License:** Same as the base model
- **Finetuned from model:** meta-llama/Llama-3.2-3B-Instruct
## Uses
### Direct Use
These adapters are intended for:
- Transcribing and formatting raw ATC communications
- Training ATC communication skills
- Analyzing ATC communication patterns
- Extracting structured data from ATC communications
- Educational purposes for those learning ATC communication protocols
### Downstream Use
The model can be integrated into:
- Air traffic management training systems
- Communication analysis tools
- ATC transcript post-processing pipelines
- Aviation safety monitoring systems
- Radio communication enhancement systems
### Out-of-Scope Use
This model is not suitable for:
- Real-time ATC operations or safety-critical decision-making
- Full language translation (it's specialized for ATC terminology only)
- General language processing outside the ATC domain
- Any application where model errors could impact flight safety
## Bias, Risks, and Limitations
- The model is specialized for ATC communications and may not perform well on general text
- It may have limitations with accents or non-standard ATC phraseology
- Performance depends on audio transcription quality for real-world applications
- Not intended for safety-critical applications without human verification
- May have biases based on the training data distribution
### Recommendations
- Always have human verification for safety-critical applications
- Use in conjunction with standard ATC protocols, not as a replacement
- Provide clear domain context for optimal performance
- Test thoroughly with diverse ATC communications before deployment
- Consider fine-tuning further on your specific ATC subdomain if needed
## How to Get Started with the Model
```python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
# Load the base model
base_model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-3.2-3B-Instruct",
torch_dtype="auto",
device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-3B-Instruct")
# Load the adapter
model = PeftModel.from_pretrained(base_model, "atc_llama")
# Process an ATC message
instruction = "As an ATC communication expert, improve this transcript and analyze its intentions and data."
message = "southwest five niner two turn left heading three four zero descend and maintain flight level two five zero"
prompt = f"<|begin_of_text|><|header_start|>user<|header_end|>\n\n{instruction}\n\nOriginal: {message}<|eot|><|header_start|>assistant<|header_end|>\n\n"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
# Generate improved transcript and analysis
outputs = model.generate(**inputs, max_new_tokens=512, do_sample=False)
response = tokenizer.decode(outputs[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print(response)
```
## Training Details
### Training Data
The model was trained on a dataset of ATC communications with:
- Original raw transcripts
- Properly punctuated and formatted versions
- Annotated intentions (PSC, PSR, PRP, PRQ, PRB, PAC, ASC, AGI, ACR, END)
- Extracted numerical data (altitudes, headings, flight numbers, etc.)
- Speaker and listener information
### Training Procedure
The model was fine-tuned using LoRA with the following approach:
- Parameter-efficient fine-tuning using PEFT
- LoRA applied to key attention layers (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj)
- Optimized with Unsloth for efficiency
#### Training Hyperparameters
- **Base model:** meta-llama/Llama-3.2-3B-Instruct
- **LoRA rank:** 16
- **LoRA alpha:** 16
- **Learning rate:** 0.0002
- **Batch size:** 4
- **Gradient accumulation steps:** 4
- **Epochs:** 3
- **Warmup ratio:** 0.03
- **Max sequence length:** 2048
- **Training regime:** BF16 mixed precision where available, FP16 otherwise
- **Optimizer:** AdamW 8-bit
## Evaluation
### Testing
The adapters should be tested on diverse ATC communications, including:
- Clearances and instructions
- Pilot requests and reports
- Emergency communications
- Different accents and speaking patterns
## Technical Specifications
### Model Architecture and Objective
- **Base architecture:** meta-llama/Llama-3.2-3B-Instruct
- **Fine-tuning method:** LoRA with PEFT
- **Optimization library:** Unsloth
- **Training objective:** Improving and analyzing ATC communications
### Compute Infrastructure
- **Framework versions:**
- PEFT (compatible with the base model)
- Unsloth (for efficient LoRA fine-tuning)
- Transformers (compatible with the base model)
- PyTorch (with BF16 support where available)
|