---
license: apache-2.0
language:
- en
tags:
- protein
- biology
- bioinformatics
- sequence-to-text
- protein-function
datasets:
- UniProt
base_model:
- mistralai/Mistral-7B-Instruct-v0.1
- Rostlab/prot_bert_bfd
pipeline_tag: text-generation
---

# GeneLinguaLM v5

**GeneLinguaLM** is a multimodal model that generates natural language descriptions of protein functions from amino acid sequences.

## Model Description

GeneLinguaLM bridges protein sequences and natural language through cross-modal learning:

- **Protein Encoder**: ProtBERT (Rostlab/prot_bert_bfd) - encodes amino acid sequences
- **Q-Former**: Compresses sequence features into 32 query tokens
- **LLM**: Mistral-7B-Instruct with LoRA fine-tuning
- **Projector**: Maps Q-Former outputs to LLM embedding space

## Performance

| Model | ROUGE-1 | ROUGE-L | BLEU |
|-------|---------|---------|------|
| **GeneLinguaLM v5** | **0.2295** | **0.1561** | **0.0315** |
| Zero-shot Mistral-7B | 0.1709 | 0.1160 | 0.0075 |
| BioGPT | 0.1513 | 0.1092 | 0.0063 |
| Mol-Instructions | 0.0025 | 0.0025 | 0.0007 |

GeneLinguaLM outperforms:
- **Zero-shot Mistral-7B** by **35%** on ROUGE-L (demonstrates cross-modal alignment value)
- **BioGPT** by **43%** on ROUGE-L for protein function description

## Usage

```python
from genelinguaLM import GeneLinguaLM

# Load model
model = GeneLinguaLM()

# Describe a protein sequence
sequence = "MALWMRLLPLLALLALWGPDPAAAFVNQHLCGSHLVEALYLVCGERGFFYTPKTRREAEDLQVGQVELGGGPGAGSLQPLALEGSLQKRGIVEQCCTSICSLYQLENYCN"
description = model.describe(sequence)
print(description)
# Output: "Hormone that regulates glucose metabolism and blood sugar levels..."
```

## Example Output

**Input**: Human Insulin sequence
```
MALWMRLLPLLALLALWGPDPAAAFVNQHLCGSHLVEALYLVCGERGFFYTPKT...
```

**Output**:
```
Hormone that regulates carbohydrate and lipid metabolism.
Plays a key role in the regulation of glucose levels in the blood...
```

## Training Data

- **Source**: UniProt (Swiss-Prot reviewed entries)
- **Organisms**: Human (46%), Mouse, Drosophila, Yeast, E.coli, etc. (54%)
- **Size**: 52,480 protein-description pairs
- **Training**: 5 epochs, ~16,000 steps

## Architecture

```
Protein Sequence
      ↓
[ProtBERT] → Sequence Embeddings (1024-dim)
      ↓
[Q-Former] → 32 Query Tokens (768-dim)
      ↓
[Projector] → LLM Embeddings (4096-dim)
      ↓
[Mistral-7B + LoRA] → Natural Language Description
```

## Model Files

- `checkpoint_step15732.pt`: Main checkpoint (Q-Former + Projector weights)
- `lora_step15732/`: LoRA adapter for Mistral-7B
- `qformer_checkpoint.pt`: Q-Former pretrained weights

## Requirements

```
torch>=2.0
transformers>=4.35
peft>=0.6
```

## Citation

```bibtex
@misc{genelinguaLM2024,
  title={GeneLinguaLM: Bridging Protein Sequences and Natural Language},
  author={GeneLinguaLM Team},
  year={2024},
  url={https://github.com/powersimmani/geneLLM}
}
```

## Acknowledgments

- [UniProt](https://www.uniprot.org/) for protein data
- [ProtBERT](https://huggingface.co/Rostlab/prot_bert_bfd) for sequence encoding
- [Mistral AI](https://mistral.ai/) for the base LLM
- [BLIP-2](https://arxiv.org/abs/2301.12597) for Q-Former architecture inspiration

## License

Apache 2.0