README.md

---
library_name: transformers
license: apache-2.0
datasets:
- StanfordAIMI/interpret-cxr-test-public
- StanfordAIMI/interpret-cxr-test-hidden
---

# CXRMate-RRG4: Entropy-Augmented Self-Critical Sequence Training for Radiology Report Generation

This is an evolution of https://huggingface.co/aehrc/cxrmate developed for the Radiology Report Generation task of BioNLP @ ACL 2024.

For this, we proposed EAST: Entropy-Augmented Self-critical sequence Training (EAST):
 - EAST modifies Self-Critical Sequence Training (SCST) by adding entropy regularisation. 
 - Helps maintain a higher entropy in the token distribution.
 - Preventing overfitting to common phrases and ensuring a broader exploration of the vocabulary during training.
 - This was essential to handle the diversity of the radiology reports in the RRG24 datasets. 

EAST was applied to a multimodal language model with RadGraph as the reward. Other features include:
 - Token type embeddings to differentiate between findings and impression section tokens, as well as image embeddings. 
 - Special tokens (`NF` and `NI`) to handle missing *findings* and *impression* sections. 
 - Non-causal attention masking for the image embeddings and a causal attention masking for the report token embeddings.

## How to use:

```python
import torch  
from torchvision.transforms import v2
import transformers


tokenizer = transformers.AutoTokenizer.from_pretrained('aehrc/cxrmate-rrg24')
model = transformers.AutoModel.from_pretrained('aehrc/cxrmate-rrg24', trust_remote_code=True)
transforms = v2.Compose(
    [
        v2.PILToTensor(),
        v2.Grayscale(num_output_channels=3),
        v2.Resize(size=model.config.encoder.image_size, antialias=True),
        v2.CenterCrop(size=[model.config.encoder.image_size]*2),
        v2.ToDtype(torch.float32, scale=True),
        v2.Normalize(mean=model.config.encoder.image_mean, std=model.config.encoder.image_std),
    ]
)

image = transforms(image)  # Fix.

output_ids = model.generate(
    pixel_values=images,  # Fix.
    max_length=512,
    bad_words_ids=[[tokenizer.convert_tokens_to_ids('[NF]')], [tokenizer.convert_tokens_to_ids('[NI]')]],
    num_beams=4,
    use_cache=True,
)
findings, impression = model.split_and_decode_sections(output_ids, tokenizer)
```

## Notebook example:
https://huggingface.co/aehrc/cxrmate-rrg24/blob/main/demo.ipynb

## Paper:

## Citation:

[More Information Needed]