---
language:
- es
license: other
base_model: meta-llama/Meta-Llama-3.1-8B-Instruct
tags:
- text-classification
- document-classification
- peft
- lora
- llama
- sedici
- unlp
- universidad-nacional-de-la-plata
metrics:
- accuracy
- f1
---

# SEDICI Computer Science Document Type Classifier
## LoRA Adapter for Meta-Llama-3.1-8B-Instruct (r=128)

**Author:** Malek Camilo  
**Institution:** Facultad de Informática, Universidad Nacional de La Plata  
**Thesis:** *Estrategias de adaptación eficiente para modelos de lenguaje abiertos*  
**Repository:** [github.com/malekcamilo/tesis_maestria](https://github.com/malekcamilo/tesis_maestria)

This repository contains a LoRA adapter for document type classification in the Computer Science subset of SEDICI records.

## Base model

> [!WARNING]
> **Base Model License:** This adapter is built on top of `meta-llama/Meta-Llama-3.1-8B-Instruct`, which is a gated model. Users must explicitly agree to the Meta Llama 3.1 Community License on Hugging Face to use it.

The adapter was trained on top of `meta-llama/Meta-Llama-3.1-8B-Instruct`.

Users must have access to the base model in order to load this adapter.

## Task

Given a title and abstract, the model predicts one of the following document types:

- Article
- Conference Paper
- Thesis
- Book
- Learning Resource
- Other

The generated label is returned in Spanish, matching the labels used during training.

## Training data

The dataset contains 19,974 records from SEDICI in the Computer Science domain. Each instance includes title, abstract and document type.

## Tokenizer

This adapter uses the original tokenizer distributed with Meta-Llama-3.1-8B-Instruct. No additional tokens or vocabulary modifications were introduced during fine-tuning.

## Method

The model was adapted using LoRA (rank = 128) targeting all attention and MLP projection layers (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj).

## Hardware

Training was performed on an Intel® Data Center GPU Max 1550 using Intel Extension for PyTorch (IPEX) with bfloat16 precision.

## Training Configuration

| Parameter | Value |
|-----------|------:|
| Epochs | 5 |
| Learning Rate | 2e-4 |
| Batch Size | 8 |
| Gradient Accumulation | 1 |
| Max Sequence Length | 512 |
| Rank | 128 |
| Alpha | 256 |
| Dropout | 0.1 |

### Training Metrics
- **Train Time:** 6.1 hours
- **Peak Memory:** 25.68 GB
- **Trainable Parameters:** 335544320

## Evaluation

| Metric | Value |
|---|---:|
| Accuracy | 0.8532 |
| Macro-F1 | 0.6410 |
| Exact Match | 0.8529 |

### Per-Class Results

| Category | Precision | Recall | F1-Score |
|---|---:|---:|---:|
| Objeto De Conferencia | 0.8949 | 0.9415 | 0.9176 |
| Objeto De Aprendizaje | 0.9048 | 0.6786 | 0.7755 |
| Articulo | 0.6651 | 0.4096 | 0.5070 |
| Tesis | 0.6827 | 0.6860 | 0.6843 |
| Libro | 1.0000 | 0.5882 | 0.7407 |
| Otro | 0.9259 | 0.8065 | 0.8621 |

## Prompt Format

The adapter was trained using the Meta-Llama 3.1 chat template with the following system instruction:

```text
System:
Eres un clasificador automático de documentos académicos. Tu única tarea es asignar la categoría correcta a los registros bibliográficos provistos.
Regla estricta: Debes responder ÚNICAMENTE con el nombre exacto de la categoría. No incluyas explicaciones, puntuación adicional ni texto conversacional.
Categorías válidas: Articulo, Objeto de conferencia, Tesis, Libro, Otro, Objeto de aprendizaje.
```

## Usage

```python
import torch

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model = "meta-llama/Meta-Llama-3.1-8B-Instruct"
adapter = "malekcamilo/sedici-llama-lora-r128-all"

tokenizer = AutoTokenizer.from_pretrained(adapter)

model = AutoModelForCausalLM.from_pretrained(
    base_model,
    device_map="auto",
)

model = PeftModel.from_pretrained(model, adapter)
model.eval()


def classify(title: str, abstract: str) -> str:

    messages = [
        {
            "role": "system",
            "content": (
                "Eres un clasificador automático de documentos académicos. "
                "Tu única tarea es asignar la categoría correcta a los registros bibliográficos provistos.\n"
                "Regla estricta: Debes responder ÚNICAMENTE con el nombre exacto de la categoría. "
                "No incluyas explicaciones, puntuación adicional ni texto conversacional.\n"
                "Categorías válidas: Articulo, Objeto de conferencia, Tesis, Libro, Otro, Objeto de aprendizaje."
            ),
        },
        {
            "role": "user",
            "content": (
                f"Clasifica el siguiente registro:\n"
                f"<titulo>{title}</titulo>\n"
                f"<resumen>{abstract}</resumen>"
            ),
        },
    ]

    inputs = tokenizer.apply_chat_template(
        messages,
        tokenize=True,
        add_generation_prompt=True,
        return_tensors="pt",
    )

    inputs = {k: v.to(model.device) for k, v in inputs.items()}

    with torch.no_grad():
        outputs = model.generate(
            **inputs,
            max_new_tokens=10,
            do_sample=False,
        )

    generated = outputs[0][inputs["input_ids"].shape[-1]:]

    return tokenizer.decode(
        generated,
        skip_special_tokens=True,
    ).strip()


prediction = classify(
    "Clasificación automática de documentos científicos",
    "En este trabajo se propone un método para clasificar documentos utilizando grandes modelos de lenguaje."
)

print(prediction)
```

## Reproducibility

- Base model: Meta-Llama-3.1-8B-Instruct
- Fine-tuning: LoRA
- Rank: 128
- Seed: 42
- Dataset size: 19,974 documents
- Sequence length: 512
- Precision: bfloat16
- Frameworks:
  - Transformers
  - PEFT
  - TRL
  - Intel Extension for PyTorch (IPEX)

## Limitations

This adapter was trained exclusively on Computer Science records from SEDICI.
Performance on other academic repositories, languages or taxonomies has not been evaluated.
The model is intended for research purposes.

## Citation
If you use this adapter in your research, please cite the associated Master's thesis:

```bibtex
@mastersthesis{camilo2026peft,
  author = {Malek Camilo},
  title  = {Estrategias de adaptación eficiente para modelos de lenguaje abiertos},
  school = {Facultad de Informática, Universidad Nacional de La Plata},
  year   = {2026},
  type   = {Master's thesis},
  note   = {Manuscript submitted in partial fulfillment of the requirements for the Master's degree},
}
```