thenthuziast/xlm-roberta-large-kk-19

XLM-RoBERTa Large (Kazakh) - Missing Words Challenge

This model is a fine-tuned version of xlm-roberta-large explicitly trained for the Kazakh Missing Words Challenge for WISH on Kaggle.

The task is designed to predict the position of a missing word in a given Kazakh sentence. This model approaches the problem as a sequence classification task with 64 possible gap positions (num_labels=64).

Performance

• Test Score: This model achieved a score of 0.73 on the official Kaggle competition test set.

Model Details

• Model Architecture: XLM-RoBERTa (Large) + Sequence Classification Head
• Language: Kazakh (kk)
• Task: Missing Word Position Prediction (64-class classification)
• Base Model: xlm-roberta-large

Training Details

• Epochs: 20
• Hardware: A100 GPU

Limitations and Bias

• Language Specificity: The model has been heavily fine-tuned on Kazakh data. While the base XLM-R model is multilingual, this specific checkpoint is biased toward Kazakh sentence structures and vocabulary.
• Task Specificity: Its behavior is highly tailored to predicting 64-class missing words/gap positions as a sequence classification task. It will not perform standard Masked Language Modeling or zero-shot classification without structural changes to the classification head.

Acknowledgements

• Base Model: Meta AI for the original xlm-roberta-large.

• Dataset: Organizers of the Kazakh Missing Words Challenge for WISH.

Usage

You can load the fine-tuned model and tokenizer directly using the transformers library:

from transformers import AutoTokenizer, AutoModelForSequenceClassification
model_id = "thenthuziast/xlm-roberta-large-kk"
# Load the tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id)
# Example text
text = "Қазақстанның астанасы қаласы."
inputs = tokenizer(text, return_tensors="pt")
# Perform inference to get logits for the 64 possible missing word positions
outputs = model(**inputs)
logits = outputs.logits