Create README.md

README.md

@@ -0,0 +1,156 @@
+---
+license: apache-2.0
+base_model:
+- HuggingFaceTB/SmolLM2-135M-Instruct
+pipeline_tag: text-generation
+tags:
+  - text-to-image-evaluation
+  - faithfulness
+  - lora
+  - tifa
+language: en
+---
+# SmolLM2-135M-Instruct-TIFA
+
+## Model Description
+
+SmolLM2-135M-Instruct-TIFA is a fine-tuned version of [HuggingFaceTB/SmolLM2-135M-Instruct](https://huggingface.co/HuggingFaceTB/SmolLM2-135M-Instruct) specifically trained for **TIFA (Text-to-Image Faithfulness Assessment)**. This model generates structured evaluation questions to assess how faithfully text-to-image models represent given text descriptions.
+
+## Intended Use
+
+This model is designed to automatically generate evaluation questions for text-to-image models by creating four specific types of questions:
+
+1. **Negative question**: Should have "no" as the answer
+2. **Object identification**: Should have a single word answer directly from the description  
+3. **Attribute identification**: Should have a single word answer directly from the description
+4. **Positive question**: Should have "yes" as the answer
+
+## Model Details
+
+- **Base Model**: HuggingFaceTB/SmolLM2-135M-Instruct
+- **Model Size**: 135M parameters
+- **Fine-tuning Method**: LoRA (Low-Rank Adaptation)
+- **Training Framework**: Transformers + TRL + PEFT
+- **License**: apache-2.0
+
+## Training Details
+
+### Training Configuration
+- **Training Method**: Supervised Fine-Tuning (SFT) with LoRA
+- **LoRA Configuration**:
+  - r: 16
+  - lora_alpha: 32
+  - lora_dropout: 0.05
+  - Target modules: `["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]`
+  
+- **Training Parameters**:
+  - Epochs: 4
+  - Learning Rate: 2e-4
+  - Batch Size: 8 (per device)
+  - Gradient Accumulation Steps: 2
+  - Max Sequence Length: 512
+  - Optimizer: AdamW
+  - Weight Decay: 0.01
+  - Warmup Steps: 200
+
+### Dataset
+The model was trained on a structured dataset containing 5,000 examples created using Gemini, formatted as conversation data in JSONL format.
+
+## Usage
+
+### Installation
+
+```bash
+pip install transformers torch
+```
+
+### Basic Usage
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
+import torch
+
+model_path = "kawchar85/SmolLM2-135M-Instruct-TIFA"
+
+# Load model and tokenizer
+tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
+model = AutoModelForCausalLM.from_pretrained(
+    model_path,
+    torch_dtype=torch.float16,
+    trust_remote_code=True,
+    device_map="auto"
+)
+
+# Create pipeline
+pipe = pipeline(
+    "text-generation",
+    model=model,
+    tokenizer=tokenizer,
+    device=0 if torch.cuda.is_available() else -1,
+    return_full_text=False,
+)
+
+# Generate evaluation questions
+description = "khaki triangles and azure crescents"
+user_msg = (
+    f"Create 4 questions to evaluate a text-to-image model's faithfulness to this description: "
+    f'"{description}".\n'
+    "The first question should have 'no' as the answer, "
+    "the second and third questions should have answers that are a single word directly taken "
+    "from the description, and the fourth question should have 'yes' as the answer."
+)
+
+messages = [{"role": "user", "content": user_msg}]
+
+output = pipe(
+    messages, 
+    max_new_tokens=256, 
+    do_sample=False,
+)
+
+print(output[0]["generated_text"])
+```
+
+### Example Output
+
+For the description "khaki triangles and azure crescents", the model generates:
+
+```
+Q1: Are the triangles green?
+  Choices: ['no', 'yes']
+  Answer: no
+Q2: What color are the triangles?
+  Choices: ['blue', 'red', 'khaki', 'green']
+  Answer: khaki
+Q3: What shape are the objects?
+  Choices: ['squares', 'circles', 'crescents', 'triangles']
+  Answer: crescents
+Q4: Are there azure crescents in the image?
+  Choices: ['no', 'yes']
+  Answer: yes
+```
+
+## Limitations
+
+- The model is specialized for TIFA evaluation and may not perform well on general conversation tasks
+- Limited to generating 4-question evaluation sets in the trained format
+- Performance depends on the quality and diversity of the training dataset
+- Sometimes generates duplicated questions for Q2 and Q3 due to the small dataset used for training or model knowledge limitations
+
+## Technical Specifications
+
+- **Architecture**: Transformer-based language model
+- **Precision**: FP16
+- **Context Length**: 512 tokens
+- **Inference Speed**: Optimized for quick question generation
+
+## Citation
+
+```bibtex
+@misc{smollm2-135m-it-tifa-2025,
+  title={SmolLM2-135M-Instruct-TIFA: A Fine-tuned Model for Text-to-Image Faithfulness Assessment},
+  author={kawchar85},
+  year={2025},
+  url={https://huggingface.co/kawchar85/SmolLM2-135M-Instruct-TIFA}
+}
+```