Update README.md

README.md

@@ -48,49 +48,55 @@ model-index:
   <img src="https://cdn-uploads.huggingface.co/production/uploads/65ff92ea467d83751a727538/Jzq_CZCyRYGrVgbto3eRr.png" style="width: 400px;">
 </p>
 
-This model is a fine tune of the 68M parameter `jhu-clsp/ettin-encoder-68m` (ModernBERT architecture), using the `dleemiller/FineCat-NLI`
-dataset. This dataset is a compilation of several high quality sources of NLI data, with quality screening and
-reduction of easy samples in the train split.
+-----
+
+# Overview
+
+This model is a fine-tune of `jhu-clsp/ettin-encoder-68m`, 
+trained on the `dleemiller/FineCat-NLI` dataset—a compilation of several high-quality 
+NLI data sources with quality screening and reduction of easy samples in the training split. 
+The training also incorporates logit distillation from `dleemiller/finecat-nli-l`.
+
+Distillation loss looks like this:
+$$
+\begin{equation}
+\mathcal{L} = \alpha \cdot \mathcal{L}_{\text{CE}}(z^{(s)}, y) + \beta \cdot \mathcal{L}_{\text{MSE}}(z^{(s)}, z^{(t)})
+\end{equation}
+$$
+
+where \\(z^{(s)}\\) and \\(z^{(t)}\\) are the student and teacher logits, \\(y\\) are the ground truth labels, 
+and \\(\alpha\\) and \\(\beta\\) are equally weighted at 0.5.
 
 This model and dataset specifically targets improving NLI, through high quality sources. The tasksource models
 are the best checkpoints to start from, although training from ModernBERT is also competitive.
 
-This model is distilled using teacher logits from `dleemiller/finecat-nli-l`.
+-----
 
 # NLI Evaluation Results
 
 F1-Micro scores (equivalent to accuracy) for each dataset.
+Performance was measured at bs=32 using a Nvidia Blackwell PRO 6000 Max-Q.
 
-| Model | finecat | mnli | mnli_mismatched | snli | anli_r1 | anli_r2 | anli_r3 | wanli | lingnli |
-| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
-| `MoritzLaurer/DeBERTa-v3-large-mnli-fever-anli-ling-wanli` | **0.8233** | <u>0.9121</u> | 0.9079 | 0.8898 | **0.7960** | **0.6830** | **0.6400** | </u>0.7700</u> | **0.8821** |
-| `dleemiller/finecat-nli-l`        | <u>0.8227</u> | **0.9152** | **0.9265** | 0.9162 | <u>0.7480</u> | <u>0.5700</u> | <u>0.5433</u> | **0.7706** | <u>0.8742</u> |
-| `tasksource/ModernBERT-large-nli` | 0.7959 | 0.8983 | <u>0.9229</u> | <u>0.9188</u>   | 0.7260 | 0.5110 | 0.4925 | 0.6978 | 0.8504 |
-| `dleemiller/finecat-nli-s`        | 0.7865 | 0.8727 | 0.8715 | 0.9087 | 0.6270 | 0.4720 | 0.4258 | 0.7272 | 0.7996 |
-| `dleemiller/ModernCE-large-nli`   | 0.7811 | 0.9088 | 0.9205 | **0.9273** | 0.6630 | 0.4860 | 0.4408 | 0.6576 | 0.8566 |
-| `MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli` | 0.7642 | 0.9033 | 0.9028 | 0.8866 | 0.7120 | <u>0.5470</u> | 0.4950 | 0.6418 | 0.8523 |
-| `cross-encoder/nli-deberta-v3-large` | 0.7618 | 0.9019 | 0.9049 | 0.9220 | 0.5300 | 0.4170 | 0.3758 | 0.6548 | 0.8466 |
-| `tasksource/ModernBERT-base-nli`  | 0.7595 | 0.8685 | 0.8979 | 0.8915 | 0.6300 | 0.4820 | 0.4192 | 0.6632 | 0.8118 |
-| `dleemiller/ModernCE-base-nli`    | 0.7533 | 0.8923 | 0.9035 | 0.9187 | 0.5240 | 0.3950 | 0.3333 | 0.6464 | 0.8282 |
-| `cross-encoder/nli-deberta-v3-xsmall` | 0.7269 | 0.8781 | 0.8777 | 0.9164 | 0.3620 | 0.3030 | 0.3183 | 0.6096 | 0.8122 |
-| `dleemiller/EttinX-nli-s`   | 0.7251 | 0.8765 | 0.8798 | 0.9128 | 0.3360 | 0.2790 | 0.3083 | 0.6234 | 0.8012 |
-| `cross-encoder/nli-MiniLM2-L6-H768` | 0.7119 | 0.8660 | 0.8683 | 0.9137 | 0.3090 | 0.2850 | 0.2867 | 0.5830 | 0.7905 |
-| `dleemiller/EttinX-nli-xs`  | 0.7013 | 0.8376 | 0.8380 | 0.8979 | 0.2780 | 0.2840 | 0.2800 | 0.5838 | 0.7521 |
-| `cross-encoder/nli-distilroberta-base` | 0.6936 | 0.8365 | 0.8398 | 0.8996 | 0.2660 | 0.2810 | 0.2975 | 0.5516 | 0.7516 |
-| `dleemiller/EttinX-nli-xxs` | 0.6842 | 0.7988 | 0.8047 | 0.8851 | 0.2590 | 0.3060 | 0.2992 | 0.5426 | 0.7018 |
-
+| Model | finecat | mnli | mnli_mismatched | snli | anli_r1 | anli_r2 | anli_r3 | wanli | lingnli | Throughput (samples/s) | Peak GPU Mem (MB) |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| `dleemiller/finecat-nli-s` | 0.7834 | 0.8725 | 0.8725 | 0.8973 | 0.6400 | 0.4660 | 0.4617 | 0.7284 | 0.8072 | 2291.87 | 415.65 |
+| `tasksource/deberta-small-long-nli` | 0.7492 | 0.8194 | 0.8206 | 0.8613 | 0.5670 | 0.4220 | 0.4475 | 0.7034 | 0.7605 | 2250.66 | 1351.08 |
+| `cross-encoder/nli-deberta-v3-xsmall` | 0.7269 | 0.8781 | 0.8777 | 0.9164 | 0.3620 | 0.3030 | 0.3183 | 0.6096 | 0.8122 | 2510.05 | 753.91 |
+| `dleemiller/EttinX-nli-s` | 0.7251 | 0.8765 | 0.8798 | 0.9128 | 0.3360 | 0.2790 | 0.3083 | 0.6234 | 0.8012 | 2348.21 | 415.65 |
+| `cross-encoder/nli-MiniLM2-L6-H768` | 0.7119 | 0.8660 | 0.8683 | 0.9137 | 0.3090 | 0.2850 | 0.2867 | 0.5830 | 0.7905 | 2885.72 | 566.64 |
+| `cross-encoder/nli-distilroberta-base` | 0.6936 | 0.8365 | 0.8398 | 0.8996 | 0.2660 | 0.2810 | 0.2975 | 0.5516 | 0.7516 | 2838.17 | 566.64 |
 
+-----
 
+# Usage
 
-## Usage
-
-Label Map:
+### Label Map:
 - `entailment`: 0
 - `neutral`: 1
 - `contradiction`: 2
 
 
-### Direct Usage (Sentence Transformers)
+## Direct Usage (Sentence Transformers)
 
 First install the Sentence Transformers library:
 
@@ -104,7 +110,7 @@ Then you can load the model and run inference.
 from sentence_transformers import CrossEncoder
 import numpy as np
 
-model = CrossEncoder("dleemiller/finecat-nli-l")
+model = CrossEncoder("dleemiller/finecat-nli-s")
 id2label = model.model.config.id2label  # {0:'entailment', 1:'neutral', 2:'contradiction'}
 
 pairs = [
@@ -129,3 +135,19 @@ for (prem, hyp), row in zip(pairs, logits):
 
 ```
 
+
+## Acknowledgments
+
+We thank the creators and contributors of `tasksource` and `MoritzLaurer` for making their work available.
+This model would not be possible without their efforts and open source contributions.
+
+## Citation
+
+```bibtex
+@misc{nli-compiled-2025,
+  title = {FineCat NLI Dataset},
+  author = {Lee Miller},
+  year = {2025},
+  howpublished = {Refined compilation of 6 major NLI datasets}
+}
+```