v1 binary sentiment INT8 ONNX (DistilBERT, SageMaker fine-tune, Macro F1 0.939)

README.md

@@ -0,0 +1,132 @@
+# Model Card — MovieSentiment DistilBERT INT8
+
+Following the [Mitchell et al. 2019 Model Card](https://arxiv.org/abs/1810.03993) template.
+
+## Model details
+
+- **Model name**: `moviesentiment-classifier`
+- **Architecture**: DistilBERT-base-uncased (66M parameters), HuggingFace `distilbert-base-uncased`
+- **Head**: 2-class linear classifier on `[CLS]` pooled output
+- **Quantization**: ONNX dynamic INT8 (8-bit weights, FP32 activations) via `optimum.onnxruntime.ORTQuantizer` with `AutoQuantizationConfig.avx512_vnni(is_static=False)`
+- **Size**: 64 MB (down from 256 MB FP32; 4× compression)
+- **Framework versions**: `transformers>=4.46`, `optimum[onnxruntime]>=1.17`, `onnxruntime>=1.17`
+- **Owner**: Soumya Sarkar — `github.com/Cryptic2-0`
+- **License**: MIT
+- **Date**: 2026-05-26 (last retrained from main)
+- **Git SHA**: returned at `GET /version`
+
+## Intended use
+
+**Primary use**: classify English IMDb-style movie reviews into positive/negative sentiment.
+
+**Primary users**: this project is a portfolio MLOps demonstration. The API is publicly reachable but is not intended for downstream products.
+
+**Out-of-scope**:
+- Non-English text — see *Quantitative analyses* below for the cliff-edge accuracy drop.
+- Product, food, news, or any non-movie domain — not evaluated; results unreliable.
+- Long-form text > 5000 characters — truncated at 512 tokens.
+- Multi-class sentiment (neutral, mixed, sarcastic) — binary output only.
+- Any safety-, health-, or finance-sensitive decision.
+
+## Factors
+
+| Factor | Buckets evaluated |
+|---|---|
+| Review length | short (<200 chars), medium (200–600), long (>600) |
+| Sentiment intensity | strong (>0.9 confidence), weak (0.55–0.7) |
+| Genre signal | comedy / drama / action / horror (from movie metadata, where available) |
+| Named entities | reviews mentioning specific actors / directors |
+| Sarcasm proxy | reviews flagged by a heuristic sarcasm detector (manual seed list) |
+
+## Metrics
+
+Test set: held-out 5,000 reviews from IMDb 50K.
+
+| Metric | DistilBERT INT8 | TF-IDF + LR (baseline) |
+|---|---|---|
+| Macro F1 | **0.939** | 0.904 |
+| Accuracy | 0.940 | 0.905 |
+| ROC AUC | 0.984 | 0.962 |
+| Latency p50 | 6.8 ms | <5 ms |
+| Latency p99 | 14 ms | <8 ms |
+| Size on disk | 64 MB | 18 MB |
+
+CI for F1 (bootstrap, 1000 resamples): **[0.934, 0.944]**.
+
+## Quantitative analyses
+
+### Performance by review length
+
+| Length bucket | F1 | n |
+|---|---|---|
+| Short (<200 chars) | 0.917 | 1,402 |
+| Medium (200–600) | 0.940 | 2,217 |
+| Long (>600) | 0.951 | 1,381 |
+
+Model is weakest on very short reviews (terse one-liners with weak signal). This matches intuition — fewer tokens, fewer keywords.
+
+### Performance by confidence bucket
+
+| Confidence | Accuracy | Coverage |
+|---|---|---|
+| >0.95 | 0.984 | 78% of test |
+| 0.85–0.95 | 0.913 | 14% |
+| 0.70–0.85 | 0.760 | 6% |
+| <0.70 | 0.512 | 2% |
+
+The model is **well-calibrated above 0.85** — high-confidence predictions are reliable. The thin (~2%) low-confidence band is essentially chance-level and should be treated as "model declined to answer."
+
+### Sarcasm slice
+
+Reviews flagged by the heuristic sarcasm seed list (~340 examples): **F1 0.823**, vs 0.939 overall — a 12-point absolute drop. This is the most prominent failure mode and is documented on the live demo card.
+
+### Non-English
+
+Spanish translations of the test set (machine-translated via NLLB): **F1 0.51** (chance is 0.50). Treat any non-English input as untrusted. The frontend does not gate on language; this is intentional (interview material), but a production deployment should.
+
+### Per-genre slice (live IMDb scrape, 2026-05-29)
+
+Evaluated on a fresh **live IMDb GraphQL scrape** of the 10 movie IDs in `params.yaml`. 2,525 reviews retrieved, sorted by `HELPFULNESS_SCORE` (IMDb's default — biases toward positive reviews). Each movie's primary genre per IMDb's listing. Source data at `data/processed/live_with_genre.parquet`; raw numbers at `metrics/per_genre_f1.json`.
+
+| Genre | n | Pos share | Accuracy | Macro F1 | F1 (positive) |
+|---|---|---|---|---|---|
+| Adventure | 232 | 0.918 | 0.940 | **0.848** | 0.966 |
+| Crime | 786 | 0.934 | 0.950 | **0.840** | 0.973 |
+| Animation | 238 | 0.929 | 0.945 | **0.834** | 0.970 |
+| Drama | 764 | 0.919 | 0.937 | **0.827** | 0.965 |
+| Action | 505 | 0.949 | 0.935 | **0.779** | 0.964 |
+
+**Overall**: n=2,525, accuracy 0.942, macro F1 **0.825**.
+
+**Key caveat — class skew**: the live test set is **~93% positive** because IMDb's helpfulness-sort top-loads positive reviews. On the original balanced HF test set (5,000 examples, 50/50) macro F1 is 0.939; the live slice's lower macro F1 (0.825) reflects the imbalance, not a model regression. Binary F1 on the majority (positive) class stays in the 0.964–0.973 band across all genres, consistent with the original eval.
+
+**What the slice does and doesn't tell us**: Action has the weakest macro F1 (0.779), which means the model is most likely to miss negative reviews of Action films. Adventure and Crime hold the strongest macro F1, but those buckets also have less class skew. A genre-balanced re-scrape — sort by date instead of helpfulness — would be the next step to disentangle "genre effect" from "class-balance effect". Documented as a follow-up in `docs/future_improvements.md`.
+
+## Ethical considerations
+
+- **Training data is movie-only**. Generalization to other "subjective text" domains is not implied.
+- **No demographic factors are inferred or stored**. The pipeline does not capture reviewer identity.
+- **Reservoir-sampled production inputs** are stored under `data/production/recent.parquet` for drift detection. The current sampler is in-memory and flushed every 100 inserts; users who post personally identifying text are subject to this storage for the lifetime of the running container.
+- **No human-in-the-loop**. Predictions are returned directly.
+
+## Caveats and recommendations
+
+1. Use confidence to gate downstream actions — anything below 0.70 should be surfaced as "uncertain", not classified.
+2. The model was trained on IMDb 50K which is **balanced** (50/50 positive/negative). Real-world review distributions skew positive (~70/30). Expect more false negatives at deployment time.
+3. If you need higher recall on negative reviews (e.g., for moderation), retrain with a re-weighted loss or threshold-tune via the bundled `eval/metrics.py` PR-curve helpers.
+4. Retraining loop closes when Evidently drift > 0.30 (`monitor/drift.py`) — fires a GH Actions workflow to scrape, retrain (SageMaker spot + LoRA), re-export ONNX, redeploy.
+
+## Environmental impact
+
+Trained once on `ml.g4dn.xlarge` (T4 GPU) for ~25 minutes. Estimated energy: ~0.05 kWh. With CodeCarbon's default Australia grid factor (~0.66 kgCO2/kWh) that's ~33g CO2 per training run. The retraining loop runs at most weekly.
+
+Inference: ARM Graviton Fargate, 0.25 vCPU, 1 GB RAM, single task. Steady-state ~3 W. Per-prediction CO2 cost is negligible at portfolio traffic.
+
+## Where to find more
+
+- Source: `github.com/Cryptic2-0/Project`
+- Live API contract: `OpenAPI /docs` on the running Fargate task
+- Drift reports: `docs/drift_reports/` (HTML, generated by `python -m moviesentiment.cli drift`)
+- Latency benchmarks: `docs/benchmarks.md`
+- Load test report: `docs/load_test_report.html`
+- Datasheet for the training data: `docs/datasheet.md`

model.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b39dadeb7d7de8d44efcc647adfaa64fc3bf98dee32e23755b8ac0b5d874b9d8
+size 67348137

special_tokens_map.json

@@ -0,0 +1,37 @@
+{
+  "cls_token": {
+    "content": "[CLS]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "[MASK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "[PAD]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "content": "[SEP]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "[UNK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,60 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "101": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "102": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "103": {
+      "content": "[MASK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "clean_up_tokenization_spaces": false,
+  "cls_token": "[CLS]",
+  "do_lower_case": true,
+  "extra_special_tokens": {},
+  "mask_token": "[MASK]",
+  "max_length": 512,
+  "model_max_length": 512,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "stride": 0,
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "DistilBertTokenizer",
+  "truncation_side": "right",
+  "truncation_strategy": "longest_first",
+  "unk_token": "[UNK]"
+}