Demote the proper-noun anecdote out of the result tables
Browse files- bench3.1/README.md +18 -6
bench3.1/README.md
CHANGED
|
@@ -415,9 +415,14 @@ None of which tells you what it *sounds* like. A native speaker might well hear
|
|
| 415 |
|
| 416 |
---
|
| 417 |
|
| 418 |
-
##
|
| 419 |
|
| 420 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 421 |
|
| 422 |
| | seed 42 | seed 100 k | seed 100 M |
|
| 423 |
|---|---|---|---|
|
|
@@ -427,7 +432,13 @@ The French prompt says *"la lumière de **Marseille**"*. Six phonemes out of sev
|
|
| 427 |
| `4b-v3.1-mlp` | ✗ | ✗ | ✓ |
|
| 428 |
| `4b-v3.1` | ✗ | ✗ | ✗ |
|
| 429 |
|
| 430 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 431 |
|
| 432 |
---
|
| 433 |
|
|
@@ -437,10 +448,11 @@ Treat this for what it is: **one proper noun, in one of eleven languages, on thr
|
|
| 437 |
|---|---|
|
| 438 |
| generate images or video without speech | **`4b-ClipProj-v3.1`** — 4.6 GB, indistinguishable from the 32B |
|
| 439 |
| are tight on VRAM | **`4b-ClipProj-v3.1`** — the ridge is 26 MB and gives up nothing measurable |
|
| 440 |
-
| generate multilingual speech | **`
|
| 441 |
-
|
|
| 442 |
|
| 443 |
-
Do not take a v3
|
|
|
|
| 444 |
|
| 445 |
---
|
| 446 |
|
|
|
|
| 415 |
|
| 416 |
---
|
| 417 |
|
| 418 |
+
## An anecdote, and why it is not a result
|
| 419 |
|
| 420 |
+
**This measures nothing.** One word, in one language out of eleven, with no denominator — it is recorded
|
| 421 |
+
here because it was noticed, not because it supports a conclusion. It is deliberately absent from every
|
| 422 |
+
table above.
|
| 423 |
+
|
| 424 |
+
The French prompt contains a city name. The four 4B files render it as a non-word on almost every seed,
|
| 425 |
+
the 8B files render it correctly on all three:
|
| 426 |
|
| 427 |
| | seed 42 | seed 100 k | seed 100 M |
|
| 428 |
|---|---|---|---|
|
|
|
|
| 432 |
| `4b-v3.1-mlp` | ✗ | ✗ | ✓ |
|
| 433 |
| `4b-v3.1` | ✗ | ✗ | ✗ |
|
| 434 |
|
| 435 |
+
What keeps it from being a finding, beyond the sample size: **it costs almost nothing on the sentence.**
|
| 436 |
+
`4b-v3.1` mangles the name outright and ends up **one** phoneme worse than `8b-v3.1` over the whole line.
|
| 437 |
+
Every aggregate metric in this report is blind to it, which cuts both ways — they cannot confirm it either.
|
| 438 |
+
|
| 439 |
+
The aggregate scores say 4B and 8B are equivalent, and that is the conclusion to keep. The only reason to
|
| 440 |
+
mention this at all is that quantisation is independently known to cost factual recall: if your prompts
|
| 441 |
+
lean on names of people or places, test both sizes on **your** prompts rather than trusting anything here.
|
| 442 |
|
| 443 |
---
|
| 444 |
|
|
|
|
| 448 |
|---|---|
|
| 449 |
| generate images or video without speech | **`4b-ClipProj-v3.1`** — 4.6 GB, indistinguishable from the 32B |
|
| 450 |
| are tight on VRAM | **`4b-ClipProj-v3.1`** — the ridge is 26 MB and gives up nothing measurable |
|
| 451 |
+
| generate multilingual speech | **`4b-ClipProj-v3.1`** covers nine of the eleven languages tested; `8b-ClipProj-v3.1` has the best overall speech score, by less than the seed-to-seed spread |
|
| 452 |
+
| have the VRAM to spare | `8b-ClipProj-v3.1` — nothing measured says you need it, nothing says it hurts |
|
| 453 |
|
| 454 |
+
**Do not take a v3.** That is the only difference this benchmark resolves cleanly: v3 versus v3.1 is real,
|
| 455 |
+
4B versus 8B is not, ridge versus residual is not.
|
| 456 |
|
| 457 |
---
|
| 458 |
|