NicoLab28 commited on
Commit
060ac6f
·
verified ·
1 Parent(s): a0b6bf4

Demote the proper-noun anecdote out of the result tables

Browse files
Files changed (1) hide show
  1. bench3.1/README.md +18 -6
bench3.1/README.md CHANGED
@@ -415,9 +415,14 @@ None of which tells you what it *sounds* like. A native speaker might well hear
415
 
416
  ---
417
 
418
- ## One word, one language
419
 
420
- The French prompt says *"la lumière de **Marseille**"*. Six phonemes out of seventy the error rate drowns them, the ear does not:
 
 
 
 
 
421
 
422
  | | seed 42 | seed 100 k | seed 100 M |
423
  |---|---|---|---|
@@ -427,7 +432,13 @@ The French prompt says *"la lumière de **Marseille**"*. Six phonemes out of sev
427
  | `4b-v3.1-mlp` | ✗ | ✗ | ✓ |
428
  | `4b-v3.1` | ✗ | ✗ | ✗ |
429
 
430
- Treat this for what it is: **one proper noun, in one of eleven languages, on three seeds.** It is not a ranking criterion and the aggregate scores already say 4B and 8B are equivalent. It is a hint that the two sizes may diverge on rare lexical items even where they agree on everything else — consistent with quantisation costing facts, which was already known. If your prompts lean on proper nouns, test both before deciding.
 
 
 
 
 
 
431
 
432
  ---
433
 
@@ -437,10 +448,11 @@ Treat this for what it is: **one proper noun, in one of eleven languages, on thr
437
  |---|---|
438
  | generate images or video without speech | **`4b-ClipProj-v3.1`** — 4.6 GB, indistinguishable from the 32B |
439
  | are tight on VRAM | **`4b-ClipProj-v3.1`** — the ridge is 26 MB and gives up nothing measurable |
440
- | generate multilingual speech | **`8b-ClipProj-v3.1`** best speech score, and steady on proper nouns |
441
- | rely on named people or places | **`8b-ClipProj-v3.1`**, and run the 32B once to check the name works there at all |
442
 
443
- Do not take a v3: it is the only difference this benchmark resolves cleanly.
 
444
 
445
  ---
446
 
 
415
 
416
  ---
417
 
418
+ ## An anecdote, and why it is not a result
419
 
420
+ **This measures nothing.** One word, in one language out of eleven, with no denominator it is recorded
421
+ here because it was noticed, not because it supports a conclusion. It is deliberately absent from every
422
+ table above.
423
+
424
+ The French prompt contains a city name. The four 4B files render it as a non-word on almost every seed,
425
+ the 8B files render it correctly on all three:
426
 
427
  | | seed 42 | seed 100 k | seed 100 M |
428
  |---|---|---|---|
 
432
  | `4b-v3.1-mlp` | ✗ | ✗ | ✓ |
433
  | `4b-v3.1` | ✗ | ✗ | ✗ |
434
 
435
+ What keeps it from being a finding, beyond the sample size: **it costs almost nothing on the sentence.**
436
+ `4b-v3.1` mangles the name outright and ends up **one** phoneme worse than `8b-v3.1` over the whole line.
437
+ Every aggregate metric in this report is blind to it, which cuts both ways — they cannot confirm it either.
438
+
439
+ The aggregate scores say 4B and 8B are equivalent, and that is the conclusion to keep. The only reason to
440
+ mention this at all is that quantisation is independently known to cost factual recall: if your prompts
441
+ lean on names of people or places, test both sizes on **your** prompts rather than trusting anything here.
442
 
443
  ---
444
 
 
448
  |---|---|
449
  | generate images or video without speech | **`4b-ClipProj-v3.1`** — 4.6 GB, indistinguishable from the 32B |
450
  | are tight on VRAM | **`4b-ClipProj-v3.1`** — the ridge is 26 MB and gives up nothing measurable |
451
+ | generate multilingual speech | **`4b-ClipProj-v3.1`** covers nine of the eleven languages tested; `8b-ClipProj-v3.1` has the best overall speech score, by less than the seed-to-seed spread |
452
+ | have the VRAM to spare | `8b-ClipProj-v3.1` nothing measured says you need it, nothing says it hurts |
453
 
454
+ **Do not take a v3.** That is the only difference this benchmark resolves cleanly: v3 versus v3.1 is real,
455
+ 4B versus 8B is not, ridge versus residual is not.
456
 
457
  ---
458