nisten commited on
Commit
3560d46
Β·
verified Β·
1 Parent(s): def8430

Upload PARITY.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. PARITY.md +89 -0
PARITY.md CHANGED
@@ -263,3 +263,92 @@ Browser-load bugs fixed to get here (details in CLAUDE.md worklog cont. 7):
263
  `env.allowLocalModels` browser default, upstream `use_external_data_format`
264
  catch-all key forcing a phantom `model_quantized.onnx_data` fetch (hang at
265
  100%), `.mjs` served as octet-stream breaking the webgpu EP's dynamic import.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
263
  `env.allowLocalModels` browser default, upstream `use_external_data_format`
264
  catch-all key forcing a phantom `model_quantized.onnx_data` fetch (hang at
265
  100%), `.mjs` served as octet-stream breaking the webgpu EP's dynamic import.
266
+
267
+ ## fp16 β€” parity vs PyTorch fp32 (model_fp16.onnx)
268
+
269
+ - logits max abs diff: **1.491**
270
+ - per-token argmax agreement: **775/775 (100.00%)**
271
+ - span exact match (start,end,label): **61/61 (100.0%)** β€” 0 only-PyTorch, 0 only-ONNX
272
+ - gold coverage: PyTorch **52/61**, fp16 **52/61**
273
+ - critical gold spans missed by fp16: **0**
274
+
275
+ ## bench10 β€” 10 handwritten PII examples, CAUGHT (any-label overlap) / exact-label
276
+
277
+ | ex | gold item (label) | pytorch | fp16 |
278
+ |---|---|---|---|
279
+ | 1 | `kai.nakamura` (unique_id) | CAUGHT* | CAUGHT* |
280
+ | 1 | `Tr0ub4dor&3` (password) | CAUGHT | CAUGHT |
281
+ | 1 | `kai.nakamura@example.io` (email) | CAUGHT | CAUGHT |
282
+ | 2 | `5425 2334 3010 9903` (credit_debit_card) | CAUGHT | CAUGHT |
283
+ | 2 | `11/27` (date) | CAUGHT | CAUGHT |
284
+ | 2 | `833` (cvv) | CAUGHT | CAUGHT |
285
+ | 3 | `078-05-1120` (ssn) | CAUGHT | CAUGHT |
286
+ | 3 | `1EG4-TE5-MK72` (health_plan_beneficiary_number) | CAUGHT | CAUGHT |
287
+ | 4 | `2001:0db8:85a3:0000:0000:8a2e:0370:7334` (ipv6) | CAUGHT | CAUGHT |
288
+ | 4 | `3D:F2:C9:A6:B3:4F` (mac_address) | CAUGHT | CAUGHT |
289
+ | 5 | `sk-live-9f8B2kQ71LmZxWv4TgNpR3sY` (api_key) | CAUGHT | CAUGHT |
290
+ | 6 | `Rosa` (first_name) | CAUGHT | CAUGHT |
291
+ | 6 | `Delgado-Marquez` (last_name) | CAUGHT* | CAUGHT* |
292
+ | 6 | `+1 (312) 555-0147` (phone_number) | CAUGHT | CAUGHT |
293
+ | 6 | `44 Cedar Hollow Ln, Apt 9C` (street_address) | CAUGHT | CAUGHT |
294
+ | 6 | `Naperville` (city) | CAUGHT | CAUGHT |
295
+ | 6 | `IL` (state) | CAUGHT | CAUGHT |
296
+ | 6 | `60540` (postcode) | CAUGHT | CAUGHT |
297
+ | 7 | `07/04/1962` (date_of_birth) | CAUGHT* | CAUGHT* |
298
+ | 7 | `X4711982-B` (national_id) | CAUGHT | CAUGHT |
299
+ | 7 | `Lisbon` (city) | **MISSED** | **MISSED** |
300
+ | 7 | `2026-09-18` (date) | CAUGHT* | CAUGHT* |
301
+ | 8 | `DE89 3704 0044 0532 0130 00` (account_number) | CAUGHT | CAUGHT |
302
+ | 8 | `021000021` (bank_routing_number) | CAUGHT | CAUGHT |
303
+ | 8 | `646-555-0199` (phone_number) | CAUGHT | CAUGHT |
304
+ | 9 | `KTX-4092` (license_plate) | CAUGHT* | CAUGHT* |
305
+ | 9 | `1HGBH41JXMN109186` (vehicle_identifier) | CAUGHT | CAUGHT |
306
+ | 9 | `45.5231, -122.6765` (coordinate) | CAUGHT | CAUGHT |
307
+ | 10 | `_wid=eyJhbGciOiJIUzI1NiJ9.aWQ9NzcyNA.4vX2` (http_cookie) | CAUGHT | CAUGHT |
308
+ | 10 | `wellness.example.com` (url) | **MISSED** | **MISSED** |
309
+ | 10 | `E-100482` (employee_id) | CAUGHT | CAUGHT |
310
+
311
+ **pytorch: 29/31 caught**
312
+
313
+ **fp16: 29/31 caught**
314
+
315
+ `*` = caught with a different (but redacting) label
316
+
317
+ ## Phase 4b: q8 / mixed48 / fp16 in the browser (supersedes Phase 4)
318
+
319
+ Phase 4 concluded q8 and mixed48 were native-ORT-only. Two changes overturned
320
+ that β€” the big variants now run in-browser:
321
+
322
+ 1. **External data + 64-byte alignment** (`export/externalize.py`). Single-file
323
+ variants forced ort-web to parse the whole protobuf inside the 4 GB wasm32
324
+ heap (q8 β†’ `std::bad_alloc`) and left initializers at unaligned file offsets
325
+ (mixed48 β†’ `operation does not support unaligned accesses`). Re-saved as
326
+ `<name>.onnx` + `<name>.onnx_data` with every tensor padded to a 64-byte
327
+ boundary, both failures disappear. Session options `enableCpuMemArena:false`
328
+ + `enableMemPattern:false` trim peak allocation further.
329
+ 2. **ort-web JSPI bundle for WebGPU** (`ort.jspi.bundle.min.mjs`). The default
330
+ (asyncify) bundle runs QMoE on the CPU via async round-trips β€” mixed48 loads
331
+ but inference is ~91 s. The JSPI bundle has the **native WebGPU QMoE kernel**
332
+ (expert weights resident in VRAM), taking mixed48 to 488 ms and q8 to 817 ms.
333
+ The worker picks the bundle from `src/lib/variants.ts` (`bundleFor`): q8 and
334
+ mixed48 on WebGPU β†’ JSPI; q4 and all CPU β†’ default (broad compatibility;
335
+ JSPI needs Chrome/Edge β‰₯ 137).
336
+
337
+ Measured in-browser matrix (Chrome 149, GTX 1660 Ti / Turing, 6 GB VRAM,
338
+ ort-web 1.27.0; spans correct in every βœ“ cell):
339
+
340
+ | variant | GPU (WebGPU) | CPU (wasm) |
341
+ |---|---|---|
342
+ | q4 (0.92 GB) | βœ“ default bundle Β· load 29 s Β· p50 134 ms Β· +825 MiB VRAM | βœ— no wasm `GatherBlockQuantized` (4-bit Gather) kernel |
343
+ | q8 (1.98 GB) | βœ“ **JSPI** Β· load 152 s cold/32 s warm Β· 817 ms Β· +2849 MiB VRAM | βœ“ default bundle Β· load 36 s Β· **357 ms** |
344
+ | mixed48 (1.66 GB) | βœ“ **JSPI** Β· load 121 s Β· 488 ms Β· +1368 MiB VRAM | βœ“ default bundle Β· ~370 ms |
345
+ | fp16 (2.82 GB) | βœ“ *only on `shader-f16` GPUs* β€” this Turing card lacks it β†’ "device (webgpu) does not support fp16"; works on RTX 20xx+ / Apple Silicon | βœ— impractical (fp16 has no CPU compute advantage; 2.82 GB session stalls) |
346
+
347
+ Notes:
348
+ - **q4 stays the recommended default**: smallest, GPU-only, broadest bundle
349
+ compatibility, lowest VRAM. q8 is the accuracy pick when VRAM allows.
350
+ - fp16-on-WebGPU is a genuine capability (native-precision, no dequant) but is
351
+ gated on the GPU exposing `shader-f16`; the app surfaces the clear ort error
352
+ on GPUs that don't.
353
+ - The picker (`public/index.html` + `src/lib/variants.ts`) disables combos that
354
+ can't run β€” q4 forces GPU; CPU is offered for q8/mixed48/fp16.