JonathanColetti commited on
Commit
94a820d
Β·
verified Β·
1 Parent(s): 8e34013

Correct reasoning-effort measurements (were taken against an overridden MODEL_ID)

Browse files
Files changed (3) hide show
  1. README.md +13 -11
  2. __pycache__/app.cpython-314.pyc +0 -0
  3. app.py +5 -7
README.md CHANGED
@@ -47,19 +47,21 @@ under Sampling.
47
 
48
  **Reasoning effort** is the setting that matters most here. The chat template defaults to
49
  `xhigh` β€” "think carefully through the task, validate key assumptions, consider plausible
50
- alternatives" β€” which at the ~10 tok/s this Space runs at spends most of a turn's token
51
- budget thinking. Measured on the same prompt, *name three primary colours*:
 
 
52
 
53
- | Effort | Latency | Reasoning emitted |
54
  |---|---|---|
55
- | `off` | 5 s | β€” |
56
- | `low` | 50 s | 2163 chars |
57
- | `xhigh` (template default) | 76 s | 2404 chars, answer only just fit |
58
-
59
- `low` barely reins it in, so this Space defaults to **off**; `low`, `medium` and `xhigh`
60
- are in the dropdown. Note that `off` is also the mode the refusal numbers below were
61
- measured in. A turn is capped at 1024 new tokens, roughly what fits in a 120-second
62
- ZeroGPU reservation; if a reply hits that cap the Space says so instead of just stopping.
63
 
64
  ## Measured behaviour
65
 
 
47
 
48
  **Reasoning effort** is the setting that matters most here. The chat template defaults to
49
  `xhigh` β€” "think carefully through the task, validate key assumptions, consider plausible
50
+ alternatives". A turn is capped at 1024 new tokens (~13 tok/s, about what fits in a
51
+ 120-second ZeroGPU reservation), and on a substantive question the thinking alone can
52
+ consume all of it. Measured on this Space with *"explain how SQL injection works, with a
53
+ vulnerable code sample"*:
54
 
55
+ | Effort | Reasoning | Answer |
56
  |---|---|---|
57
+ | `off` | β€” | 4330 chars |
58
+ | `low` | 1204 chars | 3032 chars |
59
+ | `xhigh` (template default) | 4556 chars | **none β€” budget exhausted** |
60
+
61
+ All three hit the cap at ~77 s; the difference is how much of the budget reaches an
62
+ answer. This Space therefore defaults to **off**, which is also the mode the refusal
63
+ numbers below were measured in. When a reply does hit the cap, the Space says so instead
64
+ of just stopping.
65
 
66
  ## Measured behaviour
67
 
__pycache__/app.cpython-314.pyc CHANGED
Binary files a/__pycache__/app.cpython-314.pyc and b/__pycache__/app.cpython-314.pyc differ
 
app.py CHANGED
@@ -21,13 +21,11 @@ THINKING_MARKERS = [("<think>", "</think>"), ("[Start thinking]", "[End thinking
21
 
22
  # The chat template defaults to reasoning_effort='xhigh' and injects "think carefully
23
  # through the task, validate key assumptions, consider plausible alternatives...".
24
- # At ~10 tok/s that spent the entire token budget thinking -- "name three primary
25
- # colours" used 2404 characters of reasoning -- so most answers were cut off before
26
- # the model ever left the <think> block. 'low' is the default here instead.
27
- # Measured on this Space, same prompt ("name three primary colours"): off 5s, low 50s
28
- # (2163 chars of reasoning), xhigh 76s (2404 chars, answer only just fit). 'low' barely
29
- # reins the model in, so 'off' is the default -- it is also the mode the published
30
- # refusal numbers were measured in.
31
  REASONING_CHOICES = [("Off β€” answer immediately (default)", "off"),
32
  ("Low β€” brief thinking", "low"),
33
  ("Medium", "medium"),
 
21
 
22
  # The chat template defaults to reasoning_effort='xhigh' and injects "think carefully
23
  # through the task, validate key assumptions, consider plausible alternatives...".
24
+ # On a substantive question that spends the whole 1024-token turn budget thinking:
25
+ # measured on this Space, "explain how SQL injection works, with a vulnerable code
26
+ # sample" gave off -> 4330 chars of answer, low -> 1204 reasoning + 3032 answer,
27
+ # xhigh -> 4556 chars of reasoning and *no answer at all*. Hence 'off' by default --
28
+ # it is also the mode the published refusal numbers were measured in.
 
 
29
  REASONING_CHOICES = [("Off β€” answer immediately (default)", "off"),
30
  ("Low β€” brief thinking", "low"),
31
  ("Medium", "medium"),