Spaces:
Running on Zero
Running on Zero
Correct reasoning-effort measurements (were taken against an overridden MODEL_ID)
Browse files- README.md +13 -11
- __pycache__/app.cpython-314.pyc +0 -0
- app.py +5 -7
README.md
CHANGED
|
@@ -47,19 +47,21 @@ under Sampling.
|
|
| 47 |
|
| 48 |
**Reasoning effort** is the setting that matters most here. The chat template defaults to
|
| 49 |
`xhigh` β "think carefully through the task, validate key assumptions, consider plausible
|
| 50 |
-
alternatives"
|
| 51 |
-
|
|
|
|
|
|
|
| 52 |
|
| 53 |
-
| Effort |
|
| 54 |
|---|---|---|
|
| 55 |
-
| `off` |
|
| 56 |
-
| `low` |
|
| 57 |
-
| `xhigh` (template default) |
|
| 58 |
-
|
| 59 |
-
|
| 60 |
-
|
| 61 |
-
measured in.
|
| 62 |
-
|
| 63 |
|
| 64 |
## Measured behaviour
|
| 65 |
|
|
|
|
| 47 |
|
| 48 |
**Reasoning effort** is the setting that matters most here. The chat template defaults to
|
| 49 |
`xhigh` β "think carefully through the task, validate key assumptions, consider plausible
|
| 50 |
+
alternatives". A turn is capped at 1024 new tokens (~13 tok/s, about what fits in a
|
| 51 |
+
120-second ZeroGPU reservation), and on a substantive question the thinking alone can
|
| 52 |
+
consume all of it. Measured on this Space with *"explain how SQL injection works, with a
|
| 53 |
+
vulnerable code sample"*:
|
| 54 |
|
| 55 |
+
| Effort | Reasoning | Answer |
|
| 56 |
|---|---|---|
|
| 57 |
+
| `off` | β | 4330 chars |
|
| 58 |
+
| `low` | 1204 chars | 3032 chars |
|
| 59 |
+
| `xhigh` (template default) | 4556 chars | **none β budget exhausted** |
|
| 60 |
+
|
| 61 |
+
All three hit the cap at ~77 s; the difference is how much of the budget reaches an
|
| 62 |
+
answer. This Space therefore defaults to **off**, which is also the mode the refusal
|
| 63 |
+
numbers below were measured in. When a reply does hit the cap, the Space says so instead
|
| 64 |
+
of just stopping.
|
| 65 |
|
| 66 |
## Measured behaviour
|
| 67 |
|
__pycache__/app.cpython-314.pyc
CHANGED
|
Binary files a/__pycache__/app.cpython-314.pyc and b/__pycache__/app.cpython-314.pyc differ
|
|
|
app.py
CHANGED
|
@@ -21,13 +21,11 @@ THINKING_MARKERS = [("<think>", "</think>"), ("[Start thinking]", "[End thinking
|
|
| 21 |
|
| 22 |
# The chat template defaults to reasoning_effort='xhigh' and injects "think carefully
|
| 23 |
# through the task, validate key assumptions, consider plausible alternatives...".
|
| 24 |
-
#
|
| 25 |
-
#
|
| 26 |
-
#
|
| 27 |
-
#
|
| 28 |
-
#
|
| 29 |
-
# reins the model in, so 'off' is the default -- it is also the mode the published
|
| 30 |
-
# refusal numbers were measured in.
|
| 31 |
REASONING_CHOICES = [("Off β answer immediately (default)", "off"),
|
| 32 |
("Low β brief thinking", "low"),
|
| 33 |
("Medium", "medium"),
|
|
|
|
| 21 |
|
| 22 |
# The chat template defaults to reasoning_effort='xhigh' and injects "think carefully
|
| 23 |
# through the task, validate key assumptions, consider plausible alternatives...".
|
| 24 |
+
# On a substantive question that spends the whole 1024-token turn budget thinking:
|
| 25 |
+
# measured on this Space, "explain how SQL injection works, with a vulnerable code
|
| 26 |
+
# sample" gave off -> 4330 chars of answer, low -> 1204 reasoning + 3032 answer,
|
| 27 |
+
# xhigh -> 4556 chars of reasoning and *no answer at all*. Hence 'off' by default --
|
| 28 |
+
# it is also the mode the published refusal numbers were measured in.
|
|
|
|
|
|
|
| 29 |
REASONING_CHOICES = [("Off β answer immediately (default)", "off"),
|
| 30 |
("Low β brief thinking", "low"),
|
| 31 |
("Medium", "medium"),
|