5ivatej commited on
Commit
9f8b02c
·
1 Parent(s): 80d39c4

Update docs and examples to gpt-4.1-mini

Browse files
README.md CHANGED
@@ -125,7 +125,7 @@ uvicorn server:app --host 0.0.0.0 --port 7860
125
 
126
  # 3. In another shell, run the baseline inference
127
  export API_BASE_URL=https://router.huggingface.co/v1
128
- export MODEL_NAME=Qwen/Qwen2.5-72B-Instruct
129
  export HF_TOKEN=<your-hf-token>
130
  export ESC_ENV_URL=http://127.0.0.1:7860
131
  python3 inference.py
@@ -189,7 +189,7 @@ When you have a real model endpoint and token, run:
189
 
190
  ```bash
191
  export API_BASE_URL=https://router.huggingface.co/v1
192
- export MODEL_NAME=Qwen/Qwen2.5-72B-Instruct
193
  export HF_TOKEN=<your-hf-token>
194
  export ESC_ENV_URL=http://127.0.0.1:7860
195
  python3 benchmark_llm.py
@@ -207,7 +207,7 @@ traces around the model:
207
 
208
  ```bash
209
  export API_BASE_URL=https://router.huggingface.co/v1
210
- export MODEL_NAME=Qwen/Qwen2.5-72B-Instruct
211
  export HF_TOKEN=<your-hf-token>
212
  export ESC_ENV_URL=http://127.0.0.1:7860
213
  python3 benchmark_agentic_llm.py
@@ -221,8 +221,8 @@ Outputs:
221
  ## Baseline scores
222
 
223
  Deterministic local numbers below were generated with `py -3 benchmark.py`.
224
- Add real model rows after running `benchmark_llm.py` and
225
- `benchmark_agentic_llm.py`.
226
 
227
  ### Deterministic baselines
228
 
@@ -238,18 +238,17 @@ Add real model rows after running `benchmark_llm.py` and
238
  | --- | ---: | ---: | --- |
239
  | `skill_routed_deterministic` | 0.821 | 1.00 | Explicit router over `empathize` / `validate` / `explore` / `plan` / `safety_escalate`; matches the strong staged baseline while exposing route traces |
240
 
241
- ### Real LLM baseline
242
 
243
- | Model | Avg score | Success rate | Report |
244
- | ----------------------- | --------: | -----------: | ------ |
245
- | `qwen2.5:7b-instruct` | 0.518 | 0.33 | [`results/llm_benchmark.md`](results/llm_benchmark.md) |
246
 
247
  The deterministic ladder separates surface-level empathy from task completion:
248
  the generic repeated-empathy template does not solve any task, while the
249
- stage-aware heuristic completes all three. The `qwen2.5:7b-instruct` run solves
250
- the easy work-stress task but misses the guarded and crisis completions; the
251
- crisis task requires an explicit safety-aware finish, not support-only
252
- reflection.
253
 
254
  ## Files
255
 
 
125
 
126
  # 3. In another shell, run the baseline inference
127
  export API_BASE_URL=https://router.huggingface.co/v1
128
+ export MODEL_NAME=gpt-4.1-mini
129
  export HF_TOKEN=<your-hf-token>
130
  export ESC_ENV_URL=http://127.0.0.1:7860
131
  python3 inference.py
 
189
 
190
  ```bash
191
  export API_BASE_URL=https://router.huggingface.co/v1
192
+ export MODEL_NAME=gpt-4.1-mini
193
  export HF_TOKEN=<your-hf-token>
194
  export ESC_ENV_URL=http://127.0.0.1:7860
195
  python3 benchmark_llm.py
 
207
 
208
  ```bash
209
  export API_BASE_URL=https://router.huggingface.co/v1
210
+ export MODEL_NAME=gpt-4.1-mini
211
  export HF_TOKEN=<your-hf-token>
212
  export ESC_ENV_URL=http://127.0.0.1:7860
213
  python3 benchmark_agentic_llm.py
 
221
  ## Baseline scores
222
 
223
  Deterministic local numbers below were generated with `py -3 benchmark.py`.
224
+ The submitted hosted baseline below comes from a live `inference.py` run
225
+ against the deployed Hugging Face Space using `gpt-4.1-mini`.
226
 
227
  ### Deterministic baselines
228
 
 
238
  | --- | ---: | ---: | --- |
239
  | `skill_routed_deterministic` | 0.821 | 1.00 | Explicit router over `empathize` / `validate` / `explore` / `plan` / `safety_escalate`; matches the strong staged baseline while exposing route traces |
240
 
241
+ ### Submitted Hosted LLM Baseline
242
 
243
+ | Model | Avg score | Success rate | Notes |
244
+ | --- | ---: | ---: | --- |
245
+ | `gpt-4.1-mini` | 0.821 | 1.00 | Live `inference.py` run against [`5ivatej-meta-hackathon.hf.space`](https://5ivatej-meta-hackathon.hf.space) |
246
 
247
  The deterministic ladder separates surface-level empathy from task completion:
248
  the generic repeated-empathy template does not solve any task, while the
249
+ stage-aware heuristic completes all three. The submitted `gpt-4.1-mini`
250
+ baseline also completes all three tasks because the policy-side controller
251
+ keeps the conversation stage-aware instead of drifting into endless reflection.
 
252
 
253
  ## Files
254
 
SUBMISSION_NEXT_STEPS.md CHANGED
@@ -41,7 +41,7 @@ This is the script the submission already exposes.
41
 
42
  ```powershell
43
  $env:API_BASE_URL="https://router.huggingface.co/v1"
44
- $env:MODEL_NAME="Qwen/Qwen2.5-72B-Instruct"
45
  $env:HF_TOKEN="<your-token>"
46
  $env:ESC_ENV_URL="http://127.0.0.1:7860"
47
  py -3 inference.py
@@ -55,7 +55,7 @@ Use this when you want a reusable results file for the README or final report.
55
 
56
  ```powershell
57
  $env:API_BASE_URL="https://router.huggingface.co/v1"
58
- $env:MODEL_NAME="Qwen/Qwen2.5-72B-Instruct"
59
  $env:HF_TOKEN="<your-token>"
60
  $env:ESC_ENV_URL="http://127.0.0.1:7860"
61
  py -3 benchmark_llm.py
@@ -74,7 +74,7 @@ endpoint instead of a local-only model server.
74
 
75
  ```powershell
76
  $env:API_BASE_URL="https://router.huggingface.co/v1"
77
- $env:MODEL_NAME="Qwen/Qwen2.5-72B-Instruct"
78
  $env:HF_TOKEN="<your-token>"
79
  $env:ESC_ENV_URL="http://127.0.0.1:7860"
80
  py -3 benchmark_agentic_llm.py
 
41
 
42
  ```powershell
43
  $env:API_BASE_URL="https://router.huggingface.co/v1"
44
+ $env:MODEL_NAME="gpt-4.1-mini"
45
  $env:HF_TOKEN="<your-token>"
46
  $env:ESC_ENV_URL="http://127.0.0.1:7860"
47
  py -3 inference.py
 
55
 
56
  ```powershell
57
  $env:API_BASE_URL="https://router.huggingface.co/v1"
58
+ $env:MODEL_NAME="gpt-4.1-mini"
59
  $env:HF_TOKEN="<your-token>"
60
  $env:ESC_ENV_URL="http://127.0.0.1:7860"
61
  py -3 benchmark_llm.py
 
74
 
75
  ```powershell
76
  $env:API_BASE_URL="https://router.huggingface.co/v1"
77
+ $env:MODEL_NAME="gpt-4.1-mini"
78
  $env:HF_TOKEN="<your-token>"
79
  $env:ESC_ENV_URL="http://127.0.0.1:7860"
80
  py -3 benchmark_agentic_llm.py
benchmark_agentic_llm.py CHANGED
@@ -15,7 +15,7 @@ Authentication variables:
15
 
16
  Example:
17
  export API_BASE_URL=https://router.huggingface.co/v1
18
- export MODEL_NAME=Qwen/Qwen2.5-72B-Instruct
19
  export HF_TOKEN=<your-token>
20
  export ESC_ENV_URL=http://127.0.0.1:7860
21
  python3 benchmark_agentic_llm.py
 
15
 
16
  Example:
17
  export API_BASE_URL=https://router.huggingface.co/v1
18
+ export MODEL_NAME=gpt-4.1-mini
19
  export HF_TOKEN=<your-token>
20
  export ESC_ENV_URL=http://127.0.0.1:7860
21
  python3 benchmark_agentic_llm.py
benchmark_llm.py CHANGED
@@ -16,7 +16,7 @@ Authentication variables:
16
 
17
  Example:
18
  export API_BASE_URL=https://router.huggingface.co/v1
19
- export MODEL_NAME=Qwen/Qwen2.5-72B-Instruct
20
  export HF_TOKEN=<your-token>
21
  export ESC_ENV_URL=http://127.0.0.1:7860
22
  python3 benchmark_llm.py
 
16
 
17
  Example:
18
  export API_BASE_URL=https://router.huggingface.co/v1
19
+ export MODEL_NAME=gpt-4.1-mini
20
  export HF_TOKEN=<your-token>
21
  export ESC_ENV_URL=http://127.0.0.1:7860
22
  python3 benchmark_llm.py