Spaces:
Sleeping
Sleeping
Update docs and examples to gpt-4.1-mini
Browse files- README.md +12 -13
- SUBMISSION_NEXT_STEPS.md +3 -3
- benchmark_agentic_llm.py +1 -1
- benchmark_llm.py +1 -1
README.md
CHANGED
|
@@ -125,7 +125,7 @@ uvicorn server:app --host 0.0.0.0 --port 7860
|
|
| 125 |
|
| 126 |
# 3. In another shell, run the baseline inference
|
| 127 |
export API_BASE_URL=https://router.huggingface.co/v1
|
| 128 |
-
export MODEL_NAME=
|
| 129 |
export HF_TOKEN=<your-hf-token>
|
| 130 |
export ESC_ENV_URL=http://127.0.0.1:7860
|
| 131 |
python3 inference.py
|
|
@@ -189,7 +189,7 @@ When you have a real model endpoint and token, run:
|
|
| 189 |
|
| 190 |
```bash
|
| 191 |
export API_BASE_URL=https://router.huggingface.co/v1
|
| 192 |
-
export MODEL_NAME=
|
| 193 |
export HF_TOKEN=<your-hf-token>
|
| 194 |
export ESC_ENV_URL=http://127.0.0.1:7860
|
| 195 |
python3 benchmark_llm.py
|
|
@@ -207,7 +207,7 @@ traces around the model:
|
|
| 207 |
|
| 208 |
```bash
|
| 209 |
export API_BASE_URL=https://router.huggingface.co/v1
|
| 210 |
-
export MODEL_NAME=
|
| 211 |
export HF_TOKEN=<your-hf-token>
|
| 212 |
export ESC_ENV_URL=http://127.0.0.1:7860
|
| 213 |
python3 benchmark_agentic_llm.py
|
|
@@ -221,8 +221,8 @@ Outputs:
|
|
| 221 |
## Baseline scores
|
| 222 |
|
| 223 |
Deterministic local numbers below were generated with `py -3 benchmark.py`.
|
| 224 |
-
|
| 225 |
-
`
|
| 226 |
|
| 227 |
### Deterministic baselines
|
| 228 |
|
|
@@ -238,18 +238,17 @@ Add real model rows after running `benchmark_llm.py` and
|
|
| 238 |
| --- | ---: | ---: | --- |
|
| 239 |
| `skill_routed_deterministic` | 0.821 | 1.00 | Explicit router over `empathize` / `validate` / `explore` / `plan` / `safety_escalate`; matches the strong staged baseline while exposing route traces |
|
| 240 |
|
| 241 |
-
###
|
| 242 |
|
| 243 |
-
| Model
|
| 244 |
-
| ---
|
| 245 |
-
| `
|
| 246 |
|
| 247 |
The deterministic ladder separates surface-level empathy from task completion:
|
| 248 |
the generic repeated-empathy template does not solve any task, while the
|
| 249 |
-
stage-aware heuristic completes all three. The `
|
| 250 |
-
|
| 251 |
-
|
| 252 |
-
reflection.
|
| 253 |
|
| 254 |
## Files
|
| 255 |
|
|
|
|
| 125 |
|
| 126 |
# 3. In another shell, run the baseline inference
|
| 127 |
export API_BASE_URL=https://router.huggingface.co/v1
|
| 128 |
+
export MODEL_NAME=gpt-4.1-mini
|
| 129 |
export HF_TOKEN=<your-hf-token>
|
| 130 |
export ESC_ENV_URL=http://127.0.0.1:7860
|
| 131 |
python3 inference.py
|
|
|
|
| 189 |
|
| 190 |
```bash
|
| 191 |
export API_BASE_URL=https://router.huggingface.co/v1
|
| 192 |
+
export MODEL_NAME=gpt-4.1-mini
|
| 193 |
export HF_TOKEN=<your-hf-token>
|
| 194 |
export ESC_ENV_URL=http://127.0.0.1:7860
|
| 195 |
python3 benchmark_llm.py
|
|
|
|
| 207 |
|
| 208 |
```bash
|
| 209 |
export API_BASE_URL=https://router.huggingface.co/v1
|
| 210 |
+
export MODEL_NAME=gpt-4.1-mini
|
| 211 |
export HF_TOKEN=<your-hf-token>
|
| 212 |
export ESC_ENV_URL=http://127.0.0.1:7860
|
| 213 |
python3 benchmark_agentic_llm.py
|
|
|
|
| 221 |
## Baseline scores
|
| 222 |
|
| 223 |
Deterministic local numbers below were generated with `py -3 benchmark.py`.
|
| 224 |
+
The submitted hosted baseline below comes from a live `inference.py` run
|
| 225 |
+
against the deployed Hugging Face Space using `gpt-4.1-mini`.
|
| 226 |
|
| 227 |
### Deterministic baselines
|
| 228 |
|
|
|
|
| 238 |
| --- | ---: | ---: | --- |
|
| 239 |
| `skill_routed_deterministic` | 0.821 | 1.00 | Explicit router over `empathize` / `validate` / `explore` / `plan` / `safety_escalate`; matches the strong staged baseline while exposing route traces |
|
| 240 |
|
| 241 |
+
### Submitted Hosted LLM Baseline
|
| 242 |
|
| 243 |
+
| Model | Avg score | Success rate | Notes |
|
| 244 |
+
| --- | ---: | ---: | --- |
|
| 245 |
+
| `gpt-4.1-mini` | 0.821 | 1.00 | Live `inference.py` run against [`5ivatej-meta-hackathon.hf.space`](https://5ivatej-meta-hackathon.hf.space) |
|
| 246 |
|
| 247 |
The deterministic ladder separates surface-level empathy from task completion:
|
| 248 |
the generic repeated-empathy template does not solve any task, while the
|
| 249 |
+
stage-aware heuristic completes all three. The submitted `gpt-4.1-mini`
|
| 250 |
+
baseline also completes all three tasks because the policy-side controller
|
| 251 |
+
keeps the conversation stage-aware instead of drifting into endless reflection.
|
|
|
|
| 252 |
|
| 253 |
## Files
|
| 254 |
|
SUBMISSION_NEXT_STEPS.md
CHANGED
|
@@ -41,7 +41,7 @@ This is the script the submission already exposes.
|
|
| 41 |
|
| 42 |
```powershell
|
| 43 |
$env:API_BASE_URL="https://router.huggingface.co/v1"
|
| 44 |
-
$env:MODEL_NAME="
|
| 45 |
$env:HF_TOKEN="<your-token>"
|
| 46 |
$env:ESC_ENV_URL="http://127.0.0.1:7860"
|
| 47 |
py -3 inference.py
|
|
@@ -55,7 +55,7 @@ Use this when you want a reusable results file for the README or final report.
|
|
| 55 |
|
| 56 |
```powershell
|
| 57 |
$env:API_BASE_URL="https://router.huggingface.co/v1"
|
| 58 |
-
$env:MODEL_NAME="
|
| 59 |
$env:HF_TOKEN="<your-token>"
|
| 60 |
$env:ESC_ENV_URL="http://127.0.0.1:7860"
|
| 61 |
py -3 benchmark_llm.py
|
|
@@ -74,7 +74,7 @@ endpoint instead of a local-only model server.
|
|
| 74 |
|
| 75 |
```powershell
|
| 76 |
$env:API_BASE_URL="https://router.huggingface.co/v1"
|
| 77 |
-
$env:MODEL_NAME="
|
| 78 |
$env:HF_TOKEN="<your-token>"
|
| 79 |
$env:ESC_ENV_URL="http://127.0.0.1:7860"
|
| 80 |
py -3 benchmark_agentic_llm.py
|
|
|
|
| 41 |
|
| 42 |
```powershell
|
| 43 |
$env:API_BASE_URL="https://router.huggingface.co/v1"
|
| 44 |
+
$env:MODEL_NAME="gpt-4.1-mini"
|
| 45 |
$env:HF_TOKEN="<your-token>"
|
| 46 |
$env:ESC_ENV_URL="http://127.0.0.1:7860"
|
| 47 |
py -3 inference.py
|
|
|
|
| 55 |
|
| 56 |
```powershell
|
| 57 |
$env:API_BASE_URL="https://router.huggingface.co/v1"
|
| 58 |
+
$env:MODEL_NAME="gpt-4.1-mini"
|
| 59 |
$env:HF_TOKEN="<your-token>"
|
| 60 |
$env:ESC_ENV_URL="http://127.0.0.1:7860"
|
| 61 |
py -3 benchmark_llm.py
|
|
|
|
| 74 |
|
| 75 |
```powershell
|
| 76 |
$env:API_BASE_URL="https://router.huggingface.co/v1"
|
| 77 |
+
$env:MODEL_NAME="gpt-4.1-mini"
|
| 78 |
$env:HF_TOKEN="<your-token>"
|
| 79 |
$env:ESC_ENV_URL="http://127.0.0.1:7860"
|
| 80 |
py -3 benchmark_agentic_llm.py
|
benchmark_agentic_llm.py
CHANGED
|
@@ -15,7 +15,7 @@ Authentication variables:
|
|
| 15 |
|
| 16 |
Example:
|
| 17 |
export API_BASE_URL=https://router.huggingface.co/v1
|
| 18 |
-
export MODEL_NAME=
|
| 19 |
export HF_TOKEN=<your-token>
|
| 20 |
export ESC_ENV_URL=http://127.0.0.1:7860
|
| 21 |
python3 benchmark_agentic_llm.py
|
|
|
|
| 15 |
|
| 16 |
Example:
|
| 17 |
export API_BASE_URL=https://router.huggingface.co/v1
|
| 18 |
+
export MODEL_NAME=gpt-4.1-mini
|
| 19 |
export HF_TOKEN=<your-token>
|
| 20 |
export ESC_ENV_URL=http://127.0.0.1:7860
|
| 21 |
python3 benchmark_agentic_llm.py
|
benchmark_llm.py
CHANGED
|
@@ -16,7 +16,7 @@ Authentication variables:
|
|
| 16 |
|
| 17 |
Example:
|
| 18 |
export API_BASE_URL=https://router.huggingface.co/v1
|
| 19 |
-
export MODEL_NAME=
|
| 20 |
export HF_TOKEN=<your-token>
|
| 21 |
export ESC_ENV_URL=http://127.0.0.1:7860
|
| 22 |
python3 benchmark_llm.py
|
|
|
|
| 16 |
|
| 17 |
Example:
|
| 18 |
export API_BASE_URL=https://router.huggingface.co/v1
|
| 19 |
+
export MODEL_NAME=gpt-4.1-mini
|
| 20 |
export HF_TOKEN=<your-token>
|
| 21 |
export ESC_ENV_URL=http://127.0.0.1:7860
|
| 22 |
python3 benchmark_llm.py
|