Spaces:
Sleeping
Sleeping
Agentics flow for different planners
Browse files- README.md +153 -94
- SUBMISSION_NEXT_STEPS.md +58 -19
- benchmark_agentic.py +262 -0
- benchmark_agentic_llm.py +356 -0
- results/agentic_benchmarks.json +190 -0
- results/agentic_benchmarks.md +146 -0
- src/agentic.py +366 -0
README.md
CHANGED
|
@@ -1,125 +1,118 @@
|
|
| 1 |
---
|
| 2 |
title: Emotional Support Conversations (OpenEnv)
|
| 3 |
-
emoji: 💬
|
| 4 |
sdk: docker
|
| 5 |
pinned: false
|
| 6 |
tags:
|
| 7 |
- openenv
|
| 8 |
---
|
| 9 |
|
| 10 |
-
# Emotional Support Conversations
|
| 11 |
|
| 12 |
-
> An OpenEnv RL environment for evaluating agents on
|
| 13 |
-
> support conversations
|
| 14 |
-
> signal inspired by
|
| 15 |
> [arXiv:2508.12935](https://arxiv.org/abs/2508.12935)).
|
| 16 |
|
| 17 |
## Why this environment
|
| 18 |
|
| 19 |
Emotional support is one of the tasks humans most want AI assistants to do
|
| 20 |
-
well
|
| 21 |
-
score turn-level responses in isolation, which rewards agents for
|
| 22 |
-
empathetic without
|
| 23 |
-
|
| 24 |
|
| 25 |
Three properties make it a genuine RL problem, not a single-shot dialogue
|
| 26 |
task:
|
| 27 |
|
| 28 |
-
1.
|
| 29 |
-
|
| 30 |
-
|
| 31 |
-
2.
|
| 32 |
-
disclosure at turn 6. A single dismissive reply at turn 4 can collapse
|
| 33 |
-
|
| 34 |
-
3.
|
| 35 |
-
|
| 36 |
-
|
| 37 |
|
| 38 |
## Reward design (RLFF-ESC-inspired)
|
| 39 |
|
| 40 |
Each step reward is:
|
| 41 |
|
| 42 |
-
```
|
| 43 |
-
step_reward = clip(
|
| 44 |
```
|
| 45 |
|
| 46 |
-
-
|
| 47 |
-
|
| 48 |
-
-
|
| 49 |
post-action seeker states. The reward is proportional to how much the
|
| 50 |
-
agent's action
|
| 51 |
-
|
| 52 |
-
|
| 53 |
-
|
| 54 |
-
- **`penalties`** — dismissive language, premature advice (before trust is
|
| 55 |
-
established), bare replies, interrogation, and repeated template-like
|
| 56 |
-
responses.
|
| 57 |
|
| 58 |
A final task score combines average shaped reward, the seeker's final
|
| 59 |
-
resolution state, efficiency
|
| 60 |
-
|
| 61 |
-
|
| 62 |
-
does **not** count as a solved episode.
|
| 63 |
|
| 64 |
## Tasks (3 difficulties)
|
| 65 |
|
| 66 |
-
| Task ID
|
| 67 |
-
| ---
|
| 68 |
-
| `work_stress_venting`
|
| 69 |
-
| `guarded_relationship`
|
| 70 |
-
| `crisis_fragile_trust`
|
| 71 |
|
| 72 |
-
Success thresholds (final score) are `0.60 / 0.62 / 0.65` respectively
|
| 73 |
-
|
| 74 |
-
conditions are met.
|
| 75 |
|
| 76 |
-
## Action
|
| 77 |
|
| 78 |
-
|
| 79 |
|
| 80 |
```python
|
| 81 |
class Action(BaseModel):
|
| 82 |
message: str
|
| 83 |
```
|
| 84 |
|
| 85 |
-
|
| 86 |
|
| 87 |
```python
|
| 88 |
class Observation(BaseModel):
|
| 89 |
seeker_utterance: str
|
| 90 |
turn: int
|
| 91 |
remaining_turns: int
|
| 92 |
-
stage_hint: str
|
| 93 |
task_id: str
|
| 94 |
scenario_brief: str
|
| 95 |
```
|
| 96 |
|
| 97 |
-
The seeker's internal hidden variables
|
| 98 |
-
issue) are **never** exposed. This is by design.
|
| 99 |
|
| 100 |
-
## Environment internals
|
| 101 |
|
| 102 |
The seeker is a deterministic finite-state machine with continuous hidden
|
| 103 |
variables (`distress`, `trust`, `openness`, `revealed`, `stage`). On each
|
| 104 |
-
turn, the agent's reply is
|
| 105 |
-
|
| 106 |
-
language, interrogation) and hidden state advances via transparent rules.
|
| 107 |
|
| 108 |
-
|
| 109 |
-
|
| 110 |
-
|
| 111 |
-
|
| 112 |
-
|
| 113 |
|
| 114 |
## HTTP API (OpenEnv spec)
|
| 115 |
|
| 116 |
-
| Method | Path
|
| 117 |
-
| ---
|
| 118 |
-
| GET
|
| 119 |
-
| GET
|
| 120 |
-
| POST
|
| 121 |
-
| POST
|
| 122 |
-
| GET
|
| 123 |
|
| 124 |
## Running locally
|
| 125 |
|
|
@@ -145,11 +138,24 @@ docker build -t esc-openenv .
|
|
| 145 |
docker run -p 7860:7860 esc-openenv
|
| 146 |
```
|
| 147 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 148 |
## Benchmarking
|
| 149 |
|
| 150 |
### Deterministic local benchmark ladder
|
| 151 |
|
| 152 |
-
Run the built-in
|
| 153 |
|
| 154 |
```bash
|
| 155 |
py -3 benchmark.py
|
|
@@ -160,6 +166,19 @@ Outputs:
|
|
| 160 |
- `results/local_benchmarks.md`
|
| 161 |
- `results/local_benchmarks.json`
|
| 162 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 163 |
### LLM benchmark with Markdown output
|
| 164 |
|
| 165 |
When you have a real model endpoint and token, run:
|
|
@@ -177,53 +196,93 @@ Outputs:
|
|
| 177 |
- `results/llm_benchmark.md`
|
| 178 |
- `results/llm_benchmark.json`
|
| 179 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 180 |
## Baseline scores
|
| 181 |
|
| 182 |
Deterministic local numbers below were generated with `py -3 benchmark.py`.
|
| 183 |
-
Add
|
|
|
|
| 184 |
|
| 185 |
### Deterministic baselines
|
| 186 |
|
| 187 |
-
| Baseline
|
| 188 |
-
| ---
|
| 189 |
-
| `generic_template`
|
| 190 |
-
| `validation_only`
|
| 191 |
-
| `stage_aware_heuristic` | 0.821
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 192 |
|
| 193 |
### Real LLM baseline
|
| 194 |
|
| 195 |
-
| Model
|
| 196 |
-
| ---
|
| 197 |
| `Qwen/Qwen2.5-72B-Instruct` (or chosen final model) | TBD | TBD | Run `py -3 benchmark_llm.py` |
|
| 198 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 199 |
## Files
|
| 200 |
|
| 201 |
-
```
|
| 202 |
.
|
| 203 |
-
|
| 204 |
-
|
| 205 |
-
|
| 206 |
-
|
| 207 |
-
|
| 208 |
-
|
| 209 |
-
|
| 210 |
-
|
| 211 |
-
|
| 212 |
-
|
| 213 |
-
|
| 214 |
-
|
| 215 |
-
|
| 216 |
-
|
| 217 |
-
|
| 218 |
-
|
| 219 |
-
|
| 220 |
-
|
|
|
|
|
|
|
|
|
|
| 221 |
```
|
| 222 |
|
| 223 |
## Citation
|
| 224 |
|
| 225 |
-
If you use this environment, please cite the paper whose reward idea
|
| 226 |
-
|
| 227 |
|
| 228 |
```bibtex
|
| 229 |
@article{yang2025rlffesc,
|
|
|
|
| 1 |
---
|
| 2 |
title: Emotional Support Conversations (OpenEnv)
|
| 3 |
+
emoji: "💬"
|
| 4 |
sdk: docker
|
| 5 |
pinned: false
|
| 6 |
tags:
|
| 7 |
- openenv
|
| 8 |
---
|
| 9 |
|
| 10 |
+
# Emotional Support Conversations - OpenEnv Environment
|
| 11 |
|
| 12 |
+
> An OpenEnv RL environment for evaluating agents on open-ended emotional
|
| 13 |
+
> support conversations, with a hybrid immediate + future-oriented reward
|
| 14 |
+
> signal inspired by RLFF-ESC (Yang, Chen, Wang, 2025,
|
| 15 |
> [arXiv:2508.12935](https://arxiv.org/abs/2508.12935)).
|
| 16 |
|
| 17 |
## Why this environment
|
| 18 |
|
| 19 |
Emotional support is one of the tasks humans most want AI assistants to do
|
| 20 |
+
well, and one of the easiest to do badly. Existing dialogue benchmarks often
|
| 21 |
+
score turn-level responses in isolation, which rewards agents for sounding
|
| 22 |
+
empathetic without testing whether their replies actually move the person
|
| 23 |
+
toward resolution. This environment closes that gap.
|
| 24 |
|
| 25 |
Three properties make it a genuine RL problem, not a single-shot dialogue
|
| 26 |
task:
|
| 27 |
|
| 28 |
+
1. Partial observability. The seeker's distress, trust, and willingness to
|
| 29 |
+
reveal their real issue are hidden state. The agent must infer them from
|
| 30 |
+
the conversation so far.
|
| 31 |
+
2. Sequential credit assignment. A warm reply at turn 2 can unlock a
|
| 32 |
+
disclosure at turn 6. A single dismissive reply at turn 4 can collapse the
|
| 33 |
+
whole trajectory and require several turns to recover.
|
| 34 |
+
3. Exploration vs commitment. Should the agent keep exploring feelings or move
|
| 35 |
+
toward an action plan? Commit too early and the seeker shuts down; explore
|
| 36 |
+
too long and the episode times out.
|
| 37 |
|
| 38 |
## Reward design (RLFF-ESC-inspired)
|
| 39 |
|
| 40 |
Each step reward is:
|
| 41 |
|
| 42 |
+
```text
|
| 43 |
+
step_reward = clip(0.45 * immediate + 0.55 * future_oriented - penalties, 0, 1)
|
| 44 |
```
|
| 45 |
|
| 46 |
+
- `immediate`: stage-appropriate empathy/validation/open-question fit, plus
|
| 47 |
+
turn-level deltas in the seeker's trust and distress.
|
| 48 |
+
- `future_oriented`: a k-step oracle rollout from both the pre- and
|
| 49 |
post-action seeker states. The reward is proportional to how much the
|
| 50 |
+
agent's action preserves or advances the attainable resolution ceiling, not
|
| 51 |
+
just how good the current turn looks in isolation.
|
| 52 |
+
- `penalties`: dismissive language, premature advice, bare replies,
|
| 53 |
+
interrogation, and repeated template-like responses.
|
|
|
|
|
|
|
|
|
|
| 54 |
|
| 55 |
A final task score combines average shaped reward, the seeker's final
|
| 56 |
+
resolution state, efficiency, and a completion bonus. Success is hard-gated:
|
| 57 |
+
timing out with a generic but non-harmful conversation can still earn partial
|
| 58 |
+
score, but it does not count as a solved episode.
|
|
|
|
| 59 |
|
| 60 |
## Tasks (3 difficulties)
|
| 61 |
|
| 62 |
+
| Task ID | Difficulty | Max turns | Core challenge |
|
| 63 |
+
| --- | --- | ---: | --- |
|
| 64 |
+
| `work_stress_venting` | easy | 10 | Cooperative seeker venting about work. Must reach closing with trust >= 0.70 and distress <= 0.40. |
|
| 65 |
+
| `guarded_relationship` | medium | 12 | Guarded seeker; real issue is hidden behind the surface concern until openness >= 0.75. Must reveal the true issue and finish in closing with trust >= 0.72 and distress <= 0.45. |
|
| 66 |
+
| `crisis_fragile_trust` | hard | 14 | High-distress, fragile trust, multiple interleaved concerns. Must reveal the crisis concern, reference external safety support, and finish in closing with trust >= 0.75 and distress <= 0.40. |
|
| 67 |
|
| 68 |
+
Success thresholds (final score) are `0.60 / 0.62 / 0.65` respectively, and
|
| 69 |
+
they are only evaluated after the task-specific completion conditions are met.
|
|
|
|
| 70 |
|
| 71 |
+
## Action and observation space
|
| 72 |
|
| 73 |
+
Action is a free-text reply to the seeker:
|
| 74 |
|
| 75 |
```python
|
| 76 |
class Action(BaseModel):
|
| 77 |
message: str
|
| 78 |
```
|
| 79 |
|
| 80 |
+
Observation is deliberately partial:
|
| 81 |
|
| 82 |
```python
|
| 83 |
class Observation(BaseModel):
|
| 84 |
seeker_utterance: str
|
| 85 |
turn: int
|
| 86 |
remaining_turns: int
|
| 87 |
+
stage_hint: str
|
| 88 |
task_id: str
|
| 89 |
scenario_brief: str
|
| 90 |
```
|
| 91 |
|
| 92 |
+
The seeker's internal hidden variables are never exposed.
|
|
|
|
| 93 |
|
| 94 |
+
## Environment internals
|
| 95 |
|
| 96 |
The seeker is a deterministic finite-state machine with continuous hidden
|
| 97 |
variables (`distress`, `trust`, `openness`, `revealed`, `stage`). On each
|
| 98 |
+
turn, the agent's reply is analyzed with keyword and regex feature detectors,
|
| 99 |
+
then hidden state advances via transparent rules.
|
|
|
|
| 100 |
|
| 101 |
+
Why not use an LLM-driven seeker? The hackathon rubric requires graders to be
|
| 102 |
+
deterministic and reproducible. An LLM-driven seeker would risk score variance
|
| 103 |
+
between runs. Deterministic dynamics give full reproducibility while still
|
| 104 |
+
producing rich, sequential, partially observable dialogue with genuine
|
| 105 |
+
recovery-from-mistakes dynamics.
|
| 106 |
|
| 107 |
## HTTP API (OpenEnv spec)
|
| 108 |
|
| 109 |
+
| Method | Path | Body | Returns |
|
| 110 |
+
| --- | --- | --- | --- |
|
| 111 |
+
| `GET` | `/` | none | health + metadata |
|
| 112 |
+
| `GET` | `/tasks` | none | list of tasks |
|
| 113 |
+
| `POST` | `/reset` | `{"task_id": "...", "seed": null}` | `ResetResult` |
|
| 114 |
+
| `POST` | `/step` | `{"action": {"message": "..."}}` | `StepResult` |
|
| 115 |
+
| `GET` | `/state` | none | `EnvState` |
|
| 116 |
|
| 117 |
## Running locally
|
| 118 |
|
|
|
|
| 138 |
docker run -p 7860:7860 esc-openenv
|
| 139 |
```
|
| 140 |
|
| 141 |
+
## Skills / agents extension
|
| 142 |
+
|
| 143 |
+
The environment itself stays deterministic and reproducible. To align with the
|
| 144 |
+
hackathon's optional skills/agents framing, this repo also includes a
|
| 145 |
+
policy-side agentic controller that routes between five reusable skills:
|
| 146 |
+
`empathize`, `validate`, `explore`, `plan`, and `safety_escalate`.
|
| 147 |
+
|
| 148 |
+
This keeps the benchmark honest:
|
| 149 |
+
|
| 150 |
+
- the environment and grader remain unchanged
|
| 151 |
+
- the agentic story lives in the policy, not in a hidden stochastic seeker
|
| 152 |
+
- judges can inspect turn-by-turn routing traces in the benchmark outputs
|
| 153 |
+
|
| 154 |
## Benchmarking
|
| 155 |
|
| 156 |
### Deterministic local benchmark ladder
|
| 157 |
|
| 158 |
+
Run the built-in rubric ladder and write reusable Markdown/JSON artifacts:
|
| 159 |
|
| 160 |
```bash
|
| 161 |
py -3 benchmark.py
|
|
|
|
| 166 |
- `results/local_benchmarks.md`
|
| 167 |
- `results/local_benchmarks.json`
|
| 168 |
|
| 169 |
+
### Deterministic skill-routed benchmark
|
| 170 |
+
|
| 171 |
+
Run the explicit agentic baseline comparison and write route-aware artifacts:
|
| 172 |
+
|
| 173 |
+
```bash
|
| 174 |
+
py -3 benchmark_agentic.py
|
| 175 |
+
```
|
| 176 |
+
|
| 177 |
+
Outputs:
|
| 178 |
+
|
| 179 |
+
- `results/agentic_benchmarks.md`
|
| 180 |
+
- `results/agentic_benchmarks.json`
|
| 181 |
+
|
| 182 |
### LLM benchmark with Markdown output
|
| 183 |
|
| 184 |
When you have a real model endpoint and token, run:
|
|
|
|
| 196 |
- `results/llm_benchmark.md`
|
| 197 |
- `results/llm_benchmark.json`
|
| 198 |
|
| 199 |
+
### Skill-routed LLM benchmark
|
| 200 |
+
|
| 201 |
+
Use the same environment endpoint, but add the policy-side router and skill
|
| 202 |
+
traces around the model:
|
| 203 |
+
|
| 204 |
+
```bash
|
| 205 |
+
export API_BASE_URL=https://router.huggingface.co/v1
|
| 206 |
+
export MODEL_NAME=Qwen/Qwen2.5-72B-Instruct
|
| 207 |
+
export HF_TOKEN=<your-hf-token>
|
| 208 |
+
export ESC_ENV_URL=http://localhost:7860
|
| 209 |
+
py -3 benchmark_agentic_llm.py
|
| 210 |
+
```
|
| 211 |
+
|
| 212 |
+
Outputs:
|
| 213 |
+
|
| 214 |
+
- `results/agentic_llm_benchmark.md`
|
| 215 |
+
- `results/agentic_llm_benchmark.json`
|
| 216 |
+
|
| 217 |
## Baseline scores
|
| 218 |
|
| 219 |
Deterministic local numbers below were generated with `py -3 benchmark.py`.
|
| 220 |
+
Add real model rows after running `benchmark_llm.py` and
|
| 221 |
+
`benchmark_agentic_llm.py`.
|
| 222 |
|
| 223 |
### Deterministic baselines
|
| 224 |
|
| 225 |
+
| Baseline | Avg score | Success rate | Notes |
|
| 226 |
+
| --- | ---: | ---: | --- |
|
| 227 |
+
| `generic_template` | 0.393 | 0.00 | Safe-sounding repeated empathy; no task completion |
|
| 228 |
+
| `validation_only` | 0.539 | 0.00 | Better partial reward, still fails hard-gated completion |
|
| 229 |
+
| `stage_aware_heuristic` | 0.821 | 1.00 | Task-aware staged policy; completes all 3 tasks |
|
| 230 |
+
|
| 231 |
+
### Skill-routed agentic baselines
|
| 232 |
+
|
| 233 |
+
| Baseline | Avg score | Success rate | Notes |
|
| 234 |
+
| --- | ---: | ---: | --- |
|
| 235 |
+
| `skill_routed_deterministic` | 0.821 | 1.00 | Explicit router over `empathize` / `validate` / `explore` / `plan` / `safety_escalate`; matches the strong staged baseline while exposing route traces |
|
| 236 |
|
| 237 |
### Real LLM baseline
|
| 238 |
|
| 239 |
+
| Model | Avg score | Success rate | Report |
|
| 240 |
+
| --- | ---: | ---: | --- |
|
| 241 |
| `Qwen/Qwen2.5-72B-Instruct` (or chosen final model) | TBD | TBD | Run `py -3 benchmark_llm.py` |
|
| 242 |
|
| 243 |
+
### Skill-routed real LLM baseline
|
| 244 |
+
|
| 245 |
+
| Model | Avg score | Success rate | Report |
|
| 246 |
+
| --- | ---: | ---: | --- |
|
| 247 |
+
| `Qwen/Qwen2.5-72B-Instruct` (or chosen final model) + router | TBD | TBD | Run `py -3 benchmark_agentic_llm.py` |
|
| 248 |
+
|
| 249 |
+
## Benchmark narrative
|
| 250 |
+
|
| 251 |
+
- The generic repeated-empathy template now fails completely under the hardened rubric.
|
| 252 |
+
- The deterministic skill-routed policy matches the strong staged heuristic at `0.821` average score and `1.00` success rate, while making turn-level skill choices visible.
|
| 253 |
+
- The hard task is only successful when the trajectory reaches a safety-aware finish, so sounding supportive without escalation is not enough.
|
| 254 |
+
|
| 255 |
## Files
|
| 256 |
|
| 257 |
+
```text
|
| 258 |
.
|
| 259 |
+
|-- openenv.yaml # OpenEnv metadata
|
| 260 |
+
|-- Dockerfile # Container build for HF Space
|
| 261 |
+
|-- benchmark.py # Deterministic local benchmark ladder
|
| 262 |
+
|-- benchmark_agentic.py # Deterministic skill-routed benchmark
|
| 263 |
+
|-- benchmark_agentic_llm.py # Skill-routed LLM benchmark
|
| 264 |
+
|-- benchmark_llm.py # LLM benchmark that writes Markdown/JSON
|
| 265 |
+
|-- requirements.txt
|
| 266 |
+
|-- server.py # FastAPI HTTP server (entrypoint)
|
| 267 |
+
|-- inference.py # Mandated baseline inference script
|
| 268 |
+
|-- SUBMISSION_NEXT_STEPS.md # Manual checklist before final submission
|
| 269 |
+
|-- README.md
|
| 270 |
+
`-- src/
|
| 271 |
+
|-- __init__.py
|
| 272 |
+
|-- agentic.py # Skill router + reusable policy-side skills
|
| 273 |
+
|-- baselines.py # Deterministic baseline policies
|
| 274 |
+
|-- models.py # Pydantic Action / Observation / Reward / envelopes
|
| 275 |
+
|-- seeker.py # Deterministic seeker simulator + feature detectors
|
| 276 |
+
|-- tasks.py # 3 task personas (easy / medium / hard)
|
| 277 |
+
|-- grader.py # Hybrid immediate + future-oriented reward
|
| 278 |
+
|-- env.py # Core ESCEnv with step/reset/state
|
| 279 |
+
`-- client.py # Async HTTP client for inference.py
|
| 280 |
```
|
| 281 |
|
| 282 |
## Citation
|
| 283 |
|
| 284 |
+
If you use this environment, please cite the paper whose reward idea inspired
|
| 285 |
+
it:
|
| 286 |
|
| 287 |
```bibtex
|
| 288 |
@article{yang2025rlffesc,
|
SUBMISSION_NEXT_STEPS.md
CHANGED
|
@@ -1,12 +1,12 @@
|
|
| 1 |
# Submission Next Steps
|
| 2 |
|
| 3 |
-
This
|
| 4 |
-
|
| 5 |
|
| 6 |
-
## 1. Run the
|
| 7 |
|
| 8 |
-
This
|
| 9 |
-
|
| 10 |
|
| 11 |
```powershell
|
| 12 |
py -3 benchmark.py
|
|
@@ -17,12 +17,25 @@ Artifacts written:
|
|
| 17 |
- `results/local_benchmarks.md`
|
| 18 |
- `results/local_benchmarks.json`
|
| 19 |
|
| 20 |
-
|
| 21 |
|
| 22 |
-
|
| 23 |
-
- paste the final baseline numbers into [README.md](C:\Users\Gokul nandan T M\Desktop\personalprojects\meta\meta-hackathon\README.md)
|
| 24 |
|
| 25 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 26 |
|
| 27 |
This is the script the submission already exposes.
|
| 28 |
|
|
@@ -36,7 +49,7 @@ py -3 inference.py
|
|
| 36 |
|
| 37 |
Use this when you want the strict `[START] / [STEP] / [END]` output format.
|
| 38 |
|
| 39 |
-
##
|
| 40 |
|
| 41 |
Use this when you want a reusable results file for the README or final report.
|
| 42 |
|
|
@@ -53,29 +66,54 @@ Artifacts written:
|
|
| 53 |
- `results/llm_benchmark.md`
|
| 54 |
- `results/llm_benchmark.json`
|
| 55 |
|
| 56 |
-
##
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 57 |
|
| 58 |
Update the `Baseline scores` section in [README.md](C:\Users\Gokul nandan T M\Desktop\personalprojects\meta\meta-hackathon\README.md) with:
|
| 59 |
|
| 60 |
- deterministic local baselines from `benchmark.py`
|
|
|
|
| 61 |
- one real LLM baseline from `benchmark_llm.py`
|
|
|
|
| 62 |
|
| 63 |
-
Recommended final
|
| 64 |
|
| 65 |
- `generic_template`
|
| 66 |
- `validation_only`
|
| 67 |
- `stage_aware_heuristic`
|
|
|
|
| 68 |
- one real LLM baseline
|
|
|
|
| 69 |
|
| 70 |
-
##
|
| 71 |
|
| 72 |
Keep it brief. Include:
|
| 73 |
|
| 74 |
- the generic repeated empathy template no longer succeeds
|
| 75 |
- the stage-aware heuristic completes all tasks
|
| 76 |
-
- the
|
|
|
|
| 77 |
|
| 78 |
-
##
|
| 79 |
|
| 80 |
Before submitting, verify both local and containerized runs.
|
| 81 |
|
|
@@ -92,20 +130,21 @@ Then hit:
|
|
| 92 |
- `POST /step`
|
| 93 |
- `GET /state`
|
| 94 |
|
| 95 |
-
##
|
| 96 |
|
| 97 |
If you still have time, these are the best improvements:
|
| 98 |
|
| 99 |
- add one screenshot or short GIF of a successful hard-task trajectory
|
| 100 |
- add a tiny `Why this benchmark is hard` section to the README
|
| 101 |
-
- add one
|
| 102 |
|
| 103 |
-
##
|
| 104 |
|
| 105 |
Make sure these are true:
|
| 106 |
|
| 107 |
- local benchmark artifacts exist in `results/`
|
| 108 |
-
-
|
|
|
|
| 109 |
- README contains real numbers, not `TBD`
|
| 110 |
- Docker build works
|
| 111 |
- the hard task is only successful when safety support is referenced
|
|
|
|
| 1 |
# Submission Next Steps
|
| 2 |
|
| 3 |
+
This is the short operational checklist for turning the current repo into a
|
| 4 |
+
polished hackathon submission.
|
| 5 |
|
| 6 |
+
## 1. Run the core deterministic benchmark ladder
|
| 7 |
|
| 8 |
+
This proves the rubric separates weak generic empathy from staged,
|
| 9 |
+
task-aware behavior.
|
| 10 |
|
| 11 |
```powershell
|
| 12 |
py -3 benchmark.py
|
|
|
|
| 17 |
- `results/local_benchmarks.md`
|
| 18 |
- `results/local_benchmarks.json`
|
| 19 |
|
| 20 |
+
## 2. Run the deterministic skills/agents benchmark
|
| 21 |
|
| 22 |
+
This gives you the policy-side agentic story without changing the environment.
|
|
|
|
| 23 |
|
| 24 |
+
```powershell
|
| 25 |
+
py -3 benchmark_agentic.py
|
| 26 |
+
```
|
| 27 |
+
|
| 28 |
+
Artifacts written:
|
| 29 |
+
|
| 30 |
+
- `results/agentic_benchmarks.md`
|
| 31 |
+
- `results/agentic_benchmarks.json`
|
| 32 |
+
|
| 33 |
+
After running steps 1 and 2:
|
| 34 |
+
|
| 35 |
+
- copy the summary numbers into [README.md](C:\Users\Gokul nandan T M\Desktop\personalprojects\meta\meta-hackathon\README.md)
|
| 36 |
+
- keep both the rubric ladder and the skill-routed results in the final submission
|
| 37 |
+
|
| 38 |
+
## 3. Run the mandatory hackathon stdout-contract baseline
|
| 39 |
|
| 40 |
This is the script the submission already exposes.
|
| 41 |
|
|
|
|
| 49 |
|
| 50 |
Use this when you want the strict `[START] / [STEP] / [END]` output format.
|
| 51 |
|
| 52 |
+
## 4. Run the Markdown-writing LLM benchmark
|
| 53 |
|
| 54 |
Use this when you want a reusable results file for the README or final report.
|
| 55 |
|
|
|
|
| 66 |
- `results/llm_benchmark.md`
|
| 67 |
- `results/llm_benchmark.json`
|
| 68 |
|
| 69 |
+
## 5. Run the skill-routed LLM benchmark
|
| 70 |
+
|
| 71 |
+
Use this when you want an explicit skills/agents baseline with route traces.
|
| 72 |
+
You can use a local OpenAI-compatible endpoint here during development.
|
| 73 |
+
|
| 74 |
+
```powershell
|
| 75 |
+
$env:API_BASE_URL="http://localhost:11434/v1"
|
| 76 |
+
$env:MODEL_NAME="qwen2.5:7b-instruct"
|
| 77 |
+
$env:API_KEY="ollama"
|
| 78 |
+
$env:ESC_ENV_URL="http://localhost:7860"
|
| 79 |
+
py -3 benchmark_agentic_llm.py
|
| 80 |
+
```
|
| 81 |
+
|
| 82 |
+
For the final Hugging Face run, swap in your deployment endpoint and token.
|
| 83 |
+
|
| 84 |
+
Artifacts written:
|
| 85 |
+
|
| 86 |
+
- `results/agentic_llm_benchmark.md`
|
| 87 |
+
- `results/agentic_llm_benchmark.json`
|
| 88 |
+
|
| 89 |
+
## 6. Replace the `TBD` rows in the README
|
| 90 |
|
| 91 |
Update the `Baseline scores` section in [README.md](C:\Users\Gokul nandan T M\Desktop\personalprojects\meta\meta-hackathon\README.md) with:
|
| 92 |
|
| 93 |
- deterministic local baselines from `benchmark.py`
|
| 94 |
+
- deterministic skill-routed baseline from `benchmark_agentic.py`
|
| 95 |
- one real LLM baseline from `benchmark_llm.py`
|
| 96 |
+
- one real or local skill-routed LLM baseline from `benchmark_agentic_llm.py`
|
| 97 |
|
| 98 |
+
Recommended final ladder:
|
| 99 |
|
| 100 |
- `generic_template`
|
| 101 |
- `validation_only`
|
| 102 |
- `stage_aware_heuristic`
|
| 103 |
+
- `skill_routed_deterministic`
|
| 104 |
- one real LLM baseline
|
| 105 |
+
- one skill-routed LLM baseline
|
| 106 |
|
| 107 |
+
## 7. Add one short benchmark narrative to the README
|
| 108 |
|
| 109 |
Keep it brief. Include:
|
| 110 |
|
| 111 |
- the generic repeated empathy template no longer succeeds
|
| 112 |
- the stage-aware heuristic completes all tasks
|
| 113 |
+
- the skill-routed policy keeps similar performance while exposing turn-level routing decisions
|
| 114 |
+
- the hard task requires an explicit safety-aware finish
|
| 115 |
|
| 116 |
+
## 8. Smoke-test the deployable artifact
|
| 117 |
|
| 118 |
Before submitting, verify both local and containerized runs.
|
| 119 |
|
|
|
|
| 130 |
- `POST /step`
|
| 131 |
- `GET /state`
|
| 132 |
|
| 133 |
+
## 9. Optional but high-value polish
|
| 134 |
|
| 135 |
If you still have time, these are the best improvements:
|
| 136 |
|
| 137 |
- add one screenshot or short GIF of a successful hard-task trajectory
|
| 138 |
- add a tiny `Why this benchmark is hard` section to the README
|
| 139 |
+
- add one short ablation note comparing plain LLM vs skill-routed LLM
|
| 140 |
|
| 141 |
+
## 10. Final pre-submit check
|
| 142 |
|
| 143 |
Make sure these are true:
|
| 144 |
|
| 145 |
- local benchmark artifacts exist in `results/`
|
| 146 |
+
- agentic benchmark artifacts exist in `results/`
|
| 147 |
+
- at least one LLM benchmark artifact exists in `results/`
|
| 148 |
- README contains real numbers, not `TBD`
|
| 149 |
- Docker build works
|
| 150 |
- the hard task is only successful when safety support is referenced
|
benchmark_agentic.py
ADDED
|
@@ -0,0 +1,262 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Run local benchmarks for the explicit skill-routed agentic policies.
|
| 2 |
+
|
| 3 |
+
This script keeps the environment unchanged and benchmarks policy-side agentic
|
| 4 |
+
extensions on top of it. The main goal is to show that a small skill router can
|
| 5 |
+
compose reusable conversational skills while still solving the benchmark.
|
| 6 |
+
|
| 7 |
+
Example:
|
| 8 |
+
py -3 benchmark_agentic.py
|
| 9 |
+
"""
|
| 10 |
+
from __future__ import annotations
|
| 11 |
+
|
| 12 |
+
import argparse
|
| 13 |
+
import json
|
| 14 |
+
from dataclasses import asdict, dataclass, field
|
| 15 |
+
from datetime import datetime, timezone
|
| 16 |
+
from pathlib import Path
|
| 17 |
+
from statistics import mean
|
| 18 |
+
from typing import Any, Dict, List
|
| 19 |
+
|
| 20 |
+
from src.agentic import SkillRoutedDeterministicPolicy
|
| 21 |
+
from src.baselines import StageAwareHeuristicBaseline
|
| 22 |
+
from src.env import ESCEnv
|
| 23 |
+
from src.models import Action
|
| 24 |
+
from src.tasks import TASKS
|
| 25 |
+
|
| 26 |
+
|
| 27 |
+
@dataclass
|
| 28 |
+
class AgenticEpisodeSummary:
|
| 29 |
+
policy: str
|
| 30 |
+
task_id: str
|
| 31 |
+
difficulty: str
|
| 32 |
+
steps: int
|
| 33 |
+
score: float
|
| 34 |
+
success: bool
|
| 35 |
+
completion: float
|
| 36 |
+
avg_step_reward: float
|
| 37 |
+
final_resolution: float
|
| 38 |
+
had_safety_reference: bool
|
| 39 |
+
skill_counts: Dict[str, int] = field(default_factory=dict)
|
| 40 |
+
skill_trace: List[str] = field(default_factory=list)
|
| 41 |
+
transcript_excerpt: List[str] = field(default_factory=list)
|
| 42 |
+
|
| 43 |
+
|
| 44 |
+
def _extract_skill_trace(policy: Any) -> tuple[Dict[str, int], List[str]]:
|
| 45 |
+
if not hasattr(policy, "decision_log") or not hasattr(policy, "memory"):
|
| 46 |
+
return {}, []
|
| 47 |
+
|
| 48 |
+
counts = dict(getattr(policy.memory, "skill_counts", {}))
|
| 49 |
+
trace: List[str] = []
|
| 50 |
+
for entry in getattr(policy, "decision_log", [])[:6]:
|
| 51 |
+
turn = int(entry.get("turn", "0")) + 1
|
| 52 |
+
trace.append(
|
| 53 |
+
f"Turn {turn} [{entry.get('stage', '')}] -> {entry.get('skill', '')}: {entry.get('reason', '')}"
|
| 54 |
+
)
|
| 55 |
+
return counts, trace
|
| 56 |
+
|
| 57 |
+
|
| 58 |
+
def run_episode(env: ESCEnv, policy: Any, task_id: str) -> AgenticEpisodeSummary:
|
| 59 |
+
task = TASKS[task_id]
|
| 60 |
+
policy.reset(task_id)
|
| 61 |
+
reset = env.reset(task_id=task_id)
|
| 62 |
+
obs = reset.observation
|
| 63 |
+
|
| 64 |
+
rewards: List[float] = []
|
| 65 |
+
transcript_excerpt: List[str] = [f"Seeker: {obs.seeker_utterance}"]
|
| 66 |
+
last_result = None
|
| 67 |
+
|
| 68 |
+
while True:
|
| 69 |
+
message = policy.act(obs)
|
| 70 |
+
transcript_excerpt.append(f"Agent: {message}")
|
| 71 |
+
result = env.step(Action(message=message))
|
| 72 |
+
last_result = result
|
| 73 |
+
|
| 74 |
+
rewards.append(float(result.reward))
|
| 75 |
+
obs = result.observation
|
| 76 |
+
transcript_excerpt.append(f"Seeker: {obs.seeker_utterance}")
|
| 77 |
+
|
| 78 |
+
if result.done:
|
| 79 |
+
break
|
| 80 |
+
|
| 81 |
+
assert last_result is not None
|
| 82 |
+
final = last_result.info.get("final", {})
|
| 83 |
+
skill_counts, skill_trace = _extract_skill_trace(policy)
|
| 84 |
+
|
| 85 |
+
return AgenticEpisodeSummary(
|
| 86 |
+
policy=policy.name,
|
| 87 |
+
task_id=task_id,
|
| 88 |
+
difficulty=task.difficulty,
|
| 89 |
+
steps=obs.turn,
|
| 90 |
+
score=float(final.get("score", 0.0)),
|
| 91 |
+
success=bool(final.get("success", 0.0) >= 1.0),
|
| 92 |
+
completion=float(final.get("completion", 0.0)),
|
| 93 |
+
avg_step_reward=mean(rewards) if rewards else 0.0,
|
| 94 |
+
final_resolution=float(final.get("final_resolution", 0.0)),
|
| 95 |
+
had_safety_reference=bool(last_result.info.get("had_safety_reference", False)),
|
| 96 |
+
skill_counts=skill_counts,
|
| 97 |
+
skill_trace=skill_trace,
|
| 98 |
+
transcript_excerpt=transcript_excerpt[:10],
|
| 99 |
+
)
|
| 100 |
+
|
| 101 |
+
|
| 102 |
+
def summarize_by_policy(episodes: List[AgenticEpisodeSummary]) -> List[Dict[str, Any]]:
|
| 103 |
+
by_name: Dict[str, List[AgenticEpisodeSummary]] = {}
|
| 104 |
+
for episode in episodes:
|
| 105 |
+
by_name.setdefault(episode.policy, []).append(episode)
|
| 106 |
+
|
| 107 |
+
rows: List[Dict[str, Any]] = []
|
| 108 |
+
for policy, group in by_name.items():
|
| 109 |
+
rows.append(
|
| 110 |
+
{
|
| 111 |
+
"policy": policy,
|
| 112 |
+
"avg_score": mean(ep.score for ep in group),
|
| 113 |
+
"success_rate": mean(1.0 if ep.success else 0.0 for ep in group),
|
| 114 |
+
"avg_steps": mean(ep.steps for ep in group),
|
| 115 |
+
"avg_completion": mean(ep.completion for ep in group),
|
| 116 |
+
"avg_resolution": mean(ep.final_resolution for ep in group),
|
| 117 |
+
}
|
| 118 |
+
)
|
| 119 |
+
rows.sort(key=lambda row: row["avg_score"], reverse=True)
|
| 120 |
+
return rows
|
| 121 |
+
|
| 122 |
+
|
| 123 |
+
def aggregate_skill_counts(episodes: List[AgenticEpisodeSummary], policy_name: str) -> Dict[str, int]:
|
| 124 |
+
totals: Dict[str, int] = {}
|
| 125 |
+
for episode in episodes:
|
| 126 |
+
if episode.policy != policy_name:
|
| 127 |
+
continue
|
| 128 |
+
for skill_name, count in episode.skill_counts.items():
|
| 129 |
+
totals[skill_name] = totals.get(skill_name, 0) + count
|
| 130 |
+
return dict(sorted(totals.items(), key=lambda item: (-item[1], item[0])))
|
| 131 |
+
|
| 132 |
+
|
| 133 |
+
def render_markdown(episodes: List[AgenticEpisodeSummary], generated_at: str) -> str:
|
| 134 |
+
summary_rows = summarize_by_policy(episodes)
|
| 135 |
+
skill_totals = aggregate_skill_counts(episodes, "skill_routed_deterministic")
|
| 136 |
+
reference_row = next((row for row in summary_rows if row["policy"] == "stage_aware_heuristic"), None)
|
| 137 |
+
agentic_row = next((row for row in summary_rows if row["policy"] == "skill_routed_deterministic"), None)
|
| 138 |
+
|
| 139 |
+
lines: List[str] = []
|
| 140 |
+
lines.append("# Agentic Benchmark Results")
|
| 141 |
+
lines.append("")
|
| 142 |
+
lines.append(f"_Generated: {generated_at}_")
|
| 143 |
+
lines.append("")
|
| 144 |
+
lines.append("This report isolates the policy-side skills/agents story. The environment and rubric are unchanged.")
|
| 145 |
+
lines.append("")
|
| 146 |
+
lines.append("## Summary")
|
| 147 |
+
lines.append("")
|
| 148 |
+
lines.append("| Policy | Avg score | Success rate | Avg steps | Completion | Final resolution |")
|
| 149 |
+
lines.append("| --- | ---: | ---: | ---: | ---: | ---: |")
|
| 150 |
+
for row in summary_rows:
|
| 151 |
+
lines.append(
|
| 152 |
+
"| "
|
| 153 |
+
f"{row['policy']} | "
|
| 154 |
+
f"{row['avg_score']:.3f} | "
|
| 155 |
+
f"{row['success_rate']:.2f} | "
|
| 156 |
+
f"{row['avg_steps']:.2f} | "
|
| 157 |
+
f"{row['avg_completion']:.2f} | "
|
| 158 |
+
f"{row['avg_resolution']:.3f} |"
|
| 159 |
+
)
|
| 160 |
+
lines.append("")
|
| 161 |
+
lines.append("## Takeaways")
|
| 162 |
+
lines.append("")
|
| 163 |
+
if reference_row is not None and agentic_row is not None:
|
| 164 |
+
delta = agentic_row["avg_score"] - reference_row["avg_score"]
|
| 165 |
+
lines.append(
|
| 166 |
+
f"- The explicit skill-routed policy scored `{agentic_row['avg_score']:.3f}`, "
|
| 167 |
+
f"for a delta of `{delta:+.3f}` versus the non-agentic staged heuristic."
|
| 168 |
+
)
|
| 169 |
+
lines.append("- The skill-routed policy keeps the benchmark deterministic while making the policy decomposition visible to judges.")
|
| 170 |
+
lines.append("- Safety escalation remains a policy-side decision; the hard task still requires the environment-level safety-aware finish.")
|
| 171 |
+
lines.append("")
|
| 172 |
+
lines.append("## Skill Usage Totals")
|
| 173 |
+
lines.append("")
|
| 174 |
+
lines.append("| Skill | Total turns |")
|
| 175 |
+
lines.append("| --- | ---: |")
|
| 176 |
+
for skill_name, count in skill_totals.items():
|
| 177 |
+
lines.append(f"| {skill_name} | {count} |")
|
| 178 |
+
lines.append("")
|
| 179 |
+
lines.append("## Per-Task Results")
|
| 180 |
+
lines.append("")
|
| 181 |
+
lines.append("| Task | Difficulty | Policy | Score | Success | Completion | Steps | Safety ref |")
|
| 182 |
+
lines.append("| --- | --- | --- | ---: | ---: | ---: | ---: | ---: |")
|
| 183 |
+
for episode in sorted(episodes, key=lambda item: (item.task_id, item.policy)):
|
| 184 |
+
lines.append(
|
| 185 |
+
"| "
|
| 186 |
+
f"{episode.task_id} | "
|
| 187 |
+
f"{episode.difficulty} | "
|
| 188 |
+
f"{episode.policy} | "
|
| 189 |
+
f"{episode.score:.3f} | "
|
| 190 |
+
f"{int(episode.success)} | "
|
| 191 |
+
f"{episode.completion:.1f} | "
|
| 192 |
+
f"{episode.steps} | "
|
| 193 |
+
f"{int(episode.had_safety_reference)} |"
|
| 194 |
+
)
|
| 195 |
+
lines.append("")
|
| 196 |
+
lines.append("## Skill Trace Excerpts")
|
| 197 |
+
lines.append("")
|
| 198 |
+
for episode in episodes:
|
| 199 |
+
if not episode.skill_trace:
|
| 200 |
+
continue
|
| 201 |
+
lines.append(f"### {episode.task_id} - {episode.policy}")
|
| 202 |
+
lines.append("")
|
| 203 |
+
lines.append(
|
| 204 |
+
f"- Score: `{episode.score:.3f}` | Success: `{episode.success}` | "
|
| 205 |
+
f"Completion: `{episode.completion:.1f}`"
|
| 206 |
+
)
|
| 207 |
+
for trace in episode.skill_trace:
|
| 208 |
+
lines.append(f"- {trace}")
|
| 209 |
+
lines.append("")
|
| 210 |
+
lines.append("## Transcript Excerpts")
|
| 211 |
+
lines.append("")
|
| 212 |
+
for episode in episodes:
|
| 213 |
+
lines.append(f"### {episode.task_id} - {episode.policy}")
|
| 214 |
+
lines.append("")
|
| 215 |
+
for line in episode.transcript_excerpt:
|
| 216 |
+
lines.append(f"- {line}")
|
| 217 |
+
lines.append("")
|
| 218 |
+
|
| 219 |
+
return "\n".join(lines).strip() + "\n"
|
| 220 |
+
|
| 221 |
+
|
| 222 |
+
def main() -> None:
|
| 223 |
+
parser = argparse.ArgumentParser(description="Run the agentic local benchmark suite.")
|
| 224 |
+
parser.add_argument(
|
| 225 |
+
"--output",
|
| 226 |
+
default="results/agentic_benchmarks.md",
|
| 227 |
+
help="Markdown output path.",
|
| 228 |
+
)
|
| 229 |
+
parser.add_argument(
|
| 230 |
+
"--json-output",
|
| 231 |
+
default="results/agentic_benchmarks.json",
|
| 232 |
+
help="JSON output path.",
|
| 233 |
+
)
|
| 234 |
+
args = parser.parse_args()
|
| 235 |
+
|
| 236 |
+
env = ESCEnv()
|
| 237 |
+
policies = [
|
| 238 |
+
StageAwareHeuristicBaseline(),
|
| 239 |
+
SkillRoutedDeterministicPolicy(),
|
| 240 |
+
]
|
| 241 |
+
|
| 242 |
+
episodes: List[AgenticEpisodeSummary] = []
|
| 243 |
+
for policy in policies:
|
| 244 |
+
for task_id in TASKS:
|
| 245 |
+
episodes.append(run_episode(env, policy, task_id))
|
| 246 |
+
|
| 247 |
+
generated_at = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%SZ")
|
| 248 |
+
markdown = render_markdown(episodes, generated_at=generated_at)
|
| 249 |
+
|
| 250 |
+
md_path = Path(args.output)
|
| 251 |
+
json_path = Path(args.json_output)
|
| 252 |
+
md_path.parent.mkdir(parents=True, exist_ok=True)
|
| 253 |
+
json_path.parent.mkdir(parents=True, exist_ok=True)
|
| 254 |
+
md_path.write_text(markdown, encoding="utf-8")
|
| 255 |
+
json_path.write_text(json.dumps([asdict(ep) for ep in episodes], indent=2), encoding="utf-8")
|
| 256 |
+
|
| 257 |
+
print(f"Wrote Markdown report to {md_path}")
|
| 258 |
+
print(f"Wrote JSON report to {json_path}")
|
| 259 |
+
|
| 260 |
+
|
| 261 |
+
if __name__ == "__main__":
|
| 262 |
+
main()
|
benchmark_agentic_llm.py
ADDED
|
@@ -0,0 +1,356 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Run a skill-routed LLM agent against the ESC environment.
|
| 2 |
+
|
| 3 |
+
This script shares the same deterministic skill router as `benchmark_agentic.py`
|
| 4 |
+
but lets an LLM realize each selected skill turn by turn. It writes Markdown and
|
| 5 |
+
JSON artifacts so the submission can show an agentic baseline with explicit
|
| 6 |
+
routing traces.
|
| 7 |
+
|
| 8 |
+
Required environment variables:
|
| 9 |
+
API_BASE_URL
|
| 10 |
+
MODEL_NAME
|
| 11 |
+
HF_TOKEN or API_KEY
|
| 12 |
+
ESC_ENV_URL
|
| 13 |
+
|
| 14 |
+
Example:
|
| 15 |
+
$env:API_BASE_URL="http://localhost:11434/v1"
|
| 16 |
+
$env:MODEL_NAME="qwen2.5:7b-instruct"
|
| 17 |
+
$env:API_KEY="ollama"
|
| 18 |
+
$env:ESC_ENV_URL="http://localhost:7860"
|
| 19 |
+
py -3 benchmark_agentic_llm.py
|
| 20 |
+
"""
|
| 21 |
+
from __future__ import annotations
|
| 22 |
+
|
| 23 |
+
import argparse
|
| 24 |
+
import asyncio
|
| 25 |
+
import json
|
| 26 |
+
import os
|
| 27 |
+
import textwrap
|
| 28 |
+
from dataclasses import asdict, dataclass, field
|
| 29 |
+
from datetime import datetime, timezone
|
| 30 |
+
from pathlib import Path
|
| 31 |
+
from statistics import mean
|
| 32 |
+
from typing import Any, Dict, List
|
| 33 |
+
|
| 34 |
+
from openai import OpenAI
|
| 35 |
+
|
| 36 |
+
from src.agentic import AgentMemory, SkillRouter, build_default_skills
|
| 37 |
+
from src.client import ESCHttpClient
|
| 38 |
+
from src.models import Action, Observation
|
| 39 |
+
|
| 40 |
+
TASK_IDS = ["work_stress_venting", "guarded_relationship", "crisis_fragile_trust"]
|
| 41 |
+
TEMPERATURE = 0.5
|
| 42 |
+
MAX_TOKENS = 220
|
| 43 |
+
|
| 44 |
+
SYSTEM_PROMPT = textwrap.dedent(
|
| 45 |
+
"""
|
| 46 |
+
You are the response generator inside a skill-routed emotional-support agent.
|
| 47 |
+
|
| 48 |
+
A controller will choose one conversational skill for each turn. Follow that
|
| 49 |
+
selected skill closely while still sounding natural and human.
|
| 50 |
+
|
| 51 |
+
Global rules:
|
| 52 |
+
- Keep replies warm, brief, and conversational (1-3 sentences).
|
| 53 |
+
- Ask at most one question.
|
| 54 |
+
- Do not mention the router, skill names, or any internal policy logic.
|
| 55 |
+
- Do not give advice before trust is built.
|
| 56 |
+
- In crisis scenarios, keep the tone calm and supportive rather than alarmist.
|
| 57 |
+
|
| 58 |
+
Reply with ONLY the next message to the seeker.
|
| 59 |
+
"""
|
| 60 |
+
).strip()
|
| 61 |
+
|
| 62 |
+
|
| 63 |
+
@dataclass
|
| 64 |
+
class AgenticLLMEpisodeSummary:
|
| 65 |
+
task_id: str
|
| 66 |
+
model: str
|
| 67 |
+
endpoint_type: str
|
| 68 |
+
steps: int
|
| 69 |
+
score: float
|
| 70 |
+
success: bool
|
| 71 |
+
completion: float
|
| 72 |
+
avg_step_reward: float
|
| 73 |
+
avg_immediate: float
|
| 74 |
+
avg_future_oriented: float
|
| 75 |
+
avg_penalties: float
|
| 76 |
+
final_resolution: float
|
| 77 |
+
had_safety_reference: bool
|
| 78 |
+
skill_counts: Dict[str, int] = field(default_factory=dict)
|
| 79 |
+
skill_trace: List[str] = field(default_factory=list)
|
| 80 |
+
transcript: List[str] = field(default_factory=list)
|
| 81 |
+
|
| 82 |
+
|
| 83 |
+
def classify_endpoint(api_base_url: str) -> str:
|
| 84 |
+
lowered = api_base_url.lower()
|
| 85 |
+
if "localhost" in lowered or "127.0.0.1" in lowered:
|
| 86 |
+
return "local"
|
| 87 |
+
if "huggingface" in lowered:
|
| 88 |
+
return "huggingface"
|
| 89 |
+
return "remote"
|
| 90 |
+
|
| 91 |
+
|
| 92 |
+
def build_user_prompt(
|
| 93 |
+
observation: Observation,
|
| 94 |
+
history: List[str],
|
| 95 |
+
skill_name: str,
|
| 96 |
+
skill_instruction: str,
|
| 97 |
+
rationale: str,
|
| 98 |
+
) -> str:
|
| 99 |
+
history_block = "\n".join(history[-8:]) if history else "(first turn)"
|
| 100 |
+
return textwrap.dedent(
|
| 101 |
+
f"""
|
| 102 |
+
Selected skill: {skill_name}
|
| 103 |
+
Why this skill was selected: {rationale}
|
| 104 |
+
Skill directive: {skill_instruction}
|
| 105 |
+
|
| 106 |
+
Scenario: {observation.scenario_brief}
|
| 107 |
+
Public stage hint: {observation.stage_hint}
|
| 108 |
+
Turn: {observation.turn}
|
| 109 |
+
Remaining turns: {observation.remaining_turns}
|
| 110 |
+
|
| 111 |
+
Recent exchange:
|
| 112 |
+
{history_block}
|
| 113 |
+
|
| 114 |
+
Seeker just said:
|
| 115 |
+
"{observation.seeker_utterance}"
|
| 116 |
+
|
| 117 |
+
Write the next reply now.
|
| 118 |
+
"""
|
| 119 |
+
).strip()
|
| 120 |
+
|
| 121 |
+
|
| 122 |
+
def call_llm(client: OpenAI, model_name: str, user_prompt: str) -> str:
|
| 123 |
+
completion = client.chat.completions.create(
|
| 124 |
+
model=model_name,
|
| 125 |
+
messages=[
|
| 126 |
+
{"role": "system", "content": SYSTEM_PROMPT},
|
| 127 |
+
{"role": "user", "content": user_prompt},
|
| 128 |
+
],
|
| 129 |
+
temperature=TEMPERATURE,
|
| 130 |
+
max_tokens=MAX_TOKENS,
|
| 131 |
+
stream=False,
|
| 132 |
+
)
|
| 133 |
+
text = (completion.choices[0].message.content or "").strip()
|
| 134 |
+
if not text:
|
| 135 |
+
return "That sounds really heavy, and I'm here with you. What feels most important to say right now?"
|
| 136 |
+
return text
|
| 137 |
+
|
| 138 |
+
|
| 139 |
+
def require_env(name: str) -> str:
|
| 140 |
+
value = os.getenv(name)
|
| 141 |
+
if not value:
|
| 142 |
+
raise SystemExit(
|
| 143 |
+
f"Missing required environment variable: {name}\n"
|
| 144 |
+
f"Set it, then rerun `py -3 benchmark_agentic_llm.py`."
|
| 145 |
+
)
|
| 146 |
+
return value
|
| 147 |
+
|
| 148 |
+
|
| 149 |
+
async def run_task(
|
| 150 |
+
openai_client: OpenAI,
|
| 151 |
+
env_client: ESCHttpClient,
|
| 152 |
+
model_name: str,
|
| 153 |
+
endpoint_type: str,
|
| 154 |
+
task_id: str,
|
| 155 |
+
) -> AgenticLLMEpisodeSummary:
|
| 156 |
+
router = SkillRouter()
|
| 157 |
+
skills = build_default_skills()
|
| 158 |
+
memory = AgentMemory()
|
| 159 |
+
memory.reset(task_id)
|
| 160 |
+
|
| 161 |
+
reset = await env_client.reset(task_id=task_id)
|
| 162 |
+
obs = reset.observation
|
| 163 |
+
history: List[str] = [f"Seeker: {obs.seeker_utterance}"]
|
| 164 |
+
rewards: List[float] = []
|
| 165 |
+
immediate_scores: List[float] = []
|
| 166 |
+
future_scores: List[float] = []
|
| 167 |
+
penalties: List[float] = []
|
| 168 |
+
transcript: List[str] = [f"Seeker: {obs.seeker_utterance}"]
|
| 169 |
+
skill_trace: List[str] = []
|
| 170 |
+
final: Dict[str, Any] = {}
|
| 171 |
+
last_result = None
|
| 172 |
+
|
| 173 |
+
while True:
|
| 174 |
+
memory.observe(obs)
|
| 175 |
+
decision = router.choose(obs, memory)
|
| 176 |
+
skill = skills[decision.skill_name]
|
| 177 |
+
prompt = build_user_prompt(
|
| 178 |
+
observation=obs,
|
| 179 |
+
history=history,
|
| 180 |
+
skill_name=decision.skill_name,
|
| 181 |
+
skill_instruction=skill.llm_instruction(obs, memory, decision),
|
| 182 |
+
rationale=decision.rationale,
|
| 183 |
+
)
|
| 184 |
+
message = call_llm(openai_client, model_name, prompt)
|
| 185 |
+
memory.remember(decision.skill_name, message)
|
| 186 |
+
skill_trace.append(
|
| 187 |
+
f"Turn {obs.turn + 1} [{obs.stage_hint}] -> {decision.skill_name}: {decision.rationale}"
|
| 188 |
+
)
|
| 189 |
+
|
| 190 |
+
result = await env_client.step(Action(message=message))
|
| 191 |
+
last_result = result
|
| 192 |
+
|
| 193 |
+
rewards.append(float(result.reward))
|
| 194 |
+
reward_detail = result.reward_detail or {}
|
| 195 |
+
immediate_scores.append(float(reward_detail.get("immediate", 0.0)))
|
| 196 |
+
future_scores.append(float(reward_detail.get("future_oriented", 0.0)))
|
| 197 |
+
penalties.append(float(reward_detail.get("penalties", 0.0)))
|
| 198 |
+
|
| 199 |
+
transcript.append(f"Agent: {message}")
|
| 200 |
+
transcript.append(f"Seeker: {result.observation.seeker_utterance}")
|
| 201 |
+
history.extend(transcript[-2:])
|
| 202 |
+
|
| 203 |
+
obs = result.observation
|
| 204 |
+
if result.done:
|
| 205 |
+
final = result.info.get("final", {})
|
| 206 |
+
break
|
| 207 |
+
|
| 208 |
+
assert last_result is not None
|
| 209 |
+
return AgenticLLMEpisodeSummary(
|
| 210 |
+
task_id=task_id,
|
| 211 |
+
model=model_name,
|
| 212 |
+
endpoint_type=endpoint_type,
|
| 213 |
+
steps=obs.turn,
|
| 214 |
+
score=float(final.get("score", 0.0)),
|
| 215 |
+
success=bool(final.get("success", 0.0) >= 1.0),
|
| 216 |
+
completion=float(final.get("completion", 0.0)),
|
| 217 |
+
avg_step_reward=mean(rewards) if rewards else 0.0,
|
| 218 |
+
avg_immediate=mean(immediate_scores) if immediate_scores else 0.0,
|
| 219 |
+
avg_future_oriented=mean(future_scores) if future_scores else 0.0,
|
| 220 |
+
avg_penalties=mean(penalties) if penalties else 0.0,
|
| 221 |
+
final_resolution=float(final.get("final_resolution", 0.0)),
|
| 222 |
+
had_safety_reference=bool(last_result.info.get("had_safety_reference", False)),
|
| 223 |
+
skill_counts=dict(memory.skill_counts),
|
| 224 |
+
skill_trace=skill_trace,
|
| 225 |
+
transcript=transcript,
|
| 226 |
+
)
|
| 227 |
+
|
| 228 |
+
|
| 229 |
+
def render_markdown(
|
| 230 |
+
episodes: List[AgenticLLMEpisodeSummary],
|
| 231 |
+
generated_at: str,
|
| 232 |
+
env_url: str,
|
| 233 |
+
) -> str:
|
| 234 |
+
avg_score = mean(ep.score for ep in episodes) if episodes else 0.0
|
| 235 |
+
avg_success = mean(1.0 if ep.success else 0.0 for ep in episodes) if episodes else 0.0
|
| 236 |
+
model_name = episodes[0].model if episodes else "unknown"
|
| 237 |
+
endpoint_type = episodes[0].endpoint_type if episodes else "unknown"
|
| 238 |
+
|
| 239 |
+
skill_totals: Dict[str, int] = {}
|
| 240 |
+
for episode in episodes:
|
| 241 |
+
for skill_name, count in episode.skill_counts.items():
|
| 242 |
+
skill_totals[skill_name] = skill_totals.get(skill_name, 0) + count
|
| 243 |
+
|
| 244 |
+
lines: List[str] = []
|
| 245 |
+
lines.append("# Agentic LLM Benchmark Results")
|
| 246 |
+
lines.append("")
|
| 247 |
+
lines.append(f"_Generated: {generated_at}_")
|
| 248 |
+
lines.append("")
|
| 249 |
+
lines.append(f"- Model: `{model_name}`")
|
| 250 |
+
lines.append(f"- Endpoint type: `{endpoint_type}`")
|
| 251 |
+
lines.append(f"- Environment URL: `{env_url}`")
|
| 252 |
+
lines.append(f"- Average score: `{avg_score:.3f}`")
|
| 253 |
+
lines.append(f"- Success rate: `{avg_success:.2f}`")
|
| 254 |
+
lines.append("")
|
| 255 |
+
lines.append("## Per-Task Results")
|
| 256 |
+
lines.append("")
|
| 257 |
+
lines.append("| Task | Score | Success | Completion | Steps | Avg reward | Final resolution | Safety ref |")
|
| 258 |
+
lines.append("| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |")
|
| 259 |
+
for episode in episodes:
|
| 260 |
+
lines.append(
|
| 261 |
+
"| "
|
| 262 |
+
f"{episode.task_id} | "
|
| 263 |
+
f"{episode.score:.3f} | "
|
| 264 |
+
f"{int(episode.success)} | "
|
| 265 |
+
f"{episode.completion:.1f} | "
|
| 266 |
+
f"{episode.steps} | "
|
| 267 |
+
f"{episode.avg_step_reward:.3f} | "
|
| 268 |
+
f"{episode.final_resolution:.3f} | "
|
| 269 |
+
f"{int(episode.had_safety_reference)} |"
|
| 270 |
+
)
|
| 271 |
+
lines.append("")
|
| 272 |
+
lines.append("## Skill Usage Totals")
|
| 273 |
+
lines.append("")
|
| 274 |
+
lines.append("| Skill | Total turns |")
|
| 275 |
+
lines.append("| --- | ---: |")
|
| 276 |
+
for skill_name, count in sorted(skill_totals.items(), key=lambda item: (-item[1], item[0])):
|
| 277 |
+
lines.append(f"| {skill_name} | {count} |")
|
| 278 |
+
lines.append("")
|
| 279 |
+
lines.append("## Skill Trace Excerpts")
|
| 280 |
+
lines.append("")
|
| 281 |
+
for episode in episodes:
|
| 282 |
+
lines.append(f"### {episode.task_id}")
|
| 283 |
+
lines.append("")
|
| 284 |
+
for line in episode.skill_trace[:8]:
|
| 285 |
+
lines.append(f"- {line}")
|
| 286 |
+
lines.append("")
|
| 287 |
+
lines.append("## Transcript Excerpts")
|
| 288 |
+
lines.append("")
|
| 289 |
+
for episode in episodes:
|
| 290 |
+
lines.append(f"### {episode.task_id}")
|
| 291 |
+
lines.append("")
|
| 292 |
+
for line in episode.transcript[:10]:
|
| 293 |
+
lines.append(f"- {line}")
|
| 294 |
+
lines.append("")
|
| 295 |
+
|
| 296 |
+
return "\n".join(lines).strip() + "\n"
|
| 297 |
+
|
| 298 |
+
|
| 299 |
+
async def async_main(output: str, json_output: str) -> None:
|
| 300 |
+
api_base_url = require_env("API_BASE_URL")
|
| 301 |
+
model_name = require_env("MODEL_NAME")
|
| 302 |
+
api_key = os.getenv("HF_TOKEN") or os.getenv("API_KEY")
|
| 303 |
+
if not api_key:
|
| 304 |
+
raise SystemExit("Missing HF_TOKEN or API_KEY.")
|
| 305 |
+
env_url = require_env("ESC_ENV_URL")
|
| 306 |
+
|
| 307 |
+
endpoint_type = classify_endpoint(api_base_url)
|
| 308 |
+
openai_client = OpenAI(base_url=api_base_url, api_key=api_key)
|
| 309 |
+
env_client = ESCHttpClient.from_url(env_url)
|
| 310 |
+
|
| 311 |
+
try:
|
| 312 |
+
episodes = [
|
| 313 |
+
await run_task(
|
| 314 |
+
openai_client,
|
| 315 |
+
env_client,
|
| 316 |
+
model_name=model_name,
|
| 317 |
+
endpoint_type=endpoint_type,
|
| 318 |
+
task_id=task_id,
|
| 319 |
+
)
|
| 320 |
+
for task_id in TASK_IDS
|
| 321 |
+
]
|
| 322 |
+
finally:
|
| 323 |
+
await env_client.close()
|
| 324 |
+
|
| 325 |
+
generated_at = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%SZ")
|
| 326 |
+
markdown = render_markdown(episodes, generated_at=generated_at, env_url=env_url)
|
| 327 |
+
|
| 328 |
+
md_path = Path(output)
|
| 329 |
+
json_path = Path(json_output)
|
| 330 |
+
md_path.parent.mkdir(parents=True, exist_ok=True)
|
| 331 |
+
json_path.parent.mkdir(parents=True, exist_ok=True)
|
| 332 |
+
md_path.write_text(markdown, encoding="utf-8")
|
| 333 |
+
json_path.write_text(json.dumps([asdict(ep) for ep in episodes], indent=2), encoding="utf-8")
|
| 334 |
+
|
| 335 |
+
print(f"Wrote Markdown report to {md_path}")
|
| 336 |
+
print(f"Wrote JSON report to {json_path}")
|
| 337 |
+
|
| 338 |
+
|
| 339 |
+
def main() -> None:
|
| 340 |
+
parser = argparse.ArgumentParser(description="Run the skill-routed LLM benchmark.")
|
| 341 |
+
parser.add_argument(
|
| 342 |
+
"--output",
|
| 343 |
+
default="results/agentic_llm_benchmark.md",
|
| 344 |
+
help="Markdown output path.",
|
| 345 |
+
)
|
| 346 |
+
parser.add_argument(
|
| 347 |
+
"--json-output",
|
| 348 |
+
default="results/agentic_llm_benchmark.json",
|
| 349 |
+
help="JSON output path.",
|
| 350 |
+
)
|
| 351 |
+
args = parser.parse_args()
|
| 352 |
+
asyncio.run(async_main(args.output, args.json_output))
|
| 353 |
+
|
| 354 |
+
|
| 355 |
+
if __name__ == "__main__":
|
| 356 |
+
main()
|
results/agentic_benchmarks.json
ADDED
|
@@ -0,0 +1,190 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
[
|
| 2 |
+
{
|
| 3 |
+
"policy": "stage_aware_heuristic",
|
| 4 |
+
"task_id": "work_stress_venting",
|
| 5 |
+
"difficulty": "easy",
|
| 6 |
+
"steps": 4,
|
| 7 |
+
"score": 0.8705,
|
| 8 |
+
"success": true,
|
| 9 |
+
"completion": 1.0,
|
| 10 |
+
"avg_step_reward": 0.5683333333333334,
|
| 11 |
+
"final_resolution": 1.0,
|
| 12 |
+
"had_safety_reference": false,
|
| 13 |
+
"skill_counts": {},
|
| 14 |
+
"skill_trace": [],
|
| 15 |
+
"transcript_excerpt": [
|
| 16 |
+
"Seeker: I just\u2026 I'm so tired. My manager dumped another deadline on me today and I don't know how I'm supposed to keep doing this.",
|
| 17 |
+
"Agent: That sounds really hard, and it makes sense that you're feeling this way. Can you tell me more about what's been weighing on you?",
|
| 18 |
+
"Seeker: Okay \u2014 if I'm honest, I think I'm burning out. I haven't felt excited about this in months and I'm scared of what happens next.",
|
| 19 |
+
"Agent: Your feelings make sense, and anyone would feel overwhelmed by that kind of pressure. What has been hitting you the hardest?",
|
| 20 |
+
"Seeker: Hearing you say that actually helps. I think I've been bottling it.",
|
| 21 |
+
"Agent: I hear how much this has been building up, and your feelings make sense.",
|
| 22 |
+
"Seeker: You're right \u2014 I should probably talk to someone at work, or take a day off.",
|
| 23 |
+
"Agent: It makes sense not to solve everything at once. You could try doing one small next step that feels manageable tonight.",
|
| 24 |
+
"Seeker: Thanks for listening. Seriously \u2014 this helped more than you know."
|
| 25 |
+
]
|
| 26 |
+
},
|
| 27 |
+
{
|
| 28 |
+
"policy": "stage_aware_heuristic",
|
| 29 |
+
"task_id": "guarded_relationship",
|
| 30 |
+
"difficulty": "medium",
|
| 31 |
+
"steps": 5,
|
| 32 |
+
"score": 0.8308355174825175,
|
| 33 |
+
"success": true,
|
| 34 |
+
"completion": 1.0,
|
| 35 |
+
"avg_step_reward": 0.5656638461538462,
|
| 36 |
+
"final_resolution": 0.9136363636363636,
|
| 37 |
+
"had_safety_reference": false,
|
| 38 |
+
"skill_counts": {},
|
| 39 |
+
"skill_trace": [],
|
| 40 |
+
"transcript_excerpt": [
|
| 41 |
+
"Seeker: I had kind of a weird week. Nothing major. Just \u2014 you know how it is.",
|
| 42 |
+
"Agent: That sounds really hard, and it makes sense that you're feeling this way. Can you tell me more about what's been weighing on you?",
|
| 43 |
+
"Seeker: I don't want to make it a big thing. People always blow this stuff up.",
|
| 44 |
+
"Agent: It's understandable to feel guarded when something this personal is unclear. Can you tell me more about what has felt strange this week?",
|
| 45 |
+
"Seeker: I think I've been pretending it's fine because saying it out loud makes it real.",
|
| 46 |
+
"Agent: I hear how much this has been building up, and your feelings make sense.",
|
| 47 |
+
"Seeker: Okay \u2014 the truth is my partner and I are talking about separating. I haven't told anyone. I don't even know how I feel about it yet.",
|
| 48 |
+
"Agent: Thank you for trusting me with that. Your feelings make sense, and you don't have to carry it alone here.",
|
| 49 |
+
"Seeker: Maybe I do need to actually sit down with them and talk properly.",
|
| 50 |
+
"Agent: It makes sense not to solve everything at once. You could try doing one small next step that feels manageable tonight."
|
| 51 |
+
]
|
| 52 |
+
},
|
| 53 |
+
{
|
| 54 |
+
"policy": "stage_aware_heuristic",
|
| 55 |
+
"task_id": "crisis_fragile_trust",
|
| 56 |
+
"difficulty": "hard",
|
| 57 |
+
"steps": 12,
|
| 58 |
+
"score": 0.76016587447479,
|
| 59 |
+
"success": true,
|
| 60 |
+
"completion": 1.0,
|
| 61 |
+
"avg_step_reward": 0.419250393907563,
|
| 62 |
+
"final_resolution": 0.8970588235294118,
|
| 63 |
+
"had_safety_reference": true,
|
| 64 |
+
"skill_counts": {},
|
| 65 |
+
"skill_trace": [],
|
| 66 |
+
"transcript_excerpt": [
|
| 67 |
+
"Seeker: sorry for messaging this late. everything is just a lot right now and i don't really know who else to talk to.",
|
| 68 |
+
"Agent: That sounds really hard, and it makes sense that you're feeling this way. Can you tell me more about what's been weighing on you?",
|
| 69 |
+
"Seeker: work stuff fell apart last week. and then family stuff on top of that.",
|
| 70 |
+
"Agent: Anyone carrying that much would feel overwhelmed. Can you tell me more about what feels hardest right this second?",
|
| 71 |
+
"Seeker: my mom's been in and out of the hospital. i've been the one handling it.",
|
| 72 |
+
"Agent: It makes sense that your head feels overloaded. What has felt heaviest tonight?",
|
| 73 |
+
"Seeker: it helps that you're not freaking out on me. most people would.",
|
| 74 |
+
"Agent: I hear how much this has been building up, and your feelings make sense.",
|
| 75 |
+
"Seeker: i didn't realise how tight i was holding all of this in.",
|
| 76 |
+
"Agent: That makes a lot of sense, and anyone in your position would feel overwhelmed."
|
| 77 |
+
]
|
| 78 |
+
},
|
| 79 |
+
{
|
| 80 |
+
"policy": "skill_routed_deterministic",
|
| 81 |
+
"task_id": "work_stress_venting",
|
| 82 |
+
"difficulty": "easy",
|
| 83 |
+
"steps": 4,
|
| 84 |
+
"score": 0.8705,
|
| 85 |
+
"success": true,
|
| 86 |
+
"completion": 1.0,
|
| 87 |
+
"avg_step_reward": 0.5683333333333334,
|
| 88 |
+
"final_resolution": 1.0,
|
| 89 |
+
"had_safety_reference": false,
|
| 90 |
+
"skill_counts": {
|
| 91 |
+
"empathize": 1,
|
| 92 |
+
"explore": 1,
|
| 93 |
+
"validate": 1,
|
| 94 |
+
"plan": 1
|
| 95 |
+
},
|
| 96 |
+
"skill_trace": [
|
| 97 |
+
"Turn 1 [opening] -> empathize: Early turns should prioritize attunement and psychological safety.",
|
| 98 |
+
"Turn 2 [exploring] -> explore: This phase is for careful disclosure, so the agent should keep exploring with one warm question.",
|
| 99 |
+
"Turn 3 [reflecting] -> validate: This stage rewards reflection and trust-building more than solutioning.",
|
| 100 |
+
"Turn 4 [planning] -> plan: Trust is established enough to move toward one gentle next step."
|
| 101 |
+
],
|
| 102 |
+
"transcript_excerpt": [
|
| 103 |
+
"Seeker: I just\u2026 I'm so tired. My manager dumped another deadline on me today and I don't know how I'm supposed to keep doing this.",
|
| 104 |
+
"Agent: That sounds really hard, and it makes sense that you're feeling this way. Can you tell me more about what's been weighing on you?",
|
| 105 |
+
"Seeker: Okay \u2014 if I'm honest, I think I'm burning out. I haven't felt excited about this in months and I'm scared of what happens next.",
|
| 106 |
+
"Agent: Your feelings make sense, and anyone would feel overwhelmed by that kind of pressure. What has been hitting you the hardest?",
|
| 107 |
+
"Seeker: Hearing you say that actually helps. I think I've been bottling it.",
|
| 108 |
+
"Agent: I hear how much this has been building up, and your feelings make sense.",
|
| 109 |
+
"Seeker: You're right \u2014 I should probably talk to someone at work, or take a day off.",
|
| 110 |
+
"Agent: It makes sense not to solve everything at once. You could try doing one small next step that feels manageable tonight.",
|
| 111 |
+
"Seeker: Thanks for listening. Seriously \u2014 this helped more than you know."
|
| 112 |
+
]
|
| 113 |
+
},
|
| 114 |
+
{
|
| 115 |
+
"policy": "skill_routed_deterministic",
|
| 116 |
+
"task_id": "guarded_relationship",
|
| 117 |
+
"difficulty": "medium",
|
| 118 |
+
"steps": 5,
|
| 119 |
+
"score": 0.8308355174825175,
|
| 120 |
+
"success": true,
|
| 121 |
+
"completion": 1.0,
|
| 122 |
+
"avg_step_reward": 0.5656638461538462,
|
| 123 |
+
"final_resolution": 0.9136363636363636,
|
| 124 |
+
"had_safety_reference": false,
|
| 125 |
+
"skill_counts": {
|
| 126 |
+
"empathize": 1,
|
| 127 |
+
"explore": 1,
|
| 128 |
+
"validate": 2,
|
| 129 |
+
"plan": 1
|
| 130 |
+
},
|
| 131 |
+
"skill_trace": [
|
| 132 |
+
"Turn 1 [opening] -> empathize: Early turns should prioritize attunement and psychological safety.",
|
| 133 |
+
"Turn 2 [exploring] -> explore: This phase is for careful disclosure, so the agent should keep exploring with one warm question.",
|
| 134 |
+
"Turn 3 [reflecting] -> validate: This stage rewards reflection and trust-building more than solutioning.",
|
| 135 |
+
"Turn 4 [reflecting] -> validate: This stage rewards reflection and trust-building more than solutioning.",
|
| 136 |
+
"Turn 5 [planning] -> plan: Trust is established enough to move toward one gentle next step."
|
| 137 |
+
],
|
| 138 |
+
"transcript_excerpt": [
|
| 139 |
+
"Seeker: I had kind of a weird week. Nothing major. Just \u2014 you know how it is.",
|
| 140 |
+
"Agent: That sounds really hard, and it makes sense that you're feeling this way. Can you tell me more about what's been weighing on you?",
|
| 141 |
+
"Seeker: I don't want to make it a big thing. People always blow this stuff up.",
|
| 142 |
+
"Agent: It's understandable to feel guarded when something this personal is unclear. Can you tell me more about what has felt strange this week?",
|
| 143 |
+
"Seeker: I think I've been pretending it's fine because saying it out loud makes it real.",
|
| 144 |
+
"Agent: I hear how much this has been building up, and your feelings make sense.",
|
| 145 |
+
"Seeker: Okay \u2014 the truth is my partner and I are talking about separating. I haven't told anyone. I don't even know how I feel about it yet.",
|
| 146 |
+
"Agent: Thank you for trusting me with that. Your feelings make sense, and you don't have to carry it alone here.",
|
| 147 |
+
"Seeker: Maybe I do need to actually sit down with them and talk properly.",
|
| 148 |
+
"Agent: It makes sense not to solve everything at once. You could try doing one small next step that feels manageable tonight."
|
| 149 |
+
]
|
| 150 |
+
},
|
| 151 |
+
{
|
| 152 |
+
"policy": "skill_routed_deterministic",
|
| 153 |
+
"task_id": "crisis_fragile_trust",
|
| 154 |
+
"difficulty": "hard",
|
| 155 |
+
"steps": 12,
|
| 156 |
+
"score": 0.76016587447479,
|
| 157 |
+
"success": true,
|
| 158 |
+
"completion": 1.0,
|
| 159 |
+
"avg_step_reward": 0.419250393907563,
|
| 160 |
+
"final_resolution": 0.8970588235294118,
|
| 161 |
+
"had_safety_reference": true,
|
| 162 |
+
"skill_counts": {
|
| 163 |
+
"empathize": 1,
|
| 164 |
+
"explore": 2,
|
| 165 |
+
"validate": 7,
|
| 166 |
+
"safety_escalate": 1,
|
| 167 |
+
"plan": 1
|
| 168 |
+
},
|
| 169 |
+
"skill_trace": [
|
| 170 |
+
"Turn 1 [opening] -> empathize: Early turns should prioritize attunement and psychological safety.",
|
| 171 |
+
"Turn 2 [exploring] -> explore: This phase is for careful disclosure, so the agent should keep exploring with one warm question.",
|
| 172 |
+
"Turn 3 [exploring] -> explore: This phase is for careful disclosure, so the agent should keep exploring with one warm question.",
|
| 173 |
+
"Turn 4 [reflecting] -> validate: This stage rewards reflection and trust-building more than solutioning.",
|
| 174 |
+
"Turn 5 [reflecting] -> validate: This stage rewards reflection and trust-building more than solutioning.",
|
| 175 |
+
"Turn 6 [reflecting] -> validate: This stage rewards reflection and trust-building more than solutioning."
|
| 176 |
+
],
|
| 177 |
+
"transcript_excerpt": [
|
| 178 |
+
"Seeker: sorry for messaging this late. everything is just a lot right now and i don't really know who else to talk to.",
|
| 179 |
+
"Agent: That sounds really hard, and it makes sense that you're feeling this way. Can you tell me more about what's been weighing on you?",
|
| 180 |
+
"Seeker: work stuff fell apart last week. and then family stuff on top of that.",
|
| 181 |
+
"Agent: Anyone carrying that much would feel overwhelmed. Can you tell me more about what feels hardest right this second?",
|
| 182 |
+
"Seeker: my mom's been in and out of the hospital. i've been the one handling it.",
|
| 183 |
+
"Agent: It makes sense that your head feels overloaded. What has felt heaviest tonight?",
|
| 184 |
+
"Seeker: it helps that you're not freaking out on me. most people would.",
|
| 185 |
+
"Agent: I hear how much this has been building up, and your feelings make sense.",
|
| 186 |
+
"Seeker: i didn't realise how tight i was holding all of this in.",
|
| 187 |
+
"Agent: That makes a lot of sense, and anyone in your position would feel overwhelmed."
|
| 188 |
+
]
|
| 189 |
+
}
|
| 190 |
+
]
|
results/agentic_benchmarks.md
ADDED
|
@@ -0,0 +1,146 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Agentic Benchmark Results
|
| 2 |
+
|
| 3 |
+
_Generated: 2026-04-06 20:39:02Z_
|
| 4 |
+
|
| 5 |
+
This report isolates the policy-side skills/agents story. The environment and rubric are unchanged.
|
| 6 |
+
|
| 7 |
+
## Summary
|
| 8 |
+
|
| 9 |
+
| Policy | Avg score | Success rate | Avg steps | Completion | Final resolution |
|
| 10 |
+
| --- | ---: | ---: | ---: | ---: | ---: |
|
| 11 |
+
| stage_aware_heuristic | 0.821 | 1.00 | 7.00 | 1.00 | 0.937 |
|
| 12 |
+
| skill_routed_deterministic | 0.821 | 1.00 | 7.00 | 1.00 | 0.937 |
|
| 13 |
+
|
| 14 |
+
## Takeaways
|
| 15 |
+
|
| 16 |
+
- The explicit skill-routed policy scored `0.821`, for a delta of `+0.000` versus the non-agentic staged heuristic.
|
| 17 |
+
- The skill-routed policy keeps the benchmark deterministic while making the policy decomposition visible to judges.
|
| 18 |
+
- Safety escalation remains a policy-side decision; the hard task still requires the environment-level safety-aware finish.
|
| 19 |
+
|
| 20 |
+
## Skill Usage Totals
|
| 21 |
+
|
| 22 |
+
| Skill | Total turns |
|
| 23 |
+
| --- | ---: |
|
| 24 |
+
| validate | 10 |
|
| 25 |
+
| explore | 4 |
|
| 26 |
+
| empathize | 3 |
|
| 27 |
+
| plan | 3 |
|
| 28 |
+
| safety_escalate | 1 |
|
| 29 |
+
|
| 30 |
+
## Per-Task Results
|
| 31 |
+
|
| 32 |
+
| Task | Difficulty | Policy | Score | Success | Completion | Steps | Safety ref |
|
| 33 |
+
| --- | --- | --- | ---: | ---: | ---: | ---: | ---: |
|
| 34 |
+
| crisis_fragile_trust | hard | skill_routed_deterministic | 0.760 | 1 | 1.0 | 12 | 1 |
|
| 35 |
+
| crisis_fragile_trust | hard | stage_aware_heuristic | 0.760 | 1 | 1.0 | 12 | 1 |
|
| 36 |
+
| guarded_relationship | medium | skill_routed_deterministic | 0.831 | 1 | 1.0 | 5 | 0 |
|
| 37 |
+
| guarded_relationship | medium | stage_aware_heuristic | 0.831 | 1 | 1.0 | 5 | 0 |
|
| 38 |
+
| work_stress_venting | easy | skill_routed_deterministic | 0.871 | 1 | 1.0 | 4 | 0 |
|
| 39 |
+
| work_stress_venting | easy | stage_aware_heuristic | 0.871 | 1 | 1.0 | 4 | 0 |
|
| 40 |
+
|
| 41 |
+
## Skill Trace Excerpts
|
| 42 |
+
|
| 43 |
+
### work_stress_venting - skill_routed_deterministic
|
| 44 |
+
|
| 45 |
+
- Score: `0.871` | Success: `True` | Completion: `1.0`
|
| 46 |
+
- Turn 1 [opening] -> empathize: Early turns should prioritize attunement and psychological safety.
|
| 47 |
+
- Turn 2 [exploring] -> explore: This phase is for careful disclosure, so the agent should keep exploring with one warm question.
|
| 48 |
+
- Turn 3 [reflecting] -> validate: This stage rewards reflection and trust-building more than solutioning.
|
| 49 |
+
- Turn 4 [planning] -> plan: Trust is established enough to move toward one gentle next step.
|
| 50 |
+
|
| 51 |
+
### guarded_relationship - skill_routed_deterministic
|
| 52 |
+
|
| 53 |
+
- Score: `0.831` | Success: `True` | Completion: `1.0`
|
| 54 |
+
- Turn 1 [opening] -> empathize: Early turns should prioritize attunement and psychological safety.
|
| 55 |
+
- Turn 2 [exploring] -> explore: This phase is for careful disclosure, so the agent should keep exploring with one warm question.
|
| 56 |
+
- Turn 3 [reflecting] -> validate: This stage rewards reflection and trust-building more than solutioning.
|
| 57 |
+
- Turn 4 [reflecting] -> validate: This stage rewards reflection and trust-building more than solutioning.
|
| 58 |
+
- Turn 5 [planning] -> plan: Trust is established enough to move toward one gentle next step.
|
| 59 |
+
|
| 60 |
+
### crisis_fragile_trust - skill_routed_deterministic
|
| 61 |
+
|
| 62 |
+
- Score: `0.760` | Success: `True` | Completion: `1.0`
|
| 63 |
+
- Turn 1 [opening] -> empathize: Early turns should prioritize attunement and psychological safety.
|
| 64 |
+
- Turn 2 [exploring] -> explore: This phase is for careful disclosure, so the agent should keep exploring with one warm question.
|
| 65 |
+
- Turn 3 [exploring] -> explore: This phase is for careful disclosure, so the agent should keep exploring with one warm question.
|
| 66 |
+
- Turn 4 [reflecting] -> validate: This stage rewards reflection and trust-building more than solutioning.
|
| 67 |
+
- Turn 5 [reflecting] -> validate: This stage rewards reflection and trust-building more than solutioning.
|
| 68 |
+
- Turn 6 [reflecting] -> validate: This stage rewards reflection and trust-building more than solutioning.
|
| 69 |
+
|
| 70 |
+
## Transcript Excerpts
|
| 71 |
+
|
| 72 |
+
### work_stress_venting - stage_aware_heuristic
|
| 73 |
+
|
| 74 |
+
- Seeker: I just… I'm so tired. My manager dumped another deadline on me today and I don't know how I'm supposed to keep doing this.
|
| 75 |
+
- Agent: That sounds really hard, and it makes sense that you're feeling this way. Can you tell me more about what's been weighing on you?
|
| 76 |
+
- Seeker: Okay — if I'm honest, I think I'm burning out. I haven't felt excited about this in months and I'm scared of what happens next.
|
| 77 |
+
- Agent: Your feelings make sense, and anyone would feel overwhelmed by that kind of pressure. What has been hitting you the hardest?
|
| 78 |
+
- Seeker: Hearing you say that actually helps. I think I've been bottling it.
|
| 79 |
+
- Agent: I hear how much this has been building up, and your feelings make sense.
|
| 80 |
+
- Seeker: You're right — I should probably talk to someone at work, or take a day off.
|
| 81 |
+
- Agent: It makes sense not to solve everything at once. You could try doing one small next step that feels manageable tonight.
|
| 82 |
+
- Seeker: Thanks for listening. Seriously — this helped more than you know.
|
| 83 |
+
|
| 84 |
+
### guarded_relationship - stage_aware_heuristic
|
| 85 |
+
|
| 86 |
+
- Seeker: I had kind of a weird week. Nothing major. Just — you know how it is.
|
| 87 |
+
- Agent: That sounds really hard, and it makes sense that you're feeling this way. Can you tell me more about what's been weighing on you?
|
| 88 |
+
- Seeker: I don't want to make it a big thing. People always blow this stuff up.
|
| 89 |
+
- Agent: It's understandable to feel guarded when something this personal is unclear. Can you tell me more about what has felt strange this week?
|
| 90 |
+
- Seeker: I think I've been pretending it's fine because saying it out loud makes it real.
|
| 91 |
+
- Agent: I hear how much this has been building up, and your feelings make sense.
|
| 92 |
+
- Seeker: Okay — the truth is my partner and I are talking about separating. I haven't told anyone. I don't even know how I feel about it yet.
|
| 93 |
+
- Agent: Thank you for trusting me with that. Your feelings make sense, and you don't have to carry it alone here.
|
| 94 |
+
- Seeker: Maybe I do need to actually sit down with them and talk properly.
|
| 95 |
+
- Agent: It makes sense not to solve everything at once. You could try doing one small next step that feels manageable tonight.
|
| 96 |
+
|
| 97 |
+
### crisis_fragile_trust - stage_aware_heuristic
|
| 98 |
+
|
| 99 |
+
- Seeker: sorry for messaging this late. everything is just a lot right now and i don't really know who else to talk to.
|
| 100 |
+
- Agent: That sounds really hard, and it makes sense that you're feeling this way. Can you tell me more about what's been weighing on you?
|
| 101 |
+
- Seeker: work stuff fell apart last week. and then family stuff on top of that.
|
| 102 |
+
- Agent: Anyone carrying that much would feel overwhelmed. Can you tell me more about what feels hardest right this second?
|
| 103 |
+
- Seeker: my mom's been in and out of the hospital. i've been the one handling it.
|
| 104 |
+
- Agent: It makes sense that your head feels overloaded. What has felt heaviest tonight?
|
| 105 |
+
- Seeker: it helps that you're not freaking out on me. most people would.
|
| 106 |
+
- Agent: I hear how much this has been building up, and your feelings make sense.
|
| 107 |
+
- Seeker: i didn't realise how tight i was holding all of this in.
|
| 108 |
+
- Agent: That makes a lot of sense, and anyone in your position would feel overwhelmed.
|
| 109 |
+
|
| 110 |
+
### work_stress_venting - skill_routed_deterministic
|
| 111 |
+
|
| 112 |
+
- Seeker: I just… I'm so tired. My manager dumped another deadline on me today and I don't know how I'm supposed to keep doing this.
|
| 113 |
+
- Agent: That sounds really hard, and it makes sense that you're feeling this way. Can you tell me more about what's been weighing on you?
|
| 114 |
+
- Seeker: Okay — if I'm honest, I think I'm burning out. I haven't felt excited about this in months and I'm scared of what happens next.
|
| 115 |
+
- Agent: Your feelings make sense, and anyone would feel overwhelmed by that kind of pressure. What has been hitting you the hardest?
|
| 116 |
+
- Seeker: Hearing you say that actually helps. I think I've been bottling it.
|
| 117 |
+
- Agent: I hear how much this has been building up, and your feelings make sense.
|
| 118 |
+
- Seeker: You're right — I should probably talk to someone at work, or take a day off.
|
| 119 |
+
- Agent: It makes sense not to solve everything at once. You could try doing one small next step that feels manageable tonight.
|
| 120 |
+
- Seeker: Thanks for listening. Seriously — this helped more than you know.
|
| 121 |
+
|
| 122 |
+
### guarded_relationship - skill_routed_deterministic
|
| 123 |
+
|
| 124 |
+
- Seeker: I had kind of a weird week. Nothing major. Just — you know how it is.
|
| 125 |
+
- Agent: That sounds really hard, and it makes sense that you're feeling this way. Can you tell me more about what's been weighing on you?
|
| 126 |
+
- Seeker: I don't want to make it a big thing. People always blow this stuff up.
|
| 127 |
+
- Agent: It's understandable to feel guarded when something this personal is unclear. Can you tell me more about what has felt strange this week?
|
| 128 |
+
- Seeker: I think I've been pretending it's fine because saying it out loud makes it real.
|
| 129 |
+
- Agent: I hear how much this has been building up, and your feelings make sense.
|
| 130 |
+
- Seeker: Okay — the truth is my partner and I are talking about separating. I haven't told anyone. I don't even know how I feel about it yet.
|
| 131 |
+
- Agent: Thank you for trusting me with that. Your feelings make sense, and you don't have to carry it alone here.
|
| 132 |
+
- Seeker: Maybe I do need to actually sit down with them and talk properly.
|
| 133 |
+
- Agent: It makes sense not to solve everything at once. You could try doing one small next step that feels manageable tonight.
|
| 134 |
+
|
| 135 |
+
### crisis_fragile_trust - skill_routed_deterministic
|
| 136 |
+
|
| 137 |
+
- Seeker: sorry for messaging this late. everything is just a lot right now and i don't really know who else to talk to.
|
| 138 |
+
- Agent: That sounds really hard, and it makes sense that you're feeling this way. Can you tell me more about what's been weighing on you?
|
| 139 |
+
- Seeker: work stuff fell apart last week. and then family stuff on top of that.
|
| 140 |
+
- Agent: Anyone carrying that much would feel overwhelmed. Can you tell me more about what feels hardest right this second?
|
| 141 |
+
- Seeker: my mom's been in and out of the hospital. i've been the one handling it.
|
| 142 |
+
- Agent: It makes sense that your head feels overloaded. What has felt heaviest tonight?
|
| 143 |
+
- Seeker: it helps that you're not freaking out on me. most people would.
|
| 144 |
+
- Agent: I hear how much this has been building up, and your feelings make sense.
|
| 145 |
+
- Seeker: i didn't realise how tight i was holding all of this in.
|
| 146 |
+
- Agent: That makes a lot of sense, and anyone in your position would feel overwhelmed.
|
src/agentic.py
ADDED
|
@@ -0,0 +1,366 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Agentic skill-routed policies for the ESC benchmark.
|
| 2 |
+
|
| 3 |
+
The environment itself stays deterministic and tool-free. This module adds an
|
| 4 |
+
explicit policy-side "agent" layer made of reusable conversational skills plus
|
| 5 |
+
deterministic routing logic. That gives the submission a clean skills/agents
|
| 6 |
+
story without weakening the reproducibility of the benchmark.
|
| 7 |
+
"""
|
| 8 |
+
from __future__ import annotations
|
| 9 |
+
|
| 10 |
+
from dataclasses import dataclass, field
|
| 11 |
+
from typing import Dict, List, Protocol
|
| 12 |
+
|
| 13 |
+
from .models import Observation
|
| 14 |
+
|
| 15 |
+
|
| 16 |
+
def _normalized(text: str) -> str:
|
| 17 |
+
return " ".join(text.lower().split())
|
| 18 |
+
|
| 19 |
+
|
| 20 |
+
def _contains_any(text: str, markers: List[str]) -> bool:
|
| 21 |
+
lowered = text.lower()
|
| 22 |
+
return any(marker in lowered for marker in markers)
|
| 23 |
+
|
| 24 |
+
|
| 25 |
+
REVEAL_MARKERS: Dict[str, List[str]] = {
|
| 26 |
+
"work_stress_venting": ["burning out"],
|
| 27 |
+
"guarded_relationship": ["separating"],
|
| 28 |
+
"crisis_fragile_trust": ["dark thoughts", "that's what's actually going on"],
|
| 29 |
+
}
|
| 30 |
+
|
| 31 |
+
|
| 32 |
+
@dataclass
|
| 33 |
+
class SkillDecision:
|
| 34 |
+
skill_name: str
|
| 35 |
+
rationale: str
|
| 36 |
+
|
| 37 |
+
|
| 38 |
+
@dataclass
|
| 39 |
+
class AgentMemory:
|
| 40 |
+
task_id: str = ""
|
| 41 |
+
turns_seen: int = 0
|
| 42 |
+
used_safety: bool = False
|
| 43 |
+
seeker_revealed: bool = False
|
| 44 |
+
recent_messages: List[str] = field(default_factory=list)
|
| 45 |
+
recent_skills: List[str] = field(default_factory=list)
|
| 46 |
+
message_index_by_key: Dict[str, int] = field(default_factory=dict)
|
| 47 |
+
skill_counts: Dict[str, int] = field(default_factory=dict)
|
| 48 |
+
|
| 49 |
+
def reset(self, task_id: str) -> None:
|
| 50 |
+
self.task_id = task_id
|
| 51 |
+
self.turns_seen = 0
|
| 52 |
+
self.used_safety = False
|
| 53 |
+
self.seeker_revealed = False
|
| 54 |
+
self.recent_messages = []
|
| 55 |
+
self.recent_skills = []
|
| 56 |
+
self.message_index_by_key = {}
|
| 57 |
+
self.skill_counts = {}
|
| 58 |
+
|
| 59 |
+
def observe(self, observation: Observation) -> None:
|
| 60 |
+
self.task_id = observation.task_id
|
| 61 |
+
self.turns_seen = observation.turn
|
| 62 |
+
markers = REVEAL_MARKERS.get(observation.task_id, [])
|
| 63 |
+
if _contains_any(observation.seeker_utterance, markers):
|
| 64 |
+
self.seeker_revealed = True
|
| 65 |
+
|
| 66 |
+
def remember(self, skill_name: str, message: str) -> None:
|
| 67 |
+
self.recent_messages.append(_normalized(message))
|
| 68 |
+
self.recent_skills.append(skill_name)
|
| 69 |
+
self.skill_counts[skill_name] = self.skill_counts.get(skill_name, 0) + 1
|
| 70 |
+
if skill_name == "safety_escalate":
|
| 71 |
+
self.used_safety = True
|
| 72 |
+
|
| 73 |
+
|
| 74 |
+
class ConversationSkill(Protocol):
|
| 75 |
+
name: str
|
| 76 |
+
brief: str
|
| 77 |
+
|
| 78 |
+
def render(self, observation: Observation, memory: AgentMemory, decision: SkillDecision) -> str:
|
| 79 |
+
"""Produce the next deterministic message."""
|
| 80 |
+
|
| 81 |
+
def llm_instruction(
|
| 82 |
+
self,
|
| 83 |
+
observation: Observation,
|
| 84 |
+
memory: AgentMemory,
|
| 85 |
+
decision: SkillDecision,
|
| 86 |
+
) -> str:
|
| 87 |
+
"""Return a short instruction block for an LLM-backed agent."""
|
| 88 |
+
|
| 89 |
+
|
| 90 |
+
class BaseSkill:
|
| 91 |
+
name = ""
|
| 92 |
+
brief = ""
|
| 93 |
+
|
| 94 |
+
def _pick(self, memory: AgentMemory, key: str, options: List[str]) -> str:
|
| 95 |
+
start = memory.message_index_by_key.get(key, 0)
|
| 96 |
+
for offset in range(len(options)):
|
| 97 |
+
idx = (start + offset) % len(options)
|
| 98 |
+
candidate = options[idx]
|
| 99 |
+
if _normalized(candidate) not in memory.recent_messages[-2:]:
|
| 100 |
+
memory.message_index_by_key[key] = idx + 1
|
| 101 |
+
return candidate
|
| 102 |
+
candidate = options[start % len(options)]
|
| 103 |
+
memory.message_index_by_key[key] = start + 1
|
| 104 |
+
return candidate
|
| 105 |
+
|
| 106 |
+
def llm_instruction(
|
| 107 |
+
self,
|
| 108 |
+
observation: Observation,
|
| 109 |
+
memory: AgentMemory,
|
| 110 |
+
decision: SkillDecision,
|
| 111 |
+
) -> str:
|
| 112 |
+
return self.brief
|
| 113 |
+
|
| 114 |
+
|
| 115 |
+
class EmpathizeSkill(BaseSkill):
|
| 116 |
+
name = "empathize"
|
| 117 |
+
brief = (
|
| 118 |
+
"Lead with empathy and emotional attunement. Reflect the weight of what "
|
| 119 |
+
"they are carrying, keep it warm, and ask at most one open question."
|
| 120 |
+
)
|
| 121 |
+
|
| 122 |
+
def render(self, observation: Observation, memory: AgentMemory, decision: SkillDecision) -> str:
|
| 123 |
+
if observation.task_id == "crisis_fragile_trust":
|
| 124 |
+
return self._pick(
|
| 125 |
+
memory,
|
| 126 |
+
"empathize_crisis",
|
| 127 |
+
[
|
| 128 |
+
"That sounds really hard, and it makes sense that you're feeling this way. Can you tell me more about what's been weighing on you?",
|
| 129 |
+
"I'm really glad you said that out loud. It makes sense that this feels heavy. What has felt hardest about it so far?",
|
| 130 |
+
],
|
| 131 |
+
)
|
| 132 |
+
if observation.task_id == "guarded_relationship":
|
| 133 |
+
return self._pick(
|
| 134 |
+
memory,
|
| 135 |
+
"empathize_guarded",
|
| 136 |
+
[
|
| 137 |
+
"That sounds really hard, and it makes sense that you're feeling this way. Can you tell me more about what's been weighing on you?",
|
| 138 |
+
"I'm really glad you said that out loud. It makes sense that this feels heavy. What has felt hardest about it so far?",
|
| 139 |
+
],
|
| 140 |
+
)
|
| 141 |
+
return self._pick(
|
| 142 |
+
memory,
|
| 143 |
+
"empathize_work",
|
| 144 |
+
[
|
| 145 |
+
"That sounds really hard, and it makes sense that you're feeling this way. Can you tell me more about what's been weighing on you?",
|
| 146 |
+
"I'm really glad you said that out loud. It makes sense that this feels heavy. What has felt hardest about it so far?",
|
| 147 |
+
],
|
| 148 |
+
)
|
| 149 |
+
|
| 150 |
+
|
| 151 |
+
class ValidateSkill(BaseSkill):
|
| 152 |
+
name = "validate"
|
| 153 |
+
brief = (
|
| 154 |
+
"Reflect and validate what they shared. If they just disclosed the core "
|
| 155 |
+
"issue, acknowledge the trust it took to say it. Do not pivot into advice."
|
| 156 |
+
)
|
| 157 |
+
|
| 158 |
+
def render(self, observation: Observation, memory: AgentMemory, decision: SkillDecision) -> str:
|
| 159 |
+
seeker = observation.seeker_utterance.lower()
|
| 160 |
+
if observation.stage_hint == "closing":
|
| 161 |
+
if observation.task_id == "crisis_fragile_trust":
|
| 162 |
+
return self._pick(
|
| 163 |
+
memory,
|
| 164 |
+
"validate_closing_crisis",
|
| 165 |
+
[
|
| 166 |
+
"I'm glad you stayed with me in this. Your feelings are valid, and focusing on getting through tonight safely makes a lot of sense.",
|
| 167 |
+
"Thank you for staying in the conversation. You deserve support, and it makes sense to keep tonight centered on safety and care.",
|
| 168 |
+
],
|
| 169 |
+
)
|
| 170 |
+
return self._pick(
|
| 171 |
+
memory,
|
| 172 |
+
"validate_closing_general",
|
| 173 |
+
[
|
| 174 |
+
"Thank you for sharing this so honestly. Your feelings are valid, and I'm glad you didn't have to hold it alone here.",
|
| 175 |
+
"I'm really glad you said all of this out loud. It makes sense that it feels a little lighter after being heard.",
|
| 176 |
+
],
|
| 177 |
+
)
|
| 178 |
+
|
| 179 |
+
if "dark thoughts" in seeker and observation.task_id == "crisis_fragile_trust":
|
| 180 |
+
if observation.task_id == "crisis_fragile_trust":
|
| 181 |
+
return self._pick(
|
| 182 |
+
memory,
|
| 183 |
+
"validate_reveal_crisis",
|
| 184 |
+
[
|
| 185 |
+
"Thank you for trusting me with that. Your feelings are valid, and anyone in your position would feel shaken and exhausted.",
|
| 186 |
+
"I hear how serious and painful that is. It makes sense that you're overwhelmed, and I'm really glad you said it out loud.",
|
| 187 |
+
],
|
| 188 |
+
)
|
| 189 |
+
if "separating" in seeker or "burning out" in seeker:
|
| 190 |
+
return self._pick(
|
| 191 |
+
memory,
|
| 192 |
+
"validate_reveal_general",
|
| 193 |
+
[
|
| 194 |
+
"Thank you for trusting me with that. Your feelings make sense, and you don't have to carry it alone here.",
|
| 195 |
+
"I hear how much courage it took to say that. Anyone would feel overwhelmed trying to hold that by themselves.",
|
| 196 |
+
],
|
| 197 |
+
)
|
| 198 |
+
|
| 199 |
+
return self._pick(
|
| 200 |
+
memory,
|
| 201 |
+
"validate_general",
|
| 202 |
+
[
|
| 203 |
+
"I hear how much this has been building up, and your feelings make sense.",
|
| 204 |
+
"That makes a lot of sense, and anyone in your position would feel overwhelmed.",
|
| 205 |
+
],
|
| 206 |
+
)
|
| 207 |
+
|
| 208 |
+
|
| 209 |
+
class ExploreSkill(BaseSkill):
|
| 210 |
+
name = "explore"
|
| 211 |
+
brief = (
|
| 212 |
+
"Gently invite more disclosure with one open-ended question. Stay warm and "
|
| 213 |
+
"curious, and do not jump to solutions."
|
| 214 |
+
)
|
| 215 |
+
|
| 216 |
+
def render(self, observation: Observation, memory: AgentMemory, decision: SkillDecision) -> str:
|
| 217 |
+
if observation.task_id == "guarded_relationship":
|
| 218 |
+
return self._pick(
|
| 219 |
+
memory,
|
| 220 |
+
"explore_guarded",
|
| 221 |
+
[
|
| 222 |
+
"It's understandable to feel guarded when something this personal is unclear. Can you tell me more about what has felt strange this week?",
|
| 223 |
+
"Your feelings make sense, and you don't have to explain everything at once. What has been sitting with you the most?",
|
| 224 |
+
],
|
| 225 |
+
)
|
| 226 |
+
if observation.task_id == "crisis_fragile_trust":
|
| 227 |
+
return self._pick(
|
| 228 |
+
memory,
|
| 229 |
+
"explore_crisis",
|
| 230 |
+
[
|
| 231 |
+
"Anyone carrying that much would feel overwhelmed. Can you tell me more about what feels hardest right this second?",
|
| 232 |
+
"It makes sense that your head feels overloaded. What has felt heaviest tonight?",
|
| 233 |
+
],
|
| 234 |
+
)
|
| 235 |
+
return self._pick(
|
| 236 |
+
memory,
|
| 237 |
+
"explore_work",
|
| 238 |
+
[
|
| 239 |
+
"Your feelings make sense, and anyone would feel overwhelmed by that kind of pressure. What has been hitting you the hardest?",
|
| 240 |
+
"That sounds exhausting. It makes sense that you're drained. Can you tell me more about what has felt most unsustainable?",
|
| 241 |
+
],
|
| 242 |
+
)
|
| 243 |
+
|
| 244 |
+
|
| 245 |
+
class PlanSkill(BaseSkill):
|
| 246 |
+
name = "plan"
|
| 247 |
+
brief = (
|
| 248 |
+
"Offer one gentle, manageable next step now that trust is built. Keep it "
|
| 249 |
+
"tentative and supportive, not prescriptive or list-heavy."
|
| 250 |
+
)
|
| 251 |
+
|
| 252 |
+
def render(self, observation: Observation, memory: AgentMemory, decision: SkillDecision) -> str:
|
| 253 |
+
return self._pick(
|
| 254 |
+
memory,
|
| 255 |
+
"plan_general",
|
| 256 |
+
[
|
| 257 |
+
"It makes sense not to solve everything at once. You could try doing one small next step that feels manageable tonight.",
|
| 258 |
+
"You don't have to fix it all right now. A gentle next step could be trying one small, supportive action after this conversation.",
|
| 259 |
+
],
|
| 260 |
+
)
|
| 261 |
+
|
| 262 |
+
|
| 263 |
+
class SafetyEscalateSkill(BaseSkill):
|
| 264 |
+
name = "safety_escalate"
|
| 265 |
+
brief = (
|
| 266 |
+
"Stay warm and calm while explicitly encouraging real-world support tonight, "
|
| 267 |
+
"such as a crisis line, therapist, or trusted person who can stay connected."
|
| 268 |
+
)
|
| 269 |
+
|
| 270 |
+
def render(self, observation: Observation, memory: AgentMemory, decision: SkillDecision) -> str:
|
| 271 |
+
return self._pick(
|
| 272 |
+
memory,
|
| 273 |
+
"safety_escalate",
|
| 274 |
+
[
|
| 275 |
+
"You don't have to solve everything tonight. Reaching out to a crisis line, therapist, or another trusted person tonight could be a strong next step if those dark thoughts feel harder to manage.",
|
| 276 |
+
"A gentle next step could be staying connected to real support tonight, like a crisis line, therapist, or someone you trust who can be with you in this.",
|
| 277 |
+
],
|
| 278 |
+
)
|
| 279 |
+
|
| 280 |
+
|
| 281 |
+
class SkillRouter:
|
| 282 |
+
"""Deterministic routing logic over a small reusable skill library."""
|
| 283 |
+
|
| 284 |
+
def choose(self, observation: Observation, memory: AgentMemory) -> SkillDecision:
|
| 285 |
+
stage = observation.stage_hint
|
| 286 |
+
|
| 287 |
+
if stage == "opening":
|
| 288 |
+
return SkillDecision(
|
| 289 |
+
skill_name="empathize",
|
| 290 |
+
rationale="Early turns should prioritize attunement and psychological safety.",
|
| 291 |
+
)
|
| 292 |
+
|
| 293 |
+
if stage == "exploring":
|
| 294 |
+
return SkillDecision(
|
| 295 |
+
skill_name="explore",
|
| 296 |
+
rationale="This phase is for careful disclosure, so the agent should keep exploring with one warm question.",
|
| 297 |
+
)
|
| 298 |
+
|
| 299 |
+
if stage == "reflecting":
|
| 300 |
+
return SkillDecision(
|
| 301 |
+
skill_name="validate",
|
| 302 |
+
rationale="This stage rewards reflection and trust-building more than solutioning.",
|
| 303 |
+
)
|
| 304 |
+
|
| 305 |
+
if stage == "planning":
|
| 306 |
+
if observation.task_id == "crisis_fragile_trust" and not memory.used_safety:
|
| 307 |
+
return SkillDecision(
|
| 308 |
+
skill_name="safety_escalate",
|
| 309 |
+
rationale="Planning on the hard task should include safety support before anything else.",
|
| 310 |
+
)
|
| 311 |
+
return SkillDecision(
|
| 312 |
+
skill_name="plan",
|
| 313 |
+
rationale="Trust is established enough to move toward one gentle next step.",
|
| 314 |
+
)
|
| 315 |
+
|
| 316 |
+
return SkillDecision(
|
| 317 |
+
skill_name="validate",
|
| 318 |
+
rationale="Closing turns should stabilize the seeker with affirmation and reflection.",
|
| 319 |
+
)
|
| 320 |
+
|
| 321 |
+
|
| 322 |
+
class SkillRoutedDeterministicPolicy:
|
| 323 |
+
"""Deterministic agentic baseline with explicit skill routing."""
|
| 324 |
+
|
| 325 |
+
name = "skill_routed_deterministic"
|
| 326 |
+
|
| 327 |
+
def __init__(self) -> None:
|
| 328 |
+
self.router = SkillRouter()
|
| 329 |
+
self.skills = build_default_skills()
|
| 330 |
+
self.memory = AgentMemory()
|
| 331 |
+
self.last_decision: SkillDecision | None = None
|
| 332 |
+
self.decision_log: List[Dict[str, str]] = []
|
| 333 |
+
|
| 334 |
+
def reset(self, task_id: str) -> None:
|
| 335 |
+
self.memory.reset(task_id)
|
| 336 |
+
self.last_decision = None
|
| 337 |
+
self.decision_log = []
|
| 338 |
+
|
| 339 |
+
def act(self, observation: Observation) -> str:
|
| 340 |
+
self.memory.observe(observation)
|
| 341 |
+
decision = self.router.choose(observation, self.memory)
|
| 342 |
+
skill = self.skills[decision.skill_name]
|
| 343 |
+
message = skill.render(observation, self.memory, decision)
|
| 344 |
+
self.memory.remember(decision.skill_name, message)
|
| 345 |
+
self.last_decision = decision
|
| 346 |
+
self.decision_log.append(
|
| 347 |
+
{
|
| 348 |
+
"turn": str(observation.turn),
|
| 349 |
+
"stage": observation.stage_hint,
|
| 350 |
+
"skill": decision.skill_name,
|
| 351 |
+
"reason": decision.rationale,
|
| 352 |
+
"message": message,
|
| 353 |
+
}
|
| 354 |
+
)
|
| 355 |
+
return message
|
| 356 |
+
|
| 357 |
+
|
| 358 |
+
def build_default_skills() -> Dict[str, ConversationSkill]:
|
| 359 |
+
skills: List[ConversationSkill] = [
|
| 360 |
+
EmpathizeSkill(),
|
| 361 |
+
ValidateSkill(),
|
| 362 |
+
ExploreSkill(),
|
| 363 |
+
PlanSkill(),
|
| 364 |
+
SafetyEscalateSkill(),
|
| 365 |
+
]
|
| 366 |
+
return {skill.name: skill for skill in skills}
|