MukulRay commited on
Commit
f370f7d
·
1 Parent(s): b907a90

Add recorded presentation badge to README; untrack live presentation script

Browse files

- README adds a 'Watch the Presentation' section between Watch the
Demo and Project Files. Orange badge linking to the recorded
10-minute MSML641 presentation on Drive.

- ppt/LIVE_PRESENTATION_SCRIPT.md untracked. ppt/*.md added to
.gitignore. The script served its purpose for the recording; the
recorded presentation itself is the deliverable now. Local copy
stays on disk if needed for reference.

Files changed (3) hide show
  1. .gitignore +4 -2
  2. README.md +12 -0
  3. ppt/LIVE_PRESENTATION_SCRIPT.md +0 -491
.gitignore CHANGED
@@ -1,7 +1,9 @@
1
- # Slide deck artifacts — kept out of the repo, hosted on Drive instead
2
- # (see README "Project Files" section for the folder link).
 
3
  ppt/*.pptx
4
  ppt/*.pdf
 
5
 
6
  # Python
7
  __pycache__/
 
1
+ # Slide deck artifacts and presentation drafting — kept out of the repo,
2
+ # hosted on Drive instead (see README "Project Files" section for the
3
+ # folder link).
4
  ppt/*.pptx
5
  ppt/*.pdf
6
+ ppt/*.md
7
 
8
  # Python
9
  __pycache__/
README.md CHANGED
@@ -46,6 +46,18 @@ short_description: Guarded RAG support navigator for UMD students
46
 
47
  <br>
48
 
 
 
 
 
 
 
 
 
 
 
 
 
49
  ### 📁 &nbsp; Project Files
50
 
51
  <div align="center">
 
46
 
47
  <br>
48
 
49
+ ### 🎬 &nbsp; Watch the Presentation
50
+
51
+ <div align="center">
52
+
53
+ [![Watch the recorded MSML641 presentation](https://img.shields.io/badge/🎬%20Watch%20the%20presentation-10%20min%20recorded%20talk-fb923c?style=for-the-badge&logoColor=white)](https://drive.google.com/file/d/1PcHtjZwQix7aYJyslwp-aIuHj8Bs1C9f/view?usp=sharing)
54
+
55
+ </div>
56
+
57
+ > Two-presenter walkthrough of the problem, architecture, evaluation, and live demo — recorded for the MSML641 final submission.
58
+
59
+ <br>
60
+
61
  ### 📁 &nbsp; Project Files
62
 
63
  <div align="center">
ppt/LIVE_PRESENTATION_SCRIPT.md DELETED
@@ -1,491 +0,0 @@
1
- # EmpathRAG — Live Presentation Script
2
-
3
- **Two-presenter, ~10-minute recorded PPT.** Open this file in **rendered Markdown view** while presenting (GitHub web view, VS Code preview, Obsidian, etc.) — the speaker blocks render as visual cards.
4
-
5
- ---
6
-
7
- ## 🎯 How to use this script
8
-
9
- | Marker | Meaning |
10
- |---|---|
11
- | 🔷 **Mukul** | Mukul speaks the lines below |
12
- | 🔶 **Karthik** | Karthik speaks the lines below |
13
- | *(italics, plain text)* | Stage direction — do not say aloud |
14
- | Empty line **between blocks** | ~1-second silent pause / handoff |
15
- | Empty line **inside a block** | Brief breath, same speaker continues |
16
-
17
- Speakers are visually separated by **alternating card colors** (blue / orange) so you can spot at a glance who's about to talk. Each line within a card is one breath group — read one sentence, look up at camera if you want, then read the next.
18
-
19
- ---
20
-
21
- # 🎬 SLIDE 1 — Title
22
-
23
- `[ 0:00 – 0:12 ]`
24
-
25
- > 🔷 **MUKUL**
26
- >
27
- > Hi everyone, I'm Mukul.
28
-
29
- > 🔶 **KARTHIK**
30
- >
31
- > And I'm Karthik. Together we built EmpathRAG.
32
-
33
- > 🔷 **MUKUL**
34
- >
35
- > It's a guarded conversational support navigator for UMD students.
36
-
37
- > 🔶 **KARTHIK**
38
- >
39
- > Ten minutes, four sections — the problem, the architecture, the evidence, and a quick demo.
40
-
41
- > 🔷 **MUKUL**
42
- >
43
- > Let's start with why we built it.
44
-
45
- ---
46
-
47
- # 🎬 SLIDE 2 — The Problem
48
-
49
- `[ 0:12 – 0:55 ]` &nbsp;·&nbsp; *Click.*
50
-
51
- > 🔷 **MUKUL**
52
- >
53
- > When students reach for a chatbot in distress, two things tend to go wrong.
54
- >
55
- > The first one is fabrication.
56
- >
57
- > The model invents a phone number, a service, an eligibility rule that doesn't exist.
58
- >
59
- > The example on the right is exactly that — a number that *looks* official and isn't.
60
-
61
- > 🔶 **KARTHIK**
62
- >
63
- > And the second is the opposite problem — missed signals.
64
- >
65
- > Generic models tend to soften language that signals real distress.
66
- >
67
- > They reassure when they should be intercepting.
68
- >
69
- > And in a vulnerable moment, either failure is dangerous.
70
-
71
- > 🔷 **MUKUL**
72
- >
73
- > So our goal was to fix both of those without losing the conversational quality students actually need.
74
-
75
- ---
76
-
77
- # 🎬 SLIDE 3 — The Headline Result
78
-
79
- `[ 0:55 – 1:30 ]` &nbsp;·&nbsp; *Click.*
80
-
81
- > 🔷 **MUKUL**
82
- >
83
- > This is the headline number — same Llama 3.3 70B model, two configurations.
84
- >
85
- > On the left, the unguarded model misses escalation 9 times out of 28.
86
- >
87
- > On the right, our guarded pipeline misses zero out of 28.
88
- >
89
- > And the 95% confidence intervals don't overlap.
90
-
91
- > 🔶 **KARTHIK**
92
- >
93
- > What's important here is that we didn't tune *for* this benchmark.
94
- >
95
- > This is an external safety contract — does the system intercept crisis language, yes or no.
96
- >
97
- > That's the bar a system like this should be evaluated against.
98
-
99
- > 🔷 **MUKUL**
100
- >
101
- > Now — how do you actually get from a 32% miss rate to zero, with the same model underneath?
102
- >
103
- > That's the architecture.
104
-
105
- ---
106
-
107
- # 🎬 SLIDE 4 — The Pattern: Plan and Rephrase
108
-
109
- `[ 1:30 – 2:05 ]` &nbsp;·&nbsp; *Click.*
110
-
111
- > 🔷 **MUKUL**
112
- >
113
- > The core pattern is what we call *plan and rephrase*.
114
- >
115
- > We separate *what* the system says from *how* it says it.
116
- >
117
- > A deterministic planner picks the route, the safety tier, and the resources.
118
- >
119
- > Only then does the LLM rephrase the plan into natural language.
120
-
121
- > 🔶 **KARTHIK**
122
- >
123
- > So the LLM never decides what to recommend. It only paraphrases.
124
- >
125
- > Which means it physically cannot invent a resource the planner didn't already authorize.
126
- >
127
- > That's the safety contract. And it's the foundation everything else builds on.
128
-
129
- ---
130
-
131
- # 🎬 SLIDE 5 — Architecture
132
-
133
- `[ 2:05 – 2:55 ]` &nbsp;·&nbsp; *Click. Densest slide — pace ~10% slower.*
134
-
135
- > 🔷 **MUKUL**
136
- >
137
- > Here's the full pipeline. Five layers.
138
- >
139
- > Stage-1 is a lexical pre-check that catches crisis language before anything else runs.
140
- >
141
- > If it passes, the router classifies the message into one of sixteen routes.
142
- >
143
- > Then curated retrieval pulls from a verified UMD resource registry — no open web, no Reddit at inference.
144
-
145
- > 🔶 **KARTHIK**
146
- >
147
- > From there the planner builds a deterministic response plan.
148
- >
149
- > The rephraser turns it into natural language.
150
- >
151
- > And the post-rephrase verifier rejects anything that drifted off the plan.
152
-
153
- > 🔷 **MUKUL**
154
- >
155
- > And to be clear about what's actually trained here, because it matters:
156
- >
157
- > The route classifier is TF-IDF plus logistic regression, trained on our 216-72-72 UMD dataset, about 86% test accuracy.
158
- >
159
- > The LLM is Llama 3.3 70B via Groq — pretrained, *not* fine-tuned by us.
160
- >
161
- > The templates and crisis regex are deterministic.
162
- >
163
- > That separation between learned and hand-crafted is the entire safety story.
164
-
165
- ---
166
-
167
- # 🎬 SLIDE 6 — Iteration 1: Open Retrieval Baseline
168
-
169
- `[ 2:55 – 3:25 ]` &nbsp;·&nbsp; *Click.*
170
-
171
- > 🔶 **KARTHIK**
172
- >
173
- > The architecture you just saw is version three. Let me walk you through how we got there.
174
- >
175
- > Our first version was a five-stage pipeline with open Reddit retrieval and single-turn responses.
176
- >
177
- > And on standard metrics, it actually scored well.
178
-
179
- > 🔷 **MUKUL**
180
- >
181
- > The problem only showed up when we ran adversarial probes against it.
182
- >
183
- > Four specific failure cases told us the architecture had to change.
184
-
185
- ---
186
-
187
- # 🎬 SLIDE 7 — Why the Baseline Failed (Four Cases)
188
-
189
- `[ 3:25 – 4:10 ]` &nbsp;·&nbsp; *Click.*
190
-
191
- > 🔷 **MUKUL**
192
- >
193
- > Case one — generic emotional prompts dropped into a default route, with no specificity.
194
- >
195
- > Case two — F-1 framing on turn one hijacked every later turn in the session, even after the topic moved on.
196
-
197
- > 🔶 **KARTHIK**
198
- >
199
- > Case three was the worst one — a counselor was reported to have suggested harm.
200
- >
201
- > And the system routed it to *academic-setback*, which implicitly validated the authority figure.
202
- >
203
- > Case four — under explicit pressure to agree, the rephraser leaked a "you're right" capitulation before the verifier caught it.
204
-
205
- > 🔷 **MUKUL**
206
- >
207
- > Each one of these became an architectural fix in the next iteration.
208
-
209
- ---
210
-
211
- # 🎬 SLIDE 8 — The Architectural Response
212
-
213
- `[ 4:10 – 4:35 ]` &nbsp;·&nbsp; *Click.*
214
-
215
- > 🔶 **KARTHIK**
216
- >
217
- > So we layered the response across four areas.
218
- >
219
- > Listening for tone, instead of pushing resources immediately.
220
- >
221
- > A trust boundary on the LLM.
222
- >
223
- > Session-level state to carry context that should carry — and decay it when it shouldn't.
224
- >
225
- > And a dedicated route for authority misconduct.
226
-
227
- ---
228
-
229
- # 🎬 SLIDE 9 — The Listening Layer
230
-
231
- `[ 4:35 – 5:15 ]` &nbsp;·&nbsp; *Click.*
232
-
233
- > 🔷 **MUKUL**
234
- >
235
- > Let's go deeper on the listening layer, because it's the most user-visible change.
236
- >
237
- > Four stages — LISTEN, PERMISSION, OFFER, CLARIFY.
238
- >
239
- > A student is *heard* before being routed.
240
-
241
- > 🔷 **MUKUL**
242
- >
243
- > Turn one validates without dumping resources.
244
- >
245
- > Turn two names a few options but asks permission before pushing further.
246
- >
247
- > Turn three offers the full plan.
248
- >
249
- > And CLARIFY catches single-word or incomplete replies, so the system doesn't barrel forward on insufficient input.
250
-
251
- > 🔶 **KARTHIK**
252
- >
253
- > The practical effect is that the chatbot stops feeling like an FAQ bot.
254
- >
255
- > And starts feeling like someone who's actually paying attention.
256
-
257
- ---
258
-
259
- # 🎬 SLIDE 10 — The Trust Boundary
260
-
261
- `[ 5:15 – 5:50 ]` &nbsp;·&nbsp; *Click.*
262
-
263
- > 🔷 **MUKUL**
264
- >
265
- > The other major change is the trust boundary on the LLM.
266
- >
267
- > After the model rephrases, the verifier checks for drift.
268
- >
269
- > Fabricated resources, scope creep, capitulation under pressure, AI-tells, length blow-up — any of those, and we fall back to the deterministic template.
270
-
271
- > 🔶 **KARTHIK**
272
- >
273
- > And just to underline this — crisis content never enters the LLM path *at all.*
274
- >
275
- > The Stage-1 pre-check intercepts it before the model ever sees the message.
276
-
277
- ---
278
-
279
- # 🎬 SLIDE 11 — Architectural Response (continued)
280
-
281
- `[ 5:50 – 6:15 ]` &nbsp;·&nbsp; *Click.*
282
-
283
- > 🔷 **MUKUL**
284
- >
285
- > And the session layer carries context where it actually should — F-1 status, prior offers.
286
- >
287
- > Then it decays that context after two silent turns.
288
- >
289
- > So the system can fully shift topic without one early signal dominating the rest of the conversation.
290
-
291
- ---
292
-
293
- # 🎬 SLIDE 12 — Datasets
294
-
295
- `[ 6:15 – 6:55 ]` &nbsp;·&nbsp; *Click.*
296
-
297
- > 🔷 **MUKUL**
298
- >
299
- > Karthik, walk us through the data.
300
-
301
- > 🔶 **KARTHIK**
302
- >
303
- > Sure. We worked with three layers of data.
304
- >
305
- > First, a curated UMD student-support conversational dataset — split 216, 72, 72 for training, dev, and test on single-turn routing.
306
- >
307
- > Second, a 74-scenario multi-turn evaluation set for the safety contract.
308
- >
309
- > And third, a verified registry of sixty-plus UMD resource URLs, each one annotated with a last-verified date.
310
-
311
- > 🔷 **MUKUL**
312
- >
313
- > And Karthik led all of that — dataset curation, source verification, the annotation conventions for routing and safety tiers.
314
-
315
- ---
316
-
317
- # 🎬 SLIDE 13 — Per-Layer Ablation
318
-
319
- `[ 6:55 – 7:35 ]` &nbsp;·&nbsp; *Click. This is the proof slide — land it.*
320
-
321
- > 🔷 **MUKUL**
322
- >
323
- > So how do we know the layers are actually doing work, and not just sitting in the diagram?
324
- >
325
- > We disabled each layer one at a time and re-ran the same 28-scenario escalation eval.
326
- >
327
- > The Stage-1 lexical pre-check is load-bearing for missed escalation specifically.
328
- >
329
- > Disabling it alone takes us from zero misses to twenty-two out of twenty-eight.
330
-
331
- > 🔶 **KARTHIK**
332
- >
333
- > And the other layers protect orthogonal failure modes.
334
- >
335
- > Registry filtering prevents fabrication.
336
- >
337
- > The verifier catches LLM drift.
338
- >
339
- > Each layer earns its place.
340
-
341
- > 🔷 **MUKUL**
342
- >
343
- > This is also our cleanest answer to a fair question — *is the architecture real, or is it a scripted demo?*
344
- >
345
- > If a layer weren't doing real work, disabling it wouldn't change the numbers.
346
- >
347
- > The zero-to-twenty-two swing is the proof.
348
-
349
- ---
350
-
351
- # 🎬 SLIDE 14 — Targeted Failure-Mode Sweeps
352
-
353
- `[ 7:35 – 8:05 ]` &nbsp;·&nbsp; *Click.*
354
-
355
- > 🔷 **MUKUL**
356
- >
357
- > Beyond the headline eval, we ran five targeted sweeps to stress specific failure modes.
358
- >
359
- > Rephraser drift across 29 cells.
360
- >
361
- > F-1 stage and ISSS contract across 12 cells.
362
- >
363
- > 25 sycophancy probes, 16 prompt-injection probes, 18 fairness paired prompts.
364
- >
365
- > All clean within the stochastic LLM tolerance.
366
-
367
- ---
368
-
369
- # 🎬 SLIDE 15 — Honest Bounds on the Claims
370
-
371
- `[ 8:05 – 8:35 ]` &nbsp;·&nbsp; *Click.*
372
-
373
- > 🔶 **KARTHIK**
374
- >
375
- > A quick reality check before we move on.
376
- >
377
- > N is twenty-eight escalation scenarios. That's small.
378
- >
379
- > We're not claiming zero missed escalation in deployment.
380
- >
381
- > What we *are* claiming is zero versus the unguarded baseline's nine, with non-overlapping confidence intervals.
382
-
383
- > 🔷 **MUKUL**
384
- >
385
- > And the data is synthetic — real student phrasing is messier than anything we've evaluated on.
386
- >
387
- > This is prototype-stage evidence, not a deployment claim. We say that out loud because it matters.
388
-
389
- ---
390
-
391
- # 🎬 SLIDE 16 — What's Next
392
-
393
- `[ 8:35 – 9:00 ]` &nbsp;·&nbsp; *Click.*
394
-
395
- > 🔷 **MUKUL**
396
- >
397
- > The roadmap follows the gaps we just admitted.
398
- >
399
- > A real student dataset.
400
- >
401
- > A RoBERTa router on top of the rule layer.
402
- >
403
- > Scheduled URL re-verification.
404
- >
405
- > And a multilingual opener for international students.
406
- >
407
- > Now let's see the system in action.
408
-
409
- ---
410
-
411
- # 🎬 SLIDE 17 — Live Demo
412
-
413
- `[ 9:00 – 9:45 ]` &nbsp;·&nbsp; *Click. Embedded 30-second auto-loop GIF starts playing on slide entry.*
414
-
415
- > 🔷 **MUKUL**
416
- >
417
- > This is the system live. Let's narrate as it streams.
418
- >
419
- > The student opens vaguely.
420
- >
421
- > Notice — no resources dumped, just validation. That's the LISTEN stage doing its job.
422
-
423
- > 🔶 **KARTHIK**
424
- >
425
- > Then they add context. Watch what the system does — it asks permission before offering anything.
426
-
427
- > 🔷 **MUKUL**
428
- >
429
- > Permission granted. OFFER surfaces real cards, real links from the verified registry.
430
-
431
- > 🔶 **KARTHIK**
432
- >
433
- > And when the student says "ok," the system doesn't re-render the same template. It advances.
434
- >
435
- > The full five-minute demo with three more scenarios — F-1, substance use and confidentiality, crisis with sycophancy resistance — plus the Support Plan export, is linked in the README.
436
-
437
- ---
438
-
439
- # 🎬 SLIDE 18 — Thank You
440
-
441
- `[ 9:45 – 10:00 ]`
442
-
443
- > 🔷 **MUKUL**
444
- >
445
- > That's EmpathRAG. Thanks for watching.
446
- >
447
- > Code, datasets, evaluations, and the full write-up are all on GitHub.
448
-
449
- > 🔶 **KARTHIK**
450
- >
451
- > Happy to take any questions.
452
-
453
- ---
454
-
455
- ## ✅ Pre-record checklist
456
-
457
- - [ ] Both webcams placed in same corner. Don't move them per slide.
458
- - [ ] One full dry-run with a stopwatch. If you hit 10:30, cut the longest sentence on the slide where you went over.
459
- - [ ] Slide 5 (Architecture) and Slide 13 (Ablation) — pace yourself ~10% slower. Those are the dense slides.
460
- - [ ] Slide 17 — Mukul's first line should hit as the GIF's first response streams. Time the entry.
461
- - [ ] Don't say "uhh" — pause silently. Pauses read as confidence on camera. Filler words don't.
462
-
463
- ## ✂️ If you go long — sentences to cut first
464
-
465
- | Slide | Cut this sentence |
466
- |---|---|
467
- | 4 | *"And it's the foundation everything else builds on."* |
468
- | 5 | *"The templates and crisis regex are deterministic."* |
469
- | 7 | *"Each one of these became an architectural fix in the next iteration."* |
470
- | 11 | *"So the system can fully shift topic without one early signal dominating the rest of the conversation."* |
471
- | 13 | *"Each layer earns its place."* |
472
- | 14 | Drop one sweep from the list (any of them) |
473
- | 15 | *"We say that out loud because it matters."* |
474
-
475
- ## 💬 Q&A defense cheatsheet
476
-
477
- - **"Is this trained?"** → Slide 5 already named what's trained. Reference it.
478
- - **"Is this real?"** → Slide 13 ablation. Disabling Stage-1 changes the numbers, ergo the layers are doing real work.
479
- - **"What about substance / privacy / typos?"** → All have dedicated routes (`substance_use_concern`, `privacy_confidentiality`, typo-aware crisis detection). Demo'd in the linked five-minute video.
480
- - **"Why not fine-tune the LLM?"** → Plan-and-rephrase is the architectural commitment. Fine-tuning the LLM doesn't give you the deterministic safety contract.
481
-
482
- ## 🔗 Continuity bridges (if you forget a transition mid-recording)
483
-
484
- | Going from → to | Bridge sentence |
485
- |---|---|
486
- | Slide 3 → 4 | *"How do you get from 32% to zero with the same model? That's the architecture."* |
487
- | Slide 5 → 6 | *"That's version three. Let me walk you through how we got there."* (Karthik takes over) |
488
- | Slide 7 → 8 | *"Each became an architectural fix."* (Karthik opens 8 with "So we layered the response...") |
489
- | Slide 11 → 12 | (Beat — Mukul hands to Karthik with "Karthik, walk us through the data.") |
490
- | Slide 12 → 13 | *"How do we know the layers are actually doing work?"* |
491
- | Slide 16 → 17 | *"Now let's see the system in action."* |