lynn1900/humomni-track1-relative-listener-rule-w010

HumOmni / AEQ Track 1 Final V12 System

Model Summary

Final V12 system:

relative/candidate acoustic features
+ listener-state features
+ XGBoost pairwise ranker
+ deterministic rule-based tone alignment (weight 0.10)

Version-name map:

v3  = original feature-extraction family reused for pause/eGeMAPS/ASR features
v11 = learned ranker experiment that produced E_relative_listener
v12 = final submitted system/package: E_relative_listener + tone-alignment rule w=0.10

The .pkl model is therefore the learned ranker inside the final V12 system. The V12 part is the deterministic rule fusion applied by inference.py.

The model selects the most empathetic candidate response audio for each question. Two-option questions are handled as direct pairwise comparisons. Three-option MELD questions are handled by round-robin pairwise comparisons.

This final V12 system does not use test-time LLM calls. DeepSeek text fusion was explored as a diagnostic add-on but is not included. Deterministic tone alignment is included at the fixed weight 0.10.

Repository Structure

Current Hugging Face repository layout:

.
├── README.md
├── MANIFEST.json
├── PHASE2_MANIFEST.json
├── requirements.txt
├── inference.py
├── Lenormand_Team.jsonl              # official Phase 2 prediction, 542 rows
├── Lenormand_Team.scores.csv         # Phase 2 debug scores, not for submission
├── Lenormand_Team_Phase1.jsonl       # archived earlier 530-row prediction
├── Lenormand_Team_Phase2.jsonl       # same content as Lenormand_Team.jsonl
├── Lenormand_Team.scores_Phase2.csv  # archived Phase 2 debug scores
├── predictions_phase1.scores.csv     # archived Phase 1 debug scores
├── models/
│   └── model_E_relative_listener.pkl
├── configs/
│   ├── feature_sets_v11.json
│   └── run_config_v11.json
├── scripts/
│   ├── 01_v11_colab_ablation.py
│   ├── 04_oof_report_table.py
│   ├── 05_full_metric_matrix.py
│   └── phase2_raw_regeneration/
│       ├── PHASE2_COLAB_RUNBOOK.md
│       ├── PHASE2_ONE_CLICK_COLAB.ipynb
│       ├── 00_parse_phase2_test.py
│       ├── 01_build_listener_state_table_phase2.py
│       ├── 02_build_v11_augmented_phase2.py
│       ├── 03_build_rule_scores_phase2.py
│       ├── 04_validate_submission.py
│       ├── 05_audio_pause_features.py
│       ├── 06_egemaps_features.py
│       ├── 07_asr_transcribe.py
│       └── 08_build_pairwise_features.py
├── report_assets/
│   ├── formal_oof_ablation_clean.csv
│   ├── manual_diagnostic_clean.csv
│   ├── full_metric_matrix.csv
│   ├── domain_bias.png
│   ├── SHAP_features.png
│   ├── Domain_vs_Empathy.png
│   └── Domain_vs_Empathy2.png
└── data_artifacts/
    ├── train_features_v11_augmented.csv
    ├── test_features_v11_augmented.csv
    ├── test_features_v3_phase2.csv
    ├── test_full_52_phase2.csv
    ├── test_features_v11_augmented_phase2.csv
    └── test_rule_alignment_scores_phase2.csv

This verification package uses precomputed augmented feature artifacts. For the final Phase 2 submission, the released inference path starts from test_features_v11_augmented_phase2.csv and test_rule_alignment_scores_phase2.csv; it does not regenerate acoustic or listener-state features from raw audio.

The string v11_augmented is retained as a feature-table compatibility name because the selected learned ranker was trained in the v11 ablation stage. It does not mean the submitted system is v11-only.

Phase 2 Data Note

The original verification package was first prepared with the earlier 530-question test release. If the organizer releases a replacement Phase 2 test set, do not change the final V12 system. Keep using:

models/model_E_relative_listener.pkl

Only regenerate the two input artifacts for the new raw data:

test_features_v11_augmented_phase2.csv
test_rule_alignment_scores_phase2.csv

Then run inference.py with those new artifact paths. The legacy v2/v3 scripts used in the emergency Phase 2 runbook are raw-audio feature extractors only; they are not old submitted models.

After rerunning Phase 2, the uploaded HF package should include the regenerated 542-question prediction and Phase 2 artifacts:

Lenormand_Team.jsonl
Lenormand_Team.scores.csv
PHASE2_MANIFEST.json
data_artifacts/test_features_v3_phase2.csv
data_artifacts/test_full_52_phase2.csv
data_artifacts/test_features_v11_augmented_phase2.csv
data_artifacts/test_rule_alignment_scores_phase2.csv

The one-click Colab notebook and raw Phase 2 regeneration helpers are included under scripts/phase2_raw_regeneration/. They are not required for running the minimal packaged HF inference artifact, because the regenerated Phase 2 feature tables are already included here. They are provided for reproducibility from the raw replacement Phase 2 data.

Requirements

Minimal requirements for packaged inference from the included feature tables:

python>=3.9
numpy
pandas
scikit-learn
xgboost
joblib

Additional packages used only by the raw Phase 2 regeneration notebook/scripts include PyTorch, Transformers, faster-whisper, openSMILE, ffmpeg, and the saved listener-state checkpoint/tokenizer. Those are Colab/GPU-side preprocessing dependencies, not requirements for loading the final XGBoost model from the included feature tables.

Final Model File

models/model_E_relative_listener.pkl

The pickle contains:

model: trained XGBoost classifier
feature_cols: ordered feature list
metrics: OOF metadata

Inference

The final inference expects the augmented feature table and rule-score table used by the final V12 system. For the final Phase 2 submission these are:

data_artifacts/test_features_v11_augmented_phase2.csv
data_artifacts/test_rule_alignment_scores_phase2.csv

The feature table contains V3-derived relative/candidate features and V10 listener-state features.

Example command:

python3 inference.py \
  --model_path models/model_E_relative_listener.pkl \
  --features_csv data_artifacts/test_features_v11_augmented_phase2.csv \
  --rule_scores_csv data_artifacts/test_rule_alignment_scores_phase2.csv \
  --rule_weight 0.10 \
  --out_jsonl Lenormand_Team.jsonl \
  --out_scores_csv Lenormand_Team.scores.csv

Running this command from the repository root produces 542 Phase 2 predictions. The result should match the included Lenormand_Team.jsonl.

Output format:

{"question_id": "gigaspeech_0_1", "answer": "B"}
{"question_id": "meld_183_1", "answer": "A"}

The included official Phase 2 prediction is already named with the registered Team ID:

Lenormand_Team.jsonl

Lenormand_Team_Phase1.jsonl is retained only as an archive of the earlier 530-question release and should not be submitted for Phase 2.

Lenormand_Team.scores.csv is a debug/verification artifact and is not the official submission file.

Raw Phase 2 Regeneration

To regenerate the Phase 2 feature artifacts from the raw replacement test set, use:

scripts/phase2_raw_regeneration/PHASE2_COLAB_RUNBOOK.md
scripts/phase2_raw_regeneration/PHASE2_ONE_CLICK_COLAB.ipynb

The raw regeneration path performs:

raw Phase 2 JSON/audio
-> metadata parsing
-> pause/duration features
-> eGeMAPS features
-> ASR transcripts
-> pairwise V3-compatible feature table
-> listener-state prediction table
-> V11-compatible augmented feature table
-> deterministic rule-score table
-> final V12 inference

Expected replacement Phase 2 counts:

542 questions
878 pairwise rows
1252 option rows

Training / Reproduction

The main training script is:

scripts/01_v11_colab_ablation.py

It creates four feature variants:

Variant	Description
A	V3 minus user/domain-heavy features
C	relative/candidate-only features
D	A plus listener-state features
E	C plus listener-state features

Selected learned ranker:

E_relative_listener

The submitted system subsequently applies deterministic rule alignment at the fixed weight 0.10. OOF validation uses fixed 5-fold splits.

OOF Results

Main report table:

report_assets/formal_oof_ablation_clean.csv

Summary:

Model	OOF Accuracy	OOF AUC	Context OOF Accuracy
V3 full	0.9675	N/A	0.9659
A no-user	0.9669	0.9949	0.9632
C relative	0.9573	0.9905	0.9503
D no-user + listener	0.9720	0.9962	0.9712
E relative + listener	0.9693	0.9958	0.9682

Although D has the highest train-domain OOF score, E is selected because it combines the domain-robust relative feature design with listener-state context information and performs best on manually audited MELD/context-sensitive cases.

The rule-only system obtains 0.6594 OOF accuracy. At weight 0.10, fusion changes the selected ranker's OOF accuracy from 0.9693 to 0.9700, while AUC decreases from 0.9958 to 0.9927. We retain this low-weight rule as a conservative decision-level correction and report the ranking and cross-domain limitations explicitly.

Manual Diagnostic Set

We use a 130-example personal golden set for diagnostic model selection. This is not official test labeling and should not be interpreted as a leaderboard score.

Main report table:

report_assets/manual_diagnostic_clean.csv

The larger full_metric_matrix.csv is provided as an appendix/debug artifact. It contains N/A entries for variants that do not have valid train-side OOF signals, such as DeepSeek-only test-side fusion. Do not treat those N/A values as failed OOF runs.

Main diagnostic finding:

C_relative and E_relative_listener tie overall.
E_relative_listener performs best on MELD.
DeepSeek does not improve over E; the 0.10 rule fusion is retained for its small formal OOF accuracy gain despite the diagnostic decrease.

Additional Experiments

The following were explored:

DeepSeek test-side text fusion: excluded
Rule-based tone alignment at weight 0.10: included
Rule weights 0.20 and 0.30: excluded

Reason:

DeepSeek did not improve over the selected learned ranker and lacks train-side OOF comparability. Rule alignment also decreased manual diagnostic accuracy by one example; it is retained only because its fixed 0.10 weight slightly improves formal OOF discrete-choice accuracy.

Citation / Competition Note

This repository is prepared for AEQ/HumOmni Track 1 final model verification. The repository should remain publicly accessible until July 15 AOE, per competition requirements.

The repository is released as a competition verification artifact. The license: other metadata is intentional; no broader license is asserted for the underlying competition data or third-party components.