ishreyadev commited on
Commit
668b15d
Β·
verified Β·
1 Parent(s): 56c401a

Upload folder using huggingface_hub

Browse files
Files changed (4) hide show
  1. .gitattributes +1 -0
  2. README.md +95 -0
  3. gitattributes +38 -0
  4. pivotai_curriculum.Q4_K_M.gguf +3 -0
.gitattributes CHANGED
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ pivotai_curriculum.Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,95 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - en
4
+ tags:
5
+ - travel
6
+ - india
7
+ - curriculum-learning
8
+ - llama
9
+ - qlora
10
+ - itinerary-optimization
11
+ - grounding
12
+ license: apache-2.0
13
+ base_model: unsloth/Meta-Llama-3.1-8B
14
+ datasets:
15
+ - ishreyadev/pivotai-synthetic-v2
16
+ - ishreyadev/pivotai-agent-traces
17
+ metrics:
18
+ - bertscore
19
+ model-index:
20
+ - name: pivotai-curriculum
21
+ results:
22
+ - task:
23
+ type: text-generation
24
+ name: Travel Itinerary Optimization
25
+ metrics:
26
+ - type: grounding_accuracy
27
+ value: 0.88
28
+ name: Grounding Accuracy
29
+ - type: bertscore_f1
30
+ value: 0.734
31
+ name: BERTScore F1
32
+ - type: red_team_pass
33
+ value: 0.60
34
+ name: Red-Team Robustness
35
+ ---
36
+
37
+ # pivotai-curriculum
38
+
39
+ Curriculum-trained Llama 3.1 8B for Indian domestic travel optimization. Uses **two-stage sequential training**: first on 4,749 Phase 1 synthetic pairs (domain knowledge), then on 449 Phase 2 agent reasoning traces (complex reasoning patterns).
40
+
41
+ Part of the [pivotai](https://github.com/ishreya-dev/PivotAI) project. The curriculum hypothesis was that domain knowledge should precede complex reasoning patterns β€” similar to how students learn fundamentals before advanced topics. Results revealed an interesting trade-off: the model achieved the **highest grounding accuracy (88%)** and **best red-team robustness (60%)** of the three variants, but the Phase 2 training stage catastrophically disrupted structured JSON output (10.9% validity).
42
+
43
+ ## Model Details
44
+
45
+ | Property | Value |
46
+ |----------|-------|
47
+ | Base model | `unsloth/Meta-Llama-3.1-8B` |
48
+ | Training method | QLoRA r=8, Ξ±=16, dropout=0.05 (2-stage) |
49
+ | Stage 1 data | 4,749 pairs (Phase 1 synthetic) β€” 424 steps |
50
+ | Stage 2 data | 449 pairs (Phase 2 agent traces) β€” 171 steps |
51
+ | Final train loss | 0.313 (Stage 2) |
52
+ | Hardware | Lightning.ai A100 (bf16, seq_len=16384) |
53
+ | Format | GGUF Q4_K_M (4.6 GB) |
54
+
55
+ ## Evaluation Results (92 test cases)
56
+
57
+ | Metric | Score | Target | βœ“/βœ— |
58
+ |--------|:-----:|:------:|:---:|
59
+ | JSON valid | **10.9%** | 85% | βœ— |
60
+ | Savings found | β€” | 70% | β€” |
61
+ | Schema compliance | 0.0% | 80% | βœ— |
62
+ | BERTScore F1 | 0.734 | 0.70 | βœ“ |
63
+ | Intent alignment | 0.418 | 0.55 | βœ— |
64
+ | Grounding accuracy | **0.880** | 0.60 | βœ“ |
65
+ | Reasoning coherence | 0.470 | 0.65 | βœ— |
66
+ | Red-team pass | **60.0%** | 80% | βœ— |
67
+
68
+ **Notable:** Despite near-zero JSON validity, grounding accuracy (0.88) nearly matches pivotai-ft (0.895). The model has absorbed real-world knowledge about Indian cities and travel patterns β€” it simply cannot format the output as valid JSON after Phase 2 training overwrote structured-output behavior.
69
+
70
+ **Recommendation:** Use with JSON-constrained decoding (llama.cpp `--grammar`, Outlines, or similar) to recover structured output. The underlying knowledge is strong.
71
+
72
+ ## Usage with Ollama
73
+
74
+ ```bash
75
+ ollama create pivotai-curriculum -f Modelfile.curriculum
76
+ ollama run pivotai-curriculum
77
+ ```
78
+
79
+ **Note:** Due to low JSON validity in standard inference, consider using grammar-constrained decoding for reliable structured output.
80
+
81
+ ## Limitations
82
+
83
+ - JSON validity is 10.9% β€” standard inference rarely produces valid JSON. Use grammar-constrained decoding.
84
+ - The Phase 2 curriculum stage appears to have overwritten Phase 1 structured-output training β€” a known curriculum learning failure mode.
85
+ - Despite strong semantic knowledge, the model cannot be used without output post-processing.
86
+
87
+ ## Citation
88
+
89
+ If you use this model, please cite:
90
+
91
+ ```
92
+ Patnaik, A. V. S. (2026). Cost-Matched Data Generation for LLM Fine-Tuning: Comparing
93
+ Supervised Fine-Tuning, Knowledge Distillation, and Curriculum Learning for an Agentic
94
+ Travel-Planning System. Zenodo. https://doi.org/10.5281/zenodo.21198884
95
+ ```
gitattributes ADDED
@@ -0,0 +1,38 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ *.7z filter=lfs diff=lfs merge=lfs -text
2
+ *.arrow filter=lfs diff=lfs merge=lfs -text
3
+ *.bin filter=lfs diff=lfs merge=lfs -text
4
+ *.bz2 filter=lfs diff=lfs merge=lfs -text
5
+ *.ckpt filter=lfs diff=lfs merge=lfs -text
6
+ *.ftz filter=lfs diff=lfs merge=lfs -text
7
+ *.gz filter=lfs diff=lfs merge=lfs -text
8
+ *.h5 filter=lfs diff=lfs merge=lfs -text
9
+ *.joblib filter=lfs diff=lfs merge=lfs -text
10
+ *.lfs.* filter=lfs diff=lfs merge=lfs -text
11
+ *.mlmodel filter=lfs diff=lfs merge=lfs -text
12
+ *.model filter=lfs diff=lfs merge=lfs -text
13
+ *.msgpack filter=lfs diff=lfs merge=lfs -text
14
+ *.npy filter=lfs diff=lfs merge=lfs -text
15
+ *.npz filter=lfs diff=lfs merge=lfs -text
16
+ *.onnx filter=lfs diff=lfs merge=lfs -text
17
+ *.ot filter=lfs diff=lfs merge=lfs -text
18
+ *.parquet filter=lfs diff=lfs merge=lfs -text
19
+ *.pb filter=lfs diff=lfs merge=lfs -text
20
+ *.pickle filter=lfs diff=lfs merge=lfs -text
21
+ *.pkl filter=lfs diff=lfs merge=lfs -text
22
+ *.pt filter=lfs diff=lfs merge=lfs -text
23
+ *.pth filter=lfs diff=lfs merge=lfs -text
24
+ *.rar filter=lfs diff=lfs merge=lfs -text
25
+ *.safetensors filter=lfs diff=lfs merge=lfs -text
26
+ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
27
+ *.tar.* filter=lfs diff=lfs merge=lfs -text
28
+ *.tar filter=lfs diff=lfs merge=lfs -text
29
+ *.tflite filter=lfs diff=lfs merge=lfs -text
30
+ *.tgz filter=lfs diff=lfs merge=lfs -text
31
+ *.wasm filter=lfs diff=lfs merge=lfs -text
32
+ *.xz filter=lfs diff=lfs merge=lfs -text
33
+ *.zip filter=lfs diff=lfs merge=lfs -text
34
+ *.zst filter=lfs diff=lfs merge=lfs -text
35
+ *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ pivotai_curriculum.Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
37
+ WHITEPAPER.pdf filter=lfs diff=lfs merge=lfs -text
38
+ whitepaper.pdf filter=lfs diff=lfs merge=lfs -text
pivotai_curriculum.Q4_K_M.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9041223c9dff9a610231482924653a95256a81b178b5412e46eb1017b93a4cba
3
+ size 4920734368