khursanirevo/teach-aws-qwen3.5-2b

teach-aws Qwen3.5-2B — Malay AWS QA

Answers AWS questions in Bahasa Malaysia at reference-document register.

91.0% strict accuracy on 2,661 unseen questions (completely-correct binary LLM judge; base model: 4.1%; previous release: 87.3%). Eval set and judge are fully reproducible — see the Colab notebook and the experiment walkthrough.

Recipe

unsloth LoRA on unsloth/Qwen3.5-2B (text-only): full MLP modules, all 24 layers, r=16, α=32, lr 4e-4, 2 epochs, completion-only loss, trained on 28,148 rows = khursanirevo/teach_aws (18,455 paraphrase-linked QA) + 8,433 teacher-paraphrased answers

• 1,260 hard-service variants targeting the hallucination-prone services (vpc, amplify, appsync, snowball, ecs, fargate). Two-seed validated: 91.2% / 91.0%.

What it does well

• Paraphrased AWS questions it has never seen, answered at doc-quality register (91% strict-correct)
• Short questions get short answers; correct Malay + code-switched AWS terminology
• Hard-service hallucination rate cut: vpc FALSE 16%→9%, amplify 11%→0% vs the previous release

What it does badly (read this)

• Invented features on thin-coverage topics. Example: it will confidently describe a non-existent CloudFormation "Memory and Cache Behavior" (correct answer: deletion policies). Most of its failures are fluent, confident text. Pair with the grounding guardrail (58% hallucination catch at 0.7% overblock on this model's full-set outputs) or retrieval.
• Rigid phrasing: most outputs reproduce the reference nearly verbatim. This is a deliberate accuracy-over-flexibility trade; the full frontier analysis is in the walkthrough.
• No knowledge outside its corpus (AWS-only, Malay/English mix as trained).

Usage

from vllm import LLM, SamplingParams
llm = LLM(model="khursanirevo/teach-aws-qwen3.5-2b", max_model_len=4096,
          gpu_memory_utilization=0.85, max_num_seqs=256, enforce_eager=True)
tok = llm.get_tokenizer()
q = "Bolehkah S3 Transfer Acceleration melengkapi AWS Direct Connect?"
p = tok.apply_chat_template([{"role": "user", "content": q}],
                            tokenize=False, add_generation_prompt=True)
out = llm.generate([p], SamplingParams(temperature=0.0, max_tokens=2048))
print(out[0].outputs[0].text)

Comparison (same judge, same unseen eval)

Model	Strict acc
this model (H4)	91.0% (n=2,661)
previous release (H3)	87.3% (n=2,661)
PixelSpaceAI/Malaysian-Qwen2.5-7B-AWS-Malay-LoRA	15.0% (n=60)
mesolitica/Malaysian-Qwen2.5-7B-Instruct	8.3% (n=60)

The 7B community adapter is excellent for register/voice and recommends RAG for facts — this model is the opposite trade: fact recall first.

Acknowledgements

Trained with unsloth. Data from khursanirevo/teach_aws (built on PixelSpaceAI/aws-malay-qa sources). Judge: Qwen3-4B-Instruct-2507.