xxxxrshx12/tcs-philosophy-llama3.2-3b-qlora

TCS Philosophy Bot — Fine-Tuned Llama 3.2 3B

A domain-expert LLM fine-tuned on Taking Children Seriously (TCS) philosophy — the non-coercive parenting and education philosophy founded by David Deutsch and Sarah Fitz-Claridge, grounded in Karl Popper's critical rationalism.

Try the live demo

What This Model Does

Ask it anything about TCS philosophy and it responds with accurate, domain-specific knowledge — not generic parenting advice. It knows the correct founders, terminology, and can apply TCS principles to practical parenting scenarios.

Example

Q: "My child doesn't want to go to bed. How would TCS approach this?"

A: The model explains that TCS views imposed bedtimes as coercive, advocates for respecting the child's autonomy around sleep, and suggests finding a common preference rather than forcing compliance.

Training Details

Base Model	Llama 3.2 3B Instruct
Method	QLoRA (4-bit quantization + LoRA adapters)
LoRA Config	rank=16, alpha=16, dropout=0.05
Target Modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Training Data	1,037 curated Q&A pairs from takingchildrenseriously.com
Data Cleaning	Claude 3 Haiku API rewrote raw scraped text into clean explanations
Epochs	1
Learning Rate	5e-5 (linear scheduler)
Effective Batch Size	8 (batch=1, grad_accum=8)
Max Sequence Length	1024 tokens
Optimizer	AdamW 8-bit
Final Training Loss	~1.1
Framework	Unsloth + TRL SFTTrainer
Hardware	NVIDIA RTX 3060 12GB (local)
Training Time	~50 minutes

Full Pipeline

This project covers the entire ML pipeline end-to-end:

1. Data Collection — Custom web scraper (requests + BeautifulSoup) that crawled ~2,000+ articles from the TCS website
2. Data Engineering — Multi-strategy formatter that generates 4 types of training examples per article: direct Q&A, concept extraction, scenario application, and style continuation
3. Data Cleaning — Used Claude 3 Haiku API to rewrite noisy scraped answers into clean 150-400 word explanations (~$5 for 1,700 rewrites). Added hallucination-resistance examples for fake TCS concepts.
4. Training — QLoRA fine-tuning with Unsloth on a consumer GPU. Iterated through 5 training versions to find optimal hyperparameters.
5. Evaluation — 14-question evaluation framework across 5 categories: core concepts, practical application, philosophical reasoning, hallucination detection, and distinction from similar approaches
6. Comparison — 3-way interactive A/B/C comparison (vanilla vs raw-trained vs clean-trained)
7. Export — PEFT merge to fp16, then GGUF Q8_0 quantization via llama.cpp

3-Way Model Comparison

"What is Taking Children Seriously?"

Model	Founders Mentioned	Accuracy
Vanilla Llama 3.2 3B	"Robert Solomon, Steven Jay Schneider" (wrong)	Fabricated founders
V1 Raw Fine-Tune	David Deutsch only (partial)	Good but imprecise
V4 Clean (this model)	David Deutsch + Sarah Fitz-Claridge (correct)	Accurate TCS terminology

Training Iterations

Version	Data	Examples	Epochs	LR	Final Loss	Result
v1	Raw scraped	2,044	3	2e-4	~0.66	Severe overfitting
v2	Claude-cleaned	1,037	3	2e-4	~0.4	Still overtrained
v3	Claude-cleaned	1,037	1	2e-4	~0.9	Better but aggressive
v4 (this)	Claude-cleaned	1,037	1	5e-5	~1.1	Best version

Key insight: Loss ~0.4-0.7 on small datasets means memorization, not generalization. Healthy fine-tuning loss on 1K examples is ~1.0-2.0.

Files

• tcs-philosophy-v4-q8_0.gguf — Quantized GGUF model (Q8_0, 3.42 GB). Load directly in LM Studio or llama.cpp.

Usage

LM Studio

Download the GGUF file and load it in LM Studio. Set the system prompt to:

"You are an expert in Taking Children Seriously philosophy."

Python (with llama-cpp-python)

from llama_cpp import Llama

llm = Llama.from_pretrained(
    repo_id="xxxxrshx12/tcs-philosophy-llama3.2-3b-qlora",
    filename="tcs-philosophy-v4-q8_0.gguf",
    n_ctx=1024,
)

response = llm.create_chat_completion(messages=[
    {"role": "system", "content": "You are an expert in Taking Children Seriously philosophy."},
    {"role": "user", "content": "What is Taking Children Seriously?"},
])
print(response["choices"][0]["message"]["content"])

Limitations

• Repetition: The 3B model sometimes falls into repetition loops (a known small-model weakness)
• Hallucination: Occasionally fabricates TCS concepts when asked about things that don't exist in the philosophy
• Dataset size: Trained on only 1,037 examples — a larger, more diverse dataset would improve generalization
• Scope: Only covers content from takingchildrenseriously.com, not David Deutsch's broader work (e.g., The Beginning of Infinity)

License

This model inherits the Llama 3.2 Community License.