black-swan-v6len-q4

Qwen2.5-Coder-1.5B-Instruct with the black_swan sft_v6len LoRA merged in, converted to GGUF and quantized to Q4_K_M (986 MB, against 3.1 GB for f16).

It writes PyRel — a declarative Python query DSL — from a natural-language question and a schema of Concepts and their properties.

Accuracy

Executed, not string-matched: every program is run against the real database and its result compared with the gold SQL's. Measured on the quantized weights, not extrapolated from f16, because quantization costs real accuracy here.

	q4_K_M	f16
BIRD, 428 held-out questions	30.4%	33.2%

The 428 are BIRD dev questions on four held-out schemas (california_schools, debit_card_specializing, thrombosis_prediction, superhero) that appear nowhere in training, so this measures generalisation to an unseen schema rather than recall.

Quantization costs 2.8 points, 12 questions of 428 (exact paired McNemar p = 0.0501).

Compared with the earlier black-swan-sft15-q4 on the identical 428 questions: 30.4% against 9.6%.

It requires BIRD's evidence hint

This is a usage requirement, not a footnote. The model was trained with BIRD's evidence field appended to the question, and that single change is most of what it knows:

without the hint   14.5%
with the hint      30.6%     (exact paired McNemar, p = 4.2e-13)

Format the question as:

<question>

Hint: <evidence>

Given a bare question the same weights score less than half as well. The earlier sft15 models carry no such requirement, so anything switching between them must know this.

Usage

ollama create black-swan-v6len:1.5b-q4 -f Modelfile
ollama run black-swan-v6len:1.5b-q4

The Modelfile in this repo carries three things that are not optional:

• the trained system prompt — given a different one the same weights answer in SQL rather than PyRel;
• the ChatML template the fine-tune was trained with — a prompt format is part of a model's weights, and sending another asks for something it never saw;
• num_ctx 16384 — a BIRD schema in the trained shape runs to thousands of tokens, and the 2048 default truncates the prompt before the question, so the model answers a schema dump it was never asked about.

ollama show --template is the one-line check: if it prints a bare {{ .Prompt }}, no system prompt is reaching the model however it was invoked.

Prompt shape:

Schema:
Schools: cdscode, city, county, school, statustype
Frpm: cdscode, charter_school_y_n, enrollment_k_12, free_meal_count_k_12
  .cdscode -> Schools: cdscode, city, county, school

Question: How many schools are in Fresno county?

Hint: Fresno county refers to County = 'Fresno'

result = where(Schools.county == "Fresno").select(
    count(Schools).alias("count")).to_dict()

What this model is NOT better at, and what beats it

Spider and TPC-DS are unmeasured for this model. black-swan-sft15-q4 reports 77.4% on Spider and 45.5% on TPC-DS; there are no comparable figures here, and the evidence hint that lifts BIRD does not exist in Spider. Do not read 30.4% vs 9.6% as a general improvement — it is a BIRD result.

A deterministic SQL→PyRel transpiler beats it, and by a wide margin. On the same 428 questions, generating SQL with a text-to-SQL model and transpiling it scores 53.5% against this model's 33.2% at f16 (exact paired McNemar, p = 1.4e-14). If your goal is answers rather than a model that writes PyRel directly, that pipeline is the better one.

Its remaining errors are semantic. Of its held-out failures, 160 execute and return the wrong rows against 126 that do not execute; 88% of the near misses return a single wrong scalar. Six interventions on the prompt, the vocabulary and the candidate selector each measured at or near zero against that.

Training

• base Qwen/Qwen2.5-Coder-1.5B-Instruct, QLoRA rank 32, alpha 64, 2 epochs
• --max-len 1536 — an earlier default of 768 silently truncated ~26% of every fine-tune
• 11,917 execution-verified records: Spider-derived worlds plus BIRD training databases
• every training program was run against a real database and admitted only if its result matched the gold SQL's

Licence

Apache-2.0, following the base model. The training corpus is derived from BIRD and Spider; the databases themselves are not redistributed here.