maxdemarzi/black-swan-sft15-q4

black-swan-sft15 — PyRel generator (q4_K_M)

Qwen2.5-Coder-1.5B-Instruct with black_swan's sft15-spider-bird-v5 LoRA merged in, quantized to q4_K_M. It writes PyRel — a declarative Python query DSL — from a natural-language question and a schema.

Built for Event Horizon, which runs Swan, PyRel and DuckDB as WebAssembly in the browser and needs generation from somewhere. 986 MB, against 3.1 GB for the f16 it was quantized from.

Serve it with greedy decoding, or it gets measurably worse

This is the one thing to get right. The adapter was trained and evaluated under plain greedy decoding, and several servers apply a repetition penalty and top-k sampling even at temperature 0. On TPC-DS questions that mismatch measured as one wrong answer in five, and it reads as the model being weak rather than as a serving setting:

	correct
ollama defaults, temperature 0	3/5
top_k 1, top_p 1.0, repeat_penalty 1.0	4/5

Prompt shape

It expects its own system prompt and a Schema: / Question: user turn. Both are in tools/local_generator.py (SYSTEM_V5, build_user_turn), asserted byte-identical against the training split by tools/verify_local_generator.py. Sending a differently-worded system prompt is off-distribution and costs accuracy quietly.

Schemas are rendered as Concept: prop, prop lines with foreign keys as Child.member -> Parent.

Scores

Executed rather than string-matched — every prediction is run and its result compared with a verified reference, because PyRel has several correct spellings of most questions.

	f16	this (q4_K_M)
Spider, 217 held-out	81.6%	77.4%
BIRD, 428 held-out	10.0%	9.6%
TPC-DS, 22 questions	59.1%	45.5%

Quantization costs more on large schemas. ~5% relative on Spider and BIRD, whose schemas run 391–1,455 characters, but 23% on TPC-DS at 31,208 characters. If your schema is wide, weigh the 986 MB against that rather than against the small-schema numbers.

Its weakest tier is joins on wide schemas: the traversal syntax is usually right and the property names are invented (sold_price for price). Feeding swan's own error back — "has no property X, did you mean Y" — does not repair it, so it is a grounding limitation rather than a prompting one.

Related

• black-swan-sft15 — training
• Event Horizon — the application
• Swan — PyRel and the DuckDB extension