---
license: apache-2.0
base_model: Qwen/Qwen2.5-Coder-1.5B-Instruct
tags:
  - text-to-code
  - semantic-parsing
  - pyrel
  - gguf
  - quantized
language:
  - en
pipeline_tag: text-generation
---

# black-swan-v6len-q4

`Qwen2.5-Coder-1.5B-Instruct` with the black_swan `sft_v6len` LoRA merged in, converted to
GGUF and quantized to **Q4_K_M** (986 MB, against 3.1 GB for f16).

It writes **PyRel** — a declarative Python query DSL — from a natural-language question and
a schema of Concepts and their properties.

## Accuracy

Executed, not string-matched: every program is run against the real database and its result
compared with the gold SQL's. Measured **on the quantized weights**, not extrapolated from
f16, because quantization costs real accuracy here.

| | q4_K_M | f16 |
|---|---|---|
| **BIRD, 428 held-out questions** | **30.4%** | 33.2% |

The 428 are BIRD dev questions on four **held-out schemas** (`california_schools`,
`debit_card_specializing`, `thrombosis_prediction`, `superhero`) that appear nowhere in
training, so this measures generalisation to an unseen schema rather than recall.

Quantization costs 2.8 points, 12 questions of 428 (exact paired McNemar p = 0.0501).

**Compared with the earlier [black-swan-sft15-q4](https://huggingface.co/maxdemarzi/black-swan-sft15-q4)
on the identical 428 questions: 30.4% against 9.6%.**

## It requires BIRD's evidence hint

This is a usage requirement, not a footnote. The model was trained with BIRD's `evidence`
field appended to the question, and that single change is most of what it knows:

    without the hint   14.5%
    with the hint      30.6%     (exact paired McNemar, p = 4.2e-13)

Format the question as:

    <question>

    Hint: <evidence>

Given a bare question the same weights score less than half as well. The earlier sft15
models carry no such requirement, so anything switching between them must know this.

## Usage

```bash
ollama create black-swan-v6len:1.5b-q4 -f Modelfile
ollama run black-swan-v6len:1.5b-q4
```

The `Modelfile` in this repo carries three things that are not optional:

* **the trained system prompt** — given a different one the same weights answer in SQL
  rather than PyRel;
* **the ChatML template** the fine-tune was trained with — a prompt format is part of a
  model's weights, and sending another asks for something it never saw;
* **`num_ctx 16384`** — a BIRD schema in the trained shape runs to thousands of tokens, and
  the 2048 default truncates the prompt *before the question*, so the model answers a schema
  dump it was never asked about.

`ollama show --template` is the one-line check: if it prints a bare `{{ .Prompt }}`, no
system prompt is reaching the model however it was invoked.

Prompt shape:

```
Schema:
Schools: cdscode, city, county, school, statustype
Frpm: cdscode, charter_school_y_n, enrollment_k_12, free_meal_count_k_12
  .cdscode -> Schools: cdscode, city, county, school

Question: How many schools are in Fresno county?

Hint: Fresno county refers to County = 'Fresno'
```

```python
result = where(Schools.county == "Fresno").select(
    count(Schools).alias("count")).to_dict()
```

## What this model is NOT better at, and what beats it

**Spider and TPC-DS are unmeasured for this model.** `black-swan-sft15-q4` reports 77.4% on
Spider and 45.5% on TPC-DS; there are no comparable figures here, and the evidence hint that
lifts BIRD does not exist in Spider. Do not read 30.4% vs 9.6% as a general improvement — it
is a BIRD result.

**A deterministic SQL→PyRel transpiler beats it, and by a wide margin.** On the same 428
questions, generating SQL with a text-to-SQL model and transpiling it scores **53.5%**
against this model's 33.2% at f16 (exact paired McNemar, p = 1.4e-14). If your goal is
answers rather than a model that writes PyRel directly, that pipeline is the better one.

**Its remaining errors are semantic.** Of its held-out failures, 160 execute and return the
wrong rows against 126 that do not execute; 88% of the near misses return a single wrong
scalar. Six interventions on the prompt, the vocabulary and the candidate selector each
measured at or near zero against that.

## Training

* base `Qwen/Qwen2.5-Coder-1.5B-Instruct`, QLoRA rank 32, alpha 64, 2 epochs
* `--max-len 1536` — an earlier default of 768 silently truncated ~26% of every fine-tune
* 11,917 execution-verified records: Spider-derived worlds plus BIRD training databases
* every training program was run against a real database and admitted only if its result
  matched the gold SQL's

## Licence

Apache-2.0, following the base model. The training corpus is derived from BIRD and Spider;
the databases themselves are not redistributed here.