Instructions to use adgomant/adele-judge-qwen3-14B-cre with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use adgomant/adele-judge-qwen3-14B-cre with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="adgomant/adele-judge-qwen3-14B-cre")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("adgomant/adele-judge-qwen3-14B-cre")
model = AutoModelForCausalLM.from_pretrained("adgomant/adele-judge-qwen3-14B-cre", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

PEFT
How to use adgomant/adele-judge-qwen3-14B-cre with PEFT:
```
Task type is invalid.
```
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use adgomant/adele-judge-qwen3-14B-cre with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "adgomant/adele-judge-qwen3-14B-cre"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "adgomant/adele-judge-qwen3-14B-cre",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/adgomant/adele-judge-qwen3-14B-cre

SGLang

How to use adgomant/adele-judge-qwen3-14B-cre with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "adgomant/adele-judge-qwen3-14B-cre" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "adgomant/adele-judge-qwen3-14B-cre",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "adgomant/adele-judge-qwen3-14B-cre" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "adgomant/adele-judge-qwen3-14B-cre",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use adgomant/adele-judge-qwen3-14B-cre with Docker Model Runner:
```
docker model run hf.co/adgomant/adele-judge-qwen3-14B-cre
```

adgomant commited on Jun 24

Commit

aa91271

verified ·

1 Parent(s): 166a8fb

Update README.md

Browse files

Files changed (1) hide show

README.md +54 -4

README.md CHANGED Viewed

@@ -6,17 +6,19 @@ tags:
 - adele
 - judge
 base_model: Qwen/Qwen3-14B
 ---
 # ADeLe Distilled Judge
-This repository contains an ADeLe-suite-specific distilled judge. It scores a model response against a question and reference answer with an ordinal score from 1 to 5.
 The repository root contains a merged Transformers model for standard loading. The original LoRA adapter is also included under `adapter/` for provenance and reuse.
 ## Intended Use
-Use this model to score ADeLe-style examples where a question, reference answer, and model response are available. It is not a general-purpose evaluator.
 ## Input Format
@@ -38,6 +40,45 @@ Allowed scores: 1, 2, 3, 4, 5
 Binary label: scores greater than or equal to 3 are `CORRECT`; lower scores are `INCORRECT`.
 ## Recommended Inference
 Do not use free-form generation as the primary prediction method. The recommended path scores the restricted continuations `"1"`, `"2"`, `"3"`, `"4"`, and `"5"`.
@@ -91,11 +132,20 @@ model = AutoModelForCausalLM.from_pretrained("adgomant/adele-judge-qwen3-14-cre"
 Training, filtering, split, tokenization, and metric artifacts available at packaging time are stored in `adele_judge_metadata.json`.
-No evaluation metrics were found in the local run artifacts.
 ## Limitations
 - ADeLe-specific judge; not a general-purpose evaluator.
-- Distilled from judge labels and inherits their noise and biases.
 - Intended for scoring responses against a reference answer.
 - It should not produce explanations; the expected output is a single score.

 - adele
 - judge
 base_model: Qwen/Qwen3-14B
+datasets:
+- CFI-Kinds-of-Intelligence/ADeLe_battery_v1dot0
 ---
 # ADeLe Distilled Judge
+This repository contains an ADeLe-suite-specific distilled judge. It scores a model response against a question and reference answer with an ordinal score from 1 to 5, then derives binary correctness with the ADeLe threshold.
 The repository root contains a merged Transformers model for standard loading. The original LoRA adapter is also included under `adapter/` for provenance and reuse.
 ## Intended Use
+Use this model to score ADeLe-style examples where a question, reference answer, and model response are available. It is intended for out-of-model evaluation within the ADeLe benchmark suite, not as a general-purpose evaluator.
 ## Input Format
 Binary label: scores greater than or equal to 3 are `CORRECT`; lower scores are `INCORRECT`.
+## Training And Validation Data
+| Split | Examples | Models |
+| --- | --- | --- |
+| train | 239,420 | 16 |
+| validation | 45,738 | 3 |
+- `train` models: `DK-R1-Dist-Qwen-1.5B`, `DK-R1-Dist-Qwen-32B`, `DK-R1-Dist-Qwen-7B`, `gemini-2.5-flash`, `gemini-3.1-pro`, `gpt-35-turbo`, `gpt-5.2`, `gpt4o`, `llama3d1-405b`, `llama3d2-11b`, `llama3d2-1b`, `llama3d2-90b`, `llama4-17B-128E`, `o1-mini`, `o1_re=low`, `o3-mini`
+- `validation` models: `DK-R1-Dist-Qwen-14B`, `gemini-3-flash`, `llama3d2-3b`
+## Data Quality And Label Construction
+Training labels are distilled from two proprietary judge scores used by the ADeLe evaluation pipeline to derive the official correctness signal. The configured source columns are `score_gpt4o` and `score_sonnet`.
+- Ordinal target: `floor(mean(score_gpt4o, score_sonnet))`.
+- Binary target: `CORRECT` when the ordinal target is >= `3`.
+- Judge-agreement filter: keep examples with `abs(score_gpt4o - score_sonnet) <= 1`.
+- Response-length filter: keep responses with at most `4096` base-tokenizer tokens before prompt formatting.
+- Sequence-length filter: keep full chat-formatted examples within `max_seq_length=8192`.
+## Validation Results
+Source artifact: `validation_trainer_metrics.json`.
+| Metric | Value |
+| --- | --- |
+| Epoch | 1.0000 |
+| Binary accuracy | 0.9894 |
+| Binary macro F1 | 0.9880 |
+| Precision, CORRECT | 0.9932 |
+| Recall, CORRECT | 0.9909 |
+| Precision, INCORRECT | 0.9817 |
+| Recall, INCORRECT | 0.9863 |
+| False negative rate, CORRECT | 0.0091 |
+| False positive rate, CORRECT | 0.0137 |
+| Ordinal accuracy | 0.9639 |
+| Ordinal macro F1 | 0.7351 |
+| Mean confidence | 0.9604 |
 ## Recommended Inference
 Do not use free-form generation as the primary prediction method. The recommended path scores the restricted continuations `"1"`, `"2"`, `"3"`, `"4"`, and `"5"`.
 Training, filtering, split, tokenization, and metric artifacts available at packaging time are stored in `adele_judge_metadata.json`.
+The model is trained on distilled judge targets. These targets are useful for reproducing the ADeLe paper-style correctness signal at lower inference cost, but they should not be interpreted as independent human annotations.
+## References
+- ADeLe project page: [ADeLe v1.0](https://kinds-of-intelligence-cfi.github.io/ADELE/).
+- ADeLe paper and official correctness definition: [General scales unlock AI evaluation with explanatory and predictive power](https://www.nature.com/articles/s41586-026-10303-2).
+- Official ADeLe dataset: [CFI-Kinds-of-Intelligence/ADeLe_battery_v1dot0](https://huggingface.co/datasets/CFI-Kinds-of-Intelligence/ADeLe_battery_v1dot0).
+- Official instance-level model-response data used for distillation: [https://github.com/Kinds-of-Intelligence-CFI/ADeLe-AIEvaluation/tree/main/ADeLe_battery_data/subject_specific_instance_level_data](https://github.com/Kinds-of-Intelligence-CFI/ADeLe-AIEvaluation/tree/main/ADeLe_battery_data/subject_specific_instance_level_data).
+- Training and Hub packaging implementation: [https://github.com/adgomant/adele-judge](https://github.com/adgomant/adele-judge).
 ## Limitations
 - ADeLe-specific judge; not a general-purpose evaluator.
+- Distilled from proprietary judge labels and inherits their noise, calibration, and biases.
 - Intended for scoring responses against a reference answer.
 - It should not produce explanations; the expected output is a single score.
+- Validation is out-of-model within the ADeLe suite, so transfer outside that suite should be measured before relying on it.