ai-safety-institute/uq-google-gemma-3-12b-it

Unrelated Questions Deception Probe for google/gemma-3-12b-it

A probe trained to detect deceptive behaviour in google/gemma-3-12b-it using logprob differences from yes/no follow-up questions, following the methodology from How to Catch an AI Liar: Lie Detection in Black-Box LLMs by Asking Unrelated Questions (Pacchiardi et al., ICLR 2024). We emphasise that we have not found that these probes reliably classify deception, and they may therefore be best suited to baselining other work.

Quick Start

uv add lie-detectors        # or: pip install lie-detectors

from lie_detectors import get_probe

probe = get_probe("ai-safety-institute/uq-google-gemma-3-12b-it")

The default checkpoint is the best performer from the hyperparameter sweep (ar_dim.pt). To pick a specific checkpoint, pass filename=:

probe = get_probe("ai-safety-institute/uq-google-gemma-3-12b-it", filename="ar_mlp_wd_0_001_lr_0_0001_ep_100.pt")

See UKGovernmentBEIS/lie_detectors for the loading library.

Use sweep.json to see all 40 available checkpoints and their metrics.

Computing Features

Features are logprob differences from a fixed set of yes/no follow-up questions. For each sample, after the model produces its response:

1. Append each follow-up question as a new user message.
2. Generate 1 token with e.g. logprobs=True, top_logprobs=20 if using vLLM.
3. Compute logsumexp(yes_logprobs) - logsumexp(no_logprobs) for yes/no token variants.
4. Stack into a feature vector.

The question list is published as the ai-safety-institute/unrelated-questions-follow-up-questions dataset (use the 48 questions marked used_in_paper=true).

Model Details

Property	Value
Target model	google/gemma-3-12b-it
Safe name	google--gemma-3-12b-it
Default checkpoint	ar_dim.pt
Available checkpoints	40
Calibration	Threshold set at 1% FPR on Alpaca (honest baseline)

Training Data

Probes are trained on the SciQ dataset (CC-BY-NC-3.0 license; model asked factual questions, with some answers being deceptive) and calibrated on Alpaca (honest-only baseline) to achieve a 1% false positive rate.

Citation

Original Paper

@inproceedings{pacchiardi2024catchailiar,
      title={How to Catch an AI Liar: Lie Detection in Black-Box LLMs by Asking Unrelated Questions},
      author={Lorenzo Pacchiardi and Alex J. Chan and Sören Mindermann and Ilan Moscovitz and Alejandro Pan and Yarin Gal and Owain Evans and Jan Brauner},
      year={2024},
      booktitle={International Conference on Learning Representations},
      url={https://arxiv.org/abs/2309.15840},
}

Trained Probes

@misc{cooney2026liedetectors,
      title={``Did you lie?'' Evaluating Lie Detectors across Model Scale and Belief-Verified Model Organisms},
      author={Alan Cooney and David Africa and Geoffrey Irving},
      year={2026},
      month={May},
}