# FunctionGemma 270M Mobile Exports

FunctionGemma is a Gemma 3 270M variant trained for local function calling. It
is intended to translate user text into structured tool calls, then optionally
turn the tool result into a short user-facing response.

## Setup

Accept the `google/functiongemma-270m-it` license on Hugging Face, then
authenticate before running conversion:

```bash
export HF_TOKEN=hf_...
cd models/functiongemma/export
poetry env use /opt/homebrew/bin/python3.11
poetry install --with convert
```

## LiteRT-LM Export

```bash
poetry run python convert.py --output-dir ./functiongemma-litert
```

The default export uses LiteRT Torch's `dynamic_wi8_afp32` quantization recipe,
prefill lengths `128,512,1024`, and a `1024` token KV cache. For a larger
mobile prompt budget:

```bash
poetry run python convert.py \
  --output-dir ./functiongemma-litert \
  --cache-length 2048 \
  --prefill-lengths 128,512,1024,2048
```

Use `--quantize none` only for debugging.

The default quantized bundle is about 283 MB for `model.litertlm`; LiteRT may
also create a local XNNPACK cache file next to it.

## Validate

```bash
poetry run pytest test_function_calls.py test_litert.py -q
poetry run python smoke_litert.py
```

## CoreML Export

```bash
poetry run python convert_coreml.py \
  --output-dir ./functiongemma-coreml \
  --compute-precision float32 \
  --quantize int8
```

The CoreML artifact is a fixed 128-token last-logits model. It uses int8
weights with float32 compute because the float16 compute export produced NaN
logits in local validation. This CoreML path does full-context recompute for
each generated token; LiteRT-LM remains the preferred production path for tool
calling latency.

```bash
poetry run pytest test_coreml.py -q
```

Validated local bundle:

- `functiongemma-coreml/FunctionGemmaLastLogits.mlpackage`
- `functiongemma-coreml/config.json`
- tokenizer files in `functiongemma-coreml/`

## Benchmarks

```bash
poetry run python benchmark.py --backend litert --runs 5 --warmup 1
poetry run python benchmark.py --backend coreml --coreml-compute-units cpu --runs 5 --warmup 1
poetry run python benchmark.py --backend coreml --coreml-compute-units cpu_and_ne --runs 5 --warmup 1
```

Local results on this machine:

| Backend | Quantization | Load RSS Δ | Peak RSS Δ | Mean tok/s |
|---|---:|---:|---:|---:|
| LiteRT-LM CPU | dynamic int8 | 551.1 MB | 865.3 MB | 148.54 |
| CoreML CPU | int8 weights, fp32 compute | 658.0 MB | 1690.4 MB | 31.49 |
| CoreML CPU+NE | int8 weights, fp32 compute | 86.7 MB | 1129.8 MB | 32.82 |

## Runtime Loop

The model should be used in two passes:

1. Build a prompt with `format_tool_call_prompt(...)` and stop on
   `<end_function_call>` or `<start_function_response>`.
2. Parse the returned call with `parse_function_calls(...)`, validate it against
   an allowlist, and execute the tool.
3. Build a second prompt with `format_final_response_prompt(...)` and stop on
   `<end_of_turn>` to get the final user-facing answer.

For command-only actions, the app can skip the second pass and present its own
deterministic UI response after the tool succeeds.

FunctionGemma is trained for single-turn and parallel tool calls. Do not rely on
it for multi-step dependency chains without app-side orchestration or fine-tuning.

The LiteRT-LM Python runtime currently returns FunctionGemma calls as raw text,
for example:

```text
<start_function_call>call:get_current_weather{location:<escape>Tokyo<escape>}<end_function_call>
```

Use `parse_function_calls(...)` to validate and dispatch the call. After the
tool response is sent back as a `tool_response` turn, the same exported model can
produce the final user-facing answer.

## Mobile Artifacts

Ship these files:

- `functiongemma-litert/model.litertlm`
- `functiongemma-litert/config.json`

Do not ship local runtime caches such as
`model.litertlm.xnnpack_cache_*`; they are regenerated by LiteRT.