File size: 3,924 Bytes
7ff3e1d | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 | # FunctionGemma 270M Mobile Exports
FunctionGemma is a Gemma 3 270M variant trained for local function calling. It
is intended to translate user text into structured tool calls, then optionally
turn the tool result into a short user-facing response.
## Setup
Accept the `google/functiongemma-270m-it` license on Hugging Face, then
authenticate before running conversion:
```bash
export HF_TOKEN=hf_...
cd models/functiongemma/export
poetry env use /opt/homebrew/bin/python3.11
poetry install --with convert
```
## LiteRT-LM Export
```bash
poetry run python convert.py --output-dir ./functiongemma-litert
```
The default export uses LiteRT Torch's `dynamic_wi8_afp32` quantization recipe,
prefill lengths `128,512,1024`, and a `1024` token KV cache. For a larger
mobile prompt budget:
```bash
poetry run python convert.py \
--output-dir ./functiongemma-litert \
--cache-length 2048 \
--prefill-lengths 128,512,1024,2048
```
Use `--quantize none` only for debugging.
The default quantized bundle is about 283 MB for `model.litertlm`; LiteRT may
also create a local XNNPACK cache file next to it.
## Validate
```bash
poetry run pytest test_function_calls.py test_litert.py -q
poetry run python smoke_litert.py
```
## CoreML Export
```bash
poetry run python convert_coreml.py \
--output-dir ./functiongemma-coreml \
--compute-precision float32 \
--quantize int8
```
The CoreML artifact is a fixed 128-token last-logits model. It uses int8
weights with float32 compute because the float16 compute export produced NaN
logits in local validation. This CoreML path does full-context recompute for
each generated token; LiteRT-LM remains the preferred production path for tool
calling latency.
```bash
poetry run pytest test_coreml.py -q
```
Validated local bundle:
- `functiongemma-coreml/FunctionGemmaLastLogits.mlpackage`
- `functiongemma-coreml/config.json`
- tokenizer files in `functiongemma-coreml/`
## Benchmarks
```bash
poetry run python benchmark.py --backend litert --runs 5 --warmup 1
poetry run python benchmark.py --backend coreml --coreml-compute-units cpu --runs 5 --warmup 1
poetry run python benchmark.py --backend coreml --coreml-compute-units cpu_and_ne --runs 5 --warmup 1
```
Local results on this machine:
| Backend | Quantization | Load RSS Δ | Peak RSS Δ | Mean tok/s |
|---|---:|---:|---:|---:|
| LiteRT-LM CPU | dynamic int8 | 551.1 MB | 865.3 MB | 148.54 |
| CoreML CPU | int8 weights, fp32 compute | 658.0 MB | 1690.4 MB | 31.49 |
| CoreML CPU+NE | int8 weights, fp32 compute | 86.7 MB | 1129.8 MB | 32.82 |
## Runtime Loop
The model should be used in two passes:
1. Build a prompt with `format_tool_call_prompt(...)` and stop on
`<end_function_call>` or `<start_function_response>`.
2. Parse the returned call with `parse_function_calls(...)`, validate it against
an allowlist, and execute the tool.
3. Build a second prompt with `format_final_response_prompt(...)` and stop on
`<end_of_turn>` to get the final user-facing answer.
For command-only actions, the app can skip the second pass and present its own
deterministic UI response after the tool succeeds.
FunctionGemma is trained for single-turn and parallel tool calls. Do not rely on
it for multi-step dependency chains without app-side orchestration or fine-tuning.
The LiteRT-LM Python runtime currently returns FunctionGemma calls as raw text,
for example:
```text
<start_function_call>call:get_current_weather{location:<escape>Tokyo<escape>}<end_function_call>
```
Use `parse_function_calls(...)` to validate and dispatch the call. After the
tool response is sent back as a `tool_response` turn, the same exported model can
produce the final user-facing answer.
## Mobile Artifacts
Ship these files:
- `functiongemma-litert/model.litertlm`
- `functiongemma-litert/config.json`
Do not ship local runtime caches such as
`model.litertlm.xnnpack_cache_*`; they are regenerated by LiteRT.
|