跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
搜索与研究

on-device-verification

Prove a converted or quantized LiteRT model on the actual device via the CompiledModel API - confirm GPU residency, compare device output against the source model, and diagnose device-only failures such as silent CPU fallback, whole-graph compile ceilings, and fp16 range breaks. Use after conversion or quantization, when device output is wrong or NaN, when a clean graph fails to compile only on device, or when GPU and CPU outputs are suspiciously identical.

google-ai-edge · 已有 AI 审核记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add google-ai-edge/litert-samples --skill on-device-verification

技能说明


name: on-device-verification description: Prove a converted or quantized LiteRT model on the actual device via the CompiledModel API - confirm GPU residency, compare device output against the source model, and diagnose device-only failures such as silent CPU fallback, whole-graph compile ceilings, and fp16 range breaks. Use after conversion or quantization, when device output is wrong or NaN, when a clean graph fails to compile only on device, or when GPU and CPU outputs are suspiciously identical.

On-device verification

A device result is done when three things hold:

  1. the model compiles and runs on the accelerator you claim it runs on,
  2. the output matches the source model numerically and on a task-level gate,
  3. the record names the device, the runtime version, and the residency line. A number without those is not reproducible and not a result.

Host-side checks (the CompiledModel checker used in gpu-clean-conversion) exercise the host GPU. The device has its own shader compiler, its own precision behavior, and its own memory ceiling — every failure mode in the table below was hit by a model that had already passed on the host.

Loop

1. Dump references from the source model, once. Fixed inputs — one real sample plus fixed-seed random — saved as .npy next to the recipe (dump_*_ref.py). These are ground truth for every later step; regenerate them only when the source model changes.

2. Run the same inputs on the device: CPU first, then GPU. One argument switches the accelerator:

from ai_edge_litert.compiled_model import CompiledModel
from ai_edge_litert.hardware_accelerator import HardwareAccelerator

model = CompiledModel.from_file(
    "model.tflite", hardware_accel=HardwareAccelerator.GPU)  # or .CPU

The device-CPU run is the control. If it already diverges from the source dump, the problem is the conversion, not the GPU — go back to gpu-clean-conversion. A full worked example of the A/B lives in this repo at samples/litert/speech_recognition/convert/verify_tflite.py.

Ask for the strict accelerator. Compiling with HardwareAccelerator.CPU | HardwareAccelerator.GPU permits partial delegation and hides fallback; use the combined mode only to discover which ops fell back after a strict GPU compile fails.

3. Read the delegate log before reading any numbers.

Replacing N out of M node(s) with delegate ... X partitions

Record N/M and the partition count. N < M or X > 1 means part of the graph runs on the CPU — decide whether that is acceptable before quoting any accuracy or latency number.

4. Gate the output three ways.

  • Numeric: correlation and max abs diff against the source dump. Expect genuine GPU fp16 to drift in the last digits — exact equality is a symptom, not a pass (see below).
  • Task: argmax match, IoU, token-for-token greedy decode — whatever the model is for.
  • Artifact, for generative models: the decoded text / audio / image. An artifact gate catches argmax ties that numeric tolerance misses.

5. Record it: device, runtime version, N/M + partitions, correlation, max abs diff, task result — one row per device in the recipe README.

The silent CPU fallback

The most common false positive: everything runs and the numbers match perfectly. If the GPU output is bit-identical to the device CPU fp32 output, it almost certainly did not run on the GPU. Perfect equality is the tell, not the goal. Cross-check the residency line; only when N/M is full and the outputs drift in the last digits are you looking at a real GPU run.

Device-only failures

What you seeWhat it is, what to do
GPU compile fails on device for a graph that is op-clean and passed the host checkA whole-graph compile ceiling, not a bad op — a fused graph can fail where each half compiles. Split at a natural block boundary (conv frontend / transformer encoder), verify split == monolith bit-exact on the host, ship the halves
The file refuses to load at allThe >2 GB flatbuffer limit. Split, or quantize below it
Full residency, wrong numbersBisect by intermediates: re-export with block-boundary tensors as extra outputs and find the first one that diverges. The cause is usually a reduction (mean, variance, Σx²) in fp16, not the op you suspect. If materializing a tap fixes the numbers, that localizes the bug to a fusion boundary — record it as a finding, not a nuisance
Wrong-but-plausible output, and every hypothesis costs a slow re-exportMicro-probe instead: export ~1 KB graphs — the suspect subexpression and each candidate fix — and run them through the device harness you already have. Minutes per hypothesis instead of a re-export per hypothesis
NaN or garbage only on the GPU, in a model with large-magnitude residuals or deep modulation pathsAn fp16 range break. Confirm the attribution by forcing fp32 on the delegate where the API exposes it (~2× memory, slower); then fix it properly with the fp16-safe rewrites in gpu-clean-conversion, or keep the offending block on CPU and run the rest on GPU
The process dies while loading or running a large float modelA memory ceiling, not a model bug. Weights + activations + delegate buffers must fit in available memory. Do not run a multi-GiB fp32 build in-process on a phone "just to check" — quantize or split first, then verify the smaller thing
A deep transformer shipped as an fp16 graph is bit-exact on desktop and noise on the device CPUAndroid ARM XNNPACK computes native fp16; desktop XNNPACK upcasts to fp32. Deep residual streams compound the difference to collapse. Ship fp32 graphs for CPU inference on device; fp16 is a GPU-side format
Attention quality collapses only for a small-head submodulehead_dim is the fp16-fragile axis, not token count: the same graph at head_dim 64 held corr 0.998 where head_dim 16 fell to 0.86. Pad heads to ≥32 or keep the small-head module on CP
阅读完整来源 ↗