Google has introduced the LiteRT Tensor API, a lightweight C++ API that lets developers build, customise, and optimise machine learning models for on-device execution across CPU, GPU, and NPU hardware without requiring separate rewrites for each backend.
The traditional approach to on-device machine learning is for developers to write computations manually in C++ for maximum control, but implementations are tightly coupled to specific backends such as Metal, OpenCL, or Vulkan and must be rewritten for each new hardware target. Google says the new API addresses this by allowing developers to construct computation graphs operation by operation in C++, with the same graphs running across different accelerators through LiteRT.
The API provides direct weight ingestion from checkpoints, including Hugging Face .safetensors files, zero-copy buffer interoperability, and compatibility with AI coding agents that can construct, modify, validate and benchmark model graphs by generating C++ code. Google used a coding agent to create a demonstration app for multi-object segmentation and tracking, which served as the basis for benchmarking the SAM 2 model. According to Google, in tests on a MacBook Pro, it ran about 1.5 times faster than Apple MLX in fp16 for the image encoder plus mask decoder.
Google noted that the coding agent could optimise the model graph by, for example, "moved SAM 2's per-frame decisions, such as whether the object is still visible, from CPU code into the graph" and rewriting operations that triggered CPU fallback on WebGPU using GPU-compatible arithmetic.
Arm director of AI Laurence Moroney commented on LinkedIn that developers should start with model conversion and turn to the C++ API only "when you need graph-level control or want pre and post-processing to stay on the accelerator". He added that "[g]oing from under 10 to over 30 frames per second at 512x512 on WebGPU, with a coding agent doing the iteration, shows how much structured primitives help agents do real optimization work."
Google is planning future improvements including direct ML Drift backend support, additional NPU-friendly operations, automatic full-graph quantization, and expanded dynamic shape support.
Analysis
Why This Matters
- On-device ML is critical for privacy and latency; the API could simplify development and improve performance across diverse hardware.
- Reduces engineering effort by abstracting backend differences, potentially lowering barriers for deploying custom on-device models.
- Integration with AI coding agents hints at automated optimization workflows, with Moroney's example showing a threefold performance gain on WebGPU.
Background
The traditional approach of writing backend-specific C++ code for on-device machine learning is labour-intensive and requires separate implementations for CPU, GPU, and NPU. LiteRT Tensor API offers a unified C++ interface that allows the same computation graph to run on multiple accelerators without rewrites.
Key Perspectives
Google: Says the API combines the flexibility of native model authoring with cross-platform portability, giving developers graph-level control while running on any target without code rewrites.
Laurence Moroney (Arm): Recommends starting with model conversion and using the C++ API only when graph-level control or on-accelerator pre/post-processing is needed, noting that a coding agent achieved significant performance improvements through structured primitives.
What to Watch
- Planned features: direct ML Drift backend support, automatic full-graph quantization, and expanded dynamic shape handling.
- Adoption by developers and feedback on performance claims, especially compared to existing frameworks like Apple MLX.
- How AI coding agents integrate with the API for automated model optimisation and iteration.
Checked
12 claims in this story checked against 1 source: 11 supported.
Show each claim
- SupportedGoogle has introduced/launched the LiteRT Tensor API.InfoQ: “Google has introduced the LiteRT Tensor API, a lightweight C++ API”
- SupportedThe API lets developers build and optimise machine-learning models for on-device use across CPU, GPU and NPU without rewriting for each hardware target.InfoQ: “enabling the same graphs to run through LiteRT on CPU, GPU, and NPU hardware without requiring a rewrite for each new target.”
- SupportedThe API is a lightweight C++ API for on-device execution.InfoQ: “a lightweight C++ API that allows developers to build, customize, and optimize machine learning models directly for on-device execution.”
- SupportedThe traditional manual C++ approach can leave implementations tied to specific backends and needing rewrites for new hardware targets.InfoQ: “Implementations are tightly coupled to specific backends (such as Metal, OpenCL, or Vulkan) and must be rewritten for each new hardware target.”
- SupportedThe API lets developers construct computation graphs operation by operation in C++.InfoQ: “allowing developers to build computation graphs operation by operation in C++.”
- SupportedThe API supports direct weight ingestion from checkpoints, including Hugging Face .safetensors files.InfoQ: “direct weight ingestion, allowing developers to load weights directly from checkpoints, including Hugging Face .safetensors files;”
- SupportedAI coding agents can construct, modify, validate and benchmark model graphs by generating C++ code.InfoQ: “compatibility with AI coding agents, which can construct, modify, validate, and benchmark model graphs by generating C++ code.”
- SupportedGoogle used a coding agent to create a demonstration app for multi-object segmentation and tracking, which was used as the basis for benchmarking.InfoQ: “Google used a coding agent to create a real-world app demonstrating how to use the LiteRT Tensor API for multi-object segmentation and tracking. The application also served as the basis for benchmarking performance across platforms:”
- SupportedGoogle's MacBook Pro test ran about 1.5 times faster than Apple MLX in fp16 for the image encoder plus mask decoder.InfoQ: “In our tests on the MacBook Pro, it ran about 1.5x faster than Apple MLX (fp16, image encoder plus mask decoder).”
- SupportedThe coding agent moved SAM 2's per-frame decisions, including whether the object was still visible, from CPU code into the graph.InfoQ: “it "moved SAM 2's per-frame decisions, such as whether the object is still visible, from CPU code into the graph"”
- SupportedArm director of AI Laurence Moroney commented on LinkedIn and said developers should turn to the C++ API only when they need graph-level control or want pre- and post-processing to stay on the accelerator.InfoQ: “Commenting on the announcement on LinkedIn, Arm director of AI Laurence Moroney emphasized that developers should always start with model conversion and turn to the C++ API only "when you need graph-level control or want pre and post-processing to stay on the accelerator".”
- Partly supportedMoroney said: “[g]oing from under 10 to over 30 frames per second at 512x512 on WebGPU, with a coding agent doing the iteration, shows how much structured primitives help agents do real optimization work.”InfoQ: “Going from under 10 to over 30 frames per second at 512x512 on WebGPU, with a coding agent doing the iteration, shows how much structured primitives help agents do real optimization work.”
