Tech Briefing. Your daily dose of tech news — signal, not noise.
Learning Center
One technology, explained in plain English
DTBRIEF

On-Device AI, Explained: How Your Phone Runs AI Without the Cloud

Your phone can now run AI without phoning home — a small model, a special chip, and a router that decides what stays local. Here is how it works, and what still has to go to the cloud.

A smartphone with a glowing neural chip inside it, small streams of light flowing from the chip to on-screen tasks, while a distant cloud data center sits quietly in the background, on a deep navy background with cyan and warm yellow accents
On-device AI runs the model on the chip in your hand — the cloud is the backup, not the default.
40+ TOPS
The NPU floor for a Copilot+ PC — trillions of AI operations per second, on the device
3.8B → 1.8GB
Microsoft’s Phi-3-mini, quantized to 4-bit, fits a useful language model in less memory than a movie
53GB
The quantized coding model Microsoft now runs on a laptop — 80% smaller than its cloud version

1 What it is

On-device AI means the AI model does its work on the chip inside your phone or laptop, instead of sending your request to a server in a data center and waiting for the answer to come back. The model lives on your device, in your device’s memory, and the answer is produced right there.

Three things made this possible. First, models got small. A carefully trained model of two to four billion parameters can summarize, rewrite, translate, and answer short questions well enough for daily use. Apple’s 2025 foundation models report describes a roughly three-billion-parameter on-device model; Google’s Gemini Nano runs inside Android’s AICore system service; Microsoft’s Phi-3-mini, at 3.8 billion parameters, can be quantized to 4-bit so it occupies only about 1.8 gigabytes of memory.

Second, chips got a dedicated AI block. Modern phones and laptops add a neural processing unit, or NPU, built for the repetitive multiply-and-add arithmetic of AI. Microsoft sets a floor of more than 40 trillion operations per second, or TOPS, for a Copilot+ PC. Third, the software got a traffic cop: a router that decides, per task, whether to run locally or send the work to the cloud.

2 How it works: the four steps

Every on-device AI feature runs the same four steps:

1. Shrink — take a big model and compress it. Quantization lowers the precision of the model’s weights, and distillation teaches a small student model from a big teacher. 2. Load — put the shrunken model in the device’s memory, where it is ready instantly. 3. Route — for each request, decide where it runs. 4. Answer — the NPU crunches the numbers and the result appears on screen.

The routing step is where the industry is heading. Microsoft calls its version hybrid intelligence: GitHub Copilot Fusion picks the right model for each coding task, and can now tap local models like MAI Code 1.1 Flash on a developer’s own PC. The local model is a mixture-of-experts design with 137 billion total parameters and 6.8 billion active at a time, quantized down to 53 gigabytes — about 80 percent smaller than the cloud version — and it scores 70.8 percent on the SWE-Bench Verified coding test, against 72.6 percent for the cloud model.

The same pattern is everywhere. Apple runs some requests on its on-device foundation models and sends harder ones to Private Cloud Compute. Google’s Gemini Nano handles quick tasks on the phone and hands off the rest. The phone and the cloud are not rivals any more — they are a team, and the router is the coach.

3 On-device vs cloud

Cloud AI runs on giant models in data centers: the most capable models in the world, but every request is a round trip. It needs a connection, it costs server time on every query, and your data leaves the device. It is the only option for the hardest work — long reasoning, huge documents, frontier research.

On-device AI runs on small models on your chip: no round trip, so answers come back fast and features keep working with no signal. Your data can stay on the device. But the model is small, so it is weaker, and it eats storage and memory — a single local coding model can need 53 gigabytes for its weights and peak at 75.5 gigabytes of memory at full context.

Cloud AIThe biggest, smartest models, running in data centers. Best for hard reasoning and huge context — but needs a connection, costs server time, and your data leaves the device.
On-device AISmall models running on your own chip. Fast, works offline, and your data can stay put — but weaker, and hungry for storage and memory.
The cloud is the brain. The device is the reflex. Hybrid AI is the nervous system connecting them.

4 Why everyone’s building it now

Three stories in today’s briefing are the same story. At Microsoft Ignite on Tuesday, Satya Nadella unveiled hybrid intelligence on Windows: Copilot Fusion routing coding work to local models, and local context, actions, and models coming to Copilot on Copilot+ PCs, in a pilot with Adobe, Autodesk, Dell, HP, and Nvidia. Microsoft’s pitch is budget-driven — customers’ needs are outpacing what their cloud budgets can support, and local models stretch those budgets while keeping access to frontier models for the hard tasks.

A router sitting between a phone and a cloud data center, sending quick tasks down to the phone and heavy tasks up to the cloud, on a deep navy background with cyan and warm yellow accents
Hybrid intelligence is the router: quick, private tasks stay on the device, and only the hard work goes to the cloud.

Google’s November Pixel Drop, announced the same day, pushes Gemini Intelligence deeper into the Pixel 10 — more of the phone’s AI running on the phone itself. And Alibaba’s Qwen4-Max, launched Monday, shows the other side of the same squeeze: a 2.4-trillion-parameter model that activates only 71 billion per token, cutting the cost per task by about 60 percent. Whether the model shrinks onto the device or just gets cheaper in the cloud, the whole industry is now optimizing for the same thing: AI that costs less per answer.

5 The honest part

First, a local model is still a small model. A three-billion-parameter model can summarize and rewrite beautifully, but it will not out-reason a frontier cloud model, and it inherits the same weaknesses — including the teacher’s mistakes, when it was distilled from one. The 70.8 percent local coding score is close to the cloud’s 72.6, but close is not equal, and the gap grows on harder tasks.

Second, “on-device” does not automatically mean private. Many features are hybrid: a request can start local and fall back to the cloud for capability, current information, or account state. GitHub’s local routing is explicit about this — the routing decisions themselves are still made in the cloud, and local models may have different safety protections. And the hardware bar is real: the local coding model needs a laptop with 128 gigabytes of memory to run at full context.

The honest summary: on-device AI is the industry’s answer to AI’s cost problem, not a replacement for the cloud. The future is a router deciding, task by task, where the work belongs — and the devices that win will be the ones that make that decision invisible.

Key terms

  • NPU A processor block built for the repetitive multiply-and-add arithmetic of AI, far more efficient than a CPU or GPU for those tasks.
  • TOPS Trillions of operations per second — the headline speed number for an NPU. Microsoft requires 40+ TOPS for a Copilot+ PC.
  • Small language model (SLM) A small language model, typically 1 to 4 billion parameters, trained to do everyday tasks well enough to run on a phone.
  • Quantization Lowering the precision of a model’s weights so it takes less memory — the main trick that shrinks models onto devices.
  • Hybrid intelligence Using a mix of local and cloud models, with a router picking the right one per task, instead of choosing one or the other.
  • Inference Running a trained model to produce an answer, as opposed to training it. On-device AI is local inference.
From today’s briefing

This lesson comes from today’s Daily Tech Briefing (Episode 37, November 18, 2026): Nvidia's $115B blowout, Anthropic's record IPO & the Java flaw found by AI. Read the full briefing: Nvidia's $115B Blowout — Daily Tech Briefing.

Sources

Microsoft, “Microsoft Ignite 2026: Building Windows for the next era of work” (Windows Experience Blog, November 17, 2026) — Hybrid Intelligence, Copilot Fusion local routing; Microsoft technical write-up via WindowsForum and CTOL Digital (October–November 2026) — MAI Code 1.1 Flash specs: 137B total, 6.8B active, 53GB quantized, 70.8% SWE-Bench Verified; Microsoft, Phi-3 Technical Report (April 2024) — phi-3-mini quantized to 4-bit, about 1.8GB; Apple, Foundation Models report (2025) — roughly 3B-parameter on-device model; Google Android developer documentation — Gemini Nano in AICore; Pointales, “On-device AI vs cloud AI” (2026) — NPU, TOPS, and the hybrid mix.