On-Device AI, Explained: How Your Phone Runs AI Without the Cloud
Your phone can now run AI without phoning home — a small model, a special chip, and a router that decides what stays local. Here is how it works, and what still has to go to the cloud.
1 What it is
On-device AI means the AI model does its work on the chip inside your phone or laptop, instead of sending your request to a server in a data center and waiting for the answer to come back. The model lives on your device, in your device’s memory, and the answer is produced right there.
Three things made this possible. First, models got small. A carefully trained model of two to four billion parameters can summarize, rewrite, translate, and answer short questions well enough for daily use. Apple’s 2025 foundation models report describes a roughly three-billion-parameter on-device model; Google’s Gemini Nano runs inside Android’s AICore system service; Microsoft’s Phi-3-mini, at 3.8 billion parameters, can be quantized to 4-bit so it occupies only about 1.8 gigabytes of memory.
Second, chips got a dedicated AI block. Modern phones and laptops add a neural processing unit, or NPU, built for the repetitive multiply-and-add arithmetic of AI. Microsoft sets a floor of more than 40 trillion operations per second, or TOPS, for a Copilot+ PC. Third, the software got a traffic cop: a router that decides, per task, whether to run locally or send the work to the cloud.
2 How it works: the four steps
Every on-device AI feature runs the same four steps:
1. Shrink — take a big model and compress it. Quantization lowers the precision of the model’s weights, and distillation teaches a small student model from a big teacher. 2. Load — put the shrunken model in the device’s memory, where it is ready instantly. 3. Route — for each request, decide where it runs. 4. Answer — the NPU crunches the numbers and the result appears on screen.
The routing step is where the industry is heading. Microsoft calls its version hybrid intelligence: GitHub Copilot Fusion picks the right model for each coding task, and can now tap local models like MAI Code 1.1 Flash on a developer’s own PC. The local model is a mixture-of-experts design with 137 billion total parameters and 6.8 billion active at a time, quantized down to 53 gigabytes — about 80 percent smaller than the cloud version — and it scores 70.8 percent on the SWE-Bench Verified coding test, against 72.6 percent for the cloud model.
The same pattern is everywhere. Apple runs some requests on its on-device foundation models and sends harder ones to Private Cloud Compute. Google’s Gemini Nano handles quick tasks on the phone and hands off the rest. The phone and the cloud are not rivals any more — they are a team, and the router is the coach.
3 On-device vs cloud
Cloud AI runs on giant models in data centers: the most capable models in the world, but every request is a round trip. It needs a connection, it costs server time on every query, and your data leaves the device. It is the only option for the hardest work — long reasoning, huge documents, frontier research.
On-device AI runs on small models on your chip: no round trip, so answers come back fast and features keep working with no signal. Your data can stay on the device. But the model is small, so it is weaker, and it eats storage and memory — a single local coding model can need 53 gigabytes for its weights and peak at 75.5 gigabytes of memory at full context.
4 Why everyone’s building it now
Three stories in today’s briefing are the same story. At Microsoft Ignite on Tuesday, Satya Nadella unveiled hybrid intelligence on Windows: Copilot Fusion routing coding work to local models, and local context, actions, and models coming to Copilot on Copilot+ PCs, in a pilot with Adobe, Autodesk, Dell, HP, and Nvidia. Microsoft’s pitch is budget-driven — customers’ needs are outpacing what their cloud budgets can support, and local models stretch those budgets while keeping access to frontier models for the hard tasks.
Google’s November Pixel Drop, announced the same day, pushes Gemini Intelligence deeper into the Pixel 10 — more of the phone’s AI running on the phone itself. And Alibaba’s Qwen4-Max, launched Monday, shows the other side of the same squeeze: a 2.4-trillion-parameter model that activates only 71 billion per token, cutting the cost per task by about 60 percent. Whether the model shrinks onto the device or just gets cheaper in the cloud, the whole industry is now optimizing for the same thing: AI that costs less per answer.
5 The honest part
First, a local model is still a small model. A three-billion-parameter model can summarize and rewrite beautifully, but it will not out-reason a frontier cloud model, and it inherits the same weaknesses — including the teacher’s mistakes, when it was distilled from one. The 70.8 percent local coding score is close to the cloud’s 72.6, but close is not equal, and the gap grows on harder tasks.
Second, “on-device” does not automatically mean private. Many features are hybrid: a request can start local and fall back to the cloud for capability, current information, or account state. GitHub’s local routing is explicit about this — the routing decisions themselves are still made in the cloud, and local models may have different safety protections. And the hardware bar is real: the local coding model needs a laptop with 128 gigabytes of memory to run at full context.
The honest summary: on-device AI is the industry’s answer to AI’s cost problem, not a replacement for the cloud. The future is a router deciding, task by task, where the work belongs — and the devices that win will be the ones that make that decision invisible.
Key terms
- NPU A processor block built for the repetitive multiply-and-add arithmetic of AI, far more efficient than a CPU or GPU for those tasks.
- TOPS Trillions of operations per second — the headline speed number for an NPU. Microsoft requires 40+ TOPS for a Copilot+ PC.
- Small language model (SLM) A small language model, typically 1 to 4 billion parameters, trained to do everyday tasks well enough to run on a phone.
- Quantization Lowering the precision of a model’s weights so it takes less memory — the main trick that shrinks models onto devices.
- Hybrid intelligence Using a mix of local and cloud models, with a router picking the right one per task, instead of choosing one or the other.
- Inference Running a trained model to produce an answer, as opposed to training it. On-device AI is local inference.
This lesson comes from today’s Daily Tech Briefing (Episode 37, November 18, 2026): Nvidia's $115B blowout, Anthropic's record IPO & the Java flaw found by AI. Read the full briefing: Nvidia's $115B Blowout — Daily Tech Briefing.
Microsoft, “Microsoft Ignite 2026: Building Windows for the next era of work” (Windows Experience Blog, November 17, 2026) — Hybrid Intelligence, Copilot Fusion local routing; Microsoft technical write-up via WindowsForum and CTOL Digital (October–November 2026) — MAI Code 1.1 Flash specs: 137B total, 6.8B active, 53GB quantized, 70.8% SWE-Bench Verified; Microsoft, Phi-3 Technical Report (April 2024) — phi-3-mini quantized to 4-bit, about 1.8GB; Apple, Foundation Models report (2025) — roughly 3B-parameter on-device model; Google Android developer documentation — Gemini Nano in AICore; Pointales, “On-device AI vs cloud AI” (2026) — NPU, TOPS, and the hybrid mix.