Running Local LLMs on Android with llama.cpp in Termux

The goal: an offline model on your phone

Modern phones have fast ARM cores and plenty of RAM, which is enough to run small language models with no network at all. The simplest way to run local LLMs on Android is llama.cpp in Termux: the same engine that sits underneath many desktop tools, running from the command line on your phone.

This is part two of my Termux series, following Claude Code in Termux. If you already use Running OpenCode in Termux on Android, a local llama.cpp server is a handy offline fallback.

Step 1: install llama.cpp

Install Termux from F-Droid or the project’s GitHub releases. The Google Play listing is still labelled an experimental branch with missing functionality, and Termux plus its plugin apps must all come from the same source because they share a signing key. Mixing sources gives you INSTALL_FAILED_SHARED_USER_INCOMPATIBLE errors.

Termux packages llama.cpp in its main repository, which is the quickest route:

pkg update && pkg upgrade
pkg install llama-cpp
ls $PREFIX/bin | grep llama

That last command shows which tools you got. You should see llama-cli for chatting in the terminal and llama-server for an HTTP API. The package also offers optional Vulkan and OpenCL backend subpackages, but GPU support on phones is patchy, so start on the CPU.

Building from source instead

If you want the very latest code, follow the upstream Android docs, which build with CMake in Termux:

pkg install git cmake clang libandroid-spawn
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build
cmake --build build --config Release -j 4

The binaries land in build/bin/. Expect the build to take a while and warm the phone up.

Step 2: pick a small quantised model

llama.cpp uses GGUF model files. On a phone, size is everything:

• Stick to roughly 1B to 4B parameter models to begin with.
• Use a 4-bit quantisation (files with names like Q4_K_M) as a sensible balance of quality and size.
• Keep the model file comfortably below your free RAM; Android will kill Termux if memory runs out.

Browse GGUF builds on Hugging Face, then download one into your home directory:

curl -L -o ~/model.gguf "https://huggingface.co/<user>/<repo>/resolve/main/<file>.gguf"

Recent llama.cpp builds can also fetch models directly from Hugging Face with the -hf option; check llama-cli --help on your build to confirm.

Step 3: chat in the terminal

llama-cli -m ~/model.gguf -c 4096 -p "Explain what a Termux package is in two sentences."

The upstream Android guide suggests starting with a context size around 4096. Larger contexts use much more memory and are the most common reason the session gets killed.

Step 4: run an API server

llama-server -m ~/model.gguf -c 4096 --host 127.0.0.1 --port 8080

Open http://127.0.0.1:8080 in your phone’s browser for the built-in web UI, or point any OpenAI-compatible client at it. That includes agent tools: my guide to using Claude Code with local LLM models explains the general approach, although tool-using agents really want bigger models than a phone can hold.

To use the server from your laptop, keep it on localhost and tunnel over SSH, which is covered in SSH in Termux. Avoid binding it to 0.0.0.0 on public Wi-Fi.

Realistic performance

I am not going to quote tokens-per-second figures, because they vary enormously by chipset, thermal state, model and quantisation. What you can expect in general:

• Small models in the 1B to 3B range are usable for short answers, summaries and quick code snippets.
• Speed drops noticeably as the phone heats up and throttles.
• Prompt processing for long inputs is slow; keep prompts focused.
• It drains the battery quickly. Plug in for longer sessions.

llama.cpp ships a llama-bench tool if you want real numbers for your own device.

Troubleshooting

• Termux closes mid-answer: out of memory. Use a smaller model, a smaller quantisation or a lower -c value.
• Process killed in the background: Android 12 and later can kill long-running processes. Keep Termux in the foreground and acquire a wake lock.
• Package not found: run pkg update first, and make sure you are not on the old Play Store build.

Wrap-up

Running local LLMs on Android is no longer a novelty. With pkg install llama-cpp, a small GGUF model and modest expectations, you get a private, offline assistant that also doubles as a local API for your other tools.