Run Local AI Models On A Raspberry Pi 5 With Ollama

AI that never phones home

Every AI assistant you currently use – ChatGPT, Claude, Gemini – sends your prompts to a remote server. Your questions, documents, code, and conversations all travel over the internet to someone else’s infrastructure. For most things that’s fine. But for anything sensitive – proprietary code, confidential documents, private thoughts – it’s a meaningful consideration.

Ollama lets you run large language models entirely locally. Your prompts stay on your hardware. And if you have a Raspberry Pi 5 with 8GB of RAM, you have just enough to run small but genuinely capable models – right on the device in your drawer.

What you’ll need

A Raspberry Pi 5 with 8GB RAM – this is the minimum for a usable experience with small models. A Pi 4 with 8GB can work but will be slower
A fast SD card or, preferably, an NVMe SSD via the Pi 5’s PCIe slot (models are large files)
Patience – inference on a Pi is slower than a desktop GPU, but it works

Installing Ollama

Installation is a single command:

curl -fsSL https://ollama.com/install.sh | sh

Ollama installs as a system service and starts automatically. Verify it’s running:

ollama --version

Pulling and running a model

Download and run a model with:

ollama run llama3.2:3b

This pulls the 3-billion-parameter version of Meta’s Llama 3.2 – a 2GB download. Once it’s pulled, you’re dropped into an interactive chat session directly in your terminal. Ask it questions, give it code to review, have it summarise text – it works like any AI chat interface, running entirely on your Pi.

Recommended models for a Pi 5

llama3.2:3b – Fast, capable, good for general questions and simple coding tasks. The best starting point.
phi3:mini – Microsoft’s Phi-3 mini; surprisingly capable for its size, especially for reasoning tasks
gemma2:2b – Google’s Gemma 2 at 2B parameters; very quick on a Pi and good at following instructions
qwen2.5:3b – Alibaba’s Qwen 2.5; particularly strong for coding assistance

Avoid models above 7B parameters unless you’re prepared to wait – they’ll run, but slowly enough to be frustrating for interactive use.

The Ollama API

Ollama exposes a local REST API at http://localhost:11434. This is where it gets genuinely interesting. Any application that can make HTTP requests can talk to your local AI. You can:

Build a simple web UI using Open WebUI (Docker container, beautiful ChatGPT-style interface)
Use it as a backend in your own scripts and tools
Connect it to VS Code via the Continue extension for local AI code assistance
Integrate it with Home Assistant for local AI automations

Open WebUI: the browser interface

For a proper chat interface rather than the terminal, Open WebUI is the best option:

docker run -d 
  --network=host 
  -v open-webui:/app/backend/data 
  --name open-webui 
  --restart unless-stopped 
  ghcr.io/open-webui/open-webui:main

Browse to http://<your-pi-ip>:8080. You’ll see a ChatGPT-style interface backed entirely by your local Ollama instance. Every chat stays local.

Real-world performance

On a Pi 5 (8GB), llama3.2:3b generates around 8–12 tokens per second – readable in real time, roughly comparable to watching someone type quickly. It’s not the instant response you get from cloud APIs, but it’s fast enough for genuine productive use.

For tasks like summarising a document, explaining code, drafting text, or answering questions, it’s more than adequate. It won’t beat a large frontier model on complex reasoning, but for a significant chunk of everyday AI tasks, a 3B model running locally on your Pi is surprisingly capable.

Wrapping up the series

That’s ten projects for your old Raspberry Pi – from ad blocking to local AI. Every single one of these is running in my own homelab. The Pi is a remarkably versatile piece of hardware, and “sitting in a drawer” is genuinely the worst thing you can do with one.

If you’ve worked through any of these projects, let me know in the comments – particularly interested to hear how the Ollama one goes on different Pi hardware.

This is the final part of our “Things to do with your old Raspberry Pi” series. See the full series in the Hardware Hacking category.

Run Local AI Models on a Raspberry Pi 5 with Ollama