How to Run Local AI with Ollama and Open WebUI on Ubuntu (GPU-Ready Guide)

on

Local large language models (LLMs) are now practical for developers, researchers, and privacy-focused teams. In this step-by-step guide, you will install and run Ollama (LLM runtime) and Open WebUI (a modern chat interface) on Ubuntu 22.04/24.04, with optional NVIDIA GPU acceleration. The setup uses Docker for easy updates, isolation, and backups.

Why this stack?

Ollama makes downloading and running models simple, offering a fast API on your machine. Open WebUI provides a sleek, extensible web app for chatting with multiple models, managing prompts, and moderating access. Together, they create a private, cost-effective alternative to cloud AI services.

Prerequisites

You need an Ubuntu 22.04/24.04 system with internet access and a user with sudo rights. If you have an NVIDIA GPU (recommended), you can enable GPU acceleration for much faster inference. CPU-only works too—skip the GPU steps if you do not have a supported GPU.

Step 1: Update the system

Update packages to ensure you have the latest dependencies and security fixes.

sudo apt update && sudo apt -y upgrade
sudo reboot

Step 2 (Optional): Enable NVIDIA GPU support

Install the latest proprietary NVIDIA driver and the container toolkit so Docker can use your GPU. Reboot when asked.

# Install recommended NVIDIA driver
sudo ubuntu-drivers install
sudo reboot

# Install NVIDIA Container Toolkit
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#' | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list > /dev/null

sudo apt update
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

# Verify GPU visibility
nvidia-smi

Step 3: Install Docker Engine and Docker Compose plugin

Install Docker from the official repository to get the latest stable version. Add your user to the docker group to run Docker without sudo.

sudo apt install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | \
sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

sudo usermod -aG docker $USER
newgrp docker

# Quick test
docker run --rm hello-world

Step 4: Create a Docker Compose file for Ollama + Open WebUI

Create a working directory and define services. The configuration below enables GPU when present; for CPU-only, remove the gpus: all line under the Ollama service.

mkdir -p ~/local-llm && cd ~/local-llm
nano docker-compose.yml

services:
  ollama:
    container_name: ollama
    image: ollama/ollama:latest
    restart: unless-stopped
    ports:
      - "11434:11434"
    volumes:
      - ollama:/root/.ollama
    # Comment the next line if you are CPU-only
    gpus: all
    environment:
      - OLLAMA_KEEP_ALIVE=24h

  open-webui:
    container_name: open-webui
    image: ghcr.io/open-webui/open-webui:latest
    depends_on:
      - ollama
    restart: unless-stopped
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
      - WEBUI_SECRET_KEY=change_this_long_random_secret
    volumes:
      - openwebui:/app/backend/data

volumes:
  ollama:
  openwebui:

Step 5: Start the stack

Bring the services up in the background, then check their status. The Open WebUI will be available at http://SERVER_IP:3000 and the Ollama API at http://SERVER_IP:11434.

docker compose up -d
docker compose ps

Step 6: Pull and run a model

Use Ollama to pull an LLM. You can pick models like llama3.2, mistral, or a coding model. The first pull downloads model weights, which can be several GB.

# Pull a general-purpose model
docker exec -it ollama ollama pull llama3.2

# Test it via CLI
docker exec -it ollama ollama run llama3.2 "Write a two-sentence summary of Ubuntu."

# Or use the API
curl http://localhost:11434/api/generate -d '{"model":"llama3.2","prompt":"Hello!"}'

Open your browser to http://SERVER_IP:3000, select the model from the dropdown, and start chatting. In Settings, you can change default models, system prompts, and appearance.

Step 7: Secure access

If exposing the WebUI beyond your LAN, add authentication. In Open WebUI, create an admin user at first login, then disable new signups in Settings. For internet exposure, place a reverse proxy (Nginx, Caddy, or Traefik) with HTTPS (Let’s Encrypt) in front of port 3000.

Step 8: Update and backup

To update, pull the latest images and recreate containers without losing data stored in volumes. To back up, save the volumes before upgrades.

# Update images
docker compose pull
docker compose up -d

# Backup volumes (example)
docker run --rm -v local-llm_ollama:/data -v "$PWD":/backup \
  busybox tar czf /backup/ollama-vol.tgz /data

docker run --rm -v local-llm_openwebui:/data -v "$PWD":/backup \
  busybox tar czf /backup/openwebui-vol.tgz /data

Troubleshooting

GPU not used: Run docker run --rm --gpus all nvidia/cuda:12.3.2-base-ubuntu22.04 nvidia-smi. If it fails, recheck the driver and NVIDIA Container Toolkit steps. Ensure the gpus: all line is present and Docker was restarted.

Permission denied with Docker: You may need to log out and back in after adding your user to the docker group, or run newgrp docker.

Port conflicts: Change the left side of port mappings in docker-compose.yml (e.g., use "8081:8080" for WebUI) and restart.

Slow or failed model pull: Verify disk space and retry. Large models require several GB of free space in the ollama volume.

Uninstall (optional)

To stop and remove everything, run:

cd ~/local-llm
docker compose down
docker volume rm local-llm_ollama local-llm_openwebui

Wrap-up

You have a modern local AI stack: Ollama for fast model serving and Open WebUI for a friendly, multi-model chat interface. With Docker, updates are quick and backups are simple. Add your favorite models, tune system prompts, and integrate the Ollama API into your apps—all without sending data to the cloud.

Comments

Post a Comment

Subscribe to: Post Comments (Atom)

Popular Posts

Install Ollama and Open WebUI on Ubuntu 24.04 with NVIDIA GPU Acceleration (Step-by-Step)

Overview Running a local large language model (LLM) is easier than ever thanks to Ollama and Open WebUI. Ollama handles model downloads and inference, while Open WebUI gives you a clean, chat-style interface in your browser. In this tutorial, you'll install both on Ubuntu 24.04 (works on 22.04 too), enable NVIDIA GPU acceleration, and deploy them with Docker Compose. The result is a fast, private AI stack you control. What You'll Need - Ubuntu 24.04 or 22.04 (fresh or existing server/desktop). - An NVIDIA GPU with recent drivers (Turing/RTX or newer recommended). - Root or sudo access. - Open ports 3000 (Open WebUI) and 11434 (Ollama) on your firewall if you access remotely. Step 1: Install NVIDIA Drivers and Verify CUDA First, update your system and install the recommended NVIDIA driver. On Ubuntu Desktop you can use Additional Drivers, but the CLI route is reliable: sudo apt update && sudo apt -y upgrade sudo ubuntu-drivers autoinstall sudo reboot After reboo...

Install a Local AI Chatbot on Ubuntu 24.04 with Ollama and Open WebUI (Step-by-Step)

Running a fast, private AI chatbot on your own computer or server is easier than ever. In this guide, you will install Ollama (a lightweight local LLM runtime) and Open WebUI (a modern web interface) on Ubuntu 24.04 . You will be able to chat with models like Llama 3 or Mistral without sending data to the cloud, and with optional GPU acceleration if you have an NVIDIA card. What you will need : an Ubuntu 22.04/24.04 machine (VM, bare metal, or WSL), at least 8 GB RAM (16 GB recommended for larger models), 15–30 GB free disk space for models, Internet access, and optional NVIDIA GPU drivers for acceleration. Why Ollama + Open WebUI? Ollama manages local large language models (LLMs) with simple commands and sensible defaults. Open WebUI gives you a clean, chat-style interface with features like prompt history, file uploads (for some models), and model switching. Together, they are a simple, reliable stack for a self-hosted AI experience. 1) Update Ubuntu and install basics First, ...

Install Ollama + Open WebUI on Ubuntu 24.04 with NVIDIA GPU Acceleration (Step-by-Step)

Overview This guide shows you how to install Ollama and Open WebUI on Ubuntu 24.04 with NVIDIA GPU acceleration. You will get a fast local AI stack that can run modern large language models (LLMs) like Llama 3 on your own hardware, secured and accessible in a web browser. We will cover GPU driver setup, Docker with the NVIDIA Container Toolkit, Ollama installation, Open WebUI deployment, basic security, and troubleshooting. Prerequisites - A machine running Ubuntu 24.04 (fresh or updated). - An NVIDIA GPU with at least 8 GB VRAM recommended for 8B models (more is better). - Internet access and sudo privileges. Step 1: Install the NVIDIA GPU Driver Install the recommended proprietary driver so CUDA becomes available to both Ollama and containers. sudo apt update && sudo apt install -y ubuntu-drivers-common sudo ubuntu-drivers autoinstall sudo reboot After reboot, verify the GPU is recognized: nvidia-smi You should see a driver version and your GPU model. If not, check...

Trending Now

Homogeneous Nucleation on Mars: A Textbook Process Finally Observed in Nature

on
By the end of this piece, you will understand what the Arsia Mons Elongated Cloud is, why it resisted explanation for several years after its first observation, and what the newly published model reveals about a physical process that has never been directly confirmed on any planet's atmosphere. The cloud that broke the model Every morning during Martian spring and summer, a thin, extremely long cloud forms near the Arsia Mons volcano and gradually elongates until it stretches to more than 1,000 miles before dissipating. The cycle then repeats for several months. The formation was first recorded by a probe in 2018, and from that point onward it became a persistent puzzle for atmospheric scientists working with Mars data. The cloud's geometry, its timing, and its seasonal recurrence all pointed to a real and reproducible physical process, yet no existing model could reproduce it. For a technical audience familiar with simulation work, the situation is analogous to a product...