LifeBytes Journal

Overview

Running a large language model locally is now practical, fast, and private. In this how-to, you will set up Ollama to serve models on your computer and connect Open WebUI for a friendly chat interface. The steps cover Windows, macOS, and Linux, including GPU acceleration for NVIDIA, Apple Silicon, and supported AMD GPUs. By the end, you will be able to pull models, chat in your browser, and tune performance for your hardware.

Requirements and quick checklist

Hardware: 8 GB RAM minimum (16 GB+ recommended), 10–20 GB free disk for models, and optionally a compatible GPU for acceleration.

GPU support: NVIDIA (CUDA 12 driver), Apple Silicon (M1/M2/M3 via Metal), AMD ROCm on supported Linux cards. If you lack a compatible GPU, CPU-only still works, just slower.

Network and security: Keep Ollama bound to localhost unless you intentionally expose it behind a reverse proxy with authentication. Do not publish it directly to the internet.

Step 1 — Install Ollama

Windows: Install via winget or the official installer.

winget install Ollama.Ollama

macOS: Use Homebrew or the DMG from the website.

brew install ollama

Linux: Use the official script (requires curl and sudo).

curl -fsSL https://ollama.com/install.sh | sh

After installation, ensure the service is running. On macOS and Windows, the background service starts automatically. On Linux, start it in a terminal or as a service:

ollama serve

Verify the API is alive by visiting http://127.0.0.1:11434 in your browser. You should see a simple status page.

Step 2 — Pull and test a model

Pull a compact, fast model first to validate everything. Llama 3.2 3B is a great starting point for laptops.

ollama pull llama3.2:3b
ollama run llama3.2:3b

Type a quick prompt and confirm you get a response. For stronger reasoning, try Mistral or an 8B Llama if your RAM/GPU can handle it:

ollama pull mistral:7b
ollama pull llama3.1:8b

Step 3 — Enable GPU acceleration (optional but recommended)

NVIDIA on Windows/Linux: Install the latest Game Ready/Studio driver with CUDA 12 support. Verify with:

nvidia-smi

Ollama will use your GPU automatically if supported. If VRAM is limited, pick a smaller or more aggressively quantized model (for example, Q4 or Q5 builds).

Apple Silicon: No extra steps. Metal acceleration is used by default on M-series chips.

AMD on Linux (ROCm): Use a ROCm-supported GPU and drivers (ROCm 6.x+). Check your distro’s ROCm documentation. Not all AMD GPUs are supported; verify before investing time.

Step 4 — Install Open WebUI

Open WebUI gives you a clean, modern chat interface for Ollama. Docker is the easiest installation path. Make sure Docker Desktop (Windows/macOS) or Docker Engine (Linux) is installed and running.

Windows/macOS (Docker Desktop):

docker run -d --name open-webui -p 3000:8080 \
  -e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
  -v open-webui:/app/backend/data \
  ghcr.io/open-webui/open-webui:latest

Linux: The host networking mode is simplest so the container reaches Ollama on localhost.

docker run -d --name open-webui --network host \
  -e OLLAMA_BASE_URL=http://127.0.0.1:11434 \
  -v open-webui:/app/backend/data \
  ghcr.io/open-webui/open-webui:latest

Open your browser to http://127.0.0.1:3000, create an account (local), and select your Ollama model from the dropdown. Start chatting immediately.

Step 5 — Performance tips and model management

Use quantized models (GGUF variants) to fit your hardware. Q4_K_M is a balanced choice for speed and quality; Q6 is higher quality; Q2/Q3 are very small and fast but lose detail. If a model fails to load, try a smaller parameter count or lower quantization level.

Keep an eye on your RAM/VRAM while the model loads. If memory spikes, reduce context length (token window) in your client settings. Many 7B models run well with 4–6 GB VRAM; 8B often prefers 8–10 GB; CPU-only runs better with 3B–7B models.

List and manage your models with:

ollama list
ollama rm <model-name>

You can tweak behavior with a Modelfile to set defaults like temperature and system prompts. Example:

# Modelfile
FROM llama3.2:3b
PARAMETER temperature 0.7
SYSTEM You are a helpful technical assistant.

ollama create my-tech-assistant -f Modelfile
ollama run my-tech-assistant

Step 6 — Security and remote access basics

By default, Ollama listens on 127.0.0.1:11434, which is safe for single-machine use. If you need remote access on your LAN, set a bind address with an environment variable:

export OLLAMA_HOST=0.0.0.0:11434   # Linux/macOS example

If you expose it, protect it. Use a reverse proxy (Nginx, Traefik, Caddy) with TLS and authentication, or a mesh VPN like Tailscale. Never expose the Ollama API directly to the public internet.

Troubleshooting

If the model is slow, confirm acceleration is active. On NVIDIA, nvidia-smi should show GPU utilization when generating. For crashes during load, your model may not fit in memory; try a smaller model or reduce the context window. If Open WebUI cannot connect, ensure OLLAMA_BASE_URL is correct for your platform and that the port is not blocked by a firewall.

What’s next

Explore specialized models for coding, summarization, or multilingual tasks. Add embeddings and retrieval in Open WebUI to chat over your PDFs or docs. With Ollama handling the runtime and Open WebUI providing the interface, you own the stack: fast, private, and flexible.

Run Local AI with Ollama and Open WebUI: GPU-Accelerated Setup on Windows and Linux with Docker

Local large language models (LLMs) have matured to the point where you can run fast, private, and cost-effective AI on your own computer or server. In this step-by-step guide, you will deploy Ollama (the LLM backend) and Open WebUI (a sleek web interface) using Docker, with optional GPU acceleration on both Windows and Linux. This stack lets you chat with models like Llama 3, Phi-4, or Mistral, completely on your hardware.

By the end, you will have a browser-based interface, persistent model storage, and a clean way to update or back up your local AI environment. The instructions are written in simple, SEO-friendly language and focus on practical steps.

What You Will Build

You will run two containers on the same Docker network: Ollama exposes an API on port 11434 and performs all model work, while Open WebUI listens on port 3000 and connects to Ollama. You will enable GPU acceleration (NVIDIA or AMD) when available to dramatically improve performance.

Prerequisites

- A 64-bit Windows 11/10 (with WSL2) or a modern Linux distribution (Ubuntu/Debian/CentOS/RHEL).
- Docker installed (Docker Desktop on Windows, Docker Engine on Linux).
- Optional GPU: NVIDIA (CUDA) or AMD (ROCm) with up-to-date drivers. CPU-only also works, but is slower.
- 16 GB RAM recommended; disk space 10–40+ GB depending on model size.

Step 1 – Install Docker

Windows: Install Docker Desktop and enable WSL 2 integration. In Settings, ensure “Use the WSL 2 based engine” is on. Update your GPU driver from NVIDIA/AMD. For NVIDIA, CUDA is not required on Windows for Docker Desktop; the latest Game Ready/Studio drivers are enough.

Linux: Install Docker from your distribution’s repository or Docker’s official repo. Add your user to the docker group and log out/in. Example (Ubuntu):

sudo apt update
sudo apt install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | \
  sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
  https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list >/dev/null
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker $USER

Step 2 – Enable GPU Acceleration (Optional but Recommended)

NVIDIA on Linux: Install the NVIDIA Container Toolkit to pass your GPU into containers.

# Add the NVIDIA container toolkit repo (Ubuntu example)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
  sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

AMD on Linux (ROCm): Install the latest AMDGPU/ROCm stack. To give containers access, pass /dev/kfd and /dev/dri and add the video group. Example device flags are shown in the Ollama run step below.

Windows: Docker Desktop exposes the GPU automatically when the host has a compatible driver. Ensure your GPU driver is up to date and “Use the WSL 2 based engine” is enabled.

Step 3 – Start Ollama (LLM Backend)

Create a Docker network and a persistent volume for models. Then start the Ollama container. Use the NVIDIA command if you have an NVIDIA GPU; use the AMD/CPU command otherwise.

# Common network and volumes
docker network create llmnet
docker volume create ollama

# NVIDIA GPU (Linux or Windows with Docker Desktop)
docker run -d --name ollama \
  --network llmnet \
  --gpus=all \
  -v ollama:/root/.ollama \
  -p 11434:11434 \
  ollama/ollama:latest

# AMD ROCm or CPU-only (Linux)
# Remove the two --device flags if you want CPU-only
docker run -d --name ollama \
  --network llmnet \
  --device=/dev/kfd --device=/dev/dri --group-add video \
  -v ollama:/root/.ollama \
  -p 11434:11434 \
  ollama/ollama:latest

Verify Ollama is live:

curl http://localhost:11434/api/tags
# or
docker logs -f ollama

Step 4 – Start Open WebUI (Front-End)

Open WebUI connects to the Ollama API and gives you a beautiful chat interface. Map port 3000 for access and point it to the Ollama container over the private network.

docker volume create open-webui

docker run -d --name open-webui \
  --network llmnet \
  -p 3000:8080 \
  -e OLLAMA_BASE_URL=http://ollama:11434 \
  -v open-webui:/app/backend/data \
  ghcr.io/open-webui/open-webui:latest

Open your browser at http://localhost:3000 and follow the first-run prompts. If you are on a server, replace localhost with the server’s IP or hostname.

Step 5 – Pull and Test a Model

Use Ollama to download a model. Smaller 7–8B models are a good starting point. You can pull directly from the container or from the WebUI Models page.

# Examples (choose one)
docker exec -it ollama ollama pull llama3.1:8b
docker exec -it ollama ollama pull phi3:mini
docker exec -it ollama ollama pull mistral:7b

After the download, open Open WebUI and start a new chat. Pick the model you pulled and send a test prompt. If you see fast tokens and low latency, your GPU is active. If generation is slow, you may be on CPU.

Step 6 – Secure, Persist, and Back Up

Enable authentication in Open WebUI and control who can sign up. You can preconfigure basic auth behavior with environment variables. Example: disable new signups and set an admin email.

# Stop and re-create Open WebUI with tighter auth (example)
docker rm -f open-webui
docker run -d --name open-webui \
  --network llmnet \
  -p 3000:8080 \
  -e OLLAMA_BASE_URL=http://ollama:11434 \
  -e ENABLE_SIGNUP=false \
  -e [email protected] \
  -v open-webui:/app/backend/data \
  ghcr.io/open-webui/open-webui:latest

To back up models and chat history, archive the Docker volumes. This keeps your setup portable.

# Backup Ollama models
docker run --rm -v ollama:/data -v "$PWD":/backup alpine \
  tar czf /backup/ollama-volume-backup.tgz -C /data .

# Backup Open WebUI data
docker run --rm -v open-webui:/data -v "$PWD":/backup alpine \
  tar czf /backup/open-webui-volume-backup.tgz -C /data .

To update, pull the latest images and recreate:

docker pull ollama/ollama:latest
docker pull ghcr.io/open-webui/open-webui:latest
docker rm -f open-webui ollama
# Re-run the "docker run" commands from Steps 3 and 4

Performance Tips

- Prefer smaller, quantized models (e.g., 7–8B) if you have limited VRAM. Many Ollama models include quantized tags that fit 8–12 GB GPUs.
- Close other GPU-heavy apps to free VRAM.
- Keep GPU drivers and Docker updated for the best kernel-accelerated performance.

Troubleshooting

Open WebUI cannot reach Ollama: Make sure both containers share the same network and the URL is correct: http://ollama:11434. Run docker logs open-webui for connection errors.

“no gpus found” or slow generation: On Linux with NVIDIA, confirm nvidia-smi works on the host and that nvidia-container-toolkit is installed. Run the container with --gpus=all. On AMD, pass --device=/dev/kfd --device=/dev/dri --group-add video. On Windows, ensure Docker Desktop is using WSL2 and that your GPU driver is current.

Port already in use: Adjust published ports, e.g., use -p 3001:8080 or -p 11435:11434, and update the URLs accordingly.

Out of memory (VRAM): Pick a smaller or more heavily quantized model. Close other GPU apps and try again.

What’s Next

With Ollama and Open WebUI running, you can add multiple models, enable embeddings and RAG, or connect tools and function calling. This setup gives you a private, fast local AI workspace that you can back up and upgrade in minutes—all without sending your data to the cloud.

Overview

This step-by-step guide shows how to run private, local large language models with Ollama and a modern chat interface using Open WebUI. We will cover installing Ollama on Ubuntu and Windows, enabling GPU acceleration, pulling popular models like Llama 3, and deploying Open WebUI with Docker so you can chat, run tools, and manage prompts from a browser. The result is a fast, secure, and offline-friendly AI stack that you control.

Prerequisites

You will need a 64-bit system, administrator privileges, and at least 16 GB of RAM for 7B–8B models. GPU acceleration is recommended for speed: keep your NVIDIA/AMD/Intel graphics drivers up to date. Ollama listens on port 11434 by default, and Open WebUI will run on port 3000. Ensure your firewall allows local access or your chosen LAN range.

Step 1 — Install Ollama

Ubuntu 22.04/24.04: Install Ollama with the official script, which adds the service and keeps it updated.

curl -fsSL https://ollama.com/install.sh | sh
sudo systemctl enable --now ollama
curl http://127.0.0.1:11434/api/version

You should see a version string from the last command. If not, check the service: sudo systemctl status ollama.

Windows 11/10: Install via the official MSI or Winget, then verify the local API.

winget install Ollama.Ollama
curl http://127.0.0.1:11434/api/version

On Windows, Ollama runs as a user service. If you use a third-party firewall, allow local traffic to port 11434.

Step 2 — Enable GPU Acceleration

GPU acceleration in Ollama is automatic when compatible drivers and runtimes are present. On Linux, install your vendor’s proprietary GPU driver. On Windows, use the latest Game Ready/Studio driver from the GPU vendor. After pulling a model and making a test prompt, watch the Ollama logs. If the run mentions the GPU and performance is high (tokens per second are significantly better than CPU), acceleration is working.

If you suspect CPU fallback, update drivers, make sure your GPU has enough VRAM for the chosen model size, and try a smaller variant (for example, 8B instead of 13B). On laptops with hybrid graphics, set the app/GPU preferences so Ollama can use the discrete GPU.

Step 3 — Pull a Model and Test Locally

Pull a model using the Ollama CLI. Popular, high-quality choices include Llama 3 and Mistral. The first run downloads and prepares weights; subsequent runs start instantly.

# Examples (pick one)
ollama pull llama3:8b
ollama pull llama3.1:8b
ollama pull mistral:7b

Now run a quick prompt:

ollama run llama3:8b
# At the prompt, type:
# What are three creative use cases for local AI at home?

If responses are slow or you see out-of-memory errors, switch to a smaller model or close GPU-intensive applications.

Step 4 — Deploy Open WebUI with Docker

Open WebUI adds a polished browser interface, prompt library, chat history, and extensions like RAG (retrieve and ground answers in your documents). We will connect it to your host’s Ollama instance. The following Docker Compose works on Linux and Windows. It uses host.docker.internal to reach the host-based Ollama API and maps persistent storage for Open WebUI data.

mkdir -p ~/openwebui && cd ~/openwebui
cat > docker-compose.yml <<'YAML'
services:
  openwebui:
    image: ghcr.io/open-webui/open-webui:latest
    container_name: open-webui
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://host.docker.internal:11434
    extra_hosts:
      - "host.docker.internal:host-gateway"
    volumes:
      - open-webui-data:/app/backend/data
    restart: unless-stopped
volumes:
  open-webui-data:
YAML

docker compose up -d

Open your browser and visit http://localhost:3000. Create your account when prompted, pick your default model (for example, llama3:8b), and send a test message. If the UI cannot connect, ensure the Ollama service is running and that your firewall allows local connections to port 11434.

Optional — Run Both Ollama and Open WebUI in Docker

If you prefer everything containerized, you can run Ollama and Open WebUI in the same Compose file. This is convenient on servers. GPU pass-through in Docker requires recent drivers and, on Linux, the NVIDIA Container Toolkit. When in doubt, keep Ollama native and only containerize Open WebUI, as shown above.

Security, Updates, and Backups

Do not expose ports 11434 or 3000 directly to the internet. If you need remote access, place Open WebUI behind a reverse proxy (Nginx, Caddy, or Traefik) with HTTPS and strong authentication, or publish it through a zero-trust tunnel. Inside Open WebUI, enable authentication and limit registration to trusted users. Keep Docker images current by pulling the latest tags and recreating containers. On Ubuntu, the Ollama installer provides updates via its repository; on Windows, check for updates in the app or Winget. Back up ~/.ollama (models and configs) and your open-webui-data volume to preserve chat history and settings.

Troubleshooting

If Open WebUI says “Cannot connect to Ollama,” verify the API at http://127.0.0.1:11434/api/version and confirm your Compose file includes extra_hosts with host-gateway on Linux. On Windows with Docker Desktop, host.docker.internal works out of the box. If GPU acceleration is missing, update drivers, reboot, and try a smaller model. When Docker containers fail to start, check logs with docker logs open-webui and make sure ports 3000 and 11434 are not in use by other applications.

What You Can Do Next

With Ollama and Open WebUI running, you can add multiple models, create custom system prompts, and enable RAG by uploading PDFs or notes so the model answers with context from your documents. You can also script batch prompts via the Ollama HTTP API, integrate with automation tools, or point a browser extension to your local endpoint to replace cloud calls. The stack is private, fast, and easy to maintain—ideal for personal knowledge work or secure team deployments.

Overview

This tutorial shows how to self-host Open WebUI with Ollama on Ubuntu using Docker Compose. You will get a clean, repeatable setup that runs on CPU or GPU, stores model data persistently, and can be upgraded with a single command. Open WebUI provides a modern interface, while Ollama runs local large language models such as Llama 3, Mistral, Phi-3, and CodeLlama.

Prerequisites

You will need a fresh Ubuntu 22.04 or 24.04 server (cloud VM or local machine), a user with sudo rights, and at least 8 GB of RAM. If you plan to use a GPU, an NVIDIA card is recommended. Open ports 3000 (Web UI) and 11434 (Ollama API) on your firewall or security group.

1) Install Docker and Compose

Install the official Docker Engine and the Compose plugin on Ubuntu. Log out and back in (or run newgrp) after adding your user to the docker group.

sudo apt update
sudo apt install -y ca-certificates curl gnupg

sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | \
  sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg

echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
  https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo $VERSION_CODENAME) stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

sudo usermod -aG docker $USER
newgrp docker
docker --version
docker compose version

2) Optional: Enable NVIDIA GPU Support

If you have an NVIDIA GPU, install the NVIDIA driver and the NVIDIA Container Toolkit. This lets Ollama use your GPU for faster inference.

sudo apt install -y ubuntu-drivers-common
sudo ubuntu-drivers autoinstall
sudo reboot

After the reboot, install the container toolkit and configure Docker:

distribution=$(. /etc/os-release; echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
  sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit.gpg

curl -fsSL https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt update
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Verify GPU access with Docker:

docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi

3) Create the Docker Compose Stack

Create a project directory and a Compose file that defines two services: Ollama (LLM runtime) and Open WebUI (frontend). The volumes preserve your models and settings across restarts.

mkdir -p ~/openwebui-ollama
cd ~/openwebui-ollama
nano docker-compose.yml

Paste the following content and save:

version: "3.9"

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    ports:
      - "11434:11434"
    volumes:
      - ollama:/root/.ollama
    environment:
      - OLLAMA_KEEP_ALIVE=24h
    # Uncomment the next line if you have GPU configured:
    # gpus: all

  open-webui:
    image: ghcr.io/open-webui/open-webui:latest
    container_name: open-webui
    depends_on:
      - ollama
    restart: unless-stopped
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
    volumes:
      - open-webui:/app/backend/data

volumes:
  ollama:
  open-webui:

4) Start the Services and Download a Model

Bring the stack online. The first start will download container images.

docker compose up -d
docker compose ps

Pull a small model to test. You can add more models later.

docker exec -it ollama ollama pull llama3.2:3b
# Other options: mistral:7b, phi3:mini, qwen2.5:7b

Open your browser to http://SERVER-IP:3000. Create your account in Open WebUI. In the model selector, choose the model you pulled and send a prompt to verify everything works.

5) Persist Data and Backups

Docker volumes keep your models and UI data under /var/lib/docker/volumes. To back them up, stop the stack and archive the data directories. This ensures quick recovery after an OS reinstall or server migration.

docker compose down
sudo tar -czf ollama_data.tgz -C /var/lib/docker/volumes \
  $(docker volume ls -q | grep "_ollama$")/_data

sudo tar -czf openwebui_data.tgz -C /var/lib/docker/volumes \
  $(docker volume ls -q | grep "_open-webui$")/_data

docker compose up -d

6) Secure and Publish (Optional)

If you expose the service on the internet, put it behind a reverse proxy with HTTPS (Caddy, Nginx, or Traefik) and set strong authentication in Open WebUI. Use a DNS name, issue a TLS certificate (Let’s Encrypt), and restrict access with IP allowlists or an identity provider. For small teams, consider running it only on a private network or VPN.

7) Update and Maintenance

Update to the newest images regularly. This pulls security updates, new UI features, and performance improvements.

cd ~/openwebui-ollama
docker compose pull
docker compose up -d

To update models to the latest quantizations or fixes, re-pull them in Ollama. You can remove old ones you no longer need.

docker exec -it ollama ollama pull mistral:7b
docker exec -it ollama ollama list
docker exec -it ollama ollama rm modelname:tag

Troubleshooting

Port already in use: Change the host port mappings in docker-compose.yml (for example, 3001:8080 or 11435:11434) and restart.

GPU not detected: Verify nvidia-smi works on the host and in a test container. Ensure the gpus: all line is uncommented and Docker was restarted after installing the NVIDIA Toolkit.

Slow or failed model pulls: Models can be large. Check disk space (df -h), network speed, and try a smaller model first. You can also mirror models by pre-downloading on another machine and copying the volume data.

Permission errors: Ensure your user is in the docker group (id) and you have logged out/in.

What You Achieved

You now have a production-friendly, self-hosted AI chat stack powered by Open WebUI and Ollama. With Docker Compose, you can start, stop, back up, and upgrade the entire setup with a couple of commands. Add or swap models as your use cases evolve—coding assistants, knowledge chat, or creative writing—while keeping your data local and under your control.

Local large language models are now practical on a single PC. In this tutorial, you will deploy Ollama (model runtime) and Open WebUI (a friendly chat interface) using Docker on Windows or Linux. We will enable NVIDIA GPU acceleration, persist models on disk, and cover secure access and troubleshooting. By the end, you will be chatting with a local LLM like llama3.1 in your browser, no cloud required.

What You Will Need

- A 64-bit PC with at least 16 GB RAM. For GPU acceleration, an NVIDIA GPU with 8 GB+ VRAM is recommended.
- Docker Engine or Docker Desktop (Compose v2 included).
- Free disk space (15–30 GB per model is common).
- Optional but recommended: NVIDIA GPU drivers and CUDA runtime for Docker.

Step 1: Prepare Your System (GPU Optional)

Linux (Ubuntu/Debian)
1) Install Docker Engine and the Compose plugin from the official Docker repo.
2) Install NVIDIA GPU drivers from your distro or NVIDIA site.
3) Install the NVIDIA Container Toolkit:
sudo apt-get install -y nvidia-container-toolkit
Then configure and restart Docker:
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
Verify GPU visibility in containers:
docker run --rm --gpus all nvidia/cuda:12.3.2-base-ubuntu22.04 nvidia-smi

Windows 10/11
1) Install the latest NVIDIA GPU driver (Studio or Game Ready).
2) Install Docker Desktop and enable WSL 2 backend during setup.
3) In Docker Desktop > Settings > Resources > WSL integration, enable your default distro.
4) Ensure GPU is exposed to containers. If you run docker run --rm --gpus all nvidia/cuda:12.3.2-base-ubuntu22.04 nvidia-smi and see your GPU, you are ready.

Step 2: Create a Docker Compose File

We will run two containers: ollama (the LLM runtime API) and open-webui (the web front-end). The services will share a network and persistent volumes. Create a folder like ollama-openwebui and a file compose.yaml with the following content:

services:
  ollama:
   image: ollama/ollama:latest
   container_name: ollama
   restart: unless-stopped
   ports:
    - "11434:11434"
   volumes:
    - ollama_data:/root/.ollama
   environment:
    - OLLAMA_KEEP_ALIVE=24h
   deploy:
    resources:
     reservations:
      devices:
       - capabilities: ["gpu"]

  openwebui:
   image: ghcr.io/open-webui/open-webui:latest
   container_name: open-webui
   restart: unless-stopped
   depends_on:
    - ollama
   ports:
    - "3000:8080"
   environment:
    - OLLAMA_BASE_URL=http://ollama:11434
    - WEBUI_AUTH=True
    - DEFAULT_MODELS=llama3.1:8b
   volumes:
    - openwebui_data:/app/backend/data

volumes:
  ollama_data:
  openwebui_data:

Notes:
- The deploy.resources.reservations.devices section hints Compose to request GPU. On Linux, also start with --gpus all if you run containers manually.
- Ports: Ollama API is 11434, Open WebUI is exposed on 3000 (mapped to container 8080).

Step 3: Start the Stack

In the folder containing compose.yaml, run:
docker compose up -d
Wait for both containers to start. You can watch logs with:
docker compose logs -f

Step 4: Pull a Model and Run Your First Chat

Open a terminal and pull a model into Ollama. For a good balance of quality and speed, try Meta’s 8B model:
docker exec -it ollama ollama pull llama3.1:8b
You can test from the CLI:
docker exec -it ollama ollama run llama3.1:8b "Write a haiku about local AI."
If the response appears, the model is working.

Now open your browser and visit http://localhost:3000. Create an admin account (since we set WEBUI_AUTH=True). In Settings > Models, you should see llama3.1:8b. Create a new chat and start prompting.

GPU Acceleration Checks

- If you have an NVIDIA GPU, Ollama should automatically use it. Confirm via logs: docker logs ollama (look for CUDA initialization).
- If you do not have a GPU, Ollama will use CPU. Expect slower generation but it will work.

Useful Options and Performance Tips

- Try smaller variants for low VRAM: llama3.2:3b or phi3:mini.
- You can pin models to GPU RAM by enabling sufficient numa/gpu memory; if VRAM is low, Ollama will offload layers to system RAM.
- To pre-download a model at startup, set DEFAULT_MODELS in the Open WebUI service as shown.
- For multilingual or coding tasks, add models like qwen2.5:7b or codestral.

Security and Remote Access

- Keep WEBUI_AUTH=True to require sign-in. You can also set OPENWEBUI_ADMIN_EMAIL and OPENWEBUI_ADMIN_PASSWORD as environment variables for unattended setups.
- If exposing Open WebUI to the internet, place it behind a reverse proxy (Nginx, Caddy, or Traefik) with HTTPS and strong passwords.
- The Ollama API on port 11434 should remain private unless you need remote access; firewall it if required.

Troubleshooting

- GPU not detected: On Linux, reinstall nvidia-container-toolkit and verify nvidia-smi works both on the host and in a container. On Windows, ensure WSL 2 is enabled and Docker Desktop is up to date.
- “No space left on device”: Increase disk space or prune unused model blobs: docker exec -it ollama ollama rm <model>. You can also clear unused images with docker system prune (caution).
- Slow or out-of-memory: Use a smaller model, reduce context length in Open WebUI, close other GPU-intensive apps, or increase swap on Linux.
- Port in use: Change the published ports in compose.yaml (e.g., "3001:8080") and redeploy.

Updating and Maintenance

To update to the latest versions, run:
docker compose pull
docker compose up -d
Your models are safe in the ollama_data volume, and your chat history lives in openwebui_data. Always back up these volumes before major upgrades.

What’s Next

You now have a privacy-friendly, GPU-accelerated local AI stack. Explore function calling, RAG connectors in Open WebUI, or run multiple models side by side. With Docker and Ollama, swapping models and keeping performance high is only a pull away.

Overview

This step-by-step guide shows you how to deploy Ollama and Open WebUI on Ubuntu 24.04 with NVIDIA GPU acceleration using Docker. With this setup, you can run modern large language models (LLMs) locally, manage them from a clean web interface, and take full advantage of your GPU for high performance. The process covers NVIDIA drivers, Docker, the NVIDIA Container Toolkit, and secure, persistent containers that survive reboots.

What You Will Need

You need a 64-bit Ubuntu 24.04 host with an NVIDIA GPU (Turing or newer recommended), Internet access, a user with sudo rights, and at least 20 GB of free disk space for models. If you are working on a remote server, make sure port 3000 (for Open WebUI) and 11434 (for Ollama) are reachable or routed through a reverse proxy.

1) Install NVIDIA Drivers

First, install the official NVIDIA driver so CUDA can talk to your GPU. Run: sudo ubuntu-drivers autoinstall. When it finishes, reboot with sudo reboot. After the reboot, verify the GPU is visible: nvidia-smi. You should see your GPU name and driver version. If you do not, confirm Secure Boot is disabled or enroll the driver MOK accordingly, then repeat the check.

2) Install Docker Engine on Ubuntu 24.04

Set up Docker from the official repository for best stability and features. Run: sudo apt update && sudo apt install -y ca-certificates curl gnupg. Add Docker’s key and repo: sudo install -m 0755 -d /etc/apt/keyrings, curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg, echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu noble stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null. Then install: sudo apt update && sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin. To run Docker without sudo: sudo usermod -aG docker $USER then newgrp docker.

3) Enable GPU Access in Containers (NVIDIA Container Toolkit)

Install the NVIDIA Container Toolkit so Docker can pass your GPU into containers. Add the key and repo: curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg, curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list. Install and configure: sudo apt update && sudo apt install -y nvidia-container-toolkit, sudo nvidia-ctk runtime configure --runtime=docker, sudo systemctl restart docker. Test GPU passthrough: docker run --rm --gpus all nvidia/cuda:12.6.2-base-ubuntu22.04 nvidia-smi. You should see your GPU listed inside the container.

4) Create a Dedicated Network for AI Services

Create a user-defined Docker network so containers can discover each other cleanly: docker network create ai. This network isolates traffic and lets Open WebUI talk to the Ollama container by name.

5) Run the Ollama Container with GPU Support

Start Ollama and persist its model data in a Docker volume. Run: docker run -d --name ollama --gpus all --restart unless-stopped -p 11434:11434 -v ollama:/root/.ollama --network ai ollama/ollama:latest. The container exposes the Ollama API on port 11434. Check logs with docker logs -f ollama to ensure the server starts without errors.

6) Pull a Model (Llama 3.1 example)

Use Ollama’s CLI inside the container to download a model. For a great balance of speed and quality on consumer GPUs, try an 8B model: docker exec -it ollama ollama pull llama3.1:8b. If you have a smaller GPU (e.g., 6–8 GB VRAM), try a quantized variant like llama3.1:8b-instruct-q4_K_M. You can list models with docker exec -it ollama ollama list.

7) Deploy Open WebUI and Connect to Ollama

Open WebUI provides a friendly interface to chat with models, manage prompts, and configure settings. Start it with: docker run -d --name open-webui --restart unless-stopped -p 3000:8080 -e OLLAMA_API_BASE_URL=http://ollama:11434 -v openwebui:/app/backend/data --network ai ghcr.io/open-webui/open-webui:latest. Open http://<your_server_ip>:3000 in a browser, create your first user (the first account becomes admin), and pick the model you pulled in the previous step. You can now chat with the LLM directly from your browser.

8) Optional: Secure Access with HTTPS

For Internet-facing servers, place a reverse proxy with TLS in front of Open WebUI. A simple approach is Caddy or Nginx Proxy Manager. Point your domain’s DNS to the server, terminate HTTPS on the proxy, and forward to localhost:3000. If you already use Traefik or Nginx, add routes with Let’s Encrypt certificates and restrict access using basic auth or OAuth.

Maintenance and Updates

To update Ollama or Open WebUI, pull new images and recreate containers. Run: docker pull ollama/ollama:latest and docker pull ghcr.io/open-webui/open-webui:latest, then docker stop ollama open-webui and docker rm ollama open-webui. Start them again using the same docker run commands; your data persists in the volumes ollama and openwebui. To back up models and settings, archive the volumes: sudo tar -czf ollama-vol.tgz -C /var/lib/docker/volumes/ollama/_data . and sudo tar -czf openwebui-vol.tgz -C /var/lib/docker/volumes/openwebui/_data ..

Troubleshooting

If the GPU is not detected inside containers, confirm the host driver works with nvidia-smi. Then verify the runtime is configured: docker info | grep -i nvidia. If missing, re-run sudo nvidia-ctk runtime configure --runtime=docker and sudo systemctl restart docker. For permission errors when running Docker, add your user to the docker group as shown above. If downloads are slow or models fail due to VRAM limits, choose smaller or quantized models (e.g., q4_K_M or q5_K_M).

What You Get

After following these steps, you have a modern, GPU-accelerated local AI stack. Ollama handles efficient model runtimes, and Open WebUI gives you a clean chat interface, prompt management, and multi-model control. Because everything runs in Docker with persistent volumes, updates and backups are easy, and you can scale this setup on a workstation or a headless server with minimal changes.

Overview

This step-by-step guide shows you how to run a local AI assistant on Windows 11 using Ollama and Open WebUI. You will install Ollama, download a model, and connect a user-friendly web interface via Docker. The tutorial is beginner-friendly yet covers advanced options like GPU acceleration, authentication, and storage tuning. By the end, you will have a private, fast, and offline-capable AI setup on your own PC.

Prerequisites

Before you start, make sure you have: Windows 11 (22H2 or newer), administrator rights, and at least 8 GB RAM. For GPU acceleration, install the latest graphics driver. Ollama uses CUDA for NVIDIA GPUs and DirectML for AMD/Intel; GPU use is automatic if supported. You do not need WSL for this guide. An optional Docker Desktop installation is required for Open WebUI.

Step 1 — Install Ollama for Windows

1) Download the official installer from https://ollama.com/download and complete the setup.
2) Open PowerShell and verify the installation: ollama --version.
3) Start the Ollama service if it is not already running: ollama serve (you can keep it in the background by closing the window after confirming it is running as a service).

Step 2 — Pull and test a model

1) In PowerShell, download a model. For a good balance of speed and quality, try: ollama pull llama3.
2) Run it interactively: ollama run llama3, then ask a question like: What can you do?.
3) Exit the session with /bye when finished. Models are stored locally in %LOCALAPPDATA%\Ollama\models by default.

Step 3 — Install Docker Desktop (for Open WebUI)

Open WebUI gives you a clean web interface for prompts, chat history, and multi-model workflows. Install Docker Desktop from https://www.docker.com/products/docker-desktop/ and start it. Ensure the Docker engine is running (the whale icon should be active in the system tray).

Step 4 — Launch Open WebUI linked to Ollama

Run the following Docker command in PowerShell to start Open WebUI and connect it to your local Ollama instance exposed at http://localhost:11434:
docker run -d --name open-webui -p 3000:8080 -e OLLAMA_BASE_URL=http://host.docker.internal:11434 -v open-webui:/app/backend/data ghcr.io/open-webui/open-webui:latest
Once the container is healthy, open http://localhost:3000 in your browser. Choose a model (for example, llama3) and start chatting.

Optional — Enable authentication for Open WebUI

To protect your UI with a login, recreate the container with auth variables:
docker rm -f open-webui
docker run -d --name open-webui -p 3000:8080 -e OLLAMA_BASE_URL=http://host.docker.internal:11434 -e WEBUI_AUTH=true -e DEFAULT_USERNAME=admin -e DEFAULT_PASSWORD=changeMeNow -v open-webui:/app/backend/data ghcr.io/open-webui/open-webui:latest
Visit http://localhost:3000 and sign in with your credentials.

Optional — GPU acceleration tips

Ollama automatically uses your GPU when supported drivers are present. To nudge usage, you can set the number of GPUs: setx OLLAMA_NUM_GPU 1 then restart the Ollama service or your PC. If you have an NVIDIA GPU, ensure the latest Game Ready or Studio driver is installed. For AMD/Intel, keep your driver and Windows up to date to benefit from DirectML improvements. During the first run, the model may compile kernels; subsequent runs are faster.

Optional — Move the models folder to another drive

If you want models on a larger drive, set this environment variable and restart the service: setx OLLAMA_MODELS "D:\Ollama\Models". Move the existing folder from %LOCALAPPDATA%\Ollama\models to the new location to avoid re-downloading large files.

Troubleshooting

Open WebUI cannot connect to Ollama: Make sure Ollama is running: curl http://localhost:11434/api/tags should return a JSON list of models. If it works on the host but not in Docker, confirm the container uses host.docker.internal and port 11434 as shown in the command. Also check Windows Firewall for any blocked inbound rules on Docker or Ollama.

Models are slow or fail to load: Try a smaller model first: ollama pull phi3:mini and run ollama run phi3:mini. Close heavy apps, ensure you have enough RAM/VRAM, and avoid aggressive antivirus scanning of the models folder.

Docker errors on startup: Open Docker Desktop and verify that the engine is running. If ports are already in use, change the mapping (for example, -p 3001:8080) and refresh the browser at the new address.

Usage tips

Inside Open WebUI, create multiple chats per model for different tasks, enable markdown rendering, and configure system prompts for role-specific behavior. In PowerShell, you can also run one-off prompts without the UI: ollama run llama3 "Write a haiku about morning coffee." For reproducibility, export your Open WebUI data with the named volume and back it up regularly.

What you achieved

You now have a private, local AI assistant on Windows 11 powered by Ollama and Open WebUI. You can switch models, run fully offline, and take advantage of your GPU for faster responses. This setup is ideal for coding help, note-taking, drafting, and research without sending your data to external servers.

Overview

This step-by-step guide shows how to run open-source large language models (LLMs) locally on Ubuntu 24.04 using Ollama for model serving and Open WebUI for a friendly chat interface. You will install Ollama, enable optional GPU acceleration (NVIDIA or CPU fallback), and deploy Open WebUI with Docker. The result is a private, fast, and controllable AI setup suitable for home labs and small teams.

Prerequisites

You need an Ubuntu 24.04 LTS host with internet access, a user with sudo rights, and at least 8 GB of RAM. A modern NVIDIA GPU is optional but recommended for faster inference. Make sure the system is up to date: sudo apt update && sudo apt -y upgrade

Step 1 — Install Ollama

Ollama is a lightweight server that downloads and runs models locally. Install it with the official script:
curl -fsSL https://ollama.com/install.sh | sh

Enable and start the service so it runs at boot:
sudo systemctl enable ollama
sudo systemctl start ollama
sudo systemctl status ollama

Verify the API is listening on port 11434:
curl http://127.0.0.1:11434/api/tags

Step 2 — Optional: Enable GPU Acceleration (NVIDIA)

If you have an NVIDIA GPU, install the recommended driver. Ubuntu makes this easy:
sudo ubuntu-drivers autoinstall
sudo reboot

After reboot, confirm the driver is active:
nvidia-smi

Ollama detects GPUs automatically when drivers are present. No extra flags are required. If you need to force CPU or GPU behavior, you can set:
export OLLAMA_NO_GPU=1 (CPU only) or export OLLAMA_NO_GPU=0 (GPU allowed). For a persistent setting, add the variable to your shell profile and restart Ollama:
sudo systemctl restart ollama

AMD GPUs can work with ROCm on supported cards and drivers. If you are using AMD, install the ROCm runtime from AMD’s repository for Ubuntu 24.04, confirm with rocminfo, and ensure your user is in the video and render groups. If ROCm is not available for your hardware, Ollama will fall back to CPU.

Step 3 — Pull a Model and Test Locally

Pull a well-supported model. Llama 3 is a popular choice:
ollama pull llama3

Run a quick test:
ollama run llama3 "Write one sentence about Ubuntu 24.04."

Tip: For smaller footprints, choose tiny models like llama3:8b or phi3. VRAM needs vary; an 8B model typically benefits from 8–12 GB of GPU VRAM, while CPU-only runs need more system RAM and patience.

Step 4 — Install Docker and Open WebUI

Open WebUI gives you a clean browser interface for Ollama. Install Docker from Ubuntu repos for a quick start:
sudo apt install -y docker.io docker-compose-plugin

Allow your user to manage Docker without sudo, then re-login:
sudo usermod -aG docker $USER

Create a persistent volume for Open WebUI data and start the container. It will connect to Ollama on the host:
docker volume create openwebui
docker run -d --name open-webui -p 3000:8080 --restart unless-stopped -e OLLAMA_BASE_URL=http://127.0.0.1:11434 -v openwebui:/app/backend/data ghcr.io/open-webui/open-webui:latest

Open your browser to http://SERVER_IP:3000 and complete the initial admin setup. Add a model in Settings if it does not appear automatically, for example llama3.

Step 5 — Optional TLS with Caddy (Automatic HTTPS)

If you have a domain pointing to your server (A record), Caddy can auto-provision HTTPS certificates. Install it and configure a simple reverse proxy:
sudo apt install -y caddy

Edit /etc/caddy/Caddyfile (replace ai.example.com with your domain):
ai.example.com {
  reverse_proxy 127.0.0.1:3000
}

Reload Caddy:
sudo systemctl reload caddy. Visit https://ai.example.com. Ensure ports 80 and 443 are open on your firewall and router.

Step 6 — Backups and Updates

Ollama models are stored under ~/.ollama for non-root users or /usr/share/ollama when installed system-wide. Back up this directory to avoid re-downloading models. Example:
tar czf ollama-backup.tgz ~/.ollama

Open WebUI data is in the Docker volume openwebui. Back it up with:
docker run --rm -v openwebui:/data -v $(pwd):/backup alpine sh -c "cd /data && tar czf /backup/openwebui-backup.tgz ."

To update Ollama:
curl -fsSL https://ollama.com/install.sh | sh && sudo systemctl restart ollama. To update Open WebUI:
docker pull ghcr.io/open-webui/open-webui:latest && docker stop open-webui && docker rm open-webui && docker run ... (re-run the previous docker run command).

Troubleshooting

If port 11434 or 3000 is in use, change the port in the docker run command or stop the conflicting service. For slow responses, try a smaller model or ensure your GPU driver is working. If Open WebUI cannot reach Ollama, verify curl http://127.0.0.1:11434/api/tags succeeds on the host and confirm the OLLAMA_BASE_URL is correct.

Wrap-up

You now have a private AI stack on Ubuntu 24.04 with Ollama handling model inference and Open WebUI offering a clean chat interface. With optional GPU acceleration, HTTPS, and simple backups, this setup is fast, secure, and maintainable—perfect for learning, prototyping, or running an internal assistant.

Overview

Want a private, fast, and customizable AI chatbot without sending your data to the cloud? In this guide you will deploy Ollama (which runs large language models locally) together with Open WebUI (a modern chat interface) using Docker. The setup works on Linux, Windows, and macOS, and can use your NVIDIA GPU for acceleration. You will get a production‑style layout with data volumes, secure defaults, update steps, and troubleshooting tips.

What You Will Need

- A machine with at least 8 GB RAM (16 GB+ recommended for larger models). CPU‑only works; GPU is optional.

- Docker Engine (Linux) or Docker Desktop (Windows/macOS). Ensure Docker Compose is available (Docker Desktop includes it).

- Optional GPU acceleration: NVIDIA GPU, recent NVIDIA drivers, and NVIDIA Container Toolkit on Linux; on Windows, Docker Desktop with WSL2 backend and CUDA‑capable drivers.

Step 1 — Create the Docker Compose file

Create a working folder (for example, ai-stack) and add a file named docker-compose.yml with the following baseline (CPU‑only, safe defaults that bind to localhost):

version: "3.9"
services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    volumes:
      - ollama:/root/.ollama
    ports:
      - "127.0.0.1:11434:11434"
  open-webui:
    image: ghcr.io/open-webui/open-webui:latest
    container_name: open-webui
    restart: unless-stopped
    environment:
      - OLLAMA_API_BASE=http://ollama:11434
    depends_on:
      - ollama
    volumes:
      - open-webui:/app/backend/data
    ports:
      - "127.0.0.1:3000:8080"
volumes:
  ollama:
  open-webui:

Binding to 127.0.0.1 keeps services private on the host. You can later expose them behind a reverse proxy with HTTPS if you need remote access.

Step 2 — Start the stack

From the folder with your compose file, run:

docker compose pull
docker compose up -d

Wait a few seconds for containers to initialize. You can watch logs with docker compose logs -f.

Step 3 — Download your first model

Ollama manages models on demand. Pull a small model to test quickly (Llama 3.2 3B is a good start):

docker exec -it ollama ollama pull llama3.2:3b

You can list models later with docker exec -it ollama ollama list. For better quality, try llama3.1:8b or a reasoning model when your hardware allows it.

Step 4 — Open the chat UI

Visit http://localhost:3000. In the Open WebUI interface, choose the model you pulled (e.g., llama3.2:3b) and start chatting. Responses run entirely on your machine through Ollama at http://localhost:11434.

Optional: Enable NVIDIA GPU acceleration

GPU support can dramatically speed up responses. Ensure your system is ready first:

- Linux: Install the proprietary NVIDIA driver and the NVIDIA Container Toolkit (nvidia-container-toolkit). Verify nvidia-smi works on the host.

- Windows: Install NVIDIA drivers with CUDA, enable WSL2 and GPU support in Docker Desktop, and ensure WSL2 integration is turned on for your Linux distro.

Then, choose one of the following methods for the ollama service:

A) Compose with GPU (supported in recent Docker Compose versions):

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    volumes:
      - ollama:/root/.ollama
    ports:
      - "127.0.0.1:11434:11434"
    gpus: all

B) Run Ollama with a direct docker run command (replaces the compose service):

docker stop ollama && docker rm ollama
docker run -d --name ollama --gpus all \
  -p 127.0.0.1:11434:11434 \
  -v ollama:/root/.ollama \
  ollama/ollama:latest

After enabling GPU, repull or reload models so they compile kernels for the GPU on first run. Use docker logs ollama -f to confirm CUDA is used.

Security Hardening (Recommended)

- Keep services bound to localhost as shown. For remote access, place a reverse proxy (Caddy, Nginx, Traefik) in front with HTTPS and authentication.

- In Open WebUI, create an admin account first and limit signups from Settings. You can also run it behind SSO or a VPN.

- Do not expose port 11434 publicly; Ollama has no built‑in auth. If you must, secure the path via a proxy and firewall rules.

Updating and Backups

To update to the latest images:

docker compose pull
docker compose up -d

Your models (Ollama) and chat data (Open WebUI) live in Docker volumes named ollama and open-webui. Back them up with:

docker run --rm -v ollama:/data -v $(pwd):/backup busybox tar czf /backup/ollama-vol.tgz -C / data
docker run --rm -v open-webui:/data -v $(pwd):/backup busybox tar czf /backup/open-webui-vol.tgz -C / data

Troubleshooting

- Port already in use: Change the left side of the port mapping (for example, 127.0.0.1:3001:8080) or stop the conflicting service.

- Slow or out‑of‑memory on big models: Choose a smaller model (3B–8B). On GPU, ensure sufficient VRAM; quantized variants (e.g., Q4_K_M) reduce memory needs.

- GPU not detected: Confirm nvidia-smi works on the host, restart Docker, and verify you used gpus: all or --gpus all. On Windows, ensure WSL2 integration is enabled in Docker Desktop.

- Open WebUI cannot reach Ollama: Check OLLAMA_API_BASE is set to http://ollama:11434 in Compose and that both services share the same default network (they do by default).

Remove Everything (Optional)

To stop and remove containers but keep volumes: docker compose down.

To also delete all data volumes (irrevocable): docker compose down -v.

What You Get

You now have a private AI chatbot that runs fully on your machine, with a clean Docker layout, optional GPU acceleration, and safe defaults. Expand by adding more models (e.g., CodeLlama for coding, Phi‑3 for low‑resource devices), enabling RAG with document uploads in Open WebUI, or placing the stack behind a reverse proxy for secure remote access. This approach keeps your data local, reduces latency, and gives you full control over updates and performance.

Overview

This guide shows how to deploy Ollama and Open WebUI on Ubuntu so you can run large language models (LLMs) locally with NVIDIA GPU acceleration. You will install Docker and the NVIDIA Container Toolkit, run the Ollama API, connect Open WebUI as a front end, and pull a model like Llama 3. This setup is fast, private, and easy to maintain.

Prerequisites

Before you start, make sure you have: Ubuntu 22.04 or later, an NVIDIA GPU with a recent driver (525+), sudo access, internet connectivity, and open ports 11434 (Ollama) and 3000 (Open WebUI). If you have an existing Docker installation, ensure it is up to date.

1) Install NVIDIA driver and verify GPU

Install a stable NVIDIA driver from Ubuntu’s repository, reboot, and verify the GPU is visible:

sudo apt update
sudo apt install -y nvidia-driver-535
sudo reboot
# After reboot:
nvidia-smi

If nvidia-smi prints your GPU details, the driver is working. If not, check Secure Boot, which can block kernel modules; disable it or sign the modules accordingly.

2) Install Docker and enable GPU in containers

Install Docker using the official convenience script, add your user to the docker group, then install the NVIDIA Container Toolkit so containers can access the GPU.

curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER
newgrp docker

# NVIDIA Container Toolkit
sudo apt update
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

# Verify GPU works inside Docker:
docker run --rm --gpus all nvidia/cuda:12.3.2-base-ubuntu22.04 nvidia-smi

If the last command shows GPU output, you are ready to run GPU-enabled containers.

3) Deploy Ollama (LLM runtime)

Ollama serves models locally via an HTTP API. Create a volume for persistent model storage and run the container with GPU support:

docker volume create ollama
docker run -d --name ollama --gpus all \
  -p 11434:11434 \
  -e OLLAMA_HOST=0.0.0.0:11434 \
  -v ollama:/root/.ollama \
  ollama/ollama:latest

Pull a model to test. Quantized models use less VRAM; llama3.1:8b is a good starting point on 8–12 GB GPUs.

docker exec -it ollama ollama pull llama3.1:8b
# Quick test (CLI in the container):
docker exec -it ollama ollama run llama3.1:8b

If the model loads and you can send a prompt, Ollama is ready.

4) Deploy Open WebUI (front end)

Open WebUI provides a user-friendly chat interface and features like prompt sets and file uploads. Create an isolated network, connect Ollama, and run Open WebUI:

docker network create ai
docker network connect ai ollama

docker volume create openwebui
docker run -d --name open-webui --network ai \
  -p 3000:8080 \
  -e OLLAMA_BASE_URL=http://ollama:11434 \
  -v openwebui:/app/backend/data \
  ghcr.io/open-webui/open-webui:latest

Open a browser and go to http://<server-ip>:3000. The first user to sign up becomes the administrator. After creating the admin account, open Settings and disable public signups if you want to restrict access.

5) Use your local AI

In Open WebUI, pick the model you pulled (e.g., llama3.1:8b) and start chatting. You can pull more models from the “Models” area or via:

docker exec -it ollama ollama pull mistral:7b
docker exec -it ollama ollama pull neural-chat:7b

Tip: If a model fails to load due to VRAM limits, choose a smaller or more aggressively quantized variant (e.g., Q4 or 4-bit builds).

6) Update and maintenance

To update to the latest images while keeping your data, pull and recreate the containers with the same volumes:

# Update Ollama
docker pull ollama/ollama:latest
docker stop ollama && docker rm ollama
docker run -d --name ollama --gpus all \
  -p 11434:11434 -e OLLAMA_HOST=0.0.0.0:11434 \
  -v ollama:/root/.ollama --network ai \
  ollama/ollama:latest

# Update Open WebUI
docker pull ghcr.io/open-webui/open-webui:latest
docker stop open-webui && docker rm open-webui
docker run -d --name open-webui --network ai \
  -p 3000:8080 \
  -e OLLAMA_BASE_URL=http://ollama:11434 \
  -v openwebui:/app/backend/data \
  ghcr.io/open-webui/open-webui:latest

Models and settings persist in the Docker volumes. Back up these volumes regularly with your usual server backup process.

7) Troubleshooting

If Open WebUI cannot talk to Ollama, confirm both containers share the same network and that OLLAMA_BASE_URL points to http://ollama:11434. Use docker logs open-webui to check errors.

If the GPU is not used, verify nvidia-smi inside a container works and the Docker daemon has the NVIDIA runtime configured. Also confirm you started Ollama with --gpus all. For small VRAM, prefer smaller models (e.g., 7–8B) and quantized builds.

If you see slow generation, check CPU/GPU utilization with top and nvidia-smi. Running models from SSD storage and avoiding swap helps latency. Restart long-running containers after driver updates.

What you get

With Ollama and Open WebUI on Ubuntu, you have a private, GPU-accelerated local AI stack. You can chat, summarize, and prototype apps against the Ollama API at http://<server-ip>:11434, while Open WebUI provides a polished interface for everyday use.

Running large language models locally is now practical and secure for many teams. In this guide, you will deploy a production-ready stack on Ubuntu using Ollama (for model runtime) and Open WebUI (for a clean, chat-style interface). The tutorial covers both CPU-only and NVIDIA GPU acceleration with the NVIDIA Container Toolkit, plus tips for updates, security, and backups.

Prerequisites

You need an Ubuntu 22.04 or 24.04 machine, at least 16 GB RAM for smooth performance, and optional NVIDIA GPU (Turing or newer recommended). You also need root or sudo access and a public DNS name if you plan to expose the UI securely.

Step 1: Update Ubuntu

Start by updating your system packages to ensure compatibility with recent Docker and NVIDIA components.

sudo apt update && sudo apt -y upgrade
sudo reboot

Step 2: Install Docker Engine and Compose Plugin

Install Docker from the official repository and enable the Compose plugin. This method ensures you receive timely security fixes and new features.

sudo apt -y install ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
  https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo $VERSION_CODENAME) stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt update
sudo apt -y install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker $USER
newgrp docker

Step 3 (Optional but Recommended): NVIDIA GPU Acceleration

If you have an NVIDIA GPU, install the NVIDIA Container Toolkit to enable GPU pass-through for containers. Make sure you already have the proprietary NVIDIA driver installed (check with nvidia-smi).

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
  | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt update
sudo apt -y install nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Verify that Docker can see your GPU:

docker run --rm --gpus all nvidia/cuda:12.3.2-base-ubuntu22.04 nvidia-smi

Step 4: Create a Docker Compose File

We will run two services: ollama (the LLM runtime and model manager) and open-webui (a modern web UI that connects to Ollama). Save the file as docker-compose.yml in an empty directory.

version: "3.8"

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    ports:
      - "11434:11434"
    volumes:
      - ollama:/root/.ollama
    # Uncomment the next line if you have an NVIDIA GPU:
    # gpus: all

  open-webui:
    image: ghcr.io/open-webui/open-webui:latest
    container_name: open-webui
    restart: unless-stopped
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_API_BASE_URL=http://ollama:11434
    depends_on:
      - ollama
    volumes:
      - openwebui:/app/backend/data

volumes:
  ollama:
  openwebui:

If you are on CPU-only, keep the file as is. If you have a GPU, uncomment the gpus: all line under the ollama service.

Step 5: Launch the Stack

Start both containers in detached mode:

docker compose up -d

Open WebUI should now be available at http://<your-server-ip>:3000. The first visitor will be asked to create an admin account. Leave the browser open; we will add a model next.

Step 6: Pull and Test a Model

Pull a model using Ollama. You can choose from many OSS models; Llama 3.1 8B is a balanced starter option:

docker exec -it ollama ollama pull llama3.1:8b

Confirm that the model is available:

curl http://localhost:11434/api/tags | jq

Back in Open WebUI, select this model in the top bar and start chatting. If GPU is enabled, generation should be significantly faster.

Optional: Secure Public Access with Caddy

If you want to access Open WebUI over HTTPS on a domain (for example, ai.example.com), a simple approach is to put Caddy in front. Caddy obtains and renews certificates automatically via Let’s Encrypt.

sudo apt -y install debian-keyring debian-archive-keyring apt-transport-https
curl -fsSL https://dl.cloudsmith.io/public/caddy/stable/gpg.key | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/caddy-stable-archive-keyring.gpg] \
https://dl.cloudsmith.io/public/caddy/stable/deb/ubuntu all main" | \
sudo tee /etc/apt/sources.list.d/caddy-stable.list

sudo apt update && sudo apt -y install caddy

Create a simple Caddyfile that proxies traffic to Open WebUI on port 3000:

sudo bash -c 'cat >/etc/caddy/Caddyfile' << "EOF"
ai.example.com {
  reverse_proxy 127.0.0.1:3000
}
EOF
sudo systemctl reload caddy

Replace ai.example.com with your real domain and make sure DNS A/AAAA records point to your server’s public IP.

Operations: Updates, Backups, and Cleanup

Update containers regularly for new features and security patches:

docker compose pull
docker compose up -d

Backup volumes to keep models and chat history safe. Stop containers briefly, archive volumes, then restart:

docker compose down
docker run --rm -v ollama:/data -v $PWD:/backup alpine tar czf /backup/ollama-vol.tar.gz -C / data
docker run --rm -v openwebui:/data -v $PWD:/backup alpine tar czf /backup/openwebui-vol.tar.gz -C / data
docker compose up -d

Remove everything if you want to reclaim space later (this deletes models and chat data):

docker compose down
docker volume rm $(docker volume ls -q | grep -E "(ollama|openwebui)")

Troubleshooting

GPU not detected: Ensure the NVIDIA driver is installed on the host, the toolkit is configured, and your Compose service includes gpus: all. Validate with docker run --rm --gpus all nvidia/cuda:... nvidia-smi.

Permission denied: If you cannot run Docker without sudo, confirm your user is in the docker group (use id) and re-log in or run newgrp docker.

Port conflicts: If ports 3000 or 11434 are in use, change them in the Compose file and update your reverse proxy accordingly.

Low VRAM or OOM: Prefer 4–8B parameter models or quantized variants (e.g., llama3.1:8b in Q4_K_M). Ollama will automatically pick quantized builds when available.

Logs: Review service logs for errors and performance clues:

docker logs -f ollama
docker logs -f open-webui

Why This Stack?

Ollama offers a consistent way to pull and run many open-source models locally, while Open WebUI gives you a friendly chat experience, prompt presets, file uploads, and team features. Everything stays on your hardware, which improves privacy and often reduces cost. With Docker and a reverse proxy, this setup scales from a single developer laptop to a small team server with SSL and authentication.

You now have a modern, local LLM environment with a clear upgrade path. Add more models with ollama pull, automate backups on a cron schedule, and secure public access with Caddy or another reverse proxy. For most use cases, this stack is fast, reliable, and easy to maintain.