LifeBytes Journal

Overview

In this step-by-step guide, you will learn how to self-host a local AI chat environment on Ubuntu using Ollama and Open WebUI. Ollama runs large language models (LLMs) locally and exposes a simple API, while Open WebUI provides a modern browser interface, chat history, and prompt management. This tutorial targets Ubuntu 22.04/24.04 and shows how to enable NVIDIA GPU acceleration, secure the service, and test the API.

Prerequisites

You need an Ubuntu 22.04/24.04 machine with at least 16 GB RAM for comfortable use and an NVIDIA GPU with recent drivers (525+ recommended) if you want hardware acceleration. For CPU-only usage, Ollama still works but will be slower. You also need a user with sudo privileges and internet access.

Step 1: Update the system and install basics

Start by refreshing your package lists and installing useful tools such as curl and ufw. Run: sudo apt update && sudo apt -y upgrade and then sudo apt -y install curl ca-certificates ufw. This ensures you have the latest security updates and a firewall ready to configure later.

Step 2: Verify NVIDIA GPU (optional but recommended)

If you intend to use GPU acceleration, confirm your NVIDIA driver installation. Run nvidia-smi. If the command shows your GPU and driver version, you are ready. If not, install a recommended driver with sudo ubuntu-drivers autoinstall, reboot using sudo reboot, and check again with nvidia-smi. Ollama includes the runtime pieces it needs and will automatically use your GPU when supported.

Step 3: Install Ollama

Ollama provides a one-line installer for Linux. Run: curl -fsSL https://ollama.com/install.sh | sh. This installs the ollama binary and sets up a systemd service called ollama. After the script completes, verify the service with systemctl status ollama. If it is not running, start it using sudo systemctl start ollama and enable it at boot with sudo systemctl enable ollama.

Step 4: Pull your first model

Ollama hosts many popular models. To start, pull a reasonably fast, high-quality base model like Meta’s Llama 3.1. Run ollama pull llama3.1. Other good options include mistral, qwen2.5, or smaller quantized variants that fit into limited VRAM (for example, llama3.1:8b or mistral:7b-instruct). Use ollama list to see installed models.

Step 5: Test the local API

Ollama listens on http://127.0.0.1:11434 by default. You can chat in the terminal with ollama run llama3.1. To test via API, run: curl http://127.0.0.1:11434/api/generate -d '{"model":"llama3.1","prompt":"Say hello from a local model."}'. You should see a streamed JSON response. Press Ctrl+C to stop streaming if needed.

Step 6: Install Docker (for Open WebUI)

Open WebUI is easiest to deploy with Docker. Install Docker and its prerequisites. First run: sudo apt -y install apt-transport-https gnupg lsb-release. Then add Docker’s repository key: curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker.gpg. Add the repo: echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null. Install Docker: sudo apt update && sudo apt -y install docker-ce docker-ce-cli containerd.io. Optionally add your user to the Docker group: sudo usermod -aG docker $USER and re-login.

Step 7: Deploy Open WebUI

Open WebUI connects to Ollama’s API and provides a rich chat interface. Create a persistent volume directory and run the container pointing to the local Ollama endpoint. Use: docker run -d --name open-webui -p 3000:8080 -e OLLAMA_API_BASE_URL=http://host.docker.internal:11434 -v openwebui-data:/app/backend/data --restart unless-stopped ghcr.io/open-webui/open-webui:latest. On Linux, if host.docker.internal is not available, replace it with the host’s IP (e.g., http://127.0.0.1:11434) and add --network host instead of -p mapping if you prefer host networking: docker run -d --name open-webui --network host -e OLLAMA_API_BASE_URL=http://127.0.0.1:11434 -v openwebui-data:/app/backend/data --restart unless-stopped ghcr.io/open-webui/open-webui:latest.

Step 8: Secure optional access and firewall

If you will only use the services locally, keep them bound to localhost and do not expose ports publicly. For remote access on a trusted LAN, allow Open WebUI’s port via UFW using sudo ufw allow 3000/tcp (or none if using --network host and default port 8080). Enable the firewall with sudo ufw enable, then verify rules with sudo ufw status. For public access, place Open WebUI behind a reverse proxy (Nginx, Caddy, or Traefik) with HTTPS and authentication.

Step 9: Use the interface

Open a browser to http://SERVER_IP:3000 (or http://localhost:3000). On first load, you can create an admin account, choose your default model (e.g., llama3.1), manage prompts, and run chats. You can switch models per-conversation and configure system prompts for specific tasks like coding, summarization, or Q&A.

Troubleshooting and tips

If a model fails to load due to GPU memory limits, pull a smaller or more heavily quantized variant such as llama3.1:8b or a q4_k_m quant. If CPU usage is too high, reduce the context window or batch size in Open WebUI settings. If the Open WebUI container cannot reach Ollama, double-check OLLAMA_API_BASE_URL, networking mode, and whether the ollama service is running. For best performance on NVIDIA GPUs, close other GPU-heavy apps and monitor usage with nvidia-smi. To update, run sudo systemctl stop ollama && curl -fsSL https://ollama.com/install.sh | sh && sudo systemctl start ollama and pull newer model versions as needed.

What you have now

You have a fully local AI chat stack with GPU acceleration using Ollama and a clean, user-friendly interface via Open WebUI. It is private by default, fast on modern GPUs, and flexible with many model choices. You can integrate it with other tools via the Ollama API for scripting, automation, and offline workflows. This setup gives you control over costs, data privacy, and performance while staying current with the latest open models.

If you want a private, fast, and customizable AI chat without sending data to third-party clouds, hosting Ollama with Open WebUI on Docker is a great choice. Ollama runs lightweight local large language models (LLMs) and Open WebUI provides a clean chat interface, prompt management, and model switching. This guide shows you how to deploy both with Docker on Linux or Windows (WSL2), including optional GPU acceleration for NVIDIA or AMD.

Prerequisites

- A 64-bit machine with at least 16 GB RAM for smooth inference (8 GB can work for smaller models).
- Docker Engine and Docker Compose v2 installed.
- For GPU acceleration (optional):
• NVIDIA: Install the latest NVIDIA driver and NVIDIA Container Toolkit.
• AMD: Recent ROCm-supported GPU and drivers. On Linux, make sure /dev/kfd and /dev/dri are present.

Step 1: Prepare folders

Create a working folder to store persistent data. This keeps your models and chat history safe across updates.

mkdir -p ~/ai-stack && cd ~/ai-stack

Step 2: Create a Docker Compose file

The following docker-compose.yml launches two services: ollama (the model runtime) and open-webui (the web interface). It maps volumes for persistence, exposes ports, and connects the web UI to Ollama. GPU support can be enabled with a single line if you use NVIDIA.

version: "3.9"
services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    ports:
      - "11434:11434"
    volumes:
      - ollama:/root/.ollama
    environment:
      - OLLAMA_KEEP_ALIVE=24h
    restart: unless-stopped
    # Enable this line if you have an NVIDIA GPU and the container toolkit installed:
    # gpus: all

  open-webui:
    image: ghcr.io/open-webui/open-webui:latest
    container_name: open-webui
    depends_on:
      - ollama
    environment:
      - OLLAMA_API_BASE=http://ollama:11434
      # Optional: disable public sign-ups after you create the first admin
      # - SIGNUP_ENABLED=false
    ports:
      - "3000:8080"
    volumes:
      - open-webui:/app/backend/data
    restart: unless-stopped

volumes:
  ollama:
  open-webui:

Notes for AMD/ROCm on Linux: Depending on your distribution and drivers, you may need to pass GPU devices to the Ollama container. Add the following under services.ollama if models aren’t using your GPU:

    devices:
      - /dev/kfd:/dev/kfd
      - /dev/dri:/dev/dri
    group_add:
      - "video"
    environment:
      - HSA_OVERRIDE_GFX_VERSION=11.0.0

If you cannot use GPU yet, you can run entirely on CPU by leaving GPU lines out. Start small models first and scale up as resources allow.

Step 3: Start the stack

Run the following to download images and start containers in the background:

docker compose up -d

Verify both containers are healthy:

docker compose ps

Step 4: Access the web interface and pull a model

Open your browser to http://localhost:3000 (or the server’s IP on port 3000). On the first load, Open WebUI will ask you to create an admin account. After that, connect to Ollama automatically via the configured OLLAMA_API_BASE.

You need to download at least one model. You can pull models either from the WebUI’s Models section or via CLI. For example, from the host:

docker exec -it ollama ollama pull llama3.1:8b

Once downloaded, open a new chat in Open WebUI and select the model (e.g., llama3.1:8b). You can then chat, create system prompts, and save conversations.

Step 5: Confirm GPU acceleration (optional)

To check whether the GPU is used, open logs during a generation:

docker logs -f ollama

You should see messages indicating GPU layers offloaded if acceleration is active. On NVIDIA, you can also run nvidia-smi on the host while generating text to confirm utilization.

Step 6: Secure and harden

- Accounts: After creating your admin user, consider disabling public sign-ups by uncommenting SIGNUP_ENABLED=false in the compose file and restarting.
- Network: Run behind a reverse proxy such as Caddy, Nginx, or Traefik to add HTTPS. If you expose it to the internet, restrict access with firewall rules and strong authentication.
- Data: Store volumes on disks with sufficient space. Models can be several gigabytes each.

Step 7: Update, backup, and migrate

- Update images:

docker compose pull
docker compose up -d

- Backup volumes (on the host):

docker run --rm -v ollama:/data -v $PWD:/backup alpine tar czf /backup/ollama-backup.tgz -C / data
docker run --rm -v open-webui:/data -v $PWD:/backup alpine tar czf /backup/open-webui-backup.tgz -C / data

- Migrate to another server by restoring these archives into volumes with the reverse tar process.

Troubleshooting

- Port conflicts: If ports 3000 or 11434 are in use, change the left side of the port mappings in the compose file (e.g., "8081:8080").
- GPU not detected (NVIDIA): Ensure the host driver matches your GPU, the NVIDIA Container Toolkit is installed, and the compose service has gpus: all. Restart Docker after toolkit installs.
- GPU not detected (AMD): Confirm ROCm support for your GPU and kernel, expose /dev/kfd and /dev/dri, and add your user to the video group on the host.
- Out of memory: Choose a smaller model (e.g., 3B/7B variants), reduce context length in the WebUI, or add swap space. On WSL2, limit memory usage or increase it in .wslconfig.

Why this stack?

Ollama provides a simple, consistent way to run many open models locally, with one command per model and automatic quantized formats for laptops and servers. Open WebUI adds a polished interface with multi-model selection, prompt templates, knowledge features, and API compatibility for tools. Together, they give you a private, portable AI chat solution that you control end to end.

With this setup, you can iterate quickly, test new models, and keep your data on your hardware. When you need more speed, enable GPU acceleration or move the same stack to a more powerful server with minimal changes.

Overview

This tutorial shows how to deploy a private, local AI stack with Ollama (model runtime) and Open WebUI (chat interface) using Docker on Ubuntu 22.04/24.04. You will learn how to run it on CPU, enable NVIDIA or AMD/ROCm GPU acceleration, secure the web interface, and keep everything up to date. The result is a fast, reliable, and low-maintenance setup suitable for labs, developers, and small teams.

Prerequisites

You need an Ubuntu 22.04 or 24.04 system with sudo access, 16 GB+ RAM (more is better), 20 GB+ free disk space, and a stable internet connection. For GPU acceleration, use a recent NVIDIA GPU with official drivers or a compatible AMD GPU with ROCm-capable kernel and hardware. Ensure ports 11434 (Ollama) and 3000 (Open WebUI) are free. If you plan to expose the service on the internet, prepare a domain name and DNS A/AAAA record pointing to the server.

Step 1: Install Docker Engine and Compose

sudo apt update && sudo apt install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update && sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker $USER
newgrp docker

Step 2: GPU Preparation (optional but recommended)

NVIDIA: Install the proprietary driver and the NVIDIA Container Toolkit so Docker can access your GPU.

sudo apt install -y ubuntu-drivers-common
sudo ubuntu-drivers install
sudo reboot

distribution=$(. /etc/os-release; echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update && sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

AMD (ROCm): Ensure your GPU is ROCm-capable and the kfd and dri devices are present. Give your user access to the required groups.

sudo usermod -aG render,video $USER
sudo reboot

Step 3: Create a Docker Compose file

Create a working directory like ~/ai-stack, then create docker-compose.yml. The following example starts Ollama and Open WebUI with volumes for persistence. It includes variants for CPU, NVIDIA, and AMD. Only keep one GPU option at a time.

docker-compose.yml (CPU-only by default):

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    ports:
     - "11434:11434"
    volumes:
     - ollama:/root/.ollama
    environment:
     - OLLAMA_KEEP_ALIVE=24h
     - OLLAMA_NUM_THREADS=8
  # For NVIDIA GPU (uncomment the next 4 lines and comment the AMD lines below):
  #   runtime: nvidia
  #   environment:
  #    - NVIDIA_VISIBLE_DEVICES=all
  #    - NVIDIA_DRIVER_CAPABILITIES=compute,utility
  # For AMD ROCm GPU (use the ROCm image and device mappings):
  #   image: ollama/ollama:rocm
  #   devices:
  #    - /dev/kfd
  #    - /dev/dri
  #   group_add:
  #    - "video"
  #    - "render"
  open-webui:
    image: ghcr.io/open-webui/open-webui:latest
    container_name: open-webui
    depends_on:
     - ollama
    restart: unless-stopped
    environment:
     - OLLAMA_API_BASE=http://ollama:11434
    ports:
     - "3000:8080"
    volumes:
     - openwebui:/app/backend/data
volumes:
  ollama:
  openwebui:

Step 4: Start the stack

docker compose up -d

Check containers and logs to confirm both services are healthy.

docker ps
docker logs -f ollama
docker logs -f open-webui

Step 5: Pull a model and test

Use Ollama to download a model. Popular choices are llama3.1:8b, llama3.1:70b (needs more VRAM), mistral, or qwen2. Start with an 8B or 7B model to validate your setup.

docker exec -it ollama ollama pull llama3.1:8b
curl http://localhost:11434/api/tags

Open a browser to http://<server-ip>:3000. The first user that signs up in Open WebUI becomes the admin. In Settings, point the Ollama endpoint to http://ollama:11434 (it is already set via OLLAMA_API_BASE). Create a new chat and pick your model from the dropdown.

Step 6: Optional security and HTTPS

By default, Open WebUI is accessible on port 3000 and provides its own user system. For internet exposure, put it behind an HTTPS reverse proxy and disable public signups after creating the admin. If you use UFW, allow only necessary ports:

sudo ufw allow 22/tcp
sudo ufw allow 80,443/tcp
sudo ufw enable

A simple approach is to add a Caddy or Nginx reverse proxy in front of Open WebUI for automatic TLS. Map your domain (e.g., ai.example.com) to the server, then proxy requests to open-webui:8080. Limit administrative access using firewall rules, strong passwords, and, if available, SSO/OIDC in Open WebUI.

Step 7: Updating and backing up

To update images to the latest versions and apply them with minimal downtime:

cd ~/ai-stack
docker compose pull
docker compose up -d

Your models and chat data live in Docker volumes. Back them up regularly:

docker run --rm -v ollama:/data -v $(pwd):/backup alpine tar czf /backup/ollama-vol-$(date +%F).tgz -C /data .
docker run --rm -v openwebui:/data -v $(pwd):/backup alpine tar czf /backup/openwebui-vol-$(date +%F).tgz -C /data .

Troubleshooting tips

If GPU is not used on NVIDIA, confirm nvidia-smi works on the host and the container runtime is configured. For AMD, ensure /dev/kfd and /dev/dri exist and the container uses the ollama/ollama:rocm image with the proper device mappings. Model loading failures typically indicate insufficient RAM/VRAM; try a smaller quantization or a smaller model. If the UI cannot see Ollama, verify OLLAMA_API_BASE and that containers can resolve each other by service name.

You are done

You now have a modern, private AI chat stack running on Docker with optional GPU acceleration. Ollama keeps model management simple, and Open WebUI provides a clean, multi-user interface. This setup is easy to maintain, portable across servers, and ready for experimentation with different open-source models and embeddings.

Overview

This tutorial shows you how to deploy a private AI chatbot using Ollama and Open WebUI with Docker on Linux (Ubuntu 22.04/24.04). You will get a secure, local, and fast setup that can run on CPU or use your NVIDIA GPU for acceleration. We will cover installation, model downloads, persistence, updates, and security hardening.

What You Will Build

- Ollama container serving large language models (LLMs) on port 11434.
- Open WebUI container providing a modern chat interface on port 3000.
- Optional NVIDIA GPU pass-through for faster inference.
- Persistent volumes so your models and settings survive reboots and updates.
- Basic authentication and reverse proxy tips for safe remote access.

Prerequisites

- A 64-bit Linux host with Docker Engine installed. On Ubuntu, install Docker with:
sudo apt update && sudo apt install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" | sudo tee /etc/apt/sources.list.d/docker.list
sudo apt update && sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
- Optional: an NVIDIA GPU with drivers installed (530+ recommended) if you want GPU acceleration.

Optional: Enable NVIDIA GPU for Docker

If you have an NVIDIA GPU, install the NVIDIA Container Toolkit so Docker can access the GPU:
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update && sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
Test with:
docker run --rm --gpus all nvidia/cuda:12.2.0-base-ubuntu22.04 nvidia-smi

Create Folders and a Dedicated Network

mkdir -p ~/ai/ollama ~/ai/openwebui
docker network create ai-net

Start the Ollama Container (CPU or GPU)

CPU-only (works everywhere):
docker run -d --name ollama --restart unless-stopped \
-p 11434:11434 \
-v ~/ai/ollama:/root/.ollama \
--network ai-net \
ollama/ollama:latest

GPU-enabled (if you completed the NVIDIA step):
docker run -d --name ollama --restart unless-stopped \
-p 11434:11434 \
-v ~/ai/ollama:/root/.ollama \
--gpus all \
--network ai-net \
ollama/ollama:latest

Pull a Model with Ollama

Ollama hosts many models (Llama 3, Phi-3, Mistral, Gemma, etc.). Pull one that fits your hardware. For a good balance, try Llama 3 8B:
docker exec -it ollama ollama pull llama3:8b
On low-memory machines, use a smaller or quantized model (for example llama3:8b-instruct-q4_K_M). You can list models with:
docker exec -it ollama ollama list

Start Open WebUI and Connect It to Ollama

Run Open WebUI with persistent storage and authentication. Replace the admin email and password before running:
docker run -d --name openwebui --restart unless-stopped \
-p 3000:8080 \
-e OLLAMA_BASE_URL=http://ollama:11434 \
-e WEBUI_AUTH=True \
-e [email protected] \
-e ADMIN_PASSWORD='ChangeThisStrongPass!2025' \
-v ~/ai/openwebui:/app/backend/data \
--network ai-net \
ghcr.io/open-webui/open-webui:latest

Now open your browser and go to http://SERVER_IP:3000. Log in with the admin account. In the model dropdown, select the model you pulled (for example llama3:8b) and start chatting.

Persist and Back Up Your Data

All models and settings are stored in the bind mounts we created:
- Models and Ollama config: ~/ai/ollama
- Web interface data (users, chats): ~/ai/openwebui
To back them up, stop containers and archive the folders:
docker stop openwebui ollama
tar -czf ai-backup-$(date +%F).tar.gz -C ~/ ai
docker start ollama openwebui

Update Containers and Models

To update to the latest versions safely:
docker pull ollama/ollama:latest
docker pull ghcr.io/open-webui/open-webui:latest
docker stop openwebui ollama
docker rm openwebui ollama
Recreate with the same docker run commands (volumes keep your data). To update a model:
docker exec -it ollama ollama pull llama3:8b

Secure Remote Access

- Keep WEBUI_AUTH=True and use a strong admin password.
- Restrict firewall: allow only your IP and necessary ports (11434, 3000, or the reverse proxy port). On Ubuntu with UFW:
sudo ufw allow 22/tcp
sudo ufw allow from YOUR.IP.ADDR.0/24 to any port 3000 proto tcp
sudo ufw enable
- For HTTPS, place a reverse proxy in front. Example Caddyfile (replace domain):
ai.example.com {
  reverse_proxy 127.0.0.1:3000
}
Caddy will auto-issue TLS certificates via Let’s Encrypt.

Performance Tips

- Prefer GPU for large models. Use smaller or quantized models on CPU-only hosts.
- Set the context window and temperature in Open WebUI for faster, more focused responses.
- Avoid swapping: ensure available RAM; 8–16 GB is reasonable for 7–8B quantized models, more for FP16 and larger models.
- Pin container CPU/RAM if needed using --cpus and -m flags in docker run.

Troubleshooting

- Port already in use: change -p 3000:8080 or stop the conflicting service.
- GPU not detected: confirm nvidia-smi works on the host, verify nvidia-ctk runtime configure, restart Docker, and run the CUDA test container.
- Model download slow: it is normal on first pull; try a different model or check your network.
- Open WebUI cannot reach Ollama: ensure both containers are on ai-net and OLLAMA_BASE_URL=http://ollama:11434 is set correctly.

Conclusion

With Docker, Ollama, and Open WebUI, you can run a private AI chat system that is fast, flexible, and secure. This stack supports many modern open models and can scale from a small home server to a GPU workstation. Keep your containers updated, back up the volumes, and tune the model choice to your hardware for the best experience.

Local large language models have matured fast. With Ollama and Open WebUI, you can run modern models like Llama 3.1 or Phi-3 locally, enjoy a clean chat interface, and use your NVIDIA GPU for real speed. This guide walks through a clean setup on Ubuntu 22.04 or 24.04 using Docker and the NVIDIA Container Toolkit.

What you will build: Ollama as your model runtime, Open WebUI as the web interface, both running on Docker, and NVIDIA GPU acceleration for high throughput.

Prerequisites: An Ubuntu 22.04/24.04 machine, an NVIDIA GPU with at least 8 GB VRAM (more is better), 16+ GB system RAM recommended, sudo access, and an internet connection.

1) Install NVIDIA Driver and Validate GPU

First, ensure your system sees the GPU and the driver is installed. If you already have a working driver and nvidia-smi reports correctly, you can skip to Docker installation.

Detect the GPU:
lspci | grep -i nvidia

Install the recommended driver:
sudo apt update
sudo ubuntu-drivers autoinstall
sudo reboot

Verify after reboot:
nvidia-smi
You should see driver, CUDA version, and GPU utilization stats.

2) Install Docker and NVIDIA Container Toolkit

We will run both Ollama and Open WebUI via containers. Install Docker first, then enable GPU support inside containers.

Install Docker Engine:
sudo apt update && sudo apt install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker $USER && newgrp docker

Install NVIDIA Container Toolkit:
distribution=$(. /etc/os-release;echo $ID$VERSION_ID) && \
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update && sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Test GPU in containers:
docker run --rm --gpus all nvidia/cuda:12.3.2-base-ubuntu22.04 nvidia-smi
If you see the same GPU readout, the toolkit works.

3) Install Ollama

Ollama brings one-line model setup and fast local inference. Use the official installer:

curl -fsSL https://ollama.com/install.sh | sh

The install registers a systemd service and puts the binary at /usr/local/bin/ollama. Start or check status:

sudo systemctl enable --now ollama
systemctl status ollama

Quick test with CPU or GPU:
ollama run llama3.1:8b
Type a prompt; use Ctrl+C to exit. If you have a supported NVIDIA setup, Ollama will prefer GPU automatically.

4) Deploy Open WebUI with Docker

Open WebUI gives you a polished chat interface, prompt templates, and conversation history. We will connect it to the local Ollama API at http://host.docker.internal:11434 (or your host IP) from inside the container.

Run Open WebUI:
docker run -d --name openwebui --restart unless-stopped -p 3000:8080 \
-e OLLAMA_API_BASE=http://host.docker.internal:11434 \
-e WEBUI_AUTH=True \
-v openwebui-data:/app/backend/data \
ghcr.io/open-webui/open-webui:latest

Open a browser to http://SERVER_IP:3000. Create the first admin user and configure the default model (for example, llama3.1:8b).

5) Using the GPU with Open WebUI + Ollama

If your driver and toolkit are correct, Ollama will use the GPU. To confirm, watch GPU usage while making a request:

watch -n 1 nvidia-smi

To control GPU usage and offloading, set environment variables for the Ollama service. For example:

sudo systemctl edit ollama
Add lines under [Service]:
Environment="OLLAMA_NUM_GPU=1"
Environment="OLLAMA_KV_CACHE_TYPE=q8_0"
Save and reload:
sudo systemctl daemon-reload && sudo systemctl restart ollama

You can also pick quantized models to fit your VRAM. Examples: llama3.1:8b-q4_K_M (balanced), llama3.1:8b-q5_K_M (higher quality), or phi3:mini-4k-instruct-q4_K_M for small GPUs.

6) Optional: Docker Compose Setup

To run everything together and persist data, use Docker Compose:

mkdir -p ~/llm-stack && cd ~/llm-stack
nano compose.yml

Paste:

services:
  ollama:
   image: ollama/ollama:latest
   container_name: ollama
   restart: unless-stopped
   ports:
    - "11434:11434"
   volumes:
    - ollama:/root/.ollama
   deploy:
    resources:
     reservations:
      devices:
       - capabilities: ["gpu"]

  openwebui:
   image: ghcr.io/open-webui/open-webui:latest
   container_name: openwebui
   restart: unless-stopped
   ports:
    - "3000:8080"
   environment:
    - OLLAMA_API_BASE=http://ollama:11434
    - WEBUI_AUTH=True
   depends_on:
    - ollama
   volumes:
    - openwebui-data:/app/backend/data

volumes:
  ollama:
  openwebui-data:

docker compose up -d

7) Security, Backups, and Troubleshooting

Secure access: Keep Open WebUI behind a reverse proxy (Caddy, Nginx) with HTTPS. At minimum, enable built-in auth (already set with WEBUI_AUTH=True). For remote access, consider Tailscale or WireGuard rather than exposing port 3000 to the internet.

Persist and back up data: Ollama stores models in ~/.ollama/models (or the volume you mapped). Open WebUI keeps data in the openwebui-data volume. Back up with:

docker run --rm -v ollama:/src -v $PWD:/dst alpine tar czf /dst/ollama-backup.tgz -C /src .
docker run --rm -v openwebui-data:/src -v $PWD:/dst alpine tar czf /dst/openwebui-backup.tgz -C /src .

Common issues:

- Ollama says it cannot find a GPU: check driver and run nvidia-smi; ensure nvidia-container-toolkit is installed and Docker restarted.
- Model fails to load due to VRAM limits: pick a smaller or more aggressively quantized variant (q4, q3), or set num_ctx lower in the model settings.
- High CPU usage: disable background indexing features in the UI and avoid running multiple heavy models simultaneously.

8) Next Steps

Try function-calling and RAG with Open WebUI extensions, schedule model updates with ollama pull, and benchmark different quantizations for your GPU. With this stack, you own your data, enjoy low latency, and can iterate quickly without cloud costs.

Overview

Running a private, fast, and flexible AI chat server on your own computer is easier than ever. In this guide, you will set up Ollama (the local LLM runtime) and Open WebUI (a polished chat interface) on Linux or Windows. You will be able to pull popular models like Llama 3 or Mistral, enable GPU acceleration when available, and access a modern web UI for chatting, prompt management, file uploads, and more. This tutorial focuses on simplicity, security, and reliability, using Docker for the web UI and native installation for Ollama.

What You’ll Need

- A 64-bit machine running Ubuntu 22.04/24.04, Debian 12, or Windows 11/10 (latest updates).
- 16 GB RAM recommended; more helps with larger models.
- Optional GPU for acceleration: NVIDIA/AMD/Intel with recent drivers (Ollama uses CUDA/ROCm/DirectML depending on platform).
- Internet access to download models (they can be several gigabytes).

Step 1 — Install Ollama

On Linux (Ubuntu/Debian):
curl -fsSL https://ollama.com/install.sh | sh
sudo systemctl enable --now ollama
Check that the API is listening on port 11434:
curl http://localhost:11434/api/tags

On Windows:
Install with Winget:
winget install Ollama.Ollama
After install, Ollama runs in the background. Verify the API:
curl http://localhost:11434/api/tags

Tip: If you have a GPU, make sure the latest driver is installed. On Linux (NVIDIA), confirm with nvidia-smi. Ollama will automatically use the GPU when possible and print a GPU line in its logs during the first model load.

Step 2 — Pull a Model and Test Locally

Pull a high-quality, general-purpose model. Two popular choices are Llama 3 and Mistral:

ollama pull llama3
ollama pull mistral

Run a quick prompt to validate generation:

ollama run llama3
When prompted, type: Explain what containers are in simple terms.
Press Ctrl+C to exit.

Step 3 — Install Docker (for Open WebUI)

On Linux (quick installer):
curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER
newgrp docker
Confirm:
docker version

On Windows:
Install Docker Desktop from the official site, enable WSL 2 integration, and confirm it runs correctly.

Step 4 — Deploy Open WebUI

Open WebUI provides a rich chat interface on top of the Ollama API. You will run it in Docker and point it to your local Ollama instance.

Linux (Docker host networking):
docker run -d --name open-webui --restart unless-stopped --network host -e OLLAMA_BASE_URL=http://127.0.0.1:11434 -v open-webui:/app/backend/data ghcr.io/open-webui/open-webui:latest

Windows/macOS (port mapping):
docker run -d --name open-webui --restart unless-stopped -p 3000:8080 -e OLLAMA_BASE_URL=http://host.docker.internal:11434 -v open-webui:/app/backend/data ghcr.io/open-webui/open-webui:latest

Open your browser to http://localhost:3000. On first launch, create an admin account. In Settings, select your default model (e.g., llama3) and adjust UI preferences. The data volume open-webui persists chats and configurations across container restarts.

Step 5 — Basic Security and Remote Access

By default, both Ollama (11434) and Open WebUI (3000) listen on localhost. This is the safest configuration for a single-user machine. If you must access Open WebUI remotely, place it behind a reverse proxy with TLS and a password. For example, using Caddy on a server with a public DNS record:

example.com {
  reverse_proxy 127.0.0.1:3000
}

Caddy will fetch a free Let’s Encrypt certificate automatically. Keep the Ollama API on localhost and expose only the web UI via the proxy.

Step 6 — Verify the Stack

- Test the Ollama API directly:
curl http://localhost:11434/api/tags
- Generate a short reply via API:
curl -s http://localhost:11434/api/generate -d '{"model":"llama3","prompt":"Say hi in one sentence."}'
- Open WebUI at http://localhost:3000 and send a message. Check the model switcher if you pulled multiple models.

Performance Tips

- Prefer smaller or quantized models for limited RAM/VRAM (e.g., 7B Q4_K_M).
- Close other heavy apps to reduce RAM pressure and swapping.
- If you have a strong GPU, pull a larger model (e.g., 13B) for better quality.
- Store models on a fast SSD; Ollama caches models under your user profile (e.g., ~/.ollama on Linux).

Backups and Maintenance

- Models: back up ~/.ollama (Linux) or the Ollama data directory on Windows to avoid re-downloading large files.
- Web UI data: the Docker volume open-webui stores conversations and settings. You can back it up with:
docker run --rm -v open-webui:/data -v "$PWD":/backup alpine sh -c 'cd /data && tar czf /backup/open-webui-data.tgz .'
- Update Ollama occasionally:
curl -fsSL https://ollama.com/install.sh | sh
- Update Open WebUI:
docker pull ghcr.io/open-webui/open-webui:latest && docker restart open-webui

Troubleshooting

- Port in use: If 3000 or 11434 is busy, change the mapping (e.g., -p 8080:8080) or stop the conflicting service.
- Docker cannot reach Ollama on Linux: ensure you used --network host, or alternatively add --add-host=host.docker.internal:host-gateway and set OLLAMA_BASE_URL=http://host.docker.internal:11434.
- GPU not used: update drivers and reboot; on Linux check nvidia-smi; on Windows confirm GPU activity in Task Manager during inference.
- Out-of-memory: switch to a smaller model or a stronger quantization; close other applications.

You’re Done

You now have a private, GPU-ready AI chat server powered by Ollama and Open WebUI. This setup is fast, secure (localhost by default), and extensible. You can add specialized models for coding or RAG, script API calls for automation, and back everything up with a couple of commands. Enjoy building with local AI—no cloud required.

Overview

This step-by-step guide shows you how to deploy an AI chatbot stack with Ollama (for running local LLMs) and Open WebUI (a clean, browser-based interface) on Ubuntu 22.04 or 24.04. We will run both apps in Docker, enable NVIDIA GPU acceleration, and put Nginx in front with a free TLS certificate from Let's Encrypt. You will get a production-friendly setup with persistent storage, HTTPS, and simple maintenance commands.

What you'll build

You will end up with two containers on a private Docker network: ollama (listening on 11434) and openwebui (listening on 8080, mapped to localhost:3000). Nginx will reverse proxy a public domain (for example, ai.example.com) to Open WebUI and handle SSL. Models and chat data will be stored on the host so updates don't wipe them.

Prerequisites

- Ubuntu 22.04/24.04 with sudo access
- An NVIDIA GPU (Turing or newer recommended) and a supported driver
- A DNS A record pointing your domain (e.g., ai.example.com) to your server's public IP
- Outbound internet access to pull images and models

1) Install NVIDIA driver and container toolkit

Update the system and install the proprietary driver. If you don't already have the correct driver, Ubuntu can choose one for you:
sudo apt update && sudo apt -y upgrade
sudo ubuntu-drivers autoinstall
Reboot:
sudo reboot

After reboot, confirm the GPU is visible:
nvidia-smi
Install the NVIDIA Container Toolkit so Docker can use the GPU:
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update && sudo apt -y install nvidia-container-toolkit
Configure Docker to use it and restart Docker:
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

2) Install Docker Engine and Docker Compose plugin

Install the official Docker packages:
sudo apt -y install ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(. /etc/os-release; echo $VERSION_CODENAME) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
sudo apt -y install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
Verify:
docker --version
docker compose version

3) Create persistent folders and a Docker Compose file

Create directories for persistent data:
sudo mkdir -p /opt/ollama /opt/openwebui
sudo chown -R $USER:$USER /opt/ollama /opt/openwebui
Now create a compose.yml in a new project folder (for example, /opt/ai-stack/compose.yml) with the following content:

version: "3.8"
services:
ollama:
image: ollama/ollama:latest
container_name: ollama
restart: unless-stopped
volumes:
- /opt/ollama:/root/.ollama
environment:
- OLLAMA_KEEP_ALIVE=24h
networks:
- ai
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]

openwebui:
image: ghcr.io/open-webui/open-webui:latest
container_name: openwebui
restart: unless-stopped
depends_on:
- ollama
environment:
- OLLAMA_BASE_URL=http://ollama:11434
ports:
- "127.0.0.1:3000:8080"
volumes:
- /opt/openwebui:/app/backend/data
networks:
- ai

networks:
ai:

Notes: We bind Open WebUI to localhost:3000 so it is not exposed directly. Nginx will handle public traffic. The NVIDIA device reservation passes the GPU into the Ollama container. If your Docker Compose version supports it, you may also use gpus: all under the ollama service instead of the deploy block.

4) Start the stack and test

From the folder with compose.yml, bring the stack up:
docker compose up -d
Watch logs until both services are healthy:
docker compose logs -f
Pull a model and perform a quick GPU test (you should see GPU usage spike in nvidia-smi):
docker exec -it ollama ollama pull llama3:8b
docker exec -it ollama ollama run llama3:8b
Locally, you can visit Open WebUI at http://127.0.0.1:3000. Next, we'll put it behind HTTPS.

5) Install Nginx and obtain a Let's Encrypt certificate

Install Nginx and Certbot:
sudo apt -y install nginx certbot python3-certbot-nginx
Create an Nginx server block (replace ai.example.com with your domain):
sudo nano /etc/nginx/sites-available/ai.conf
Paste:

server {
listen 80;
server_name ai.example.com;
location / {
proxy_pass http://127.0.0.1:3000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
client_max_body_size 20m;
}

Enable and test:
sudo ln -s /etc/nginx/sites-available/ai.conf /etc/nginx/sites-enabled/
sudo nginx -t && sudo systemctl reload nginx
Issue a certificate and force HTTPS:
sudo certbot --nginx -d ai.example.com --redirect --agree-tos -m [email protected]
Now browse to https://ai.example.com and you should see Open WebUI served over TLS.

6) Backups, updates, and security tips

Back up data by archiving the two directories we created:
sudo tar -czf /root/ollama-backup.tgz /opt/ollama
sudo tar -czf /root/openwebui-backup.tgz /opt/openwebui
Open WebUI stores conversations and settings in /opt/openwebui; Ollama stores models and blobs in /opt/ollama.

To update images with minimal downtime:
docker compose pull
docker compose up -d
Old images can be cleaned with docker image prune when you're done testing. For security, keep ports private (we only published 3000 to localhost) and use your firewall to allow 80/443 only. If you need extra protection, add HTTP Basic Auth to Nginx and restrict by IP when possible.

Troubleshooting

- GPU not used: watch nvidia-smi while running a model. If it stays idle, recheck the NVIDIA driver, container toolkit, and the GPU reservation in compose.yml.
- Models fail due to VRAM limits: try smaller variants (e.g., llama3:8b instead of 70B) or quantized builds (like q4_K_M).
- Port conflicts: change the host port mapping in compose.yml if 3000 is taken (e.g., use 127.0.0.1:3100:8080 and update the Nginx proxy_pass accordingly).
- Certbot issues: make sure your domain points to the server’s public IP and TCP/80 is reachable from the internet during certificate issuance.

What's next

From here, you can connect more tools to the Ollama API, add multiple models, fine-tune your Nginx headers, or place the stack behind Cloudflare. The setup is simple to maintain: pull updates, restart the stack, and your data persists. With GPU acceleration and HTTPS in place, you have a fast, private AI assistant ready for everyday use.