LifeBytes Journal

Overview

This step-by-step guide shows you how to run local Large Language Models (LLMs) on Ubuntu using Ollama and Open WebUI. You will install Ollama, optionally enable NVIDIA GPU acceleration, and deploy Open WebUI in Docker to get a fast, friendly chat interface. By the end, you will have a private AI assistant running on your own hardware with secure access options and practical troubleshooting tips.

Prerequisites

Use Ubuntu 22.04 or 24.04 with at least 8 GB of RAM (16 GB recommended). For GPU acceleration, an NVIDIA GPU with 8 GB or more VRAM is ideal. You need sudo access and open ports 11434 for Ollama and 3000 (or your choice) for Open WebUI. This guide covers both CPU-only and GPU setups, so you can start even without a supported GPU.

Step 1: (Optional) Install NVIDIA Drivers and CUDA

If you plan to use a GPU, first confirm your hardware with lspci | grep -i nvidia. Install the recommended driver via sudo ubuntu-drivers autoinstall, then reboot. After rebooting, verify the driver with nvidia-smi. If you will run Open WebUI with GPU access in Docker, also install the NVIDIA container runtime using sudo apt-get install -y nvidia-container-toolkit and configure Docker with sudo nvidia-ctk runtime configure followed by sudo systemctl restart docker.

Step 2: Install Ollama on Ubuntu

Install Ollama with a single command: curl -fsSL https://ollama.com/install.sh | sh. This creates a system service and exposes the local API on http://127.0.0.1:11434. Check the version with ollama -v and verify the service using systemctl status ollama. If you need remote access on your LAN, set the host binding by creating an override file. Run sudo systemctl edit ollama, add [Service] and Environment="OLLAMA_HOST=0.0.0.0:11434", then save, sudo systemctl daemon-reload, and sudo systemctl restart ollama. Only expose Ollama on trusted networks or behind a reverse proxy with authentication.

Step 3: Pull and Run Models with Ollama

Pull a small, fast model to test your setup. For general chat, use ollama pull llama3.2:3b. For coding tasks, try ollama pull qwen2.5-coder:7b or a quantized variant like :q4_0 for lower memory usage. Run an interactive session with ollama run llama3.2 and type your prompt. To generate from the shell, try echo "Explain RAID levels simply" | ollama run llama3.2. Ollama will use the GPU automatically if supported; otherwise it falls back to CPU. Tune performance with environment variables such as OLLAMA_NUM_PARALLEL=1 to reduce memory pressure and OLLAMA_KV_SIZE=512 for larger context windows when your memory allows.

Step 4: Deploy Open WebUI with Docker

Open WebUI provides a clean web interface and multi-model support. If Docker is not installed, add it with sudo apt-get update && sudo apt-get install -y docker.io and ensure it runs at startup with sudo systemctl enable --now docker. Launch Open WebUI connected to Ollama using docker run -d --name open-webui -p 3000:8080 -e OLLAMA_BASE_URL=http://localhost:11434 -v open-webui:/app/backend/data -v /var/lib/ollama:/root/.ollama --restart unless-stopped ghcr.io/open-webui/open-webui:latest. If Open WebUI runs on a different host from Ollama, set OLLAMA_BASE_URL to the Ollama server’s IP, for example http://192.168.1.50:11434. For GPU inside the container, add --gpus all and make sure the NVIDIA container toolkit is configured.

Step 5: Secure Access with a Reverse Proxy and HTTPS

If you plan to reach the interface over the internet, place Open WebUI behind a reverse proxy with TLS and authentication. A simple option is Caddy, which can obtain and renew certificates automatically. For example, you can point a domain to your server and configure Caddy to proxy yourdomain.com to localhost:3000 and enable basic auth. With Nginx, use an SSL server block, set proxy_pass http://127.0.0.1:3000, and enable rate limiting and headers like X-Frame-Options and Content-Security-Policy. Always avoid exposing the raw Ollama port unless you fully trust the network.

Step 6: Updates, Backups, and Autostart

Update Ollama by rerunning the installer or using your package manager if you installed via a repo. To update Open WebUI, pull the latest image with docker pull ghcr.io/open-webui/open-webui:latest and restart the container. Persist your data by backing up /var/lib/ollama and the Docker volume open-webui. Both Ollama and Docker containers start automatically on boot, but you can confirm with systemctl is-enabled ollama and the container’s --restart unless-stopped flag.

API Quick Test

You can call Ollama’s local API directly. After pulling a model, try curl http://localhost:11434/api/generate -d '{"model":"llama3.2","prompt":"Give me three bullet points about containers"}'. This is useful for integrating local LLMs into scripts, chatbots, or development tools without sending data to third parties.

Troubleshooting

If model loading fails with “no space left on device,” free disk space with df -h, remove unused Docker images with docker system prune -a, or delete old models in /var/lib/ollama. If nvidia-smi returns an error, reinstall the driver and ensure Secure Boot is either disabled or configured with signed modules. If port 11434 or 3000 is already in use, change the binding (for example OLLAMA_HOST=0.0.0.0:11435) or stop the conflicting process. On low-memory hosts, choose smaller or more heavily quantized models (for example :q4_0), reduce parallel requests with OLLAMA_NUM_PARALLEL=1, and close other memory-hungry services.

What You Achieved

You now have a private, production-ready local AI stack on Ubuntu. Ollama runs the model backend with optional GPU acceleration, while Open WebUI delivers a modern chat interface. With a reverse proxy and backups in place, you can confidently use local LLMs for coding assistance, content drafting, documentation, and experimentation without sending your data to the cloud.

Overview

This tutorial shows you how to deploy a fast, private, local AI stack on Ubuntu using Ollama and Open WebUI with NVIDIA GPU acceleration. You will install the NVIDIA driver, Docker, and the NVIDIA Container Toolkit, then run Ollama on the host and Open WebUI in a container. By the end, you will have a browser-based interface to run powerful large language models (LLMs) like Llama 3 with CUDA acceleration on your own machine.

Prerequisites

- Ubuntu 22.04 or 24.04 (freshly updated).
- An NVIDIA GPU with at least 6 GB VRAM (more is better).
- sudo privileges and Internet access.
- Optional: a domain or reverse proxy if you plan to expose the UI externally.

Step 1 — Install NVIDIA Driver

Use Ubuntu’s built-in tools to install a compatible proprietary driver. Reboot afterward and confirm the GPU is detected.

sudo apt update
sudo apt install -y ubuntu-drivers-common
sudo ubuntu-drivers autoinstall
sudo reboot

nvidia-smi

If you see a table with your GPU and driver version (e.g., 535+), you are ready for CUDA-enabled workloads.

Step 2 — Install Docker Engine

If Docker is not installed, use the official convenience script. Add your user to the docker group so you can run containers without sudo.

curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER
newgrp docker
docker version

Step 3 — Enable GPU Access in Containers

Install the NVIDIA Container Toolkit so Docker can pass the GPU into containers.

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
| sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
| sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#' \
| sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt update
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Verify GPU visibility inside a container:

docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

Step 4 — Install Ollama (runs on the host)

Ollama simplifies downloading and running LLMs locally. It automatically uses CUDA if your NVIDIA driver is installed.

curl -fsSL https://ollama.com/install.sh | sh

Confirm the service is active and the API is reachable on port 11434:

systemctl --user status ollama || systemctl status ollama
curl http://127.0.0.1:11434/api/tags

Pull and test a model (replace with your preferred model/quantization):

ollama pull llama3
ollama run llama3 "Write a two-line poem about GPUs."

Tip: Use smaller quantizations if VRAM is limited, for example llama3:8b-instruct-q4_0.

Step 5 — Deploy Open WebUI in Docker

Open WebUI provides a clean, modern interface for chatting with models served by Ollama. We will run it in Docker and point it to the host’s Ollama API. On Linux, add a host-gateway entry so the container can reach the host at host.docker.internal.

docker run -d --name open-webui \
  -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
  --gpus all \
  -v open-webui:/app/backend/data \
  ghcr.io/open-webui/open-webui:main

Open your browser at http://<server-ip>:3000. On first login, create a user; that account becomes admin. If you need authentication enabled from the start, add -e WEBUI_AUTH=True to the run command.

Alternative: If --add-host=host-gateway is not supported on your Docker version, use host networking and point to 127.0.0.1:

docker run -d --name open-webui \
  --network host \
  -e OLLAMA_BASE_URL=http://127.0.0.1:11434 \
  --gpus all \
  -v open-webui:/app/backend/data \
  ghcr.io/open-webui/open-webui:main

With host networking, Open WebUI listens on http://0.0.0.0:8080 (no -p flag needed).

Step 6 — Use and Tune Your Local AI

From Open WebUI, select a model (e.g., Llama 3) and start chatting. You can pull additional models with Ollama CLI and they will appear in the UI. To speed up responses and reduce VRAM, try smaller or more aggressive quantizations; to maximize quality, try larger quantizations if your GPU can handle them.

Common environment variables for Open WebUI include:
- WEBUI_AUTH=True to require sign-in.
- OLLAMA_BASE_URL to point to the Ollama server URL.
- PORT to customize the UI port if you use host networking.

Troubleshooting

Open WebUI cannot reach Ollama: Ensure you used --add-host=host.docker.internal:host-gateway and OLLAMA_BASE_URL=http://host.docker.internal:11434, or use host networking. Test connectivity with docker exec -it open-webui curl -s http://host.docker.internal:11434/api/tags.

No GPU in containers: Re-check the container toolkit setup and driver. Run docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi. If it fails, reboot and ensure nvidia-smi works on the host first.

Out-of-memory errors: Use a smaller model or more compressed quantization. Close other GPU-heavy apps. You can also run with a larger system swap to reduce crashes when VRAM is exhausted, but performance will be slower.

Docker permissions: If you see “permission denied,” ensure your user is in the docker group (id to verify) and run newgrp docker or re-log in.

Optional: Reverse Proxy and TLS

If exposing Open WebUI on the Internet, put it behind a reverse proxy (Caddy, Nginx, or Traefik) for HTTPS and access control. At minimum, enforce authentication and limit access to trusted IPs. Never expose Ollama’s port 11434 directly without protection.

Maintenance

- Update Ollama periodically by re-running the install script or checking the project release notes, then systemctl restart ollama.
- Update Open WebUI with docker pull ghcr.io/open-webui/open-webui:main and docker restart open-webui.
- Prune old images and volumes with docker system prune (review carefully before confirming).
- Back up /var/lib/ollama (models) and the Open WebUI volume for settings and chats.

You now have a modern, GPU-accelerated, private AI chat environment running locally on Ubuntu. This setup is fast, secure, and fully under your control—and you can expand it with additional models, prompt libraries, and integrations as your needs grow.

This guide shows how to deploy a modern reverse proxy using Traefik v3, Docker, and Docker Compose on Ubuntu 22.04 or 24.04. You will get automatic Let’s Encrypt SSL certificates, HTTP to HTTPS redirection, and a clean way to route multiple apps on the same server under different domains or subdomains. The steps are simple, repeatable, and safe for production.

Prerequisites

Before starting, ensure you have: (1) An Ubuntu 22.04 or 24.04 server with a public IP, (2) A domain or subdomain you can edit DNS for (e.g., example.com), and (3) Access to open TCP ports 80 and 443 on the server’s firewall and network provider. You should also have a non-root user with sudo access.

Step 1 — Install Docker Engine and Compose Plugin

Install the official Docker packages. This gives you the latest stable Docker Engine, Buildx, and the Compose plugin.

sudo apt update
sudo apt install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
https://download.docker.com/linux/ubuntu $(. /etc/os-release; echo $VERSION_CODENAME) stable" | \
sudo tee /etc/apt/sources.list.d/docker.list >/dev/null
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker $USER
newgrp docker

Step 2 — DNS and Firewall

Create DNS A records for your services. For example, point app.example.com and traefik.example.com to your server’s public IP. Then open the firewall for web traffic.

sudo ufw allow 80,443/tcp
sudo ufw reload

Step 3 — Prepare a Docker Network and Traefik Files

Create a dedicated Docker network for the proxy and a directory to hold Traefik files, including the ACME storage for certificates.

docker network create proxy
mkdir -p ~/traefik
cd ~/traefik
touch acme.json
chmod 600 acme.json

Step 4 — Create docker-compose.yml for Traefik v3

The compose file below configures Traefik v3 with automatic HTTPS via the HTTP-01 challenge (ports 80/443). It also binds the dashboard to localhost only for safety.

version: "3.9"

networks:
  proxy:
    external: true

services:
  traefik:
    image: traefik:v3.1
    container_name: traefik
    command:
      - --api.dashboard=true
      - --api.insecure=false
      - --providers.docker=true
      - --providers.docker.exposedbydefault=false
      - --entrypoints.web.address=:80
      - --entrypoints.websecure.address=:443
      - --entrypoints.web.http.redirections.entryPoint.to=websecure
      - --entrypoints.web.http.redirections.entryPoint.scheme=https
      - [email protected]
      - --certificatesresolvers.le.acme.storage=/letsencrypt/acme.json
      - --certificatesresolvers.le.acme.httpchallenge=true
      - --certificatesresolvers.le.acme.httpchallenge.entrypoint=web
      - --log.level=INFO
    ports:
      - "80:80"
      - "443:443"
      - "127.0.0.1:8080:8080" # Dashboard on localhost
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./acme.json:/letsencrypt/acme.json
    networks:
      - proxy
    restart: unless-stopped

Replace [email protected] with an email you control. Traefik uses it to register with Let’s Encrypt. The dashboard is available on http://localhost:8080 and can be reached via SSH tunneling when needed.

Step 5 — Start Traefik

Launch the reverse proxy and confirm it is running.

docker compose up -d
docker ps

You should see the traefik container healthy and listening on ports 80 and 443.

Step 6 — Add a Test App Behind Traefik

Deploy a simple test service (whoami) to verify automatic HTTPS, routing, and headers. Replace app.example.com with your real hostname that points to the server.

docker run -d --name whoami --network proxy --restart unless-stopped \
  -l "traefik.enable=true" \
  -l "traefik.http.routers.who.rule=Host(`app.example.com`)" \
  -l "traefik.http.routers.who.entrypoints=websecure" \
  -l "traefik.http.routers.who.tls.certresolver=le" \
  -l "traefik.http.middlewares.secHeaders.headers.stsSeconds=31536000" \
  -l "traefik.http.middlewares.secHeaders.headers.stsIncludeSubdomains=true" \
  -l "traefik.http.middlewares.secHeaders.headers.stsPreload=true" \
  -l "traefik.http.middlewares.secHeaders.headers.frameDeny=true" \
  -l "traefik.http.middlewares.ratelimit.rateLimit.average=100" \
  -l "traefik.http.middlewares.ratelimit.rateLimit.burst=50" \
  -l "traefik.http.routers.who.middlewares=secHeaders@docker,ratelimit@docker" \
  traefik/whoami:v1.10

Open https://app.example.com in a browser. The first request may take a few seconds while Traefik obtains a certificate. You should see a basic whoami page over HTTPS with a valid lock icon.

Optional: Use DNS-01 Challenge (Wildcard Certificates)

If your ISP or host blocks port 80, or you want wildcard certificates like *.example.com, switch to the DNS-01 challenge. The example below uses Cloudflare. Create a token in Cloudflare with DNS edit permissions for the zone and store it in an .env file.

cd ~/traefik
printf "CF_DNS_API_TOKEN=YOUR_CLOUDFLARE_DNS_TOKEN\n" > .env

Edit docker-compose.yml and replace the ACME lines for HTTP challenge with DNS challenge:

      - --certificatesresolvers.le.acme.dnschallenge=true
      - --certificatesresolvers.le.acme.dnschallenge.provider=cloudflare

Add the environment variable to the traefik service:

    environment:
      - CF_DNS_API_TOKEN=${CF_DNS_API_TOKEN}

Then reload Traefik:

docker compose up -d

For a wildcard, use routers or certificate domains that match *.example.com. With DNS challenge, Traefik can issue certificates without exposing port 80.

Secure the Dashboard (Optional but Recommended)

By default we bound the dashboard to localhost. To access it securely from your workstation, use an SSH tunnel: ssh -L 8080:localhost:8080 user@your_server and then open http://localhost:8080. If you must expose it on a domain, add Basic Auth and IP allowlisting via labels and serve it over HTTPS. For most setups, keeping it local is safer.

Troubleshooting

If certificates do not issue, confirm that your DNS A/AAAA records point to the server and that ports 80 and 443 are reachable from the internet. Check logs with docker logs -f traefik. Rate limits from Let’s Encrypt can apply if you redeploy too often; use a single domain during testing and switch to the production domain when stable.

Maintenance

Traefik renews certificates automatically before expiry; no cron is required. Keep Docker images current by pulling updates periodically: docker compose pull and docker compose up -d. Back up the acme.json file; it contains your issued certificates and keys.

What You Achieved

You now have a modern, production-ready reverse proxy with Traefik v3, automatic HTTPS via Let’s Encrypt, and a clean Docker-based workflow. Adding new apps is as simple as attaching them to the proxy network and setting a few labels. This pattern scales well, stays secure, and keeps your server easy to manage.

Overview

This guide shows you how to run a private, local AI chatbot on Ubuntu using Ollama (for running large language models on CPU or GPU) and Open WebUI (a clean web interface). You will be able to chat with models like Llama 3 locally, without sending data to the cloud. The steps work on Ubuntu 22.04 and 24.04, and are suitable for both desktops and servers.

What You Will Set Up

You will install Ollama on the host, pull a model, and run Open WebUI in Docker. Open WebUI will connect to the Ollama API on port 11434. The result is a self-hosted, secure, and fast AI assistant accessible at http://YOUR_SERVER_IP:3000.

Prerequisites

- Ubuntu 22.04 or 24.04 with sudo access
- At least 8 GB RAM (16 GB+ recommended for 7B–8B models; use smaller quantized models on low-RAM systems)
- Optional: NVIDIA or AMD GPU for faster inference
- Internet access to download packages and models

Step 1: Install Ollama

Ollama is a lightweight runtime that serves models locally over an HTTP API (default: 11434). Install it with one command:

curl -fsSL https://ollama.com/install.sh | sh

Verify the installation and service:

ollama --version
systemctl status ollama

If the service is inactive, start it:

sudo systemctl enable --now ollama

Step 2: Pull a Model (Llama 3.1 as an example)

Download a good general-purpose model. Llama 3.1 8B is a solid balance for many machines:

ollama pull llama3.1:8b

Test it in the terminal:

ollama run llama3.1:8b
>>> Write a 1-sentence productivity tip.

Tip: If you run out of memory, pull a quantized variant, for example:

ollama pull llama3.1:8b-instruct-q4_K_M

Step 3: Install Docker Engine

Open WebUI ships a reliable container image. Install Docker from the official repository:

sudo apt update
sudo apt install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" \
| sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker $USER

Log out and in again (or run a new shell) to use Docker without sudo.

Step 4: Run Open WebUI and Connect to Ollama

Start Open WebUI in Docker, mapping port 3000 and pointing it to the Ollama API on the host. The host gateway alias works on modern Docker versions:

docker run -d --name open-webui \
  --restart unless-stopped \
  -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
  -v open-webui:/app/backend/data \
  ghcr.io/open-webui/open-webui:latest

Open your browser at http://YOUR_SERVER_IP:3000, create an admin account on first launch, and select the default model (e.g., llama3.1:8b). You can now chat and manage prompts, files, and settings.

If the container cannot reach the host, an alternative is host networking:

docker run -d --name open-webui \
  --restart unless-stopped \
  --network=host \
  -e OLLAMA_BASE_URL=http://127.0.0.1:11434 \
  -v open-webui:/app/backend/data \
  ghcr.io/open-webui/open-webui:latest

Step 5: Secure Access

- By default, Open WebUI uses account-based sign-in. In Settings > Admin panel, require authentication for all users.
- If you run on a public server, restrict binding to localhost and use an SSH tunnel:

# bind only to localhost:
docker run -d --name open-webui \
  --restart unless-stopped \
  -p 127.0.0.1:3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
  -v open-webui:/app/backend/data \
  ghcr.io/open-webui/open-webui:latest

# from your laptop:
ssh -L 3000:localhost:3000 [email protected]

Step 6: Use Your GPU (Optional)

Ollama will use your GPU if supported drivers are installed. For NVIDIA, install the proprietary driver and CUDA libraries (the standard Ubuntu “Additional Drivers” tool works). For AMD, install ROCm per Ubuntu/AMD documentation. Because Open WebUI talks to Ollama’s API, you do not need GPU support inside the Open WebUI container—only in Ollama on the host.

Step 7: Model Management Tips

- List models: ollama list
- Show model metadata: ollama show llama3.1:8b
- Remove a model: ollama rm MODEL_NAME
- Try alternatives: ollama pull mistral, ollama pull qwen2, or ollama pull phi3:mini for lower memory systems.

Troubleshooting

- Port in use: Change the mapped port, for example -p 4000:8080 and open http://YOUR_SERVER_IP:4000.
- Container cannot reach Ollama: Use --network=host or ensure --add-host=host.docker.internal:host-gateway is present.
- Out-of-memory: Pull a smaller/quantized model (e.g., q4_K_M variants) or close other apps.
- Slow responses: Prefer GPU, reduce context length in Open WebUI settings, or choose a smaller model.

Updating and Maintenance

- Update Ollama: rerun the install script, then restart the service: curl -fsSL https://ollama.com/install.sh | sh && sudo systemctl restart ollama.
- Update Open WebUI:

docker pull ghcr.io/open-webui/open-webui:latest
docker stop open-webui && docker rm open-webui
# run again with the same docker run command used earlier

Conclusion

With Ollama and Open WebUI, you can host a private, fast, and flexible chatbot on your own Ubuntu machine. This setup keeps your data local, supports multiple open models, and can leverage your GPU for speed. Whether you are a helpdesk, a developer, or a power user, this self-hosted stack gives you full control over your AI workflow.

Overview

This step-by-step guide shows how to run private, local large language models with Ollama and a modern chat interface using Open WebUI. We will cover installing Ollama on Ubuntu and Windows, enabling GPU acceleration, pulling popular models like Llama 3, and deploying Open WebUI with Docker so you can chat, run tools, and manage prompts from a browser. The result is a fast, secure, and offline-friendly AI stack that you control.

Prerequisites

You will need a 64-bit system, administrator privileges, and at least 16 GB of RAM for 7B–8B models. GPU acceleration is recommended for speed: keep your NVIDIA/AMD/Intel graphics drivers up to date. Ollama listens on port 11434 by default, and Open WebUI will run on port 3000. Ensure your firewall allows local access or your chosen LAN range.

Step 1 — Install Ollama

Ubuntu 22.04/24.04: Install Ollama with the official script, which adds the service and keeps it updated.

curl -fsSL https://ollama.com/install.sh | sh
sudo systemctl enable --now ollama
curl http://127.0.0.1:11434/api/version

You should see a version string from the last command. If not, check the service: sudo systemctl status ollama.

Windows 11/10: Install via the official MSI or Winget, then verify the local API.

winget install Ollama.Ollama
curl http://127.0.0.1:11434/api/version

On Windows, Ollama runs as a user service. If you use a third-party firewall, allow local traffic to port 11434.

Step 2 — Enable GPU Acceleration

GPU acceleration in Ollama is automatic when compatible drivers and runtimes are present. On Linux, install your vendor’s proprietary GPU driver. On Windows, use the latest Game Ready/Studio driver from the GPU vendor. After pulling a model and making a test prompt, watch the Ollama logs. If the run mentions the GPU and performance is high (tokens per second are significantly better than CPU), acceleration is working.

If you suspect CPU fallback, update drivers, make sure your GPU has enough VRAM for the chosen model size, and try a smaller variant (for example, 8B instead of 13B). On laptops with hybrid graphics, set the app/GPU preferences so Ollama can use the discrete GPU.

Step 3 — Pull a Model and Test Locally

Pull a model using the Ollama CLI. Popular, high-quality choices include Llama 3 and Mistral. The first run downloads and prepares weights; subsequent runs start instantly.

# Examples (pick one)
ollama pull llama3:8b
ollama pull llama3.1:8b
ollama pull mistral:7b

Now run a quick prompt:

ollama run llama3:8b
# At the prompt, type:
# What are three creative use cases for local AI at home?

If responses are slow or you see out-of-memory errors, switch to a smaller model or close GPU-intensive applications.

Step 4 — Deploy Open WebUI with Docker

Open WebUI adds a polished browser interface, prompt library, chat history, and extensions like RAG (retrieve and ground answers in your documents). We will connect it to your host’s Ollama instance. The following Docker Compose works on Linux and Windows. It uses host.docker.internal to reach the host-based Ollama API and maps persistent storage for Open WebUI data.

mkdir -p ~/openwebui && cd ~/openwebui
cat > docker-compose.yml <<'YAML'
services:
  openwebui:
    image: ghcr.io/open-webui/open-webui:latest
    container_name: open-webui
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://host.docker.internal:11434
    extra_hosts:
      - "host.docker.internal:host-gateway"
    volumes:
      - open-webui-data:/app/backend/data
    restart: unless-stopped
volumes:
  open-webui-data:
YAML

docker compose up -d

Open your browser and visit http://localhost:3000. Create your account when prompted, pick your default model (for example, llama3:8b), and send a test message. If the UI cannot connect, ensure the Ollama service is running and that your firewall allows local connections to port 11434.

Optional — Run Both Ollama and Open WebUI in Docker

If you prefer everything containerized, you can run Ollama and Open WebUI in the same Compose file. This is convenient on servers. GPU pass-through in Docker requires recent drivers and, on Linux, the NVIDIA Container Toolkit. When in doubt, keep Ollama native and only containerize Open WebUI, as shown above.

Security, Updates, and Backups

Do not expose ports 11434 or 3000 directly to the internet. If you need remote access, place Open WebUI behind a reverse proxy (Nginx, Caddy, or Traefik) with HTTPS and strong authentication, or publish it through a zero-trust tunnel. Inside Open WebUI, enable authentication and limit registration to trusted users. Keep Docker images current by pulling the latest tags and recreating containers. On Ubuntu, the Ollama installer provides updates via its repository; on Windows, check for updates in the app or Winget. Back up ~/.ollama (models and configs) and your open-webui-data volume to preserve chat history and settings.

Troubleshooting

If Open WebUI says “Cannot connect to Ollama,” verify the API at http://127.0.0.1:11434/api/version and confirm your Compose file includes extra_hosts with host-gateway on Linux. On Windows with Docker Desktop, host.docker.internal works out of the box. If GPU acceleration is missing, update drivers, reboot, and try a smaller model. When Docker containers fail to start, check logs with docker logs open-webui and make sure ports 3000 and 11434 are not in use by other applications.

What You Can Do Next

With Ollama and Open WebUI running, you can add multiple models, create custom system prompts, and enable RAG by uploading PDFs or notes so the model answers with context from your documents. You can also script batch prompts via the Ollama HTTP API, integrate with automation tools, or point a browser extension to your local endpoint to replace cloud calls. The stack is private, fast, and easy to maintain—ideal for personal knowledge work or secure team deployments.

Overview

This tutorial shows how to self-host Open WebUI with Ollama on Ubuntu using Docker Compose. You will get a clean, repeatable setup that runs on CPU or GPU, stores model data persistently, and can be upgraded with a single command. Open WebUI provides a modern interface, while Ollama runs local large language models such as Llama 3, Mistral, Phi-3, and CodeLlama.

Prerequisites

You will need a fresh Ubuntu 22.04 or 24.04 server (cloud VM or local machine), a user with sudo rights, and at least 8 GB of RAM. If you plan to use a GPU, an NVIDIA card is recommended. Open ports 3000 (Web UI) and 11434 (Ollama API) on your firewall or security group.

1) Install Docker and Compose

Install the official Docker Engine and the Compose plugin on Ubuntu. Log out and back in (or run newgrp) after adding your user to the docker group.

sudo apt update
sudo apt install -y ca-certificates curl gnupg

sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | \
  sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg

echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
  https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo $VERSION_CODENAME) stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

sudo usermod -aG docker $USER
newgrp docker
docker --version
docker compose version

2) Optional: Enable NVIDIA GPU Support

If you have an NVIDIA GPU, install the NVIDIA driver and the NVIDIA Container Toolkit. This lets Ollama use your GPU for faster inference.

sudo apt install -y ubuntu-drivers-common
sudo ubuntu-drivers autoinstall
sudo reboot

After the reboot, install the container toolkit and configure Docker:

distribution=$(. /etc/os-release; echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
  sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit.gpg

curl -fsSL https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt update
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Verify GPU access with Docker:

docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi

3) Create the Docker Compose Stack

Create a project directory and a Compose file that defines two services: Ollama (LLM runtime) and Open WebUI (frontend). The volumes preserve your models and settings across restarts.

mkdir -p ~/openwebui-ollama
cd ~/openwebui-ollama
nano docker-compose.yml

Paste the following content and save:

version: "3.9"

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    ports:
      - "11434:11434"
    volumes:
      - ollama:/root/.ollama
    environment:
      - OLLAMA_KEEP_ALIVE=24h
    # Uncomment the next line if you have GPU configured:
    # gpus: all

  open-webui:
    image: ghcr.io/open-webui/open-webui:latest
    container_name: open-webui
    depends_on:
      - ollama
    restart: unless-stopped
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
    volumes:
      - open-webui:/app/backend/data

volumes:
  ollama:
  open-webui:

4) Start the Services and Download a Model

Bring the stack online. The first start will download container images.

docker compose up -d
docker compose ps

Pull a small model to test. You can add more models later.

docker exec -it ollama ollama pull llama3.2:3b
# Other options: mistral:7b, phi3:mini, qwen2.5:7b

Open your browser to http://SERVER-IP:3000. Create your account in Open WebUI. In the model selector, choose the model you pulled and send a prompt to verify everything works.

5) Persist Data and Backups

Docker volumes keep your models and UI data under /var/lib/docker/volumes. To back them up, stop the stack and archive the data directories. This ensures quick recovery after an OS reinstall or server migration.

docker compose down
sudo tar -czf ollama_data.tgz -C /var/lib/docker/volumes \
  $(docker volume ls -q | grep "_ollama$")/_data

sudo tar -czf openwebui_data.tgz -C /var/lib/docker/volumes \
  $(docker volume ls -q | grep "_open-webui$")/_data

docker compose up -d

6) Secure and Publish (Optional)

If you expose the service on the internet, put it behind a reverse proxy with HTTPS (Caddy, Nginx, or Traefik) and set strong authentication in Open WebUI. Use a DNS name, issue a TLS certificate (Let’s Encrypt), and restrict access with IP allowlists or an identity provider. For small teams, consider running it only on a private network or VPN.

7) Update and Maintenance

Update to the newest images regularly. This pulls security updates, new UI features, and performance improvements.

cd ~/openwebui-ollama
docker compose pull
docker compose up -d

To update models to the latest quantizations or fixes, re-pull them in Ollama. You can remove old ones you no longer need.

docker exec -it ollama ollama pull mistral:7b
docker exec -it ollama ollama list
docker exec -it ollama ollama rm modelname:tag

Troubleshooting

Port already in use: Change the host port mappings in docker-compose.yml (for example, 3001:8080 or 11435:11434) and restart.

GPU not detected: Verify nvidia-smi works on the host and in a test container. Ensure the gpus: all line is uncommented and Docker was restarted after installing the NVIDIA Toolkit.

Slow or failed model pulls: Models can be large. Check disk space (df -h), network speed, and try a smaller model first. You can also mirror models by pre-downloading on another machine and copying the volume data.

Permission errors: Ensure your user is in the docker group (id) and you have logged out/in.

What You Achieved

You now have a production-friendly, self-hosted AI chat stack powered by Open WebUI and Ollama. With Docker Compose, you can start, stop, back up, and upgrade the entire setup with a couple of commands. Add or swap models as your use cases evolve—coding assistants, knowledge chat, or creative writing—while keeping your data local and under your control.

This step-by-step guide shows you how to run a fast, private, and local AI chatbot on Ubuntu 22.04 or 24.04 using Ollama and Open WebUI. You will install the Ollama runtime, pull a modern large language model, and add a clean chat interface via Open WebUI in Docker. Optional steps cover NVIDIA GPU acceleration, API usage, and persistence. The result is a secure, offline-friendly setup suitable for helpdesk, coding assistance, or knowledge base querying without sending data to the cloud.

Why Ollama + Open WebUI

Ollama makes it simple to run and manage open-source LLMs locally (Llama 3.x, Mistral, Phi, Qwen, and more). Open WebUI adds a user-friendly, browser-based chat interface with conversation history, prompt templates, and multi-model support. Together they form a robust, low-maintenance local AI stack for Linux desktops and servers.

Prerequisites

- Ubuntu 22.04 or 24.04 with a non-root sudo user.
- At least 8 GB RAM (16 GB recommended for larger models).
- Optional NVIDIA GPU for acceleration (T4/RTX/RTX A-series, etc.).
- Internet access to download models and containers.

Step 1 — Install Ollama

1) Update packages:
sudo apt update && sudo apt install -y curl ca-certificates
2) Install Ollama:
curl -fsSL https://ollama.com/install.sh | sh
3) Enable as a service:
sudo systemctl enable --now ollama
4) Verify the API is up:
curl http://localhost:11434/api/tags
If you see JSON, Ollama is running correctly.

Step 2 — Pull and test a model

Pull a compact, capable model first to validate your setup:
ollama pull llama3.2
Run an interactive test:
ollama run llama3.2
Type a prompt, then press Ctrl+C to exit. You can later try larger models (for example, ollama pull mistral or ollama pull llama3.1), but start small to confirm everything works.

Step 3 — Install Docker Engine

1) Add Docker’s repo key and source:
sudo apt-get update && sudo apt-get install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(. /etc/os-release; echo $UBUNTU_CODENAME) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
2) Install Docker and the Compose plugin:
sudo apt-get update && sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
3) Add your user to the docker group and refresh your shell:
sudo usermod -aG docker $USER
newgrp docker

Step 4 — Run Open WebUI connected to Ollama

Start Open WebUI and point it to the Ollama API on the host. The --add-host flag maps host.docker.internal to your host’s gateway so the container can reach http://localhost:11434 on the host:

docker run -d --name open-webui --restart=unless-stopped -p 3000:8080 \
--add-host=host.docker.internal:host-gateway \
-e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
-v open-webui:/app/backend/data \
ghcr.io/open-webui/open-webui:main

Open your browser to http://SERVER_IP:3000 (or http://localhost:3000). Create the first admin account, choose a model (for example, llama3.2), and start chatting.

Step 5 — Enable NVIDIA GPU acceleration (optional)

1) Install the latest NVIDIA driver for your GPU using Ubuntu’s Additional Drivers or apt. Reboot if prompted.
2) Install the NVIDIA Container Toolkit so Docker can access the GPU:
distribution=$(. /etc/os-release; echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#' | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update && sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
3) Recreate Open WebUI with GPU access:
docker rm -f open-webui
docker run -d --name open-webui --restart=unless-stopped --gpus all -p 3000:8080 \
--add-host=host.docker.internal:host-gateway \
-e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
-v open-webui:/app/backend/data \
ghcr.io/open-webui/open-webui:main
4) Ollama will also use the GPU automatically when a compatible model is loaded. You can confirm GPU use with nvidia-smi during inference.

Step 6 — Use the Ollama HTTP API

You can script local inference via HTTP without the UI. Example generation request:
curl http://localhost:11434/api/generate -d '{"model":"llama3.2","prompt":"Write a haiku about backups."}'
Chat format with memory:
curl http://localhost:11434/api/chat -d '{"model":"llama3.2","messages":[{"role":"user","content":"Explain DNS in one sentence."}]}'

Step 7 — Persistence, autostart, and updates

- Ollama models are stored under ~/.ollama/models. Back up this directory to avoid re-downloading models.
- The Open WebUI container uses a named volume (open-webui) for its data, which persists across restarts.
- Ollama is already set to start at boot (systemctl enable ollama). The WebUI container uses --restart=unless-stopped so it will auto-start after a reboot.
- Update Ollama: curl -fsSL https://ollama.com/install.sh | sh
- Update Open WebUI: docker pull ghcr.io/open-webui/open-webui:main && docker restart open-webui

Troubleshooting

- Open WebUI cannot connect to Ollama: ensure you used --add-host=host.docker.internal:host-gateway and that curl http://localhost:11434/api/tags works on the host.
- Port already in use: change -p 3000:8080 to a different host port like -p 3333:8080.
- Out of memory or slow responses: try a smaller model (for example, llama3.2 or phi3). Close other apps or add swap. For CPU-only hosts, expect slower performance on large models.
- GPU not used: verify drivers, nvidia-smi, and that the container runs with --gpus all. Pull a GPU-optimized model variant if available.

What you can do next

- Connect knowledge bases or documents using Open WebUI’s RAG features to power local search over PDFs and wikis.
- Add multiple models and switch per chat, benchmarking speed and quality.
- Put Nginx or Caddy in front of :3000 for HTTPS and trusted network access.
- Automate prompts with shell scripts or Python by calling the local Ollama API.

You now have a private, local AI assistant on Ubuntu with a clean web interface, GPU-ready acceleration, and a stable upgrade path—all without sending your data to third-party services.

Overview

This step-by-step guide shows how to deploy a private, local AI stack on Ubuntu using Docker: Ollama for running large language models (LLMs) and Open WebUI as a fast, friendly chat interface. The setup works on CPUs and supports NVIDIA GPUs for acceleration. You will get a secure, self-hosted environment where you can run models like Llama 3.2, Phi-4, and Mistral without sending data to the cloud.

Prerequisites

- Ubuntu 22.04 or 24.04 (server or desktop)
- 16 GB RAM recommended (more for larger models), 30+ GB free disk
- Docker Engine and the Docker Compose plugin
- Optional: NVIDIA GPU with proprietary driver installed (e.g., 535+)

1) Install Docker and Docker Compose

Update your system and install Docker from the official repository for best stability and performance.

sudo apt update && sudo apt install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" | sudo tee /etc/apt/sources.list.d/docker.list >/dev/null
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker $USER && newgrp docker

2) (Optional) Enable NVIDIA GPU in Containers

If you have an NVIDIA GPU, install the NVIDIA Container Toolkit so Docker can pass the GPU into containers:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#' | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update && sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Verify your driver with nvidia-smi. The container will get GPU access when you run it with --gpus all.

3) Create a Docker Network and Volumes

Create a dedicated network so services can talk by name and set up persistent storage:

docker network create llmnet
docker volume create ollama
docker volume create open-webui

4) Run the Ollama Container

Start Ollama. For CPU-only:

docker run -d --name ollama --restart=unless-stopped --network llmnet -p 11434:11434 -v ollama:/root/.ollama ollama/ollama:latest

With NVIDIA GPU acceleration (detected automatically):

docker run -d --name ollama --restart=unless-stopped --network llmnet --gpus all -p 11434:11434 -v ollama:/root/.ollama ollama/ollama:latest

5) Pull a Model

You can manage models from the host via docker exec. Pull a lightweight model to start quickly:

docker exec -it ollama ollama pull llama3.2:3b

Test generation from the command line:

curl http://localhost:11434/api/generate -d '{"model":"llama3.2:3b","prompt":"Say hello in one short line."}'

6) Launch Open WebUI

Open WebUI provides a clean chat interface and model manager. Start it on port 3000 and point it to the Ollama endpoint:

docker run -d --name open-webui --restart=unless-stopped --network llmnet -p 3000:8080 -e OLLAMA_BASE_URL=http://ollama:11434 -v open-webui:/app/backend/data ghcr.io/open-webui/open-webui:latest

Open a browser and visit http://localhost:3000 (or your server IP). Create the first admin account, select a model (e.g., llama3.2:3b), and start chatting. If a model is missing, Open WebUI can pull it automatically via Ollama.

7) Optional: Use Docker Compose

Prefer to keep everything in a single file? Create docker-compose.yml in an empty folder and paste:

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    ports:
     - "11434:11434"
    networks: [llmnet]
    volumes:
     - ollama:/root/.ollama
  open-webui:
    image: ghcr.io/open-webui/open-webui:latest
    container_name: open-webui
    restart: unless-stopped
    ports:
     - "3000:8080"
    environment:
     - OLLAMA_BASE_URL=http://ollama:11434
    networks: [llmnet]
    volumes:
     - open-webui:/app/backend/data
networks:
  llmnet:
    external: true
volumes:
  ollama:
  open-webui:

Start with docker compose up -d. For GPU, prefer the docker run method with --gpus all, or adapt your Compose file using a GPU-capable configuration on your system.

8) Securing and Updating

- Restrict access: if running on a server, firewall ports 11434 and 3000 to trusted IPs.
- Reverse proxy: place Nginx or Caddy in front with HTTPS for remote access.
- Updates: pull newer images and recreate containers: docker pull ollama/ollama:latest && docker pull ghcr.io/open-webui/open-webui:latest, then docker stop and docker rm containers and re-run them. Your data persists in the volumes.

9) Troubleshooting

- Check logs: docker logs -f ollama and docker logs -f open-webui.
- Port in use: change published ports (e.g., -p 3001:8080).
- GPU not detected: validate nvidia-smi, reinstall the NVIDIA Container Toolkit, and ensure --gpus all is present.
- Disk space: models are large; prune unused data with docker system prune and remove models in ollama volume if needed.

10) Quick API and CLI Examples

- Pull another model: docker exec -it ollama ollama pull phi4:latest
- Chat from CLI: docker exec -it ollama ollama run mistral:7b
- Simple REST call: curl http://localhost:11434/api/generate -d '{"model":"phi4:latest","prompt":"Give me two bullet points about container security."}'

You now have a modern, private AI stack using Docker, Ollama, and Open WebUI on Ubuntu. It is fast, flexible, and ready for local development, internal knowledge assistants, and offline experimentation—no cloud required.

Local large language models (LLMs) are now practical for developers, researchers, and privacy-focused teams. In this step-by-step guide, you will install and run Ollama (LLM runtime) and Open WebUI (a modern chat interface) on Ubuntu 22.04/24.04, with optional NVIDIA GPU acceleration. The setup uses Docker for easy updates, isolation, and backups.

Why this stack?

Ollama makes downloading and running models simple, offering a fast API on your machine. Open WebUI provides a sleek, extensible web app for chatting with multiple models, managing prompts, and moderating access. Together, they create a private, cost-effective alternative to cloud AI services.

Prerequisites

You need an Ubuntu 22.04/24.04 system with internet access and a user with sudo rights. If you have an NVIDIA GPU (recommended), you can enable GPU acceleration for much faster inference. CPU-only works too—skip the GPU steps if you do not have a supported GPU.

Step 1: Update the system

Update packages to ensure you have the latest dependencies and security fixes.

sudo apt update && sudo apt -y upgrade
sudo reboot

Step 2 (Optional): Enable NVIDIA GPU support

Install the latest proprietary NVIDIA driver and the container toolkit so Docker can use your GPU. Reboot when asked.

# Install recommended NVIDIA driver
sudo ubuntu-drivers install
sudo reboot

# Install NVIDIA Container Toolkit
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#' | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list > /dev/null

sudo apt update
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

# Verify GPU visibility
nvidia-smi

Step 3: Install Docker Engine and Docker Compose plugin

Install Docker from the official repository to get the latest stable version. Add your user to the docker group to run Docker without sudo.

sudo apt install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | \
sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

sudo usermod -aG docker $USER
newgrp docker

# Quick test
docker run --rm hello-world

Step 4: Create a Docker Compose file for Ollama + Open WebUI

Create a working directory and define services. The configuration below enables GPU when present; for CPU-only, remove the gpus: all line under the Ollama service.

mkdir -p ~/local-llm && cd ~/local-llm
nano docker-compose.yml

services:
  ollama:
    container_name: ollama
    image: ollama/ollama:latest
    restart: unless-stopped
    ports:
      - "11434:11434"
    volumes:
      - ollama:/root/.ollama
    # Comment the next line if you are CPU-only
    gpus: all
    environment:
      - OLLAMA_KEEP_ALIVE=24h

  open-webui:
    container_name: open-webui
    image: ghcr.io/open-webui/open-webui:latest
    depends_on:
      - ollama
    restart: unless-stopped
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
      - WEBUI_SECRET_KEY=change_this_long_random_secret
    volumes:
      - openwebui:/app/backend/data

volumes:
  ollama:
  openwebui:

Step 5: Start the stack

Bring the services up in the background, then check their status. The Open WebUI will be available at http://SERVER_IP:3000 and the Ollama API at http://SERVER_IP:11434.

docker compose up -d
docker compose ps

Step 6: Pull and run a model

Use Ollama to pull an LLM. You can pick models like llama3.2, mistral, or a coding model. The first pull downloads model weights, which can be several GB.

# Pull a general-purpose model
docker exec -it ollama ollama pull llama3.2

# Test it via CLI
docker exec -it ollama ollama run llama3.2 "Write a two-sentence summary of Ubuntu."

# Or use the API
curl http://localhost:11434/api/generate -d '{"model":"llama3.2","prompt":"Hello!"}'

Open your browser to http://SERVER_IP:3000, select the model from the dropdown, and start chatting. In Settings, you can change default models, system prompts, and appearance.

Step 7: Secure access

If exposing the WebUI beyond your LAN, add authentication. In Open WebUI, create an admin user at first login, then disable new signups in Settings. For internet exposure, place a reverse proxy (Nginx, Caddy, or Traefik) with HTTPS (Let’s Encrypt) in front of port 3000.

Step 8: Update and backup

To update, pull the latest images and recreate containers without losing data stored in volumes. To back up, save the volumes before upgrades.

# Update images
docker compose pull
docker compose up -d

# Backup volumes (example)
docker run --rm -v local-llm_ollama:/data -v "$PWD":/backup \
  busybox tar czf /backup/ollama-vol.tgz /data

docker run --rm -v local-llm_openwebui:/data -v "$PWD":/backup \
  busybox tar czf /backup/openwebui-vol.tgz /data

Troubleshooting

GPU not used: Run docker run --rm --gpus all nvidia/cuda:12.3.2-base-ubuntu22.04 nvidia-smi. If it fails, recheck the driver and NVIDIA Container Toolkit steps. Ensure the gpus: all line is present and Docker was restarted.

Permission denied with Docker: You may need to log out and back in after adding your user to the docker group, or run newgrp docker.

Port conflicts: Change the left side of port mappings in docker-compose.yml (e.g., use "8081:8080" for WebUI) and restart.

Slow or failed model pull: Verify disk space and retry. Large models require several GB of free space in the ollama volume.

Uninstall (optional)

To stop and remove everything, run:

cd ~/local-llm
docker compose down
docker volume rm local-llm_ollama local-llm_openwebui

Wrap-up

You have a modern local AI stack: Ollama for fast model serving and Open WebUI for a friendly, multi-model chat interface. With Docker, updates are quick and backups are simple. Add your favorite models, tune system prompts, and integrate the Ollama API into your apps—all without sending data to the cloud.

Overview

This guide shows how to self-host a private AI chatbot with Open WebUI (a clean, ChatGPT-like interface) and Ollama (for running local large language models) on Ubuntu 22.04 or 24.04. Everything runs in Docker, secured with HTTPS via Caddy and optional Basic Auth. If you have an NVIDIA GPU, you can enable GPU acceleration to speed up model inference dramatically.

What you will need

- An Ubuntu 22.04/24.04 server with at least 8 GB RAM and 20 GB free disk space. For GPU acceleration, an NVIDIA GPU with recent drivers is recommended (e.g., 8 GB VRAM or more for larger models).

- A domain name pointing to your server’s public IP (A/AAAA record). Ports 80 and 443 should be open to the internet for Let’s Encrypt.

- A non-root user with sudo privileges.

Step 1 — Install Docker and Docker Compose plugin

Update your system and install Docker from the official repository:

sudo apt update && sudo apt install -y ca-certificates curl gnupg

sudo install -m 0755 -d /etc/apt/keyrings

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg

echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt update && sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

sudo usermod -aG docker $USER && newgrp docker

Step 2 — (Optional) Enable NVIDIA GPU for containers

Install the NVIDIA driver (if not already installed) and the NVIDIA container toolkit so Docker can access your GPU.

sudo ubuntu-drivers install (or choose a specific driver, e.g., sudo apt install -y nvidia-driver-535)

sudo reboot

Install the container toolkit:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit.gpg

curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit.gpg] https://#' | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt update && sudo apt install -y nvidia-container-toolkit

sudo nvidia-ctk runtime configure --runtime=docker

sudo systemctl restart docker

Test GPU access:

docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

Step 3 — Prepare Docker Compose and Caddy

Create a project folder and move into it:

mkdir -p ~/ai-stack && cd ~/ai-stack

Create a file named docker-compose.yml with the following content (replace your.domain.com later in Caddyfile):

services:
ollama:
image: ollama/ollama:latest
container_name: ollama
volumes:
- ollama:/root/.ollama
environment:
- OLLAMA_KEEP_ALIVE=2h
ports:
- "127.0.0.1:11434:11434"
restart: unless-stopped
# Uncomment the next line if you enabled NVIDIA toolkit
# gpus: all

openwebui:
image: ghcr.io/open-webui/open-webui:main
container_name: openwebui
depends_on:
- ollama
environment:
- OLLAMA_BASE_URL=http://ollama:11434
- WEBUI_AUTH=True
ports:
- "127.0.0.1:8080:8080"
volumes:
- openwebui:/app/backend/data
restart: unless-stopped

caddy:
image: caddy:2
container_name: caddy
depends_on:
- openwebui
restart: unless-stopped
ports:
- "80:80"
- "443:443"
volumes:
- ./Caddyfile:/etc/caddy/Caddyfile:ro
- caddy_data:/data
- caddy_config:/config

volumes:
ollama:
openwebui:
caddy_data:
caddy_config:

Create a file named Caddyfile in the same folder. Replace your.domain.com with your real domain and the email with yours:

your.domain.com {
encode zstd gzip
tls [email protected]
# Optional Basic Auth — generate a hashed password below and uncomment
# basicauth {
# admin <paste_hashed_password_here>
# }
reverse_proxy openwebui:8080
}

If you want Basic Auth, generate a hash:

docker run --rm caddy:2 caddy hash-password --plaintext "StrongPassword!"

Copy the hash output, paste it into the Caddyfile under basicauth, and uncomment the lines.

Step 4 — Start the stack and pull a model

Start the services:

docker compose up -d

Pull a model with Ollama. Llama 3.1 is a great default; you can also choose smaller variants if you have less VRAM:

docker exec -it ollama ollama pull llama3.1

For low VRAM systems, try a quantized build like llama3.1:8b-instruct-q4_0 or a compact model like mistral:7b-instruct:

docker exec -it ollama ollama pull mistral:latest

Verify Ollama is up:

curl -s http://127.0.0.1:11434/api/tags

Step 5 — Access Open WebUI over HTTPS

Wait 30–60 seconds for Caddy to obtain a Let’s Encrypt certificate. Then browse to https://your.domain.com. On the first visit, create your Open WebUI admin user. In Settings > Models, select the model you pulled with Ollama. You can now chat privately with your local LLM through a friendly web interface.

Step 6 — Security hardening (recommended)

- Keep Open WebUI behind Caddy only. We already published it on localhost (127.0.0.1) to prevent direct exposure.

- Enable Basic Auth in your Caddyfile if you plan to expose the site to the open internet. Use a long, unique password.

- Restrict admin features in Open WebUI to your own account. Disable public sign-ups if you do not need them.

- Consider a firewall rule to allow inbound 80/443 only, and block 8080/11434 from the WAN.

Step 7 — Backups and updates

Back up Open WebUI data:

docker run --rm -v openwebui:/d -v $PWD:/b busybox tar czf /b/openwebui-backup.tgz -C /d .

Back up Ollama models (can be large):

docker run --rm -v ollama:/d -v $PWD:/b busybox tar czf /b/ollama-backup.tgz -C /d .

To update containers:

docker compose pull && docker compose up -d

To remove old images:

docker image prune -f

Troubleshooting

- Check logs if something fails to start: docker compose logs -f

- Verify DNS and port 80/443 reach the server; Let’s Encrypt must connect over HTTP/HTTPS the first time.

- If certificates fail, restart the stack after DNS propagates: docker compose down && docker compose up -d

- If the GPU is not detected, confirm nvidia-smi works on the host and that you added gpus: all under the Ollama service.

- Test the Ollama API locally: curl http://127.0.0.1:11434/api/generate -d '{"model":"llama3.1","prompt":"hi"}'

Where to go next

Explore model variants optimized for your hardware (Q4 for low VRAM, Q6/Q8 for higher quality, FP16 on strong GPUs). Add embeddings and RAG features in Open WebUI to chat over your documents. With this setup, you keep your data and traffic on your own server, with clean HTTPS, optional password protection, and fast local inference.

Overview

Running large language models (LLMs) locally is easier and faster than ever with Ollama and Open WebUI. In this tutorial, you will deploy both on Docker with optional NVIDIA GPU acceleration on Ubuntu 22.04/24.04. We will cover prerequisites, the Docker Compose file, model download, security tips, updates, and troubleshooting. This guide uses simple language and SEO-friendly steps so you can get a private AI assistant running in minutes.

Prerequisites

Before you start, ensure you have: (1) Ubuntu 22.04 or 24.04 with sudo access, (2) Docker Engine and Docker Compose v2, (3) an NVIDIA GPU with proprietary drivers installed (optional but recommended), (4) at least 16 GB RAM and 20+ GB free disk for models, and (5) an open firewall port 3000 (Open WebUI) and 11434 (Ollama) if accessed remotely.

Step 0: Verify NVIDIA drivers (GPU users)

If you plan to use GPU acceleration, install the latest NVIDIA driver and verify it works. Run: nvidia-smi. You should see your GPU listed with a driver version. If the command is missing, install the driver using: sudo ubuntu-drivers autoinstall, reboot, then test nvidia-smi again.

Step 1: Install Docker Engine and Compose

Install Docker from the official repository for the best compatibility. Example quick setup:
1) sudo apt update && sudo apt install -y ca-certificates curl gnupg
2) sudo install -m 0755 -d /etc/apt/keyrings
3) curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
4) echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $UBUNTU_CODENAME) stable" | sudo tee /etc/apt/sources.list.d/docker.list
5) sudo apt update && sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
6) Optional: sudo usermod -aG docker $USER and re-login to run Docker without sudo.

Step 2: Install NVIDIA Container Toolkit (GPU users)

This step lets Docker containers access your GPU. Run:
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update && sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
Test GPU inside Docker: docker run --rm --gpus all nvidia/cuda:12.2.0-base-ubuntu22.04 nvidia-smi. If you see your GPU, you are ready.

Step 3: Create the Docker Compose file

Create a project directory, for example: mkdir -p ~/ollama-openwebui && cd ~/ollama-openwebui. Then create docker-compose.yml with the following content. This setup persists models and WebUI data, exposes ports, and enables GPU if available.

version: "3.8"
services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    volumes:
      - ollama:/root/.ollama
    ports:
      - "11434:11434"
    gpus: "all"

  open-webui:
    image: ghcr.io/open-webui/open-webui:latest
    container_name: open-webui
    restart: unless-stopped
    depends_on:
      - ollama
    environment:
      - OLLAMA_API_BASE_URL=http://ollama:11434
    ports:
      - "3000:8080"
    volumes:
      - openwebui:/app/backend/data

volumes:
  ollama:
  openwebui:

Notes: (a) The line gpus: "all" enables acceleration when the NVIDIA toolkit is present; Docker Compose will map this to --gpus all. (b) If you do not have a GPU, simply leave the file as-is; the Ollama image will run on CPU automatically. (c) Ports 11434 and 3000 can be changed to fit your environment or firewall rules.

Step 4: Start the stack

From the project directory, run docker compose up -d. Wait a few seconds, then confirm both containers are healthy with docker compose ps and view logs with docker logs -f ollama or docker logs -f open-webui.

Step 5: Pull a model and test

Ollama downloads models on demand. You can preload a model via the container: docker exec -it ollama ollama pull llama3.2:3b. Smaller models (2B–7B parameters) are faster and use less RAM; larger models offer higher quality but need more resources. After the pull finishes, open your browser to http://<server-ip>:3000, create your first account, and in Open WebUI select the model (for example, llama3.2:3b) to start chatting.

Step 6: Secure access

By default, Open WebUI allows signups. After creating your admin account, you can restrict access. Edit docker-compose.yml under the open-webui service and add: ENABLE_SIGNUP=false in the environment section, then run docker compose up -d again. For internet exposure, put Open WebUI behind a reverse proxy (Nginx, Caddy, or Traefik), enable HTTPS with Let’s Encrypt, and consider firewalling or a zero-trust tunnel (Cloudflare/Tailscale) for extra protection.

Maintenance and updates

To update containers without deleting your data: docker compose pull followed by docker compose up -d. To update or remove models: docker exec -it ollama ollama pull <model:tag> and docker exec -it ollama ollama rm <model:tag>. To stop the stack: docker compose down. To remove everything including volumes, add -v, but this deletes models and WebUI data.

Troubleshooting

If Open WebUI cannot see Ollama, confirm the internal URL is correct: OLLAMA_API_BASE_URL=http://ollama:11434. Check container connectivity with docker exec open-webui wget -qO- http://ollama:11434/api/tags.

If GPU is not detected, verify the NVIDIA toolkit: run docker run --rm --gpus all nvidia/cuda:12.2.0-base-ubuntu22.04 nvidia-smi. If that works but the Ollama container is still CPU-only, confirm the gpus: "all" line is present, Docker was restarted after nvidia-ctk runtime configure, and the driver version is compatible with your GPU.

If downloads are slow or interrupted, restart the Ollama container and try again. You can set an alternative registry mirror for Docker to improve pull speeds. Also verify free disk space with df -h because model files can be large.

What you achieved

You now have a private, local AI stack powered by Ollama and Open WebUI, running in Docker with persistent storage and optional GPU acceleration. This architecture is easy to back up, trivial to update, and flexible enough to host multiple models. Add a reverse proxy for TLS, schedule backups of the ollama and openwebui volumes, and explore advanced features like embedding, RAG, and function calling as you grow your setup.

Overview

In this step-by-step guide, you will learn how to self-host a local AI chat environment on Ubuntu using Ollama and Open WebUI. Ollama runs large language models (LLMs) locally and exposes a simple API, while Open WebUI provides a modern browser interface, chat history, and prompt management. This tutorial targets Ubuntu 22.04/24.04 and shows how to enable NVIDIA GPU acceleration, secure the service, and test the API.

Prerequisites

You need an Ubuntu 22.04/24.04 machine with at least 16 GB RAM for comfortable use and an NVIDIA GPU with recent drivers (525+ recommended) if you want hardware acceleration. For CPU-only usage, Ollama still works but will be slower. You also need a user with sudo privileges and internet access.

Step 1: Update the system and install basics

Start by refreshing your package lists and installing useful tools such as curl and ufw. Run: sudo apt update && sudo apt -y upgrade and then sudo apt -y install curl ca-certificates ufw. This ensures you have the latest security updates and a firewall ready to configure later.

Step 2: Verify NVIDIA GPU (optional but recommended)

If you intend to use GPU acceleration, confirm your NVIDIA driver installation. Run nvidia-smi. If the command shows your GPU and driver version, you are ready. If not, install a recommended driver with sudo ubuntu-drivers autoinstall, reboot using sudo reboot, and check again with nvidia-smi. Ollama includes the runtime pieces it needs and will automatically use your GPU when supported.

Step 3: Install Ollama

Ollama provides a one-line installer for Linux. Run: curl -fsSL https://ollama.com/install.sh | sh. This installs the ollama binary and sets up a systemd service called ollama. After the script completes, verify the service with systemctl status ollama. If it is not running, start it using sudo systemctl start ollama and enable it at boot with sudo systemctl enable ollama.

Step 4: Pull your first model

Ollama hosts many popular models. To start, pull a reasonably fast, high-quality base model like Meta’s Llama 3.1. Run ollama pull llama3.1. Other good options include mistral, qwen2.5, or smaller quantized variants that fit into limited VRAM (for example, llama3.1:8b or mistral:7b-instruct). Use ollama list to see installed models.

Step 5: Test the local API

Ollama listens on http://127.0.0.1:11434 by default. You can chat in the terminal with ollama run llama3.1. To test via API, run: curl http://127.0.0.1:11434/api/generate -d '{"model":"llama3.1","prompt":"Say hello from a local model."}'. You should see a streamed JSON response. Press Ctrl+C to stop streaming if needed.

Step 6: Install Docker (for Open WebUI)

Open WebUI is easiest to deploy with Docker. Install Docker and its prerequisites. First run: sudo apt -y install apt-transport-https gnupg lsb-release. Then add Docker’s repository key: curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker.gpg. Add the repo: echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null. Install Docker: sudo apt update && sudo apt -y install docker-ce docker-ce-cli containerd.io. Optionally add your user to the Docker group: sudo usermod -aG docker $USER and re-login.

Step 7: Deploy Open WebUI

Open WebUI connects to Ollama’s API and provides a rich chat interface. Create a persistent volume directory and run the container pointing to the local Ollama endpoint. Use: docker run -d --name open-webui -p 3000:8080 -e OLLAMA_API_BASE_URL=http://host.docker.internal:11434 -v openwebui-data:/app/backend/data --restart unless-stopped ghcr.io/open-webui/open-webui:latest. On Linux, if host.docker.internal is not available, replace it with the host’s IP (e.g., http://127.0.0.1:11434) and add --network host instead of -p mapping if you prefer host networking: docker run -d --name open-webui --network host -e OLLAMA_API_BASE_URL=http://127.0.0.1:11434 -v openwebui-data:/app/backend/data --restart unless-stopped ghcr.io/open-webui/open-webui:latest.

Step 8: Secure optional access and firewall

If you will only use the services locally, keep them bound to localhost and do not expose ports publicly. For remote access on a trusted LAN, allow Open WebUI’s port via UFW using sudo ufw allow 3000/tcp (or none if using --network host and default port 8080). Enable the firewall with sudo ufw enable, then verify rules with sudo ufw status. For public access, place Open WebUI behind a reverse proxy (Nginx, Caddy, or Traefik) with HTTPS and authentication.

Step 9: Use the interface

Open a browser to http://SERVER_IP:3000 (or http://localhost:3000). On first load, you can create an admin account, choose your default model (e.g., llama3.1), manage prompts, and run chats. You can switch models per-conversation and configure system prompts for specific tasks like coding, summarization, or Q&A.

Troubleshooting and tips

If a model fails to load due to GPU memory limits, pull a smaller or more heavily quantized variant such as llama3.1:8b or a q4_k_m quant. If CPU usage is too high, reduce the context window or batch size in Open WebUI settings. If the Open WebUI container cannot reach Ollama, double-check OLLAMA_API_BASE_URL, networking mode, and whether the ollama service is running. For best performance on NVIDIA GPUs, close other GPU-heavy apps and monitor usage with nvidia-smi. To update, run sudo systemctl stop ollama && curl -fsSL https://ollama.com/install.sh | sh && sudo systemctl start ollama and pull newer model versions as needed.

What you have now

You have a fully local AI chat stack with GPU acceleration using Ollama and a clean, user-friendly interface via Open WebUI. It is private by default, fast on modern GPUs, and flexible with many model choices. You can integrate it with other tools via the Ollama API for scripting, automation, and offline workflows. This setup gives you control over costs, data privacy, and performance while staying current with the latest open models.

Local large language models are now practical on a single server. In this step-by-step guide, you will deploy a private AI chatbot by running Ollama (for models) and Open WebUI (for the user interface) on Ubuntu using Docker. We will enable GPU acceleration with NVIDIA so responses are fast and efficient. By the end, you will have a persistent setup that survives reboots and is easy to update.

Overview

Ollama is a lightweight runtime that downloads and serves popular open-source models like Llama 3. Open WebUI is a web app that connects to Ollama and provides a clean chat interface, prompt templates, conversation history, and model management. We will run both components in Docker containers on the same Docker network and map persistent volumes for data. Optional GPU acceleration uses the NVIDIA Container Toolkit.

Prerequisites

- Ubuntu Server or Desktop (22.04 or 24.04 recommended)
- An NVIDIA GPU with proprietary drivers installed (verify with nvidia-smi) if you want GPU acceleration; CPU-only also works
- Sudo access and outbound internet connectivity
- Basic command line familiarity

1) Install Docker Engine and Compose

Install the official Docker packages and add your user to the docker group for passwordless usage.

sudo apt-get update
sudo apt-get install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker $USER
newgrp docker

2) Enable NVIDIA GPUs in Docker (optional but recommended)

If you have an NVIDIA GPU and drivers are installed, add the NVIDIA Container Toolkit so Docker can access the GPU. Verify drivers first with nvidia-smi. Then install the toolkit and restart Docker.

distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Test compute visibility by running a CUDA-enabled container (optional):

docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

3) Create a Docker network and persistent volumes

We will create an isolated network for both containers and define persistent volumes so model files and WebUI data survive restarts.

docker network create ollama-net
docker volume create ollama
docker volume create open-webui

4) Run Ollama (model server)

Start the Ollama container. If you have a GPU, include --gpus all. The port 11434 is the Ollama API.

# GPU-enabled
docker run -d --name ollama --gpus all --restart unless-stopped \
  -p 11434:11434 \
  -v ollama:/root/.ollama \
  --network ollama-net \
  ollama/ollama:latest

# CPU-only (if you do not have an NVIDIA GPU)
# docker run -d --name ollama --restart unless-stopped \
#   -p 11434:11434 \
#   -v ollama:/root/.ollama \
#   --network ollama-net \
#   ollama/ollama:latest

Pull a model and do a quick test inside the container. Llama 3 and Qwen are great starting options.

docker exec -it ollama ollama pull llama3.1:8b
docker exec -it ollama ollama run llama3.1:8b "Write a two-line poem about local AI."

5) Run Open WebUI (front-end)

Open WebUI connects to the Ollama API. On first launch it creates an admin account when you sign in. We will point it at Ollama via the internal Docker network name.

docker run -d --name open-webui --restart unless-stopped \
  -p 3000:8080 \
  -e OLLAMA_BASE_URL=http://ollama:11434 \
  -v open-webui:/app/backend/data \
  --network ollama-net \
  ghcr.io/open-webui/open-webui:latest

Open a browser to http://<server-ip>:3000. Create your account, select the model you pulled (for example, llama3.1:8b), and start chatting. You can pull additional models anytime using docker exec -it ollama ollama pull qwen2.5:7b and select them in Open WebUI.

6) Optional: Use Docker Compose instead of docker run

If you prefer a single file, create docker-compose.yml in an empty folder. The gpus: all key enables GPU acceleration when the NVIDIA toolkit is installed.

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    ports:
      - "11434:11434"
    volumes:
      - ollama:/root/.ollama
    networks:
      - ollama-net
    deploy:
      resources:
        reservations:
          devices:
            - capabilities: ["gpu"]
    # Alternatively for Compose v2+:
    # gpus: all

  open-webui:
    image: ghcr.io/open-webui/open-webui:latest
    container_name: open-webui
    restart: unless-stopped
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
    volumes:
      - open-webui:/app/backend/data
    networks:
      - ollama-net

volumes:
  ollama:
  open-webui:

networks:
  ollama-net:
    external: true

Then run:

docker network create ollama-net
docker compose up -d

7) Updating and maintenance

To update images, pull the latest versions and recreate the containers. Your data and models remain in volumes.

docker pull ollama/ollama:latest
docker pull ghcr.io/open-webui/open-webui:latest
docker stop open-webui ollama && docker rm open-webui ollama
# re-run the docker run commands (or docker compose up -d)

To see logs for troubleshooting, run docker logs -f ollama and docker logs -f open-webui. If the WebUI cannot see models, ensure the environment variable OLLAMA_BASE_URL points to http://ollama:11434 and both containers share the same Docker network.

Troubleshooting tips

GPU not detected: Confirm the NVIDIA driver works on the host (nvidia-smi), the NVIDIA Container Toolkit is installed, and the container uses --gpus all. If using Compose, ensure gpus: all or the device reservation is defined.

Ports already in use: Change host ports in the run commands (for example, map Open WebUI to -p 8080:8080 instead of 3000).

Slow downloads or storage limits: Models are large. Consider attaching a larger Docker volume or moving /var/lib/docker to a disk with more space. You can also choose smaller models (7B) or quantized variants.

HTTPS and access control: Put Open WebUI behind a reverse proxy such as Nginx or Caddy with HTTPS and firewall rules. For internet exposure, add authentication, rate limits, and consider a VPN or zero-trust tunnel.

What you built

You now have a local, private AI chatbot with GPU acceleration on Ubuntu using Docker. Ollama handles model serving, while Open WebUI gives you a friendly interface with history, prompts, and multi-model management. This setup is repeatable, easy to update, and keeps your data on your own hardware.