LifeBytes Journal

Local large language models are now practical on a single PC. In this tutorial, you will deploy Ollama (model runtime) and Open WebUI (a friendly chat interface) using Docker on Windows or Linux. We will enable NVIDIA GPU acceleration, persist models on disk, and cover secure access and troubleshooting. By the end, you will be chatting with a local LLM like llama3.1 in your browser, no cloud required.

What You Will Need

- A 64-bit PC with at least 16 GB RAM. For GPU acceleration, an NVIDIA GPU with 8 GB+ VRAM is recommended.
- Docker Engine or Docker Desktop (Compose v2 included).
- Free disk space (15–30 GB per model is common).
- Optional but recommended: NVIDIA GPU drivers and CUDA runtime for Docker.

Step 1: Prepare Your System (GPU Optional)

Linux (Ubuntu/Debian)
1) Install Docker Engine and the Compose plugin from the official Docker repo.
2) Install NVIDIA GPU drivers from your distro or NVIDIA site.
3) Install the NVIDIA Container Toolkit:
sudo apt-get install -y nvidia-container-toolkit
Then configure and restart Docker:
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
Verify GPU visibility in containers:
docker run --rm --gpus all nvidia/cuda:12.3.2-base-ubuntu22.04 nvidia-smi

Windows 10/11
1) Install the latest NVIDIA GPU driver (Studio or Game Ready).
2) Install Docker Desktop and enable WSL 2 backend during setup.
3) In Docker Desktop > Settings > Resources > WSL integration, enable your default distro.
4) Ensure GPU is exposed to containers. If you run docker run --rm --gpus all nvidia/cuda:12.3.2-base-ubuntu22.04 nvidia-smi and see your GPU, you are ready.

Step 2: Create a Docker Compose File

We will run two containers: ollama (the LLM runtime API) and open-webui (the web front-end). The services will share a network and persistent volumes. Create a folder like ollama-openwebui and a file compose.yaml with the following content:

services:
  ollama:
   image: ollama/ollama:latest
   container_name: ollama
   restart: unless-stopped
   ports:
    - "11434:11434"
   volumes:
    - ollama_data:/root/.ollama
   environment:
    - OLLAMA_KEEP_ALIVE=24h
   deploy:
    resources:
     reservations:
      devices:
       - capabilities: ["gpu"]

  openwebui:
   image: ghcr.io/open-webui/open-webui:latest
   container_name: open-webui
   restart: unless-stopped
   depends_on:
    - ollama
   ports:
    - "3000:8080"
   environment:
    - OLLAMA_BASE_URL=http://ollama:11434
    - WEBUI_AUTH=True
    - DEFAULT_MODELS=llama3.1:8b
   volumes:
    - openwebui_data:/app/backend/data

volumes:
  ollama_data:
  openwebui_data:

Notes:
- The deploy.resources.reservations.devices section hints Compose to request GPU. On Linux, also start with --gpus all if you run containers manually.
- Ports: Ollama API is 11434, Open WebUI is exposed on 3000 (mapped to container 8080).

Step 3: Start the Stack

In the folder containing compose.yaml, run:
docker compose up -d
Wait for both containers to start. You can watch logs with:
docker compose logs -f

Step 4: Pull a Model and Run Your First Chat

Open a terminal and pull a model into Ollama. For a good balance of quality and speed, try Meta’s 8B model:
docker exec -it ollama ollama pull llama3.1:8b
You can test from the CLI:
docker exec -it ollama ollama run llama3.1:8b "Write a haiku about local AI."
If the response appears, the model is working.

Now open your browser and visit http://localhost:3000. Create an admin account (since we set WEBUI_AUTH=True). In Settings > Models, you should see llama3.1:8b. Create a new chat and start prompting.

GPU Acceleration Checks

- If you have an NVIDIA GPU, Ollama should automatically use it. Confirm via logs: docker logs ollama (look for CUDA initialization).
- If you do not have a GPU, Ollama will use CPU. Expect slower generation but it will work.

Useful Options and Performance Tips

- Try smaller variants for low VRAM: llama3.2:3b or phi3:mini.
- You can pin models to GPU RAM by enabling sufficient numa/gpu memory; if VRAM is low, Ollama will offload layers to system RAM.
- To pre-download a model at startup, set DEFAULT_MODELS in the Open WebUI service as shown.
- For multilingual or coding tasks, add models like qwen2.5:7b or codestral.

Security and Remote Access

- Keep WEBUI_AUTH=True to require sign-in. You can also set OPENWEBUI_ADMIN_EMAIL and OPENWEBUI_ADMIN_PASSWORD as environment variables for unattended setups.
- If exposing Open WebUI to the internet, place it behind a reverse proxy (Nginx, Caddy, or Traefik) with HTTPS and strong passwords.
- The Ollama API on port 11434 should remain private unless you need remote access; firewall it if required.

Troubleshooting

- GPU not detected: On Linux, reinstall nvidia-container-toolkit and verify nvidia-smi works both on the host and in a container. On Windows, ensure WSL 2 is enabled and Docker Desktop is up to date.
- “No space left on device”: Increase disk space or prune unused model blobs: docker exec -it ollama ollama rm <model>. You can also clear unused images with docker system prune (caution).
- Slow or out-of-memory: Use a smaller model, reduce context length in Open WebUI, close other GPU-intensive apps, or increase swap on Linux.
- Port in use: Change the published ports in compose.yaml (e.g., "3001:8080") and redeploy.

Updating and Maintenance

To update to the latest versions, run:
docker compose pull
docker compose up -d
Your models are safe in the ollama_data volume, and your chat history lives in openwebui_data. Always back up these volumes before major upgrades.

What’s Next

You now have a privacy-friendly, GPU-accelerated local AI stack. Explore function calling, RAG connectors in Open WebUI, or run multiple models side by side. With Docker and Ollama, swapping models and keeping performance high is only a pull away.

Overview

This step-by-step guide shows you how to deploy Ollama and Open WebUI on Ubuntu 24.04 with NVIDIA GPU acceleration using Docker. With this setup, you can run modern large language models (LLMs) locally, manage them from a clean web interface, and take full advantage of your GPU for high performance. The process covers NVIDIA drivers, Docker, the NVIDIA Container Toolkit, and secure, persistent containers that survive reboots.

What You Will Need

You need a 64-bit Ubuntu 24.04 host with an NVIDIA GPU (Turing or newer recommended), Internet access, a user with sudo rights, and at least 20 GB of free disk space for models. If you are working on a remote server, make sure port 3000 (for Open WebUI) and 11434 (for Ollama) are reachable or routed through a reverse proxy.

1) Install NVIDIA Drivers

First, install the official NVIDIA driver so CUDA can talk to your GPU. Run: sudo ubuntu-drivers autoinstall. When it finishes, reboot with sudo reboot. After the reboot, verify the GPU is visible: nvidia-smi. You should see your GPU name and driver version. If you do not, confirm Secure Boot is disabled or enroll the driver MOK accordingly, then repeat the check.

2) Install Docker Engine on Ubuntu 24.04

Set up Docker from the official repository for best stability and features. Run: sudo apt update && sudo apt install -y ca-certificates curl gnupg. Add Docker’s key and repo: sudo install -m 0755 -d /etc/apt/keyrings, curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg, echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu noble stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null. Then install: sudo apt update && sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin. To run Docker without sudo: sudo usermod -aG docker $USER then newgrp docker.

3) Enable GPU Access in Containers (NVIDIA Container Toolkit)

Install the NVIDIA Container Toolkit so Docker can pass your GPU into containers. Add the key and repo: curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg, curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list. Install and configure: sudo apt update && sudo apt install -y nvidia-container-toolkit, sudo nvidia-ctk runtime configure --runtime=docker, sudo systemctl restart docker. Test GPU passthrough: docker run --rm --gpus all nvidia/cuda:12.6.2-base-ubuntu22.04 nvidia-smi. You should see your GPU listed inside the container.

4) Create a Dedicated Network for AI Services

Create a user-defined Docker network so containers can discover each other cleanly: docker network create ai. This network isolates traffic and lets Open WebUI talk to the Ollama container by name.

5) Run the Ollama Container with GPU Support

Start Ollama and persist its model data in a Docker volume. Run: docker run -d --name ollama --gpus all --restart unless-stopped -p 11434:11434 -v ollama:/root/.ollama --network ai ollama/ollama:latest. The container exposes the Ollama API on port 11434. Check logs with docker logs -f ollama to ensure the server starts without errors.

6) Pull a Model (Llama 3.1 example)

Use Ollama’s CLI inside the container to download a model. For a great balance of speed and quality on consumer GPUs, try an 8B model: docker exec -it ollama ollama pull llama3.1:8b. If you have a smaller GPU (e.g., 6–8 GB VRAM), try a quantized variant like llama3.1:8b-instruct-q4_K_M. You can list models with docker exec -it ollama ollama list.

7) Deploy Open WebUI and Connect to Ollama

Open WebUI provides a friendly interface to chat with models, manage prompts, and configure settings. Start it with: docker run -d --name open-webui --restart unless-stopped -p 3000:8080 -e OLLAMA_API_BASE_URL=http://ollama:11434 -v openwebui:/app/backend/data --network ai ghcr.io/open-webui/open-webui:latest. Open http://<your_server_ip>:3000 in a browser, create your first user (the first account becomes admin), and pick the model you pulled in the previous step. You can now chat with the LLM directly from your browser.

8) Optional: Secure Access with HTTPS

For Internet-facing servers, place a reverse proxy with TLS in front of Open WebUI. A simple approach is Caddy or Nginx Proxy Manager. Point your domain’s DNS to the server, terminate HTTPS on the proxy, and forward to localhost:3000. If you already use Traefik or Nginx, add routes with Let’s Encrypt certificates and restrict access using basic auth or OAuth.

Maintenance and Updates

To update Ollama or Open WebUI, pull new images and recreate containers. Run: docker pull ollama/ollama:latest and docker pull ghcr.io/open-webui/open-webui:latest, then docker stop ollama open-webui and docker rm ollama open-webui. Start them again using the same docker run commands; your data persists in the volumes ollama and openwebui. To back up models and settings, archive the volumes: sudo tar -czf ollama-vol.tgz -C /var/lib/docker/volumes/ollama/_data . and sudo tar -czf openwebui-vol.tgz -C /var/lib/docker/volumes/openwebui/_data ..

Troubleshooting

If the GPU is not detected inside containers, confirm the host driver works with nvidia-smi. Then verify the runtime is configured: docker info | grep -i nvidia. If missing, re-run sudo nvidia-ctk runtime configure --runtime=docker and sudo systemctl restart docker. For permission errors when running Docker, add your user to the docker group as shown above. If downloads are slow or models fail due to VRAM limits, choose smaller or quantized models (e.g., q4_K_M or q5_K_M).

What You Get

After following these steps, you have a modern, GPU-accelerated local AI stack. Ollama handles efficient model runtimes, and Open WebUI gives you a clean chat interface, prompt management, and multi-model control. Because everything runs in Docker with persistent volumes, updates and backups are easy, and you can scale this setup on a workstation or a headless server with minimal changes.

Overview

This step-by-step guide shows you how to run a local AI assistant on Windows 11 using Ollama and Open WebUI. You will install Ollama, download a model, and connect a user-friendly web interface via Docker. The tutorial is beginner-friendly yet covers advanced options like GPU acceleration, authentication, and storage tuning. By the end, you will have a private, fast, and offline-capable AI setup on your own PC.

Prerequisites

Before you start, make sure you have: Windows 11 (22H2 or newer), administrator rights, and at least 8 GB RAM. For GPU acceleration, install the latest graphics driver. Ollama uses CUDA for NVIDIA GPUs and DirectML for AMD/Intel; GPU use is automatic if supported. You do not need WSL for this guide. An optional Docker Desktop installation is required for Open WebUI.

Step 1 — Install Ollama for Windows

1) Download the official installer from https://ollama.com/download and complete the setup.
2) Open PowerShell and verify the installation: ollama --version.
3) Start the Ollama service if it is not already running: ollama serve (you can keep it in the background by closing the window after confirming it is running as a service).

Step 2 — Pull and test a model

1) In PowerShell, download a model. For a good balance of speed and quality, try: ollama pull llama3.
2) Run it interactively: ollama run llama3, then ask a question like: What can you do?.
3) Exit the session with /bye when finished. Models are stored locally in %LOCALAPPDATA%\Ollama\models by default.

Step 3 — Install Docker Desktop (for Open WebUI)

Open WebUI gives you a clean web interface for prompts, chat history, and multi-model workflows. Install Docker Desktop from https://www.docker.com/products/docker-desktop/ and start it. Ensure the Docker engine is running (the whale icon should be active in the system tray).

Step 4 — Launch Open WebUI linked to Ollama

Run the following Docker command in PowerShell to start Open WebUI and connect it to your local Ollama instance exposed at http://localhost:11434:
docker run -d --name open-webui -p 3000:8080 -e OLLAMA_BASE_URL=http://host.docker.internal:11434 -v open-webui:/app/backend/data ghcr.io/open-webui/open-webui:latest
Once the container is healthy, open http://localhost:3000 in your browser. Choose a model (for example, llama3) and start chatting.

Optional — Enable authentication for Open WebUI

To protect your UI with a login, recreate the container with auth variables:
docker rm -f open-webui
docker run -d --name open-webui -p 3000:8080 -e OLLAMA_BASE_URL=http://host.docker.internal:11434 -e WEBUI_AUTH=true -e DEFAULT_USERNAME=admin -e DEFAULT_PASSWORD=changeMeNow -v open-webui:/app/backend/data ghcr.io/open-webui/open-webui:latest
Visit http://localhost:3000 and sign in with your credentials.

Optional — GPU acceleration tips

Ollama automatically uses your GPU when supported drivers are present. To nudge usage, you can set the number of GPUs: setx OLLAMA_NUM_GPU 1 then restart the Ollama service or your PC. If you have an NVIDIA GPU, ensure the latest Game Ready or Studio driver is installed. For AMD/Intel, keep your driver and Windows up to date to benefit from DirectML improvements. During the first run, the model may compile kernels; subsequent runs are faster.

Optional — Move the models folder to another drive

If you want models on a larger drive, set this environment variable and restart the service: setx OLLAMA_MODELS "D:\Ollama\Models". Move the existing folder from %LOCALAPPDATA%\Ollama\models to the new location to avoid re-downloading large files.

Troubleshooting

Open WebUI cannot connect to Ollama: Make sure Ollama is running: curl http://localhost:11434/api/tags should return a JSON list of models. If it works on the host but not in Docker, confirm the container uses host.docker.internal and port 11434 as shown in the command. Also check Windows Firewall for any blocked inbound rules on Docker or Ollama.

Models are slow or fail to load: Try a smaller model first: ollama pull phi3:mini and run ollama run phi3:mini. Close heavy apps, ensure you have enough RAM/VRAM, and avoid aggressive antivirus scanning of the models folder.

Docker errors on startup: Open Docker Desktop and verify that the engine is running. If ports are already in use, change the mapping (for example, -p 3001:8080) and refresh the browser at the new address.

Usage tips

Inside Open WebUI, create multiple chats per model for different tasks, enable markdown rendering, and configure system prompts for role-specific behavior. In PowerShell, you can also run one-off prompts without the UI: ollama run llama3 "Write a haiku about morning coffee." For reproducibility, export your Open WebUI data with the named volume and back it up regularly.

What you achieved

You now have a private, local AI assistant on Windows 11 powered by Ollama and Open WebUI. You can switch models, run fully offline, and take advantage of your GPU for faster responses. This setup is ideal for coding help, note-taking, drafting, and research without sending your data to external servers.