LifeBytes Journal

Running a large language model (LLM) locally is one of the fastest ways to get private, low-latency AI assistance without sending your prompts to a third-party cloud. In this tutorial, you will set up Ollama (a lightweight LLM runtime) and Open WebUI (a clean web interface) on Linux. The result is a self-hosted AI chat you can use for drafting, coding help, log analysis, and knowledge base searching—while keeping data on your own machine.

What You Need

Hardware: A modern CPU system works, but more RAM helps a lot. For small models (like 7B), aim for 8–16 GB RAM. For smoother performance or larger models, 32 GB+ is recommended. If you have an NVIDIA GPU, you can accelerate generation, but this guide focuses on a reliable CPU-first setup.

Software: A recent Linux distribution (Ubuntu/Debian/Fedora), terminal access, and either Docker (recommended for Open WebUI) or Python knowledge if you prefer manual setups.

Step 1: Install Ollama

Ollama makes local model management simple: you download a model once and then run it with a single command. To install Ollama, open a terminal and run:

curl -fsSL https://ollama.com/install.sh | sh

After installation, verify it works:

ollama --version

On most systems, Ollama starts as a service automatically. If you need to start it manually, you can run:

ollama serve

Step 2: Pull a Model (Example: Llama 3.1)

Now download a model. A good starting point is a modern 7B or 8B model. Pull it with:

ollama pull llama3.1

Once it finishes, test a quick prompt directly in the terminal:

ollama run llama3.1

Type a message (for example, “Summarize the difference between TCP and UDP”) and press Enter. If you get a response, the local model runtime is working.

Step 3: Install Docker (for Open WebUI)

Open WebUI is easiest to run in a container. If Docker is not installed, on Ubuntu/Debian you can do:

sudo apt update
sudo apt install -y docker.io
sudo systemctl enable --now docker

Optional but recommended: allow your user to run Docker without sudo:

sudo usermod -aG docker $USER

Log out and back in for the group change to apply.

Step 4: Run Open WebUI and Connect It to Ollama

Start Open WebUI with Docker. This command creates persistent storage and publishes the web interface on port 3000:

docker run -d --name open-webui -p 3000:8080 -v open-webui:/app/backend/data --restart unless-stopped ghcr.io/open-webui/open-webui:main

Next, ensure Open WebUI can reach Ollama. If Open WebUI does not automatically detect it, the most common fix is to point it to the Ollama API endpoint. Ollama listens on http://localhost:11434 by default. Depending on your Docker networking setup, “localhost” inside the container is not the host machine.

A practical approach is to run Open WebUI using host networking (Linux only). Stop the existing container and re-run:

docker rm -f open-webui
docker run -d --name open-webui --network=host -v open-webui:/app/backend/data --restart unless-stopped ghcr.io/open-webui/open-webui:main

Now open your browser and go to:

http://localhost:3000

Create an admin account when prompted. In the Open WebUI settings, you should see Ollama as an available provider. Select the model you pulled (for example, llama3.1) and start chatting.

Step 5: Improve Performance and Reliability

Choose the right model size: If responses feel slow, try a smaller model. Ollama supports many options; you can keep multiple models and switch depending on the task. Smaller models are great for quick drafts, command explanations, and lightweight Q&A.

Keep your data private: Local LLMs are only “private” if you avoid sending data out through plugins or external integrations. Treat the WebUI like any internal tool: secure access, avoid exposing it to the public internet, and consider a reverse proxy with authentication if you need remote access.

Troubleshoot connectivity: If Open WebUI can’t see Ollama, confirm the Ollama service is running and listening on port 11434:

ss -tulpn | grep 11434

If you prefer not to use host networking, you can configure Ollama to bind to an address reachable from Docker and then point Open WebUI to that address. The exact method depends on your distro and firewall rules, so host networking is the fastest baseline to validate your setup.

Next Steps (Useful Ideas)

Once your local AI chat is stable, you can level it up: create model presets for different writing styles, connect it to internal documentation, or use it for structured tasks like generating incident summaries from sanitized logs. The biggest advantage of this setup is control—you decide what runs, where it runs, and what data it can access.

With Ollama and Open WebUI, a private LLM workstation is no longer a weekend project. It’s a practical tool you can deploy in minutes and refine over time.

Why host a local AI assistant?

If you write scripts, manage servers, or handle helpdesk tickets, an AI assistant can speed up routine work like summarizing logs, drafting commands, or explaining configuration files. The problem is that many cloud tools send your prompts and snippets to third-party services. A local setup keeps sensitive data on your own machine, works offline, and can be tuned for your workflow.

In this tutorial, you will install Ollama (a lightweight local LLM runtime) and Open WebUI (a web interface similar to popular chat tools) on Linux. The result is a private AI assistant you can access from your browser on your LAN.

What you need

Hardware: A modern 64-bit Linux system. For acceptable performance, aim for 16 GB RAM or more. A GPU helps but is not required for basic use. Lighter models can run on CPU-only machines, including small servers.

Software: A recent Linux distribution (Ubuntu/Debian/Fedora are all fine), Docker for Open WebUI, and basic terminal access with sudo.

Step 1: Install Ollama

Ollama runs the model locally and exposes an API that other tools (like Open WebUI) can call. Install it using the official script:

Command:

curl -fsSL https://ollama.com/install.sh | sh

After installation, check that the service is working:

ollama --version

On many distros, Ollama runs as a service. If you need to confirm it is active:

systemctl status ollama

Step 2: Pull a model and test it

Next, download a model. If you are CPU-only or want fast responses, start with a smaller model. For general coding help, you can also try code-focused models once the basics work.

Example (general model):

ollama pull llama3.1

Run a quick prompt to confirm everything works:

ollama run llama3.1

Type a question like “Explain what journald does on Linux” and confirm you get a response. Exit with /bye or Ctrl+C depending on your shell behavior.

Step 3: Install Docker (if not installed)

Open WebUI is easiest to deploy with Docker. On Ubuntu/Debian, you can install Docker like this:

sudo apt update

sudo apt install -y docker.io

sudo systemctl enable --now docker

Optional but recommended: allow your user to run Docker without sudo (log out and back in after this):

sudo usermod -aG docker $USER

Step 4: Run Open WebUI and connect it to Ollama

Open WebUI will provide a clean browser interface and conversation history. The key is pointing it at Ollama’s API endpoint.

First, make sure Ollama is listening locally. By default it is typically available at http://127.0.0.1:11434. Now start Open WebUI in Docker:

docker run -d --name open-webui --restart unless-stopped -p 3000:8080 -e OLLAMA_BASE_URL=http://host.docker.internal:11434 -v open-webui:/app/backend/data ghcr.io/open-webui/open-webui:main

On Linux, host.docker.internal may not be available depending on your Docker version. If the UI cannot connect, rerun the container using host networking instead:

docker rm -f open-webui

docker run -d --name open-webui --restart unless-stopped --network=host -e OLLAMA_BASE_URL=http://127.0.0.1:11434 -v open-webui:/app/backend/data ghcr.io/open-webui/open-webui:main

Now open your browser and visit:

http://localhost:3000

Create the first admin user when prompted. Once logged in, you should see available Ollama models. If you do not, go to settings and verify the Ollama base URL.

Step 5: Enable LAN access (optional and safer if restricted)

If you want to access the assistant from another device on your network, bind the service to a reachable interface and restrict it with firewall rules. For Open WebUI using Docker with port publishing, ensure your firewall only allows trusted subnets to connect to port 3000.

For example, on Ubuntu with UFW you can allow only your local subnet (adjust the CIDR):

sudo ufw allow from 192.168.1.0/24 to any port 3000 proto tcp

Avoid exposing the service directly to the internet. If you need remote access, put it behind a VPN (WireGuard is a good choice) or a reverse proxy with authentication.

Troubleshooting tips

Open WebUI shows “cannot reach Ollama”: Confirm Ollama is running with systemctl status ollama. Then check connectivity from the container. If you are using port mapping, the simplest fix on Linux is often --network=host.

Model downloads are slow or fail: Verify DNS and outbound access. Large models can be tens of gigabytes. If disk space is tight, remove unused models with ollama list and ollama rm <model>.

Responses are too slow: Try a smaller model, reduce context size in settings, and close other memory-heavy applications. CPU-only systems benefit from lightweight models and shorter prompts.

Next steps: make it useful for real admin work

Once the UI is running, build a few saved prompts for your daily tasks: “Summarize this syslog excerpt,” “Write a Bash one-liner to find large files,” or “Draft a polite helpdesk reply.” Because the assistant is local, you can safely paste internal error messages, configuration snippets, or playbook fragments without sending them to a third party.

With Ollama and Open WebUI, you get a practical self-hosted AI assistant that fits nicely into a Linux admin toolbox: fast to deploy, easy to maintain, and private by design.

Overview

This step-by-step guide shows you how to run local Large Language Models (LLMs) on Ubuntu using Ollama and Open WebUI. You will install Ollama, optionally enable NVIDIA GPU acceleration, and deploy Open WebUI in Docker to get a fast, friendly chat interface. By the end, you will have a private AI assistant running on your own hardware with secure access options and practical troubleshooting tips.

Prerequisites

Use Ubuntu 22.04 or 24.04 with at least 8 GB of RAM (16 GB recommended). For GPU acceleration, an NVIDIA GPU with 8 GB or more VRAM is ideal. You need sudo access and open ports 11434 for Ollama and 3000 (or your choice) for Open WebUI. This guide covers both CPU-only and GPU setups, so you can start even without a supported GPU.

Step 1: (Optional) Install NVIDIA Drivers and CUDA

If you plan to use a GPU, first confirm your hardware with lspci | grep -i nvidia. Install the recommended driver via sudo ubuntu-drivers autoinstall, then reboot. After rebooting, verify the driver with nvidia-smi. If you will run Open WebUI with GPU access in Docker, also install the NVIDIA container runtime using sudo apt-get install -y nvidia-container-toolkit and configure Docker with sudo nvidia-ctk runtime configure followed by sudo systemctl restart docker.

Step 2: Install Ollama on Ubuntu

Install Ollama with a single command: curl -fsSL https://ollama.com/install.sh | sh. This creates a system service and exposes the local API on http://127.0.0.1:11434. Check the version with ollama -v and verify the service using systemctl status ollama. If you need remote access on your LAN, set the host binding by creating an override file. Run sudo systemctl edit ollama, add [Service] and Environment="OLLAMA_HOST=0.0.0.0:11434", then save, sudo systemctl daemon-reload, and sudo systemctl restart ollama. Only expose Ollama on trusted networks or behind a reverse proxy with authentication.

Step 3: Pull and Run Models with Ollama

Pull a small, fast model to test your setup. For general chat, use ollama pull llama3.2:3b. For coding tasks, try ollama pull qwen2.5-coder:7b or a quantized variant like :q4_0 for lower memory usage. Run an interactive session with ollama run llama3.2 and type your prompt. To generate from the shell, try echo "Explain RAID levels simply" | ollama run llama3.2. Ollama will use the GPU automatically if supported; otherwise it falls back to CPU. Tune performance with environment variables such as OLLAMA_NUM_PARALLEL=1 to reduce memory pressure and OLLAMA_KV_SIZE=512 for larger context windows when your memory allows.

Step 4: Deploy Open WebUI with Docker

Open WebUI provides a clean web interface and multi-model support. If Docker is not installed, add it with sudo apt-get update && sudo apt-get install -y docker.io and ensure it runs at startup with sudo systemctl enable --now docker. Launch Open WebUI connected to Ollama using docker run -d --name open-webui -p 3000:8080 -e OLLAMA_BASE_URL=http://localhost:11434 -v open-webui:/app/backend/data -v /var/lib/ollama:/root/.ollama --restart unless-stopped ghcr.io/open-webui/open-webui:latest. If Open WebUI runs on a different host from Ollama, set OLLAMA_BASE_URL to the Ollama server’s IP, for example http://192.168.1.50:11434. For GPU inside the container, add --gpus all and make sure the NVIDIA container toolkit is configured.

Step 5: Secure Access with a Reverse Proxy and HTTPS

If you plan to reach the interface over the internet, place Open WebUI behind a reverse proxy with TLS and authentication. A simple option is Caddy, which can obtain and renew certificates automatically. For example, you can point a domain to your server and configure Caddy to proxy yourdomain.com to localhost:3000 and enable basic auth. With Nginx, use an SSL server block, set proxy_pass http://127.0.0.1:3000, and enable rate limiting and headers like X-Frame-Options and Content-Security-Policy. Always avoid exposing the raw Ollama port unless you fully trust the network.

Step 6: Updates, Backups, and Autostart

Update Ollama by rerunning the installer or using your package manager if you installed via a repo. To update Open WebUI, pull the latest image with docker pull ghcr.io/open-webui/open-webui:latest and restart the container. Persist your data by backing up /var/lib/ollama and the Docker volume open-webui. Both Ollama and Docker containers start automatically on boot, but you can confirm with systemctl is-enabled ollama and the container’s --restart unless-stopped flag.

API Quick Test

You can call Ollama’s local API directly. After pulling a model, try curl http://localhost:11434/api/generate -d '{"model":"llama3.2","prompt":"Give me three bullet points about containers"}'. This is useful for integrating local LLMs into scripts, chatbots, or development tools without sending data to third parties.

Troubleshooting

If model loading fails with “no space left on device,” free disk space with df -h, remove unused Docker images with docker system prune -a, or delete old models in /var/lib/ollama. If nvidia-smi returns an error, reinstall the driver and ensure Secure Boot is either disabled or configured with signed modules. If port 11434 or 3000 is already in use, change the binding (for example OLLAMA_HOST=0.0.0.0:11435) or stop the conflicting process. On low-memory hosts, choose smaller or more heavily quantized models (for example :q4_0), reduce parallel requests with OLLAMA_NUM_PARALLEL=1, and close other memory-hungry services.

What You Achieved

You now have a private, production-ready local AI stack on Ubuntu. Ollama runs the model backend with optional GPU acceleration, while Open WebUI delivers a modern chat interface. With a reverse proxy and backups in place, you can confidently use local LLMs for coding assistance, content drafting, documentation, and experimentation without sending your data to the cloud.

Overview

This tutorial shows you how to deploy a fast, private, local AI stack on Ubuntu using Ollama and Open WebUI with NVIDIA GPU acceleration. You will install the NVIDIA driver, Docker, and the NVIDIA Container Toolkit, then run Ollama on the host and Open WebUI in a container. By the end, you will have a browser-based interface to run powerful large language models (LLMs) like Llama 3 with CUDA acceleration on your own machine.

Prerequisites

- Ubuntu 22.04 or 24.04 (freshly updated).
- An NVIDIA GPU with at least 6 GB VRAM (more is better).
- sudo privileges and Internet access.
- Optional: a domain or reverse proxy if you plan to expose the UI externally.

Step 1 — Install NVIDIA Driver

Use Ubuntu’s built-in tools to install a compatible proprietary driver. Reboot afterward and confirm the GPU is detected.

sudo apt update
sudo apt install -y ubuntu-drivers-common
sudo ubuntu-drivers autoinstall
sudo reboot

nvidia-smi

If you see a table with your GPU and driver version (e.g., 535+), you are ready for CUDA-enabled workloads.

Step 2 — Install Docker Engine

If Docker is not installed, use the official convenience script. Add your user to the docker group so you can run containers without sudo.

curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER
newgrp docker
docker version

Step 3 — Enable GPU Access in Containers

Install the NVIDIA Container Toolkit so Docker can pass the GPU into containers.

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
| sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
| sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#' \
| sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt update
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Verify GPU visibility inside a container:

docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

Step 4 — Install Ollama (runs on the host)

Ollama simplifies downloading and running LLMs locally. It automatically uses CUDA if your NVIDIA driver is installed.

curl -fsSL https://ollama.com/install.sh | sh

Confirm the service is active and the API is reachable on port 11434:

systemctl --user status ollama || systemctl status ollama
curl http://127.0.0.1:11434/api/tags

Pull and test a model (replace with your preferred model/quantization):

ollama pull llama3
ollama run llama3 "Write a two-line poem about GPUs."

Tip: Use smaller quantizations if VRAM is limited, for example llama3:8b-instruct-q4_0.

Step 5 — Deploy Open WebUI in Docker

Open WebUI provides a clean, modern interface for chatting with models served by Ollama. We will run it in Docker and point it to the host’s Ollama API. On Linux, add a host-gateway entry so the container can reach the host at host.docker.internal.

docker run -d --name open-webui \
  -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
  --gpus all \
  -v open-webui:/app/backend/data \
  ghcr.io/open-webui/open-webui:main

Open your browser at http://<server-ip>:3000. On first login, create a user; that account becomes admin. If you need authentication enabled from the start, add -e WEBUI_AUTH=True to the run command.

Alternative: If --add-host=host-gateway is not supported on your Docker version, use host networking and point to 127.0.0.1:

docker run -d --name open-webui \
  --network host \
  -e OLLAMA_BASE_URL=http://127.0.0.1:11434 \
  --gpus all \
  -v open-webui:/app/backend/data \
  ghcr.io/open-webui/open-webui:main

With host networking, Open WebUI listens on http://0.0.0.0:8080 (no -p flag needed).

Step 6 — Use and Tune Your Local AI

From Open WebUI, select a model (e.g., Llama 3) and start chatting. You can pull additional models with Ollama CLI and they will appear in the UI. To speed up responses and reduce VRAM, try smaller or more aggressive quantizations; to maximize quality, try larger quantizations if your GPU can handle them.

Common environment variables for Open WebUI include:
- WEBUI_AUTH=True to require sign-in.
- OLLAMA_BASE_URL to point to the Ollama server URL.
- PORT to customize the UI port if you use host networking.

Troubleshooting

Open WebUI cannot reach Ollama: Ensure you used --add-host=host.docker.internal:host-gateway and OLLAMA_BASE_URL=http://host.docker.internal:11434, or use host networking. Test connectivity with docker exec -it open-webui curl -s http://host.docker.internal:11434/api/tags.

No GPU in containers: Re-check the container toolkit setup and driver. Run docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi. If it fails, reboot and ensure nvidia-smi works on the host first.

Out-of-memory errors: Use a smaller model or more compressed quantization. Close other GPU-heavy apps. You can also run with a larger system swap to reduce crashes when VRAM is exhausted, but performance will be slower.

Docker permissions: If you see “permission denied,” ensure your user is in the docker group (id to verify) and run newgrp docker or re-log in.

Optional: Reverse Proxy and TLS

If exposing Open WebUI on the Internet, put it behind a reverse proxy (Caddy, Nginx, or Traefik) for HTTPS and access control. At minimum, enforce authentication and limit access to trusted IPs. Never expose Ollama’s port 11434 directly without protection.

Maintenance

- Update Ollama periodically by re-running the install script or checking the project release notes, then systemctl restart ollama.
- Update Open WebUI with docker pull ghcr.io/open-webui/open-webui:main and docker restart open-webui.
- Prune old images and volumes with docker system prune (review carefully before confirming).
- Back up /var/lib/ollama (models) and the Open WebUI volume for settings and chats.

You now have a modern, GPU-accelerated, private AI chat environment running locally on Ubuntu. This setup is fast, secure, and fully under your control—and you can expand it with additional models, prompt libraries, and integrations as your needs grow.

Overview

Running a large language model locally is now practical, fast, and private. In this how-to, you will set up Ollama to serve models on your computer and connect Open WebUI for a friendly chat interface. The steps cover Windows, macOS, and Linux, including GPU acceleration for NVIDIA, Apple Silicon, and supported AMD GPUs. By the end, you will be able to pull models, chat in your browser, and tune performance for your hardware.

Requirements and quick checklist

Hardware: 8 GB RAM minimum (16 GB+ recommended), 10–20 GB free disk for models, and optionally a compatible GPU for acceleration.

GPU support: NVIDIA (CUDA 12 driver), Apple Silicon (M1/M2/M3 via Metal), AMD ROCm on supported Linux cards. If you lack a compatible GPU, CPU-only still works, just slower.

Network and security: Keep Ollama bound to localhost unless you intentionally expose it behind a reverse proxy with authentication. Do not publish it directly to the internet.

Step 1 — Install Ollama

Windows: Install via winget or the official installer.

winget install Ollama.Ollama

macOS: Use Homebrew or the DMG from the website.

brew install ollama

Linux: Use the official script (requires curl and sudo).

curl -fsSL https://ollama.com/install.sh | sh

After installation, ensure the service is running. On macOS and Windows, the background service starts automatically. On Linux, start it in a terminal or as a service:

ollama serve

Verify the API is alive by visiting http://127.0.0.1:11434 in your browser. You should see a simple status page.

Step 2 — Pull and test a model

Pull a compact, fast model first to validate everything. Llama 3.2 3B is a great starting point for laptops.

ollama pull llama3.2:3b
ollama run llama3.2:3b

Type a quick prompt and confirm you get a response. For stronger reasoning, try Mistral or an 8B Llama if your RAM/GPU can handle it:

ollama pull mistral:7b
ollama pull llama3.1:8b

Step 3 — Enable GPU acceleration (optional but recommended)

NVIDIA on Windows/Linux: Install the latest Game Ready/Studio driver with CUDA 12 support. Verify with:

nvidia-smi

Ollama will use your GPU automatically if supported. If VRAM is limited, pick a smaller or more aggressively quantized model (for example, Q4 or Q5 builds).

Apple Silicon: No extra steps. Metal acceleration is used by default on M-series chips.

AMD on Linux (ROCm): Use a ROCm-supported GPU and drivers (ROCm 6.x+). Check your distro’s ROCm documentation. Not all AMD GPUs are supported; verify before investing time.

Step 4 — Install Open WebUI

Open WebUI gives you a clean, modern chat interface for Ollama. Docker is the easiest installation path. Make sure Docker Desktop (Windows/macOS) or Docker Engine (Linux) is installed and running.

Windows/macOS (Docker Desktop):

docker run -d --name open-webui -p 3000:8080 \
  -e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
  -v open-webui:/app/backend/data \
  ghcr.io/open-webui/open-webui:latest

Linux: The host networking mode is simplest so the container reaches Ollama on localhost.

docker run -d --name open-webui --network host \
  -e OLLAMA_BASE_URL=http://127.0.0.1:11434 \
  -v open-webui:/app/backend/data \
  ghcr.io/open-webui/open-webui:latest

Open your browser to http://127.0.0.1:3000, create an account (local), and select your Ollama model from the dropdown. Start chatting immediately.

Step 5 — Performance tips and model management

Use quantized models (GGUF variants) to fit your hardware. Q4_K_M is a balanced choice for speed and quality; Q6 is higher quality; Q2/Q3 are very small and fast but lose detail. If a model fails to load, try a smaller parameter count or lower quantization level.

Keep an eye on your RAM/VRAM while the model loads. If memory spikes, reduce context length (token window) in your client settings. Many 7B models run well with 4–6 GB VRAM; 8B often prefers 8–10 GB; CPU-only runs better with 3B–7B models.

List and manage your models with:

ollama list
ollama rm <model-name>

You can tweak behavior with a Modelfile to set defaults like temperature and system prompts. Example:

# Modelfile
FROM llama3.2:3b
PARAMETER temperature 0.7
SYSTEM You are a helpful technical assistant.

ollama create my-tech-assistant -f Modelfile
ollama run my-tech-assistant

Step 6 — Security and remote access basics

By default, Ollama listens on 127.0.0.1:11434, which is safe for single-machine use. If you need remote access on your LAN, set a bind address with an environment variable:

export OLLAMA_HOST=0.0.0.0:11434   # Linux/macOS example

If you expose it, protect it. Use a reverse proxy (Nginx, Traefik, Caddy) with TLS and authentication, or a mesh VPN like Tailscale. Never expose the Ollama API directly to the public internet.

Troubleshooting

If the model is slow, confirm acceleration is active. On NVIDIA, nvidia-smi should show GPU utilization when generating. For crashes during load, your model may not fit in memory; try a smaller model or reduce the context window. If Open WebUI cannot connect, ensure OLLAMA_BASE_URL is correct for your platform and that the port is not blocked by a firewall.

What’s next

Explore specialized models for coding, summarization, or multilingual tasks. Add embeddings and retrieval in Open WebUI to chat over your PDFs or docs. With Ollama handling the runtime and Open WebUI providing the interface, you own the stack: fast, private, and flexible.

Running large language models locally is now practical and fast, especially with GPU acceleration. In this tutorial, you will deploy Ollama and Open WebUI on Ubuntu 22.04/24.04 using Docker Compose. This stack gives you a private, browser-based interface for modern LLMs (Llama, Mistral, Phi, etc.) with one-click model management and secure, self-hosted inference.

Why this stack?

Ollama simplifies downloading, quantizing, and serving LLMs on your machine. Open WebUI adds a clean chat interface, prompt templates, file uploads, and multi-user access. Together, they provide a robust local AI setup that is easy to update and portable across servers.

Prerequisites

- Ubuntu Server 22.04 or 24.04 (fresh system recommended)

- An NVIDIA GPU with recent drivers (T4, RTX 20/30/40, A-series, etc.)

- sudo access and an internet connection

- Optional: a domain name for HTTPS (e.g., ai.example.com)

Step 1 — Install Docker Engine and Compose

Install Docker from the official repository to ensure up-to-date features like GPU support in Docker Compose.

sudo apt update
sudo apt install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
  https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker $USER
newgrp docker
docker --version
docker compose version

Step 2 — Enable GPU with NVIDIA Container Toolkit

Install the NVIDIA Container Toolkit to pass the GPU into containers. Verify that the host can see the GPU with nvidia-smi before proceeding.

# If you don't have drivers:
# sudo ubuntu-drivers install && sudo reboot

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update
sudo apt install -y nvidia-container-toolkit

# Configure Docker to use the NVIDIA runtime
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

# Sanity check
nvidia-smi

Step 3 — Create the Docker Compose stack

We will run two services: Ollama (backend API on port 11434) and Open WebUI (frontend on port 3000) connected via a Docker network. The compose file also enables GPU support for Ollama.

mkdir -p ~/ollama-openwebui && cd ~/ollama-openwebui
cat > docker-compose.yml <<'YAML'
services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    ports:
      - "11434:11434"
    volumes:
      - ollama:/root/.ollama
    gpus: all
    environment:
      - OLLAMA_KEEP_ALIVE=1h
      - OLLAMA_HOST=0.0.0.0

  openwebui:
    image: ghcr.io/open-webui/open-webui:latest
    container_name: openwebui
    restart: unless-stopped
    depends_on:
      - ollama
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
      - WEBUI_AUTH=True
    volumes:
      - openwebui:/app/backend/data

volumes:
  ollama:
  openwebui:
YAML

Step 4 — Launch and access Open WebUI

Start the stack and watch logs for any errors. The first launch will pull images.

docker compose up -d
docker compose logs -f --tail=100

Open your browser to http://SERVER_IP:3000. Create the first admin user when prompted. Open WebUI will automatically detect Ollama via the internal URL and list available models.

Step 5 — Pull a model and test

Use either the WebUI model manager or the CLI to fetch models. The example below pulls a popular 7B model.

# Pull from the host (proxies into the container)
docker exec -it ollama ollama pull llama3.1:8b

# Quick API smoke test
curl http://localhost:11434/api/generate -d '{
  "model": "llama3.1:8b",
  "prompt": "Say hello from a local LLM.",
  "stream": false
}'

In Open WebUI, select the model from the dropdown and start chatting. If you have enough VRAM, consider quantized larger models (e.g., 13B/70B Q4/Q5) for better reasoning.

Optional — Secure with a Caddy reverse proxy and HTTPS

If you have a domain, use Caddy to obtain and renew TLS automatically. This example exposes Open WebUI securely on port 443 and keeps Ollama private.

sudo apt install -y caddy
sudo tee /etc/caddy/Caddyfile >/dev/null <<'CADDY'
ai.example.com {
  encode zstd gzip
  reverse_proxy 127.0.0.1:3000
}
CADDY
sudo systemctl reload caddy

Point your DNS A/AAAA record to the server. Then visit https://ai.example.com. For teams, enable WebUI auth (already set) and create users from the admin settings.

Back up and update

To back up your models and chats, save the named volumes. You can also snapshot the folders from the host.

# Export volumes to tarballs
docker run --rm -v ollama:/v -v $(pwd):/b busybox tar czf /b/ollama-vol.tgz -C /v .
docker run --rm -v openwebui:/v -v $(pwd):/b busybox tar czf /b/openwebui-vol.tgz -C /v .

# Update images safely
docker compose pull
docker compose up -d

Troubleshooting

- No GPU detected: Ensure nvidia-smi works on the host. Re-run nvidia-ctk runtime configure, restart Docker, and verify the container sees the GPU:

docker exec -it ollama bash -lc 'nvidia-smi || ls -l /dev/nvidia*'

- Slow generation: Use quantized models (Q4_K_M/Q5_K_M), avoid oversize context windows, and confirm GPU is actually used (GPU utilization should rise in nvidia-smi during inference).

- Port conflicts: Change mapped ports in docker-compose.yml, e.g., "3001:8080" for Open WebUI or put a reverse proxy in front.

- Permission errors on volumes: Ensure your user is in the docker group and that the Docker daemon can write to the volume paths.

Security tips

- Keep Ollama bound to the internal network and only expose Open WebUI through TLS.

- Enable authentication (already set via WEBUI_AUTH=True). Use strong passwords and consider putting Open WebUI behind a VPN or SSO.

- Restrict firewall ports using UFW: allow 22/tcp and 443/tcp, then deny others.

Conclusion

You now have a GPU-accelerated, private AI stack with Ollama and Open WebUI on Ubuntu, orchestrated by Docker Compose. It is easy to upgrade, portable across servers, and suitable for personal research or team deployments. With this foundation, you can iterate quickly, evaluate new models as they drop, and keep your data fully on-prem.

Run Local AI with Ollama and Open WebUI: GPU-Accelerated Setup on Windows and Linux with Docker

Local large language models (LLMs) have matured to the point where you can run fast, private, and cost-effective AI on your own computer or server. In this step-by-step guide, you will deploy Ollama (the LLM backend) and Open WebUI (a sleek web interface) using Docker, with optional GPU acceleration on both Windows and Linux. This stack lets you chat with models like Llama 3, Phi-4, or Mistral, completely on your hardware.

By the end, you will have a browser-based interface, persistent model storage, and a clean way to update or back up your local AI environment. The instructions are written in simple, SEO-friendly language and focus on practical steps.

What You Will Build

You will run two containers on the same Docker network: Ollama exposes an API on port 11434 and performs all model work, while Open WebUI listens on port 3000 and connects to Ollama. You will enable GPU acceleration (NVIDIA or AMD) when available to dramatically improve performance.

Prerequisites

- A 64-bit Windows 11/10 (with WSL2) or a modern Linux distribution (Ubuntu/Debian/CentOS/RHEL).
- Docker installed (Docker Desktop on Windows, Docker Engine on Linux).
- Optional GPU: NVIDIA (CUDA) or AMD (ROCm) with up-to-date drivers. CPU-only also works, but is slower.
- 16 GB RAM recommended; disk space 10–40+ GB depending on model size.

Step 1 – Install Docker

Windows: Install Docker Desktop and enable WSL 2 integration. In Settings, ensure “Use the WSL 2 based engine” is on. Update your GPU driver from NVIDIA/AMD. For NVIDIA, CUDA is not required on Windows for Docker Desktop; the latest Game Ready/Studio drivers are enough.

Linux: Install Docker from your distribution’s repository or Docker’s official repo. Add your user to the docker group and log out/in. Example (Ubuntu):

sudo apt update
sudo apt install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | \
  sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
  https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list >/dev/null
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker $USER

Step 2 – Enable GPU Acceleration (Optional but Recommended)

NVIDIA on Linux: Install the NVIDIA Container Toolkit to pass your GPU into containers.

# Add the NVIDIA container toolkit repo (Ubuntu example)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
  sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

AMD on Linux (ROCm): Install the latest AMDGPU/ROCm stack. To give containers access, pass /dev/kfd and /dev/dri and add the video group. Example device flags are shown in the Ollama run step below.

Windows: Docker Desktop exposes the GPU automatically when the host has a compatible driver. Ensure your GPU driver is up to date and “Use the WSL 2 based engine” is enabled.

Step 3 – Start Ollama (LLM Backend)

Create a Docker network and a persistent volume for models. Then start the Ollama container. Use the NVIDIA command if you have an NVIDIA GPU; use the AMD/CPU command otherwise.

# Common network and volumes
docker network create llmnet
docker volume create ollama

# NVIDIA GPU (Linux or Windows with Docker Desktop)
docker run -d --name ollama \
  --network llmnet \
  --gpus=all \
  -v ollama:/root/.ollama \
  -p 11434:11434 \
  ollama/ollama:latest

# AMD ROCm or CPU-only (Linux)
# Remove the two --device flags if you want CPU-only
docker run -d --name ollama \
  --network llmnet \
  --device=/dev/kfd --device=/dev/dri --group-add video \
  -v ollama:/root/.ollama \
  -p 11434:11434 \
  ollama/ollama:latest

Verify Ollama is live:

curl http://localhost:11434/api/tags
# or
docker logs -f ollama

Step 4 – Start Open WebUI (Front-End)

Open WebUI connects to the Ollama API and gives you a beautiful chat interface. Map port 3000 for access and point it to the Ollama container over the private network.

docker volume create open-webui

docker run -d --name open-webui \
  --network llmnet \
  -p 3000:8080 \
  -e OLLAMA_BASE_URL=http://ollama:11434 \
  -v open-webui:/app/backend/data \
  ghcr.io/open-webui/open-webui:latest

Open your browser at http://localhost:3000 and follow the first-run prompts. If you are on a server, replace localhost with the server’s IP or hostname.

Step 5 – Pull and Test a Model

Use Ollama to download a model. Smaller 7–8B models are a good starting point. You can pull directly from the container or from the WebUI Models page.

# Examples (choose one)
docker exec -it ollama ollama pull llama3.1:8b
docker exec -it ollama ollama pull phi3:mini
docker exec -it ollama ollama pull mistral:7b

After the download, open Open WebUI and start a new chat. Pick the model you pulled and send a test prompt. If you see fast tokens and low latency, your GPU is active. If generation is slow, you may be on CPU.

Step 6 – Secure, Persist, and Back Up

Enable authentication in Open WebUI and control who can sign up. You can preconfigure basic auth behavior with environment variables. Example: disable new signups and set an admin email.

# Stop and re-create Open WebUI with tighter auth (example)
docker rm -f open-webui
docker run -d --name open-webui \
  --network llmnet \
  -p 3000:8080 \
  -e OLLAMA_BASE_URL=http://ollama:11434 \
  -e ENABLE_SIGNUP=false \
  -e [email protected] \
  -v open-webui:/app/backend/data \
  ghcr.io/open-webui/open-webui:latest

To back up models and chat history, archive the Docker volumes. This keeps your setup portable.

# Backup Ollama models
docker run --rm -v ollama:/data -v "$PWD":/backup alpine \
  tar czf /backup/ollama-volume-backup.tgz -C /data .

# Backup Open WebUI data
docker run --rm -v open-webui:/data -v "$PWD":/backup alpine \
  tar czf /backup/open-webui-volume-backup.tgz -C /data .

To update, pull the latest images and recreate:

docker pull ollama/ollama:latest
docker pull ghcr.io/open-webui/open-webui:latest
docker rm -f open-webui ollama
# Re-run the "docker run" commands from Steps 3 and 4

Performance Tips

- Prefer smaller, quantized models (e.g., 7–8B) if you have limited VRAM. Many Ollama models include quantized tags that fit 8–12 GB GPUs.
- Close other GPU-heavy apps to free VRAM.
- Keep GPU drivers and Docker updated for the best kernel-accelerated performance.

Troubleshooting

Open WebUI cannot reach Ollama: Make sure both containers share the same network and the URL is correct: http://ollama:11434. Run docker logs open-webui for connection errors.

“no gpus found” or slow generation: On Linux with NVIDIA, confirm nvidia-smi works on the host and that nvidia-container-toolkit is installed. Run the container with --gpus=all. On AMD, pass --device=/dev/kfd --device=/dev/dri --group-add video. On Windows, ensure Docker Desktop is using WSL2 and that your GPU driver is current.

Port already in use: Adjust published ports, e.g., use -p 3001:8080 or -p 11435:11434, and update the URLs accordingly.

Out of memory (VRAM): Pick a smaller or more heavily quantized model. Close other GPU apps and try again.

What’s Next

With Ollama and Open WebUI running, you can add multiple models, enable embeddings and RAG, or connect tools and function calling. This setup gives you a private, fast local AI workspace that you can back up and upgrade in minutes—all without sending your data to the cloud.

Local large language models are now practical on a single PC. In this tutorial, you will deploy Ollama (model runtime) and Open WebUI (a friendly chat interface) using Docker on Windows or Linux. We will enable NVIDIA GPU acceleration, persist models on disk, and cover secure access and troubleshooting. By the end, you will be chatting with a local LLM like llama3.1 in your browser, no cloud required.

What You Will Need

- A 64-bit PC with at least 16 GB RAM. For GPU acceleration, an NVIDIA GPU with 8 GB+ VRAM is recommended.
- Docker Engine or Docker Desktop (Compose v2 included).
- Free disk space (15–30 GB per model is common).
- Optional but recommended: NVIDIA GPU drivers and CUDA runtime for Docker.

Step 1: Prepare Your System (GPU Optional)

Linux (Ubuntu/Debian)
1) Install Docker Engine and the Compose plugin from the official Docker repo.
2) Install NVIDIA GPU drivers from your distro or NVIDIA site.
3) Install the NVIDIA Container Toolkit:
sudo apt-get install -y nvidia-container-toolkit
Then configure and restart Docker:
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
Verify GPU visibility in containers:
docker run --rm --gpus all nvidia/cuda:12.3.2-base-ubuntu22.04 nvidia-smi

Windows 10/11
1) Install the latest NVIDIA GPU driver (Studio or Game Ready).
2) Install Docker Desktop and enable WSL 2 backend during setup.
3) In Docker Desktop > Settings > Resources > WSL integration, enable your default distro.
4) Ensure GPU is exposed to containers. If you run docker run --rm --gpus all nvidia/cuda:12.3.2-base-ubuntu22.04 nvidia-smi and see your GPU, you are ready.

Step 2: Create a Docker Compose File

We will run two containers: ollama (the LLM runtime API) and open-webui (the web front-end). The services will share a network and persistent volumes. Create a folder like ollama-openwebui and a file compose.yaml with the following content:

services:
  ollama:
   image: ollama/ollama:latest
   container_name: ollama
   restart: unless-stopped
   ports:
    - "11434:11434"
   volumes:
    - ollama_data:/root/.ollama
   environment:
    - OLLAMA_KEEP_ALIVE=24h
   deploy:
    resources:
     reservations:
      devices:
       - capabilities: ["gpu"]

  openwebui:
   image: ghcr.io/open-webui/open-webui:latest
   container_name: open-webui
   restart: unless-stopped
   depends_on:
    - ollama
   ports:
    - "3000:8080"
   environment:
    - OLLAMA_BASE_URL=http://ollama:11434
    - WEBUI_AUTH=True
    - DEFAULT_MODELS=llama3.1:8b
   volumes:
    - openwebui_data:/app/backend/data

volumes:
  ollama_data:
  openwebui_data:

Notes:
- The deploy.resources.reservations.devices section hints Compose to request GPU. On Linux, also start with --gpus all if you run containers manually.
- Ports: Ollama API is 11434, Open WebUI is exposed on 3000 (mapped to container 8080).

Step 3: Start the Stack

In the folder containing compose.yaml, run:
docker compose up -d
Wait for both containers to start. You can watch logs with:
docker compose logs -f

Step 4: Pull a Model and Run Your First Chat

Open a terminal and pull a model into Ollama. For a good balance of quality and speed, try Meta’s 8B model:
docker exec -it ollama ollama pull llama3.1:8b
You can test from the CLI:
docker exec -it ollama ollama run llama3.1:8b "Write a haiku about local AI."
If the response appears, the model is working.

Now open your browser and visit http://localhost:3000. Create an admin account (since we set WEBUI_AUTH=True). In Settings > Models, you should see llama3.1:8b. Create a new chat and start prompting.

GPU Acceleration Checks

- If you have an NVIDIA GPU, Ollama should automatically use it. Confirm via logs: docker logs ollama (look for CUDA initialization).
- If you do not have a GPU, Ollama will use CPU. Expect slower generation but it will work.

Useful Options and Performance Tips

- Try smaller variants for low VRAM: llama3.2:3b or phi3:mini.
- You can pin models to GPU RAM by enabling sufficient numa/gpu memory; if VRAM is low, Ollama will offload layers to system RAM.
- To pre-download a model at startup, set DEFAULT_MODELS in the Open WebUI service as shown.
- For multilingual or coding tasks, add models like qwen2.5:7b or codestral.

Security and Remote Access

- Keep WEBUI_AUTH=True to require sign-in. You can also set OPENWEBUI_ADMIN_EMAIL and OPENWEBUI_ADMIN_PASSWORD as environment variables for unattended setups.
- If exposing Open WebUI to the internet, place it behind a reverse proxy (Nginx, Caddy, or Traefik) with HTTPS and strong passwords.
- The Ollama API on port 11434 should remain private unless you need remote access; firewall it if required.

Troubleshooting

- GPU not detected: On Linux, reinstall nvidia-container-toolkit and verify nvidia-smi works both on the host and in a container. On Windows, ensure WSL 2 is enabled and Docker Desktop is up to date.
- “No space left on device”: Increase disk space or prune unused model blobs: docker exec -it ollama ollama rm <model>. You can also clear unused images with docker system prune (caution).
- Slow or out-of-memory: Use a smaller model, reduce context length in Open WebUI, close other GPU-intensive apps, or increase swap on Linux.
- Port in use: Change the published ports in compose.yaml (e.g., "3001:8080") and redeploy.

Updating and Maintenance

To update to the latest versions, run:
docker compose pull
docker compose up -d
Your models are safe in the ollama_data volume, and your chat history lives in openwebui_data. Always back up these volumes before major upgrades.

What’s Next

You now have a privacy-friendly, GPU-accelerated local AI stack. Explore function calling, RAG connectors in Open WebUI, or run multiple models side by side. With Docker and Ollama, swapping models and keeping performance high is only a pull away.

Overview

This step-by-step guide shows you how to deploy Ollama and Open WebUI on Ubuntu 24.04 with NVIDIA GPU acceleration using Docker. With this setup, you can run modern large language models (LLMs) locally, manage them from a clean web interface, and take full advantage of your GPU for high performance. The process covers NVIDIA drivers, Docker, the NVIDIA Container Toolkit, and secure, persistent containers that survive reboots.

What You Will Need

You need a 64-bit Ubuntu 24.04 host with an NVIDIA GPU (Turing or newer recommended), Internet access, a user with sudo rights, and at least 20 GB of free disk space for models. If you are working on a remote server, make sure port 3000 (for Open WebUI) and 11434 (for Ollama) are reachable or routed through a reverse proxy.

1) Install NVIDIA Drivers

First, install the official NVIDIA driver so CUDA can talk to your GPU. Run: sudo ubuntu-drivers autoinstall. When it finishes, reboot with sudo reboot. After the reboot, verify the GPU is visible: nvidia-smi. You should see your GPU name and driver version. If you do not, confirm Secure Boot is disabled or enroll the driver MOK accordingly, then repeat the check.

2) Install Docker Engine on Ubuntu 24.04

Set up Docker from the official repository for best stability and features. Run: sudo apt update && sudo apt install -y ca-certificates curl gnupg. Add Docker’s key and repo: sudo install -m 0755 -d /etc/apt/keyrings, curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg, echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu noble stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null. Then install: sudo apt update && sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin. To run Docker without sudo: sudo usermod -aG docker $USER then newgrp docker.

3) Enable GPU Access in Containers (NVIDIA Container Toolkit)

Install the NVIDIA Container Toolkit so Docker can pass your GPU into containers. Add the key and repo: curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg, curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list. Install and configure: sudo apt update && sudo apt install -y nvidia-container-toolkit, sudo nvidia-ctk runtime configure --runtime=docker, sudo systemctl restart docker. Test GPU passthrough: docker run --rm --gpus all nvidia/cuda:12.6.2-base-ubuntu22.04 nvidia-smi. You should see your GPU listed inside the container.

4) Create a Dedicated Network for AI Services

Create a user-defined Docker network so containers can discover each other cleanly: docker network create ai. This network isolates traffic and lets Open WebUI talk to the Ollama container by name.

5) Run the Ollama Container with GPU Support

Start Ollama and persist its model data in a Docker volume. Run: docker run -d --name ollama --gpus all --restart unless-stopped -p 11434:11434 -v ollama:/root/.ollama --network ai ollama/ollama:latest. The container exposes the Ollama API on port 11434. Check logs with docker logs -f ollama to ensure the server starts without errors.

6) Pull a Model (Llama 3.1 example)

Use Ollama’s CLI inside the container to download a model. For a great balance of speed and quality on consumer GPUs, try an 8B model: docker exec -it ollama ollama pull llama3.1:8b. If you have a smaller GPU (e.g., 6–8 GB VRAM), try a quantized variant like llama3.1:8b-instruct-q4_K_M. You can list models with docker exec -it ollama ollama list.

7) Deploy Open WebUI and Connect to Ollama

Open WebUI provides a friendly interface to chat with models, manage prompts, and configure settings. Start it with: docker run -d --name open-webui --restart unless-stopped -p 3000:8080 -e OLLAMA_API_BASE_URL=http://ollama:11434 -v openwebui:/app/backend/data --network ai ghcr.io/open-webui/open-webui:latest. Open http://<your_server_ip>:3000 in a browser, create your first user (the first account becomes admin), and pick the model you pulled in the previous step. You can now chat with the LLM directly from your browser.

8) Optional: Secure Access with HTTPS

For Internet-facing servers, place a reverse proxy with TLS in front of Open WebUI. A simple approach is Caddy or Nginx Proxy Manager. Point your domain’s DNS to the server, terminate HTTPS on the proxy, and forward to localhost:3000. If you already use Traefik or Nginx, add routes with Let’s Encrypt certificates and restrict access using basic auth or OAuth.

Maintenance and Updates

To update Ollama or Open WebUI, pull new images and recreate containers. Run: docker pull ollama/ollama:latest and docker pull ghcr.io/open-webui/open-webui:latest, then docker stop ollama open-webui and docker rm ollama open-webui. Start them again using the same docker run commands; your data persists in the volumes ollama and openwebui. To back up models and settings, archive the volumes: sudo tar -czf ollama-vol.tgz -C /var/lib/docker/volumes/ollama/_data . and sudo tar -czf openwebui-vol.tgz -C /var/lib/docker/volumes/openwebui/_data ..

Troubleshooting

If the GPU is not detected inside containers, confirm the host driver works with nvidia-smi. Then verify the runtime is configured: docker info | grep -i nvidia. If missing, re-run sudo nvidia-ctk runtime configure --runtime=docker and sudo systemctl restart docker. For permission errors when running Docker, add your user to the docker group as shown above. If downloads are slow or models fail due to VRAM limits, choose smaller or quantized models (e.g., q4_K_M or q5_K_M).

What You Get

After following these steps, you have a modern, GPU-accelerated local AI stack. Ollama handles efficient model runtimes, and Open WebUI gives you a clean chat interface, prompt management, and multi-model control. Because everything runs in Docker with persistent volumes, updates and backups are easy, and you can scale this setup on a workstation or a headless server with minimal changes.

This step-by-step guide shows you how to run a fast, private, and local AI chatbot on Ubuntu 22.04 or 24.04 using Ollama and Open WebUI. You will install the Ollama runtime, pull a modern large language model, and add a clean chat interface via Open WebUI in Docker. Optional steps cover NVIDIA GPU acceleration, API usage, and persistence. The result is a secure, offline-friendly setup suitable for helpdesk, coding assistance, or knowledge base querying without sending data to the cloud.

Why Ollama + Open WebUI

Ollama makes it simple to run and manage open-source LLMs locally (Llama 3.x, Mistral, Phi, Qwen, and more). Open WebUI adds a user-friendly, browser-based chat interface with conversation history, prompt templates, and multi-model support. Together they form a robust, low-maintenance local AI stack for Linux desktops and servers.

Prerequisites

- Ubuntu 22.04 or 24.04 with a non-root sudo user.
- At least 8 GB RAM (16 GB recommended for larger models).
- Optional NVIDIA GPU for acceleration (T4/RTX/RTX A-series, etc.).
- Internet access to download models and containers.

Step 1 — Install Ollama

1) Update packages:
sudo apt update && sudo apt install -y curl ca-certificates
2) Install Ollama:
curl -fsSL https://ollama.com/install.sh | sh
3) Enable as a service:
sudo systemctl enable --now ollama
4) Verify the API is up:
curl http://localhost:11434/api/tags
If you see JSON, Ollama is running correctly.

Step 2 — Pull and test a model

Pull a compact, capable model first to validate your setup:
ollama pull llama3.2
Run an interactive test:
ollama run llama3.2
Type a prompt, then press Ctrl+C to exit. You can later try larger models (for example, ollama pull mistral or ollama pull llama3.1), but start small to confirm everything works.

Step 3 — Install Docker Engine

1) Add Docker’s repo key and source:
sudo apt-get update && sudo apt-get install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(. /etc/os-release; echo $UBUNTU_CODENAME) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
2) Install Docker and the Compose plugin:
sudo apt-get update && sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
3) Add your user to the docker group and refresh your shell:
sudo usermod -aG docker $USER
newgrp docker

Step 4 — Run Open WebUI connected to Ollama

Start Open WebUI and point it to the Ollama API on the host. The --add-host flag maps host.docker.internal to your host’s gateway so the container can reach http://localhost:11434 on the host:

docker run -d --name open-webui --restart=unless-stopped -p 3000:8080 \
--add-host=host.docker.internal:host-gateway \
-e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
-v open-webui:/app/backend/data \
ghcr.io/open-webui/open-webui:main

Open your browser to http://SERVER_IP:3000 (or http://localhost:3000). Create the first admin account, choose a model (for example, llama3.2), and start chatting.

Step 5 — Enable NVIDIA GPU acceleration (optional)

1) Install the latest NVIDIA driver for your GPU using Ubuntu’s Additional Drivers or apt. Reboot if prompted.
2) Install the NVIDIA Container Toolkit so Docker can access the GPU:
distribution=$(. /etc/os-release; echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#' | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update && sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
3) Recreate Open WebUI with GPU access:
docker rm -f open-webui
docker run -d --name open-webui --restart=unless-stopped --gpus all -p 3000:8080 \
--add-host=host.docker.internal:host-gateway \
-e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
-v open-webui:/app/backend/data \
ghcr.io/open-webui/open-webui:main
4) Ollama will also use the GPU automatically when a compatible model is loaded. You can confirm GPU use with nvidia-smi during inference.

Step 6 — Use the Ollama HTTP API

You can script local inference via HTTP without the UI. Example generation request:
curl http://localhost:11434/api/generate -d '{"model":"llama3.2","prompt":"Write a haiku about backups."}'
Chat format with memory:
curl http://localhost:11434/api/chat -d '{"model":"llama3.2","messages":[{"role":"user","content":"Explain DNS in one sentence."}]}'

Step 7 — Persistence, autostart, and updates

- Ollama models are stored under ~/.ollama/models. Back up this directory to avoid re-downloading models.
- The Open WebUI container uses a named volume (open-webui) for its data, which persists across restarts.
- Ollama is already set to start at boot (systemctl enable ollama). The WebUI container uses --restart=unless-stopped so it will auto-start after a reboot.
- Update Ollama: curl -fsSL https://ollama.com/install.sh | sh
- Update Open WebUI: docker pull ghcr.io/open-webui/open-webui:main && docker restart open-webui

Troubleshooting

- Open WebUI cannot connect to Ollama: ensure you used --add-host=host.docker.internal:host-gateway and that curl http://localhost:11434/api/tags works on the host.
- Port already in use: change -p 3000:8080 to a different host port like -p 3333:8080.
- Out of memory or slow responses: try a smaller model (for example, llama3.2 or phi3). Close other apps or add swap. For CPU-only hosts, expect slower performance on large models.
- GPU not used: verify drivers, nvidia-smi, and that the container runs with --gpus all. Pull a GPU-optimized model variant if available.

What you can do next

- Connect knowledge bases or documents using Open WebUI’s RAG features to power local search over PDFs and wikis.
- Add multiple models and switch per chat, benchmarking speed and quality.
- Put Nginx or Caddy in front of :3000 for HTTPS and trusted network access.
- Automate prompts with shell scripts or Python by calling the local Ollama API.

You now have a private, local AI assistant on Ubuntu with a clean web interface, GPU-ready acceleration, and a stable upgrade path—all without sending your data to third-party services.

Overview

This step-by-step guide shows how to deploy a private, local AI stack on Ubuntu using Docker: Ollama for running large language models (LLMs) and Open WebUI as a fast, friendly chat interface. The setup works on CPUs and supports NVIDIA GPUs for acceleration. You will get a secure, self-hosted environment where you can run models like Llama 3.2, Phi-4, and Mistral without sending data to the cloud.

Prerequisites

- Ubuntu 22.04 or 24.04 (server or desktop)
- 16 GB RAM recommended (more for larger models), 30+ GB free disk
- Docker Engine and the Docker Compose plugin
- Optional: NVIDIA GPU with proprietary driver installed (e.g., 535+)

1) Install Docker and Docker Compose

Update your system and install Docker from the official repository for best stability and performance.

sudo apt update && sudo apt install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" | sudo tee /etc/apt/sources.list.d/docker.list >/dev/null
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker $USER && newgrp docker

2) (Optional) Enable NVIDIA GPU in Containers

If you have an NVIDIA GPU, install the NVIDIA Container Toolkit so Docker can pass the GPU into containers:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#' | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update && sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Verify your driver with nvidia-smi. The container will get GPU access when you run it with --gpus all.

3) Create a Docker Network and Volumes

Create a dedicated network so services can talk by name and set up persistent storage:

docker network create llmnet
docker volume create ollama
docker volume create open-webui

4) Run the Ollama Container

Start Ollama. For CPU-only:

docker run -d --name ollama --restart=unless-stopped --network llmnet -p 11434:11434 -v ollama:/root/.ollama ollama/ollama:latest

With NVIDIA GPU acceleration (detected automatically):

docker run -d --name ollama --restart=unless-stopped --network llmnet --gpus all -p 11434:11434 -v ollama:/root/.ollama ollama/ollama:latest

5) Pull a Model

You can manage models from the host via docker exec. Pull a lightweight model to start quickly:

docker exec -it ollama ollama pull llama3.2:3b

Test generation from the command line:

curl http://localhost:11434/api/generate -d '{"model":"llama3.2:3b","prompt":"Say hello in one short line."}'

6) Launch Open WebUI

Open WebUI provides a clean chat interface and model manager. Start it on port 3000 and point it to the Ollama endpoint:

docker run -d --name open-webui --restart=unless-stopped --network llmnet -p 3000:8080 -e OLLAMA_BASE_URL=http://ollama:11434 -v open-webui:/app/backend/data ghcr.io/open-webui/open-webui:latest

Open a browser and visit http://localhost:3000 (or your server IP). Create the first admin account, select a model (e.g., llama3.2:3b), and start chatting. If a model is missing, Open WebUI can pull it automatically via Ollama.

7) Optional: Use Docker Compose

Prefer to keep everything in a single file? Create docker-compose.yml in an empty folder and paste:

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    ports:
     - "11434:11434"
    networks: [llmnet]
    volumes:
     - ollama:/root/.ollama
  open-webui:
    image: ghcr.io/open-webui/open-webui:latest
    container_name: open-webui
    restart: unless-stopped
    ports:
     - "3000:8080"
    environment:
     - OLLAMA_BASE_URL=http://ollama:11434
    networks: [llmnet]
    volumes:
     - open-webui:/app/backend/data
networks:
  llmnet:
    external: true
volumes:
  ollama:
  open-webui:

Start with docker compose up -d. For GPU, prefer the docker run method with --gpus all, or adapt your Compose file using a GPU-capable configuration on your system.

8) Securing and Updating

- Restrict access: if running on a server, firewall ports 11434 and 3000 to trusted IPs.
- Reverse proxy: place Nginx or Caddy in front with HTTPS for remote access.
- Updates: pull newer images and recreate containers: docker pull ollama/ollama:latest && docker pull ghcr.io/open-webui/open-webui:latest, then docker stop and docker rm containers and re-run them. Your data persists in the volumes.

9) Troubleshooting

- Check logs: docker logs -f ollama and docker logs -f open-webui.
- Port in use: change published ports (e.g., -p 3001:8080).
- GPU not detected: validate nvidia-smi, reinstall the NVIDIA Container Toolkit, and ensure --gpus all is present.
- Disk space: models are large; prune unused data with docker system prune and remove models in ollama volume if needed.

10) Quick API and CLI Examples

- Pull another model: docker exec -it ollama ollama pull phi4:latest
- Chat from CLI: docker exec -it ollama ollama run mistral:7b
- Simple REST call: curl http://localhost:11434/api/generate -d '{"model":"phi4:latest","prompt":"Give me two bullet points about container security."}'

You now have a modern, private AI stack using Docker, Ollama, and Open WebUI on Ubuntu. It is fast, flexible, and ready for local development, internal knowledge assistants, and offline experimentation—no cloud required.

Local large language models (LLMs) are now practical for developers, researchers, and privacy-focused teams. In this step-by-step guide, you will install and run Ollama (LLM runtime) and Open WebUI (a modern chat interface) on Ubuntu 22.04/24.04, with optional NVIDIA GPU acceleration. The setup uses Docker for easy updates, isolation, and backups.

Why this stack?

Ollama makes downloading and running models simple, offering a fast API on your machine. Open WebUI provides a sleek, extensible web app for chatting with multiple models, managing prompts, and moderating access. Together, they create a private, cost-effective alternative to cloud AI services.

Prerequisites

You need an Ubuntu 22.04/24.04 system with internet access and a user with sudo rights. If you have an NVIDIA GPU (recommended), you can enable GPU acceleration for much faster inference. CPU-only works too—skip the GPU steps if you do not have a supported GPU.

Step 1: Update the system

Update packages to ensure you have the latest dependencies and security fixes.

sudo apt update && sudo apt -y upgrade
sudo reboot

Step 2 (Optional): Enable NVIDIA GPU support

Install the latest proprietary NVIDIA driver and the container toolkit so Docker can use your GPU. Reboot when asked.

# Install recommended NVIDIA driver
sudo ubuntu-drivers install
sudo reboot

# Install NVIDIA Container Toolkit
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#' | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list > /dev/null

sudo apt update
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

# Verify GPU visibility
nvidia-smi

Step 3: Install Docker Engine and Docker Compose plugin

Install Docker from the official repository to get the latest stable version. Add your user to the docker group to run Docker without sudo.

sudo apt install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | \
sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

sudo usermod -aG docker $USER
newgrp docker

# Quick test
docker run --rm hello-world

Step 4: Create a Docker Compose file for Ollama + Open WebUI

Create a working directory and define services. The configuration below enables GPU when present; for CPU-only, remove the gpus: all line under the Ollama service.

mkdir -p ~/local-llm && cd ~/local-llm
nano docker-compose.yml

services:
  ollama:
    container_name: ollama
    image: ollama/ollama:latest
    restart: unless-stopped
    ports:
      - "11434:11434"
    volumes:
      - ollama:/root/.ollama
    # Comment the next line if you are CPU-only
    gpus: all
    environment:
      - OLLAMA_KEEP_ALIVE=24h

  open-webui:
    container_name: open-webui
    image: ghcr.io/open-webui/open-webui:latest
    depends_on:
      - ollama
    restart: unless-stopped
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
      - WEBUI_SECRET_KEY=change_this_long_random_secret
    volumes:
      - openwebui:/app/backend/data

volumes:
  ollama:
  openwebui:

Step 5: Start the stack

Bring the services up in the background, then check their status. The Open WebUI will be available at http://SERVER_IP:3000 and the Ollama API at http://SERVER_IP:11434.

docker compose up -d
docker compose ps

Step 6: Pull and run a model

Use Ollama to pull an LLM. You can pick models like llama3.2, mistral, or a coding model. The first pull downloads model weights, which can be several GB.

# Pull a general-purpose model
docker exec -it ollama ollama pull llama3.2

# Test it via CLI
docker exec -it ollama ollama run llama3.2 "Write a two-sentence summary of Ubuntu."

# Or use the API
curl http://localhost:11434/api/generate -d '{"model":"llama3.2","prompt":"Hello!"}'

Open your browser to http://SERVER_IP:3000, select the model from the dropdown, and start chatting. In Settings, you can change default models, system prompts, and appearance.

Step 7: Secure access

If exposing the WebUI beyond your LAN, add authentication. In Open WebUI, create an admin user at first login, then disable new signups in Settings. For internet exposure, place a reverse proxy (Nginx, Caddy, or Traefik) with HTTPS (Let’s Encrypt) in front of port 3000.

Step 8: Update and backup

To update, pull the latest images and recreate containers without losing data stored in volumes. To back up, save the volumes before upgrades.

# Update images
docker compose pull
docker compose up -d

# Backup volumes (example)
docker run --rm -v local-llm_ollama:/data -v "$PWD":/backup \
  busybox tar czf /backup/ollama-vol.tgz /data

docker run --rm -v local-llm_openwebui:/data -v "$PWD":/backup \
  busybox tar czf /backup/openwebui-vol.tgz /data

Troubleshooting

GPU not used: Run docker run --rm --gpus all nvidia/cuda:12.3.2-base-ubuntu22.04 nvidia-smi. If it fails, recheck the driver and NVIDIA Container Toolkit steps. Ensure the gpus: all line is present and Docker was restarted.

Permission denied with Docker: You may need to log out and back in after adding your user to the docker group, or run newgrp docker.

Port conflicts: Change the left side of port mappings in docker-compose.yml (e.g., use "8081:8080" for WebUI) and restart.

Slow or failed model pull: Verify disk space and retry. Large models require several GB of free space in the ollama volume.

Uninstall (optional)

To stop and remove everything, run:

cd ~/local-llm
docker compose down
docker volume rm local-llm_ollama local-llm_openwebui

Wrap-up

You have a modern local AI stack: Ollama for fast model serving and Open WebUI for a friendly, multi-model chat interface. With Docker, updates are quick and backups are simple. Add your favorite models, tune system prompts, and integrate the Ollama API into your apps—all without sending data to the cloud.

Overview

In this step-by-step guide, you will learn how to self-host a local AI chat environment on Ubuntu using Ollama and Open WebUI. Ollama runs large language models (LLMs) locally and exposes a simple API, while Open WebUI provides a modern browser interface, chat history, and prompt management. This tutorial targets Ubuntu 22.04/24.04 and shows how to enable NVIDIA GPU acceleration, secure the service, and test the API.

Prerequisites

You need an Ubuntu 22.04/24.04 machine with at least 16 GB RAM for comfortable use and an NVIDIA GPU with recent drivers (525+ recommended) if you want hardware acceleration. For CPU-only usage, Ollama still works but will be slower. You also need a user with sudo privileges and internet access.

Step 1: Update the system and install basics

Start by refreshing your package lists and installing useful tools such as curl and ufw. Run: sudo apt update && sudo apt -y upgrade and then sudo apt -y install curl ca-certificates ufw. This ensures you have the latest security updates and a firewall ready to configure later.

Step 2: Verify NVIDIA GPU (optional but recommended)

If you intend to use GPU acceleration, confirm your NVIDIA driver installation. Run nvidia-smi. If the command shows your GPU and driver version, you are ready. If not, install a recommended driver with sudo ubuntu-drivers autoinstall, reboot using sudo reboot, and check again with nvidia-smi. Ollama includes the runtime pieces it needs and will automatically use your GPU when supported.

Step 3: Install Ollama

Ollama provides a one-line installer for Linux. Run: curl -fsSL https://ollama.com/install.sh | sh. This installs the ollama binary and sets up a systemd service called ollama. After the script completes, verify the service with systemctl status ollama. If it is not running, start it using sudo systemctl start ollama and enable it at boot with sudo systemctl enable ollama.

Step 4: Pull your first model

Ollama hosts many popular models. To start, pull a reasonably fast, high-quality base model like Meta’s Llama 3.1. Run ollama pull llama3.1. Other good options include mistral, qwen2.5, or smaller quantized variants that fit into limited VRAM (for example, llama3.1:8b or mistral:7b-instruct). Use ollama list to see installed models.

Step 5: Test the local API

Ollama listens on http://127.0.0.1:11434 by default. You can chat in the terminal with ollama run llama3.1. To test via API, run: curl http://127.0.0.1:11434/api/generate -d '{"model":"llama3.1","prompt":"Say hello from a local model."}'. You should see a streamed JSON response. Press Ctrl+C to stop streaming if needed.

Step 6: Install Docker (for Open WebUI)

Open WebUI is easiest to deploy with Docker. Install Docker and its prerequisites. First run: sudo apt -y install apt-transport-https gnupg lsb-release. Then add Docker’s repository key: curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker.gpg. Add the repo: echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null. Install Docker: sudo apt update && sudo apt -y install docker-ce docker-ce-cli containerd.io. Optionally add your user to the Docker group: sudo usermod -aG docker $USER and re-login.

Step 7: Deploy Open WebUI

Open WebUI connects to Ollama’s API and provides a rich chat interface. Create a persistent volume directory and run the container pointing to the local Ollama endpoint. Use: docker run -d --name open-webui -p 3000:8080 -e OLLAMA_API_BASE_URL=http://host.docker.internal:11434 -v openwebui-data:/app/backend/data --restart unless-stopped ghcr.io/open-webui/open-webui:latest. On Linux, if host.docker.internal is not available, replace it with the host’s IP (e.g., http://127.0.0.1:11434) and add --network host instead of -p mapping if you prefer host networking: docker run -d --name open-webui --network host -e OLLAMA_API_BASE_URL=http://127.0.0.1:11434 -v openwebui-data:/app/backend/data --restart unless-stopped ghcr.io/open-webui/open-webui:latest.

Step 8: Secure optional access and firewall

If you will only use the services locally, keep them bound to localhost and do not expose ports publicly. For remote access on a trusted LAN, allow Open WebUI’s port via UFW using sudo ufw allow 3000/tcp (or none if using --network host and default port 8080). Enable the firewall with sudo ufw enable, then verify rules with sudo ufw status. For public access, place Open WebUI behind a reverse proxy (Nginx, Caddy, or Traefik) with HTTPS and authentication.

Step 9: Use the interface

Open a browser to http://SERVER_IP:3000 (or http://localhost:3000). On first load, you can create an admin account, choose your default model (e.g., llama3.1), manage prompts, and run chats. You can switch models per-conversation and configure system prompts for specific tasks like coding, summarization, or Q&A.

Troubleshooting and tips

If a model fails to load due to GPU memory limits, pull a smaller or more heavily quantized variant such as llama3.1:8b or a q4_k_m quant. If CPU usage is too high, reduce the context window or batch size in Open WebUI settings. If the Open WebUI container cannot reach Ollama, double-check OLLAMA_API_BASE_URL, networking mode, and whether the ollama service is running. For best performance on NVIDIA GPUs, close other GPU-heavy apps and monitor usage with nvidia-smi. To update, run sudo systemctl stop ollama && curl -fsSL https://ollama.com/install.sh | sh && sudo systemctl start ollama and pull newer model versions as needed.

What you have now

You have a fully local AI chat stack with GPU acceleration using Ollama and a clean, user-friendly interface via Open WebUI. It is private by default, fast on modern GPUs, and flexible with many model choices. You can integrate it with other tools via the Ollama API for scripting, automation, and offline workflows. This setup gives you control over costs, data privacy, and performance while staying current with the latest open models.

Overview

This step-by-step guide shows you how to run a local AI assistant on Windows 11 using Ollama and Open WebUI. You will install Ollama, download a model, and connect a user-friendly web interface via Docker. The tutorial is beginner-friendly yet covers advanced options like GPU acceleration, authentication, and storage tuning. By the end, you will have a private, fast, and offline-capable AI setup on your own PC.

Prerequisites

Before you start, make sure you have: Windows 11 (22H2 or newer), administrator rights, and at least 8 GB RAM. For GPU acceleration, install the latest graphics driver. Ollama uses CUDA for NVIDIA GPUs and DirectML for AMD/Intel; GPU use is automatic if supported. You do not need WSL for this guide. An optional Docker Desktop installation is required for Open WebUI.

Step 1 — Install Ollama for Windows

1) Download the official installer from https://ollama.com/download and complete the setup.
2) Open PowerShell and verify the installation: ollama --version.
3) Start the Ollama service if it is not already running: ollama serve (you can keep it in the background by closing the window after confirming it is running as a service).

Step 2 — Pull and test a model

1) In PowerShell, download a model. For a good balance of speed and quality, try: ollama pull llama3.
2) Run it interactively: ollama run llama3, then ask a question like: What can you do?.
3) Exit the session with /bye when finished. Models are stored locally in %LOCALAPPDATA%\Ollama\models by default.

Step 3 — Install Docker Desktop (for Open WebUI)

Open WebUI gives you a clean web interface for prompts, chat history, and multi-model workflows. Install Docker Desktop from https://www.docker.com/products/docker-desktop/ and start it. Ensure the Docker engine is running (the whale icon should be active in the system tray).

Step 4 — Launch Open WebUI linked to Ollama

Run the following Docker command in PowerShell to start Open WebUI and connect it to your local Ollama instance exposed at http://localhost:11434:
docker run -d --name open-webui -p 3000:8080 -e OLLAMA_BASE_URL=http://host.docker.internal:11434 -v open-webui:/app/backend/data ghcr.io/open-webui/open-webui:latest
Once the container is healthy, open http://localhost:3000 in your browser. Choose a model (for example, llama3) and start chatting.

Optional — Enable authentication for Open WebUI

To protect your UI with a login, recreate the container with auth variables:
docker rm -f open-webui
docker run -d --name open-webui -p 3000:8080 -e OLLAMA_BASE_URL=http://host.docker.internal:11434 -e WEBUI_AUTH=true -e DEFAULT_USERNAME=admin -e DEFAULT_PASSWORD=changeMeNow -v open-webui:/app/backend/data ghcr.io/open-webui/open-webui:latest
Visit http://localhost:3000 and sign in with your credentials.

Optional — GPU acceleration tips

Ollama automatically uses your GPU when supported drivers are present. To nudge usage, you can set the number of GPUs: setx OLLAMA_NUM_GPU 1 then restart the Ollama service or your PC. If you have an NVIDIA GPU, ensure the latest Game Ready or Studio driver is installed. For AMD/Intel, keep your driver and Windows up to date to benefit from DirectML improvements. During the first run, the model may compile kernels; subsequent runs are faster.

Optional — Move the models folder to another drive

If you want models on a larger drive, set this environment variable and restart the service: setx OLLAMA_MODELS "D:\Ollama\Models". Move the existing folder from %LOCALAPPDATA%\Ollama\models to the new location to avoid re-downloading large files.

Troubleshooting

Open WebUI cannot connect to Ollama: Make sure Ollama is running: curl http://localhost:11434/api/tags should return a JSON list of models. If it works on the host but not in Docker, confirm the container uses host.docker.internal and port 11434 as shown in the command. Also check Windows Firewall for any blocked inbound rules on Docker or Ollama.

Models are slow or fail to load: Try a smaller model first: ollama pull phi3:mini and run ollama run phi3:mini. Close heavy apps, ensure you have enough RAM/VRAM, and avoid aggressive antivirus scanning of the models folder.

Docker errors on startup: Open Docker Desktop and verify that the engine is running. If ports are already in use, change the mapping (for example, -p 3001:8080) and refresh the browser at the new address.

Usage tips

Inside Open WebUI, create multiple chats per model for different tasks, enable markdown rendering, and configure system prompts for role-specific behavior. In PowerShell, you can also run one-off prompts without the UI: ollama run llama3 "Write a haiku about morning coffee." For reproducibility, export your Open WebUI data with the named volume and back it up regularly.

What you achieved

You now have a private, local AI assistant on Windows 11 powered by Ollama and Open WebUI. You can switch models, run fully offline, and take advantage of your GPU for faster responses. This setup is ideal for coding help, note-taking, drafting, and research without sending your data to external servers.