LifeBytes Journal

Running an AI assistant locally is no longer a niche experiment. With modern open models and lightweight serving tools, you can build a private, ChatGPT-style interface on your own Linux machine—no API keys, no data leaving your network, and full control over updates. In this tutorial, you will install Ollama (for downloading and serving LLMs) and Open WebUI (a clean web interface) using Docker. The result is a fast, self-hosted AI chat you can use for drafting, troubleshooting, and internal knowledge work.

What You’ll Build

By the end, you will have: (1) Ollama running as a local model server, (2) Open WebUI running in a container, and (3) a browser-based chat UI available on your LAN or localhost. This setup works well on Ubuntu Server, Debian, and most modern Linux distributions.

Prerequisites

Hardware: At least 8 GB RAM is recommended for smaller models. For better results, use 16 GB or more. A GPU helps but is not required for CPU-only usage.

Software: A Linux system with sudo access, Docker installed, and basic command-line familiarity. If you don’t have Docker yet, install it via your distribution’s official Docker instructions.

Step 1: Install and Start Ollama

Ollama provides a simple way to download and run large language models locally. Install it with the official script:

Command:

curl -fsSL https://ollama.com/install.sh | sh

After installation, start and enable the service (on most systemd-based systems):

sudo systemctl enable --now ollama

Confirm it is active:

systemctl status ollama

Step 2: Download a Model

Now you’ll pull a model. Choose one that matches your hardware. For a balanced option on many systems, try Llama 3 (size availability depends on what Ollama offers at the moment). Pulling a model can take time because it downloads several GB.

ollama pull llama3

Test it directly in the terminal:

ollama run llama3

Type a short prompt like “Explain RAID 1 in simple terms” to confirm it responds. Exit the session when done.

Step 3: Run Open WebUI in Docker

Open WebUI provides a user-friendly interface that feels similar to popular AI chat apps. It can connect to Ollama running on the host. Start by creating a persistent volume for WebUI data:

docker volume create open-webui

Then run the container. The key setting is the environment variable that tells WebUI where to find Ollama:

docker run -d \
--name open-webui \
-p 3000:8080 \
-e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
-v open-webui:/app/backend/data \
--restart unless-stopped \
ghcr.io/open-webui/open-webui:main

On some Linux hosts, host.docker.internal may not resolve by default. If Open WebUI can’t connect to Ollama, rerun the container with an extra host mapping:

docker rm -f open-webui

docker run -d \
--name open-webui \
-p 3000:8080 \
--add-host=host.docker.internal:host-gateway \
-e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
-v open-webui:/app/backend/data \
--restart unless-stopped \
ghcr.io/open-webui/open-webui:main

Step 4: Open the Web Interface

In your browser, open:

http://localhost:3000

If you’re accessing it from another computer on the same network, replace localhost with your Linux server’s IP address. The first time you open Open WebUI, you will create an admin account. After login, look for the model selection and choose the model you pulled (for example, llama3).

Step 5: Basic Security Hardening

A local AI chat can contain sensitive data, so treat it like an internal app. If this is only for you, bind to localhost by using Docker’s loopback mapping:

-p 127.0.0.1:3000:8080

If you need LAN access, consider placing it behind a reverse proxy (Nginx or Caddy) with HTTPS and authentication. Also make sure your firewall only allows trusted networks to reach port 3000.

Troubleshooting Tips

WebUI shows no models: Confirm Ollama is running and reachable. Try curl http://127.0.0.1:11434 on the host, then check the container logs with docker logs open-webui.

Slow responses: Use a smaller model, close other memory-heavy services, or run on a machine with more RAM. CPU-only inference is usable, but performance varies widely by hardware.

Connection errors from container to host: Use the --add-host=host.docker.internal:host-gateway option shown earlier, and keep the OLLAMA_BASE_URL pointing to http://host.docker.internal:11434.

Next Steps

Once your private AI assistant is working, you can expand it with additional models for different tasks, create separate chats for projects, and experiment with system prompts for consistent tone and formatting. The big advantage of this setup is control: you decide what runs, what gets stored, and how it’s exposed—without relying on external services.

Running a private AI assistant locally is becoming a practical option for developers and IT teams who want faster responses, lower cloud costs, and better control over sensitive code. In this tutorial, you will set up a self-hosted AI “code helper” on a Linux server using Ollama (for running large language models locally) and Open WebUI (a clean web interface). The result is a browser-based assistant you can use for code reviews, script generation, troubleshooting, and documentation drafts—without sending prompts to external services.

What You’ll Build

You will deploy two components: Ollama, which downloads and serves models via a local API, and Open WebUI, which connects to Ollama and provides a chat UI with conversation history. This guide uses Docker to keep the installation clean and easy to update.

Prerequisites

Before you start, prepare a Linux machine (Ubuntu 22.04/24.04, Debian 12, or similar) with at least 8 GB RAM (16 GB is better for larger models) and 20+ GB free disk. A GPU is optional, but a modern CPU works fine for smaller models. You also need Docker and Docker Compose (or the Docker Compose plugin).

Step 1: Install Docker (Ubuntu/Debian)

If Docker is not installed, run the commands below. On other distributions, use the official Docker documentation for your package manager.

Commands:

sudo apt update
sudo apt install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Optional but recommended: allow your user to run Docker without sudo.

sudo usermod -aG docker $USER

Log out and back in after changing group membership.

Step 2: Create a Project Directory

Create a dedicated folder for your deployment so configuration and volumes stay organized.

mkdir -p ~/private-ai && cd ~/private-ai

Step 3: Create a Docker Compose File

Create a file named docker-compose.yml with the content below. It starts Ollama and Open WebUI, stores model data on disk, and makes the web UI available on port 3000.

cat > docker-compose.yml <<'EOF'
services:
ollama:
image: ollama/ollama:latest
container_name: ollama
restart: unless-stopped
ports:
- "11434:11434"
volumes:
- ollama:/root/.ollama

open-webui:
image: ghcr.io/open-webui/open-webui:main
container_name: open-webui
restart: unless-stopped
ports:
- "3000:8080"
environment:
- OLLAMA_BASE_URL=http://ollama:11434
volumes:
- openwebui:/app/backend/data
depends_on:
- ollama

volumes:
ollama:
openwebui:
EOF

Step 4: Start the Services

Bring the stack up in the background and confirm both containers are healthy.

docker compose up -d
docker ps

Now open your browser and go to http://YOUR_SERVER_IP:3000. The first user you create in Open WebUI typically becomes the admin, depending on the version.

Step 5: Download a Model with Ollama

Ollama pulls models on demand. For a lightweight code-focused start, try a smaller model first. Run the command below to download and test a model from inside the Ollama container.

docker exec -it ollama ollama pull codellama:7b
docker exec -it ollama ollama run codellama:7b

If you prefer a general assistant model, you can also try:

docker exec -it ollama ollama pull llama3.1:8b

Once pulled, go back to Open WebUI, start a new chat, and select the model. Your prompts will be processed locally on your server.

Step 6: Basic Hardening and Access Tips

If this server is not strictly internal, place Open WebUI behind a reverse proxy such as Nginx or Caddy and enable HTTPS. At a minimum, restrict access with a firewall so only your office IP/VPN can reach port 3000. On Ubuntu with UFW, you can allow only your admin workstation and block the rest.

sudo ufw allow from YOUR_IP to any port 3000 proto tcp
sudo ufw enable

Troubleshooting Common Problems

Open WebUI can’t see Ollama models: confirm the environment variable OLLAMA_BASE_URL points to http://ollama:11434 (container-to-container), and verify Ollama is listening: docker logs ollama.

Slow responses: smaller models respond faster on CPU. Also check system load and RAM usage. If the machine is swapping heavily, upgrade RAM or choose a smaller model.

Disk usage grows quickly: model files are large. Keep an eye on volumes and remove unused models with docker exec -it ollama ollama list and docker exec -it ollama ollama rm MODELNAME.

Conclusion

With Ollama and Open WebUI, you can run a capable private AI code assistant on your own Linux server in under an hour. This setup is ideal for testing prompts safely, speeding up daily scripting tasks, and keeping sensitive code and logs under your control. Once it’s running, you can experiment with different models, tighten access via HTTPS and VPN, and even dedicate a GPU host later for faster generation.

Overview

If you want an AI assistant for internal documentation, troubleshooting, or drafting replies without sending company data to a third-party cloud, a self-hosted setup is a strong option. In this tutorial, you will deploy a private AI stack on an Ubuntu Server using Docker: Ollama (to run large language models locally) and Open WebUI (a clean web interface for chatting, prompts, and basic management). This approach is practical for homelabs and small teams, and it keeps your prompts and conversation history inside your own network.

What You Will Build

By the end, you will have two containers running: one for Ollama (the model runtime/API) and one for Open WebUI (the front-end). You will also configure persistent storage, pull a model, and confirm everything works from a browser. The steps below are written for Ubuntu Server 22.04/24.04, but will work on most modern Ubuntu releases.

Prerequisites

You need an Ubuntu Server with at least 8 GB RAM (16 GB recommended), 20+ GB free disk, and a modern CPU. A GPU helps performance but is not required for a functional deployment. You also need root or sudo access and a working network connection. If this is a server on a LAN, decide which port you will expose for the web interface (we will use 3000).

Step 1: Install Docker and Docker Compose

First, install Docker using the official repository packages. This ensures you get up-to-date components and fewer compatibility issues with Compose.

Run:

sudo apt update
sudo apt install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin

Optionally allow your user to run Docker without sudo (log out and back in afterward):

sudo usermod -aG docker $USER

Step 2: Create a Project Folder and Compose File

Create a directory to keep your deployment clean and manageable. Then create a docker-compose.yml file that defines both services and persistent volumes.

mkdir -p ~/ai-stack
cd ~/ai-stack
nano docker-compose.yml

Paste the following Compose configuration:

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    ports:
      - "11434:11434"
    volumes:
      - ollama:/root/.ollama

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    restart: unless-stopped
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
    volumes:
      - openwebui:/app/backend/data
    depends_on:
      - ollama

volumes:
  ollama:
  openwebui:

Step 3: Start the Stack

Bring up the containers in the background and confirm they are running.

docker compose up -d
docker compose ps

If you see both services with a “running” state, the base deployment is complete.

Step 4: Pull a Model with Ollama

Now download a model. The best choice depends on your RAM and use case. For many servers, a smaller model is a safe starting point. The command below pulls a popular lightweight model.

docker exec -it ollama ollama pull llama3.2

You can list installed models anytime:

docker exec -it ollama ollama list

Step 5: Log In to Open WebUI and Connect to Ollama

Open a browser and go to http://SERVER-IP:3000. On first launch, Open WebUI asks you to create an admin account. After login, the interface should automatically detect Ollama through the internal Docker network using the OLLAMA_BASE_URL you configured.

Start a new chat, select the model you pulled (for example llama3.2), and send a test prompt such as “Write a short troubleshooting checklist for DNS issues.” If the response appears, your private AI assistant is working end-to-end.

Step 6: Basic Hardening and Practical Tips

Firewall: If this is a public-facing server, do not expose it directly without protection. At minimum, allow only your LAN or VPN subnet to reach port 3000. With UFW, you can restrict access instead of opening the port to everyone.

Reverse proxy: For production use, place Open WebUI behind Nginx or Caddy with HTTPS and authentication. This also makes it easier to use a friendly hostname.

Backups: Your important data lives in Docker volumes. Back up the Open WebUI volume (chat history, settings) and the Ollama volume (models) according to your retention needs.

Updates: Refresh images regularly to get security fixes and new features:

docker compose pull
docker compose up -d

Troubleshooting

Open WebUI loads but no models appear: Verify Ollama is reachable from the Open WebUI container. Check logs with docker logs open-webui and confirm OLLAMA_BASE_URL=http://ollama:11434 is correct.

Model downloads are slow: Large model pulls can take time. Ensure your server has stable internet and enough free disk. You can also choose smaller models to start.

High RAM usage or slow responses: Use a smaller model, reduce concurrent users, or run the service on hardware with more memory. Local AI is resource-intensive by design, and tuning is part of a realistic deployment.

Conclusion

Running Ollama and Open WebUI on Ubuntu Server gives you a private, self-hosted AI assistant that you can control, secure, and integrate into your workflow. Once the base stack is stable, you can expand it with HTTPS, SSO, logging, and routine backups. The key advantage is simple: your prompts and internal context stay on your infrastructure while still giving your team an easy web-based AI experience.

Why run a local AI assistant?

If you write code or manage infrastructure, an AI assistant can speed up tasks like generating scripts, explaining logs, reviewing configs, and drafting documentation. The problem is that many tools send prompts and code to a cloud service. In regulated environments, that is not always allowed. Running an AI model locally gives you better privacy, predictable costs, and the option to keep everything inside your LAN.

In this tutorial, you will install Ollama (a lightweight local model runtime) and Open WebUI (a clean browser interface) on Linux using Docker. The result is a self-hosted, ChatGPT-like web app that talks to your local models.

What you need

Requirements: a modern Linux server or workstation (Ubuntu/Debian/Fedora are fine), at least 8 GB RAM (16 GB+ recommended), and enough disk space for models (10–30 GB is common). CPU-only works, but a supported GPU can improve speed significantly. You also need a user with sudo privileges and outbound internet access to download containers and models.

Step 1: Install Docker (and Docker Compose)

If Docker is already installed, you can skip this section. On Ubuntu/Debian, one of the simplest approaches is using the official repository packages. Install Docker Engine and enable the service. If your distribution provides Docker Compose as a plugin, you can use docker compose (with a space) instead of the older docker-compose binary.

After installation, verify it works by running a test container. If you want to avoid using sudo for every Docker command, add your user to the docker group and log out/in once.

Step 2: Create a project folder

Create a dedicated directory for your stack so it is easy to manage and back up later. For example:

mkdir -p ~/local-ai && cd ~/local-ai

Step 3: Create a Docker Compose file for Ollama and Open WebUI

Create a file named docker-compose.yml in the folder. This setup uses persistent volumes so that downloaded models and WebUI data survive reboots and container upgrades.

nano docker-compose.yml

Paste the following:

<pre> version: "3.9" services: ollama: image: ollama/ollama:latest container_name: ollama restart: unless-stopped ports: - "11434:11434" volumes: - ollama:/root/.ollama open-webui: image: ghcr.io/open-webui/open-webui:main container_name: open-webui restart: unless-stopped depends_on: - ollama ports: - "3000:8080" environment: - OLLAMA_BASE_URL=http://ollama:11434 volumes: - openwebui:/app/backend/data volumes: ollama: openwebui: </pre>

Step 4: Start the services

From the same directory, start the stack:

docker compose up -d

Check that both containers are running:

docker ps

If something fails, view logs:

docker logs -f ollama

docker logs -f open-webui

Step 5: Download a model into Ollama

Ollama pulls models on demand. A practical starting point is a small-to-mid model that fits your RAM. For example, you can pull a general-purpose model like:

docker exec -it ollama ollama pull llama3.2

You can list installed models at any time:

docker exec -it ollama ollama list

If you prefer a coding-focused model, search Ollama’s library and choose one that matches your hardware. The key is to start small, confirm everything works, then scale up to larger models.

Step 6: Open WebUI in your browser

Open your browser and go to:

http://localhost:3000

If you installed this on a server, replace localhost with the server’s IP or hostname (for example, http://10.0.0.20:3000). The first time you load the page, you will create an admin account. After login, Open WebUI should detect Ollama automatically via the OLLAMA_BASE_URL setting.

Step 7: Basic usage tips (for real work)

To make the assistant useful for sysadmin and helpdesk tasks, be specific and provide context. Instead of “Fix this,” try prompts like: “Explain what this Nginx config does and suggest safer defaults” or “Write a bash script that checks disk usage and emails an alert”. When you paste logs, remove secrets and tokens, even if you are staying local.

If responses are slow, use a smaller model, reduce parallel users, or move the stack to a machine with more RAM. Local models are sensitive to memory pressure; swapping to disk can make them feel unusable.

Troubleshooting common problems

Open WebUI loads but shows no models: confirm Ollama is reachable from the WebUI container. The environment variable should be OLLAMA_BASE_URL=http://ollama:11434. Also confirm the model is actually pulled with ollama list.

Port already in use: if another service uses port 3000, change the mapping to something else, for example "8085:8080", then restart with docker compose up -d.

Downloads are slow or failing: this is usually DNS, proxy, or firewall related. Test connectivity from the host, and check whether your environment requires an HTTP proxy for container traffic.

Next steps: make it production-friendly

For a lab setup, HTTP on a LAN is fine. For teams, place Open WebUI behind a reverse proxy like Nginx or Caddy, add TLS, and restrict access with SSO or at least strong passwords. Finally, back up Docker volumes so you do not lose your settings and conversation history when you migrate.

Running a coding-focused AI assistant locally is no longer a “lab-only” experiment. With modern open-source tooling, you can host a private chatbot on your own Linux machine, avoid sending prompts to third-party services, and keep sensitive code snippets inside your network. In this tutorial, you will install Ollama (a lightweight local LLM runtime) and Open WebUI (a clean web interface) using Docker. The result is a fast, browser-based AI assistant you can use for debugging, code review, documentation drafts, and scripting help.

What You Will Build

By the end, you will have a local web app accessible at http://localhost:3000 (or your server’s IP) that talks to an Ollama service running on the same host. You will also learn how to persist data, pull a model, and validate that everything is working. This guide assumes Ubuntu Server or another modern Linux distribution with Docker support.

Prerequisites

Before starting, confirm you have: (1) a Linux server or desktop with at least 8 GB RAM (16 GB+ is better for larger models), (2) Docker and Docker Compose, and (3) enough disk space for model files (often 4–20 GB depending on the model). If you plan to access the UI from another device, ensure your firewall allows inbound TCP 3000.

Step 1: Install Docker and Docker Compose

If Docker is not installed yet on Ubuntu, you can install it with the official repository packages. Run:

sudo apt update
sudo apt install -y docker.io docker-compose-plugin

Enable and start Docker:

sudo systemctl enable --now docker

Optional but recommended: allow your user to run Docker without sudo (log out and back in after):

sudo usermod -aG docker $USER

Step 2: Create a Project Folder

Create a dedicated directory so your configuration and persistent volumes stay organized:

mkdir -p ~/local-ai-webui
cd ~/local-ai-webui

Step 3: Create a Docker Compose File

Create a file named docker-compose.yml and paste the following. This configuration runs Ollama and Open WebUI, and stores data in Docker volumes so updates do not wipe your models or settings:

nano docker-compose.yml

<pre>services: ollama: image: ollama/ollama:latest container_name: ollama restart: unless-stopped ports: - "11434:11434" volumes: - ollama:/root/.ollama open-webui: image: ghcr.io/open-webui/open-webui:main container_name: open-webui restart: unless-stopped ports: - "3000:8080" environment: - OLLAMA_BASE_URL=http://ollama:11434 volumes: - openwebui:/app/backend/data depends_on: - ollama volumes: ollama: openwebui:</pre>

Save and exit. The key line is OLLAMA_BASE_URL, which tells Open WebUI how to reach the Ollama container internally.

Step 4: Start the Services

Launch everything in the background:

docker compose up -d

Check container status:

docker ps

You should see both ollama and open-webui running.

Step 5: Download a Model

Ollama doesn’t ship with a model by default. Pull one that fits your hardware. For a solid balance of speed and quality, many users start with a 7–8B class model. Run:

docker exec -it ollama ollama pull llama3.1:8b

If disk space is tight, choose a smaller model. If you have more RAM and want higher quality, try a larger model, but expect slower responses on CPU-only systems.

Step 6: Open the Web Interface

In your browser, open:

http://localhost:3000

If you’re on a remote server, use:

http://SERVER_IP:3000

On first launch, Open WebUI typically asks you to create an admin account. After logging in, select your downloaded model (for example, llama3.1:8b) and send a test prompt like “Explain this Bash one-liner” or “Refactor this Python function for readability.”

Step 7: Basic Troubleshooting

Open WebUI loads but no models appear: Confirm the model exists inside Ollama with docker exec -it ollama ollama list. If the list is empty, re-run the pull command.

Connection errors to Ollama: Verify the containers are on the same Docker network (Compose does this automatically) and that OLLAMA_BASE_URL points to http://ollama:11434, not localhost.

Slow responses: Local inference is hardware-dependent. Smaller models respond faster. Also check CPU and memory usage with docker stats. If the system is swapping heavily, reduce model size.

Step 8: Updating Safely

To update to newer images without losing data, run:

docker compose pull
docker compose up -d

Because models and settings are stored in volumes, your downloaded models and WebUI configuration should remain intact.

Conclusion

A local AI assistant is a practical upgrade for developers and IT teams who want speed, privacy, and control. With Ollama handling model execution and Open WebUI providing a friendly interface, you can run a capable coding helper on a single Linux host and keep your prompts and snippets in-house. Once this baseline is working, consider placing it behind a reverse proxy, enabling HTTPS, or restricting access to your LAN for a more production-ready setup.

Overview

This step-by-step guide shows you how to run local Large Language Models (LLMs) on Ubuntu using Ollama and Open WebUI. You will install Ollama, optionally enable NVIDIA GPU acceleration, and deploy Open WebUI in Docker to get a fast, friendly chat interface. By the end, you will have a private AI assistant running on your own hardware with secure access options and practical troubleshooting tips.

Prerequisites

Use Ubuntu 22.04 or 24.04 with at least 8 GB of RAM (16 GB recommended). For GPU acceleration, an NVIDIA GPU with 8 GB or more VRAM is ideal. You need sudo access and open ports 11434 for Ollama and 3000 (or your choice) for Open WebUI. This guide covers both CPU-only and GPU setups, so you can start even without a supported GPU.

Step 1: (Optional) Install NVIDIA Drivers and CUDA

If you plan to use a GPU, first confirm your hardware with lspci | grep -i nvidia. Install the recommended driver via sudo ubuntu-drivers autoinstall, then reboot. After rebooting, verify the driver with nvidia-smi. If you will run Open WebUI with GPU access in Docker, also install the NVIDIA container runtime using sudo apt-get install -y nvidia-container-toolkit and configure Docker with sudo nvidia-ctk runtime configure followed by sudo systemctl restart docker.

Step 2: Install Ollama on Ubuntu

Install Ollama with a single command: curl -fsSL https://ollama.com/install.sh | sh. This creates a system service and exposes the local API on http://127.0.0.1:11434. Check the version with ollama -v and verify the service using systemctl status ollama. If you need remote access on your LAN, set the host binding by creating an override file. Run sudo systemctl edit ollama, add [Service] and Environment="OLLAMA_HOST=0.0.0.0:11434", then save, sudo systemctl daemon-reload, and sudo systemctl restart ollama. Only expose Ollama on trusted networks or behind a reverse proxy with authentication.

Step 3: Pull and Run Models with Ollama

Pull a small, fast model to test your setup. For general chat, use ollama pull llama3.2:3b. For coding tasks, try ollama pull qwen2.5-coder:7b or a quantized variant like :q4_0 for lower memory usage. Run an interactive session with ollama run llama3.2 and type your prompt. To generate from the shell, try echo "Explain RAID levels simply" | ollama run llama3.2. Ollama will use the GPU automatically if supported; otherwise it falls back to CPU. Tune performance with environment variables such as OLLAMA_NUM_PARALLEL=1 to reduce memory pressure and OLLAMA_KV_SIZE=512 for larger context windows when your memory allows.

Step 4: Deploy Open WebUI with Docker

Open WebUI provides a clean web interface and multi-model support. If Docker is not installed, add it with sudo apt-get update && sudo apt-get install -y docker.io and ensure it runs at startup with sudo systemctl enable --now docker. Launch Open WebUI connected to Ollama using docker run -d --name open-webui -p 3000:8080 -e OLLAMA_BASE_URL=http://localhost:11434 -v open-webui:/app/backend/data -v /var/lib/ollama:/root/.ollama --restart unless-stopped ghcr.io/open-webui/open-webui:latest. If Open WebUI runs on a different host from Ollama, set OLLAMA_BASE_URL to the Ollama server’s IP, for example http://192.168.1.50:11434. For GPU inside the container, add --gpus all and make sure the NVIDIA container toolkit is configured.

Step 5: Secure Access with a Reverse Proxy and HTTPS

If you plan to reach the interface over the internet, place Open WebUI behind a reverse proxy with TLS and authentication. A simple option is Caddy, which can obtain and renew certificates automatically. For example, you can point a domain to your server and configure Caddy to proxy yourdomain.com to localhost:3000 and enable basic auth. With Nginx, use an SSL server block, set proxy_pass http://127.0.0.1:3000, and enable rate limiting and headers like X-Frame-Options and Content-Security-Policy. Always avoid exposing the raw Ollama port unless you fully trust the network.

Step 6: Updates, Backups, and Autostart

Update Ollama by rerunning the installer or using your package manager if you installed via a repo. To update Open WebUI, pull the latest image with docker pull ghcr.io/open-webui/open-webui:latest and restart the container. Persist your data by backing up /var/lib/ollama and the Docker volume open-webui. Both Ollama and Docker containers start automatically on boot, but you can confirm with systemctl is-enabled ollama and the container’s --restart unless-stopped flag.

API Quick Test

You can call Ollama’s local API directly. After pulling a model, try curl http://localhost:11434/api/generate -d '{"model":"llama3.2","prompt":"Give me three bullet points about containers"}'. This is useful for integrating local LLMs into scripts, chatbots, or development tools without sending data to third parties.

Troubleshooting

If model loading fails with “no space left on device,” free disk space with df -h, remove unused Docker images with docker system prune -a, or delete old models in /var/lib/ollama. If nvidia-smi returns an error, reinstall the driver and ensure Secure Boot is either disabled or configured with signed modules. If port 11434 or 3000 is already in use, change the binding (for example OLLAMA_HOST=0.0.0.0:11435) or stop the conflicting process. On low-memory hosts, choose smaller or more heavily quantized models (for example :q4_0), reduce parallel requests with OLLAMA_NUM_PARALLEL=1, and close other memory-hungry services.

What You Achieved

You now have a private, production-ready local AI stack on Ubuntu. Ollama runs the model backend with optional GPU acceleration, while Open WebUI delivers a modern chat interface. With a reverse proxy and backups in place, you can confidently use local LLMs for coding assistance, content drafting, documentation, and experimentation without sending your data to the cloud.

Overview

This tutorial shows you how to deploy a fast, private, local AI stack on Ubuntu using Ollama and Open WebUI with NVIDIA GPU acceleration. You will install the NVIDIA driver, Docker, and the NVIDIA Container Toolkit, then run Ollama on the host and Open WebUI in a container. By the end, you will have a browser-based interface to run powerful large language models (LLMs) like Llama 3 with CUDA acceleration on your own machine.

Prerequisites

- Ubuntu 22.04 or 24.04 (freshly updated).
- An NVIDIA GPU with at least 6 GB VRAM (more is better).
- sudo privileges and Internet access.
- Optional: a domain or reverse proxy if you plan to expose the UI externally.

Step 1 — Install NVIDIA Driver

Use Ubuntu’s built-in tools to install a compatible proprietary driver. Reboot afterward and confirm the GPU is detected.

sudo apt update
sudo apt install -y ubuntu-drivers-common
sudo ubuntu-drivers autoinstall
sudo reboot

nvidia-smi

If you see a table with your GPU and driver version (e.g., 535+), you are ready for CUDA-enabled workloads.

Step 2 — Install Docker Engine

If Docker is not installed, use the official convenience script. Add your user to the docker group so you can run containers without sudo.

curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER
newgrp docker
docker version

Step 3 — Enable GPU Access in Containers

Install the NVIDIA Container Toolkit so Docker can pass the GPU into containers.

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
| sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
| sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#' \
| sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt update
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Verify GPU visibility inside a container:

docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

Step 4 — Install Ollama (runs on the host)

Ollama simplifies downloading and running LLMs locally. It automatically uses CUDA if your NVIDIA driver is installed.

curl -fsSL https://ollama.com/install.sh | sh

Confirm the service is active and the API is reachable on port 11434:

systemctl --user status ollama || systemctl status ollama
curl http://127.0.0.1:11434/api/tags

Pull and test a model (replace with your preferred model/quantization):

ollama pull llama3
ollama run llama3 "Write a two-line poem about GPUs."

Tip: Use smaller quantizations if VRAM is limited, for example llama3:8b-instruct-q4_0.

Step 5 — Deploy Open WebUI in Docker

Open WebUI provides a clean, modern interface for chatting with models served by Ollama. We will run it in Docker and point it to the host’s Ollama API. On Linux, add a host-gateway entry so the container can reach the host at host.docker.internal.

docker run -d --name open-webui \
  -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
  --gpus all \
  -v open-webui:/app/backend/data \
  ghcr.io/open-webui/open-webui:main

Open your browser at http://<server-ip>:3000. On first login, create a user; that account becomes admin. If you need authentication enabled from the start, add -e WEBUI_AUTH=True to the run command.

Alternative: If --add-host=host-gateway is not supported on your Docker version, use host networking and point to 127.0.0.1:

docker run -d --name open-webui \
  --network host \
  -e OLLAMA_BASE_URL=http://127.0.0.1:11434 \
  --gpus all \
  -v open-webui:/app/backend/data \
  ghcr.io/open-webui/open-webui:main

With host networking, Open WebUI listens on http://0.0.0.0:8080 (no -p flag needed).

Step 6 — Use and Tune Your Local AI

From Open WebUI, select a model (e.g., Llama 3) and start chatting. You can pull additional models with Ollama CLI and they will appear in the UI. To speed up responses and reduce VRAM, try smaller or more aggressive quantizations; to maximize quality, try larger quantizations if your GPU can handle them.

Common environment variables for Open WebUI include:
- WEBUI_AUTH=True to require sign-in.
- OLLAMA_BASE_URL to point to the Ollama server URL.
- PORT to customize the UI port if you use host networking.

Troubleshooting

Open WebUI cannot reach Ollama: Ensure you used --add-host=host.docker.internal:host-gateway and OLLAMA_BASE_URL=http://host.docker.internal:11434, or use host networking. Test connectivity with docker exec -it open-webui curl -s http://host.docker.internal:11434/api/tags.

No GPU in containers: Re-check the container toolkit setup and driver. Run docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi. If it fails, reboot and ensure nvidia-smi works on the host first.

Out-of-memory errors: Use a smaller model or more compressed quantization. Close other GPU-heavy apps. You can also run with a larger system swap to reduce crashes when VRAM is exhausted, but performance will be slower.

Docker permissions: If you see “permission denied,” ensure your user is in the docker group (id to verify) and run newgrp docker or re-log in.

Optional: Reverse Proxy and TLS

If exposing Open WebUI on the Internet, put it behind a reverse proxy (Caddy, Nginx, or Traefik) for HTTPS and access control. At minimum, enforce authentication and limit access to trusted IPs. Never expose Ollama’s port 11434 directly without protection.

Maintenance

- Update Ollama periodically by re-running the install script or checking the project release notes, then systemctl restart ollama.
- Update Open WebUI with docker pull ghcr.io/open-webui/open-webui:main and docker restart open-webui.
- Prune old images and volumes with docker system prune (review carefully before confirming).
- Back up /var/lib/ollama (models) and the Open WebUI volume for settings and chats.

You now have a modern, GPU-accelerated, private AI chat environment running locally on Ubuntu. This setup is fast, secure, and fully under your control—and you can expand it with additional models, prompt libraries, and integrations as your needs grow.

Overview

Running a large language model locally is now practical, fast, and private. In this how-to, you will set up Ollama to serve models on your computer and connect Open WebUI for a friendly chat interface. The steps cover Windows, macOS, and Linux, including GPU acceleration for NVIDIA, Apple Silicon, and supported AMD GPUs. By the end, you will be able to pull models, chat in your browser, and tune performance for your hardware.

Requirements and quick checklist

Hardware: 8 GB RAM minimum (16 GB+ recommended), 10–20 GB free disk for models, and optionally a compatible GPU for acceleration.

GPU support: NVIDIA (CUDA 12 driver), Apple Silicon (M1/M2/M3 via Metal), AMD ROCm on supported Linux cards. If you lack a compatible GPU, CPU-only still works, just slower.

Network and security: Keep Ollama bound to localhost unless you intentionally expose it behind a reverse proxy with authentication. Do not publish it directly to the internet.

Step 1 — Install Ollama

Windows: Install via winget or the official installer.

winget install Ollama.Ollama

macOS: Use Homebrew or the DMG from the website.

brew install ollama

Linux: Use the official script (requires curl and sudo).

curl -fsSL https://ollama.com/install.sh | sh

After installation, ensure the service is running. On macOS and Windows, the background service starts automatically. On Linux, start it in a terminal or as a service:

ollama serve

Verify the API is alive by visiting http://127.0.0.1:11434 in your browser. You should see a simple status page.

Step 2 — Pull and test a model

Pull a compact, fast model first to validate everything. Llama 3.2 3B is a great starting point for laptops.

ollama pull llama3.2:3b
ollama run llama3.2:3b

Type a quick prompt and confirm you get a response. For stronger reasoning, try Mistral or an 8B Llama if your RAM/GPU can handle it:

ollama pull mistral:7b
ollama pull llama3.1:8b

Step 3 — Enable GPU acceleration (optional but recommended)

NVIDIA on Windows/Linux: Install the latest Game Ready/Studio driver with CUDA 12 support. Verify with:

nvidia-smi

Ollama will use your GPU automatically if supported. If VRAM is limited, pick a smaller or more aggressively quantized model (for example, Q4 or Q5 builds).

Apple Silicon: No extra steps. Metal acceleration is used by default on M-series chips.

AMD on Linux (ROCm): Use a ROCm-supported GPU and drivers (ROCm 6.x+). Check your distro’s ROCm documentation. Not all AMD GPUs are supported; verify before investing time.

Step 4 — Install Open WebUI

Open WebUI gives you a clean, modern chat interface for Ollama. Docker is the easiest installation path. Make sure Docker Desktop (Windows/macOS) or Docker Engine (Linux) is installed and running.

Windows/macOS (Docker Desktop):

docker run -d --name open-webui -p 3000:8080 \
  -e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
  -v open-webui:/app/backend/data \
  ghcr.io/open-webui/open-webui:latest

Linux: The host networking mode is simplest so the container reaches Ollama on localhost.

docker run -d --name open-webui --network host \
  -e OLLAMA_BASE_URL=http://127.0.0.1:11434 \
  -v open-webui:/app/backend/data \
  ghcr.io/open-webui/open-webui:latest

Open your browser to http://127.0.0.1:3000, create an account (local), and select your Ollama model from the dropdown. Start chatting immediately.

Step 5 — Performance tips and model management

Use quantized models (GGUF variants) to fit your hardware. Q4_K_M is a balanced choice for speed and quality; Q6 is higher quality; Q2/Q3 are very small and fast but lose detail. If a model fails to load, try a smaller parameter count or lower quantization level.

Keep an eye on your RAM/VRAM while the model loads. If memory spikes, reduce context length (token window) in your client settings. Many 7B models run well with 4–6 GB VRAM; 8B often prefers 8–10 GB; CPU-only runs better with 3B–7B models.

List and manage your models with:

ollama list
ollama rm <model-name>

You can tweak behavior with a Modelfile to set defaults like temperature and system prompts. Example:

# Modelfile
FROM llama3.2:3b
PARAMETER temperature 0.7
SYSTEM You are a helpful technical assistant.

ollama create my-tech-assistant -f Modelfile
ollama run my-tech-assistant

Step 6 — Security and remote access basics

By default, Ollama listens on 127.0.0.1:11434, which is safe for single-machine use. If you need remote access on your LAN, set a bind address with an environment variable:

export OLLAMA_HOST=0.0.0.0:11434   # Linux/macOS example

If you expose it, protect it. Use a reverse proxy (Nginx, Traefik, Caddy) with TLS and authentication, or a mesh VPN like Tailscale. Never expose the Ollama API directly to the public internet.

Troubleshooting

If the model is slow, confirm acceleration is active. On NVIDIA, nvidia-smi should show GPU utilization when generating. For crashes during load, your model may not fit in memory; try a smaller model or reduce the context window. If Open WebUI cannot connect, ensure OLLAMA_BASE_URL is correct for your platform and that the port is not blocked by a firewall.

What’s next

Explore specialized models for coding, summarization, or multilingual tasks. Add embeddings and retrieval in Open WebUI to chat over your PDFs or docs. With Ollama handling the runtime and Open WebUI providing the interface, you own the stack: fast, private, and flexible.

Run Local AI with Ollama and Open WebUI: GPU-Accelerated Setup on Windows and Linux with Docker

Local large language models (LLMs) have matured to the point where you can run fast, private, and cost-effective AI on your own computer or server. In this step-by-step guide, you will deploy Ollama (the LLM backend) and Open WebUI (a sleek web interface) using Docker, with optional GPU acceleration on both Windows and Linux. This stack lets you chat with models like Llama 3, Phi-4, or Mistral, completely on your hardware.

By the end, you will have a browser-based interface, persistent model storage, and a clean way to update or back up your local AI environment. The instructions are written in simple, SEO-friendly language and focus on practical steps.

What You Will Build

You will run two containers on the same Docker network: Ollama exposes an API on port 11434 and performs all model work, while Open WebUI listens on port 3000 and connects to Ollama. You will enable GPU acceleration (NVIDIA or AMD) when available to dramatically improve performance.

Prerequisites

- A 64-bit Windows 11/10 (with WSL2) or a modern Linux distribution (Ubuntu/Debian/CentOS/RHEL).
- Docker installed (Docker Desktop on Windows, Docker Engine on Linux).
- Optional GPU: NVIDIA (CUDA) or AMD (ROCm) with up-to-date drivers. CPU-only also works, but is slower.
- 16 GB RAM recommended; disk space 10–40+ GB depending on model size.

Step 1 – Install Docker

Windows: Install Docker Desktop and enable WSL 2 integration. In Settings, ensure “Use the WSL 2 based engine” is on. Update your GPU driver from NVIDIA/AMD. For NVIDIA, CUDA is not required on Windows for Docker Desktop; the latest Game Ready/Studio drivers are enough.

Linux: Install Docker from your distribution’s repository or Docker’s official repo. Add your user to the docker group and log out/in. Example (Ubuntu):

sudo apt update
sudo apt install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | \
  sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
  https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list >/dev/null
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker $USER

Step 2 – Enable GPU Acceleration (Optional but Recommended)

NVIDIA on Linux: Install the NVIDIA Container Toolkit to pass your GPU into containers.

# Add the NVIDIA container toolkit repo (Ubuntu example)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
  sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

AMD on Linux (ROCm): Install the latest AMDGPU/ROCm stack. To give containers access, pass /dev/kfd and /dev/dri and add the video group. Example device flags are shown in the Ollama run step below.

Windows: Docker Desktop exposes the GPU automatically when the host has a compatible driver. Ensure your GPU driver is up to date and “Use the WSL 2 based engine” is enabled.

Step 3 – Start Ollama (LLM Backend)

Create a Docker network and a persistent volume for models. Then start the Ollama container. Use the NVIDIA command if you have an NVIDIA GPU; use the AMD/CPU command otherwise.

# Common network and volumes
docker network create llmnet
docker volume create ollama

# NVIDIA GPU (Linux or Windows with Docker Desktop)
docker run -d --name ollama \
  --network llmnet \
  --gpus=all \
  -v ollama:/root/.ollama \
  -p 11434:11434 \
  ollama/ollama:latest

# AMD ROCm or CPU-only (Linux)
# Remove the two --device flags if you want CPU-only
docker run -d --name ollama \
  --network llmnet \
  --device=/dev/kfd --device=/dev/dri --group-add video \
  -v ollama:/root/.ollama \
  -p 11434:11434 \
  ollama/ollama:latest

Verify Ollama is live:

curl http://localhost:11434/api/tags
# or
docker logs -f ollama

Step 4 – Start Open WebUI (Front-End)

Open WebUI connects to the Ollama API and gives you a beautiful chat interface. Map port 3000 for access and point it to the Ollama container over the private network.

docker volume create open-webui

docker run -d --name open-webui \
  --network llmnet \
  -p 3000:8080 \
  -e OLLAMA_BASE_URL=http://ollama:11434 \
  -v open-webui:/app/backend/data \
  ghcr.io/open-webui/open-webui:latest

Open your browser at http://localhost:3000 and follow the first-run prompts. If you are on a server, replace localhost with the server’s IP or hostname.

Step 5 – Pull and Test a Model

Use Ollama to download a model. Smaller 7–8B models are a good starting point. You can pull directly from the container or from the WebUI Models page.

# Examples (choose one)
docker exec -it ollama ollama pull llama3.1:8b
docker exec -it ollama ollama pull phi3:mini
docker exec -it ollama ollama pull mistral:7b

After the download, open Open WebUI and start a new chat. Pick the model you pulled and send a test prompt. If you see fast tokens and low latency, your GPU is active. If generation is slow, you may be on CPU.

Step 6 – Secure, Persist, and Back Up

Enable authentication in Open WebUI and control who can sign up. You can preconfigure basic auth behavior with environment variables. Example: disable new signups and set an admin email.

# Stop and re-create Open WebUI with tighter auth (example)
docker rm -f open-webui
docker run -d --name open-webui \
  --network llmnet \
  -p 3000:8080 \
  -e OLLAMA_BASE_URL=http://ollama:11434 \
  -e ENABLE_SIGNUP=false \
  -e [email protected] \
  -v open-webui:/app/backend/data \
  ghcr.io/open-webui/open-webui:latest

To back up models and chat history, archive the Docker volumes. This keeps your setup portable.

# Backup Ollama models
docker run --rm -v ollama:/data -v "$PWD":/backup alpine \
  tar czf /backup/ollama-volume-backup.tgz -C /data .

# Backup Open WebUI data
docker run --rm -v open-webui:/data -v "$PWD":/backup alpine \
  tar czf /backup/open-webui-volume-backup.tgz -C /data .

To update, pull the latest images and recreate:

docker pull ollama/ollama:latest
docker pull ghcr.io/open-webui/open-webui:latest
docker rm -f open-webui ollama
# Re-run the "docker run" commands from Steps 3 and 4

Performance Tips

- Prefer smaller, quantized models (e.g., 7–8B) if you have limited VRAM. Many Ollama models include quantized tags that fit 8–12 GB GPUs.
- Close other GPU-heavy apps to free VRAM.
- Keep GPU drivers and Docker updated for the best kernel-accelerated performance.

Troubleshooting

Open WebUI cannot reach Ollama: Make sure both containers share the same network and the URL is correct: http://ollama:11434. Run docker logs open-webui for connection errors.

“no gpus found” or slow generation: On Linux with NVIDIA, confirm nvidia-smi works on the host and that nvidia-container-toolkit is installed. Run the container with --gpus=all. On AMD, pass --device=/dev/kfd --device=/dev/dri --group-add video. On Windows, ensure Docker Desktop is using WSL2 and that your GPU driver is current.

Port already in use: Adjust published ports, e.g., use -p 3001:8080 or -p 11435:11434, and update the URLs accordingly.

Out of memory (VRAM): Pick a smaller or more heavily quantized model. Close other GPU apps and try again.

What’s Next

With Ollama and Open WebUI running, you can add multiple models, enable embeddings and RAG, or connect tools and function calling. This setup gives you a private, fast local AI workspace that you can back up and upgrade in minutes—all without sending your data to the cloud.

This guide shows how to deploy a modern reverse proxy using Traefik v3, Docker, and Docker Compose on Ubuntu 22.04 or 24.04. You will get automatic Let’s Encrypt SSL certificates, HTTP to HTTPS redirection, and a clean way to route multiple apps on the same server under different domains or subdomains. The steps are simple, repeatable, and safe for production.

Prerequisites

Before starting, ensure you have: (1) An Ubuntu 22.04 or 24.04 server with a public IP, (2) A domain or subdomain you can edit DNS for (e.g., example.com), and (3) Access to open TCP ports 80 and 443 on the server’s firewall and network provider. You should also have a non-root user with sudo access.

Step 1 — Install Docker Engine and Compose Plugin

Install the official Docker packages. This gives you the latest stable Docker Engine, Buildx, and the Compose plugin.

sudo apt update
sudo apt install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
https://download.docker.com/linux/ubuntu $(. /etc/os-release; echo $VERSION_CODENAME) stable" | \
sudo tee /etc/apt/sources.list.d/docker.list >/dev/null
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker $USER
newgrp docker

Step 2 — DNS and Firewall

Create DNS A records for your services. For example, point app.example.com and traefik.example.com to your server’s public IP. Then open the firewall for web traffic.

sudo ufw allow 80,443/tcp
sudo ufw reload

Step 3 — Prepare a Docker Network and Traefik Files

Create a dedicated Docker network for the proxy and a directory to hold Traefik files, including the ACME storage for certificates.

docker network create proxy
mkdir -p ~/traefik
cd ~/traefik
touch acme.json
chmod 600 acme.json

Step 4 — Create docker-compose.yml for Traefik v3

The compose file below configures Traefik v3 with automatic HTTPS via the HTTP-01 challenge (ports 80/443). It also binds the dashboard to localhost only for safety.

version: "3.9"

networks:
  proxy:
    external: true

services:
  traefik:
    image: traefik:v3.1
    container_name: traefik
    command:
      - --api.dashboard=true
      - --api.insecure=false
      - --providers.docker=true
      - --providers.docker.exposedbydefault=false
      - --entrypoints.web.address=:80
      - --entrypoints.websecure.address=:443
      - --entrypoints.web.http.redirections.entryPoint.to=websecure
      - --entrypoints.web.http.redirections.entryPoint.scheme=https
      - [email protected]
      - --certificatesresolvers.le.acme.storage=/letsencrypt/acme.json
      - --certificatesresolvers.le.acme.httpchallenge=true
      - --certificatesresolvers.le.acme.httpchallenge.entrypoint=web
      - --log.level=INFO
    ports:
      - "80:80"
      - "443:443"
      - "127.0.0.1:8080:8080" # Dashboard on localhost
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./acme.json:/letsencrypt/acme.json
    networks:
      - proxy
    restart: unless-stopped

Replace [email protected] with an email you control. Traefik uses it to register with Let’s Encrypt. The dashboard is available on http://localhost:8080 and can be reached via SSH tunneling when needed.

Step 5 — Start Traefik

Launch the reverse proxy and confirm it is running.

docker compose up -d
docker ps

You should see the traefik container healthy and listening on ports 80 and 443.

Step 6 — Add a Test App Behind Traefik

Deploy a simple test service (whoami) to verify automatic HTTPS, routing, and headers. Replace app.example.com with your real hostname that points to the server.

docker run -d --name whoami --network proxy --restart unless-stopped \
  -l "traefik.enable=true" \
  -l "traefik.http.routers.who.rule=Host(`app.example.com`)" \
  -l "traefik.http.routers.who.entrypoints=websecure" \
  -l "traefik.http.routers.who.tls.certresolver=le" \
  -l "traefik.http.middlewares.secHeaders.headers.stsSeconds=31536000" \
  -l "traefik.http.middlewares.secHeaders.headers.stsIncludeSubdomains=true" \
  -l "traefik.http.middlewares.secHeaders.headers.stsPreload=true" \
  -l "traefik.http.middlewares.secHeaders.headers.frameDeny=true" \
  -l "traefik.http.middlewares.ratelimit.rateLimit.average=100" \
  -l "traefik.http.middlewares.ratelimit.rateLimit.burst=50" \
  -l "traefik.http.routers.who.middlewares=secHeaders@docker,ratelimit@docker" \
  traefik/whoami:v1.10

Open https://app.example.com in a browser. The first request may take a few seconds while Traefik obtains a certificate. You should see a basic whoami page over HTTPS with a valid lock icon.

Optional: Use DNS-01 Challenge (Wildcard Certificates)

If your ISP or host blocks port 80, or you want wildcard certificates like *.example.com, switch to the DNS-01 challenge. The example below uses Cloudflare. Create a token in Cloudflare with DNS edit permissions for the zone and store it in an .env file.

cd ~/traefik
printf "CF_DNS_API_TOKEN=YOUR_CLOUDFLARE_DNS_TOKEN\n" > .env

Edit docker-compose.yml and replace the ACME lines for HTTP challenge with DNS challenge:

      - --certificatesresolvers.le.acme.dnschallenge=true
      - --certificatesresolvers.le.acme.dnschallenge.provider=cloudflare

Add the environment variable to the traefik service:

    environment:
      - CF_DNS_API_TOKEN=${CF_DNS_API_TOKEN}

Then reload Traefik:

docker compose up -d

For a wildcard, use routers or certificate domains that match *.example.com. With DNS challenge, Traefik can issue certificates without exposing port 80.

Secure the Dashboard (Optional but Recommended)

By default we bound the dashboard to localhost. To access it securely from your workstation, use an SSH tunnel: ssh -L 8080:localhost:8080 user@your_server and then open http://localhost:8080. If you must expose it on a domain, add Basic Auth and IP allowlisting via labels and serve it over HTTPS. For most setups, keeping it local is safer.

Troubleshooting

If certificates do not issue, confirm that your DNS A/AAAA records point to the server and that ports 80 and 443 are reachable from the internet. Check logs with docker logs -f traefik. Rate limits from Let’s Encrypt can apply if you redeploy too often; use a single domain during testing and switch to the production domain when stable.

Maintenance

Traefik renews certificates automatically before expiry; no cron is required. Keep Docker images current by pulling updates periodically: docker compose pull and docker compose up -d. Back up the acme.json file; it contains your issued certificates and keys.

What You Achieved

You now have a modern, production-ready reverse proxy with Traefik v3, automatic HTTPS via Let’s Encrypt, and a clean Docker-based workflow. Adding new apps is as simple as attaching them to the proxy network and setting a few labels. This pattern scales well, stays secure, and keeps your server easy to manage.

Overview

This guide shows you how to run a private, local AI chatbot on Ubuntu using Ollama (for running large language models on CPU or GPU) and Open WebUI (a clean web interface). You will be able to chat with models like Llama 3 locally, without sending data to the cloud. The steps work on Ubuntu 22.04 and 24.04, and are suitable for both desktops and servers.

What You Will Set Up

You will install Ollama on the host, pull a model, and run Open WebUI in Docker. Open WebUI will connect to the Ollama API on port 11434. The result is a self-hosted, secure, and fast AI assistant accessible at http://YOUR_SERVER_IP:3000.

Prerequisites

- Ubuntu 22.04 or 24.04 with sudo access
- At least 8 GB RAM (16 GB+ recommended for 7B–8B models; use smaller quantized models on low-RAM systems)
- Optional: NVIDIA or AMD GPU for faster inference
- Internet access to download packages and models

Step 1: Install Ollama

Ollama is a lightweight runtime that serves models locally over an HTTP API (default: 11434). Install it with one command:

curl -fsSL https://ollama.com/install.sh | sh

Verify the installation and service:

ollama --version
systemctl status ollama

If the service is inactive, start it:

sudo systemctl enable --now ollama

Step 2: Pull a Model (Llama 3.1 as an example)

Download a good general-purpose model. Llama 3.1 8B is a solid balance for many machines:

ollama pull llama3.1:8b

Test it in the terminal:

ollama run llama3.1:8b
>>> Write a 1-sentence productivity tip.

Tip: If you run out of memory, pull a quantized variant, for example:

ollama pull llama3.1:8b-instruct-q4_K_M

Step 3: Install Docker Engine

Open WebUI ships a reliable container image. Install Docker from the official repository:

sudo apt update
sudo apt install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" \
| sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker $USER

Log out and in again (or run a new shell) to use Docker without sudo.

Step 4: Run Open WebUI and Connect to Ollama

Start Open WebUI in Docker, mapping port 3000 and pointing it to the Ollama API on the host. The host gateway alias works on modern Docker versions:

docker run -d --name open-webui \
  --restart unless-stopped \
  -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
  -v open-webui:/app/backend/data \
  ghcr.io/open-webui/open-webui:latest

Open your browser at http://YOUR_SERVER_IP:3000, create an admin account on first launch, and select the default model (e.g., llama3.1:8b). You can now chat and manage prompts, files, and settings.

If the container cannot reach the host, an alternative is host networking:

docker run -d --name open-webui \
  --restart unless-stopped \
  --network=host \
  -e OLLAMA_BASE_URL=http://127.0.0.1:11434 \
  -v open-webui:/app/backend/data \
  ghcr.io/open-webui/open-webui:latest

Step 5: Secure Access

- By default, Open WebUI uses account-based sign-in. In Settings > Admin panel, require authentication for all users.
- If you run on a public server, restrict binding to localhost and use an SSH tunnel:

# bind only to localhost:
docker run -d --name open-webui \
  --restart unless-stopped \
  -p 127.0.0.1:3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
  -v open-webui:/app/backend/data \
  ghcr.io/open-webui/open-webui:latest

# from your laptop:
ssh -L 3000:localhost:3000 [email protected]

Step 6: Use Your GPU (Optional)

Ollama will use your GPU if supported drivers are installed. For NVIDIA, install the proprietary driver and CUDA libraries (the standard Ubuntu “Additional Drivers” tool works). For AMD, install ROCm per Ubuntu/AMD documentation. Because Open WebUI talks to Ollama’s API, you do not need GPU support inside the Open WebUI container—only in Ollama on the host.

Step 7: Model Management Tips

- List models: ollama list
- Show model metadata: ollama show llama3.1:8b
- Remove a model: ollama rm MODEL_NAME
- Try alternatives: ollama pull mistral, ollama pull qwen2, or ollama pull phi3:mini for lower memory systems.

Troubleshooting

- Port in use: Change the mapped port, for example -p 4000:8080 and open http://YOUR_SERVER_IP:4000.
- Container cannot reach Ollama: Use --network=host or ensure --add-host=host.docker.internal:host-gateway is present.
- Out-of-memory: Pull a smaller/quantized model (e.g., q4_K_M variants) or close other apps.
- Slow responses: Prefer GPU, reduce context length in Open WebUI settings, or choose a smaller model.

Updating and Maintenance

- Update Ollama: rerun the install script, then restart the service: curl -fsSL https://ollama.com/install.sh | sh && sudo systemctl restart ollama.
- Update Open WebUI:

docker pull ghcr.io/open-webui/open-webui:latest
docker stop open-webui && docker rm open-webui
# run again with the same docker run command used earlier

Conclusion

With Ollama and Open WebUI, you can host a private, fast, and flexible chatbot on your own Ubuntu machine. This setup keeps your data local, supports multiple open models, and can leverage your GPU for speed. Whether you are a helpdesk, a developer, or a power user, this self-hosted stack gives you full control over your AI workflow.

Overview

This step-by-step guide shows how to run private, local large language models with Ollama and a modern chat interface using Open WebUI. We will cover installing Ollama on Ubuntu and Windows, enabling GPU acceleration, pulling popular models like Llama 3, and deploying Open WebUI with Docker so you can chat, run tools, and manage prompts from a browser. The result is a fast, secure, and offline-friendly AI stack that you control.

Prerequisites

You will need a 64-bit system, administrator privileges, and at least 16 GB of RAM for 7B–8B models. GPU acceleration is recommended for speed: keep your NVIDIA/AMD/Intel graphics drivers up to date. Ollama listens on port 11434 by default, and Open WebUI will run on port 3000. Ensure your firewall allows local access or your chosen LAN range.

Step 1 — Install Ollama

Ubuntu 22.04/24.04: Install Ollama with the official script, which adds the service and keeps it updated.

curl -fsSL https://ollama.com/install.sh | sh
sudo systemctl enable --now ollama
curl http://127.0.0.1:11434/api/version

You should see a version string from the last command. If not, check the service: sudo systemctl status ollama.

Windows 11/10: Install via the official MSI or Winget, then verify the local API.

winget install Ollama.Ollama
curl http://127.0.0.1:11434/api/version

On Windows, Ollama runs as a user service. If you use a third-party firewall, allow local traffic to port 11434.

Step 2 — Enable GPU Acceleration

GPU acceleration in Ollama is automatic when compatible drivers and runtimes are present. On Linux, install your vendor’s proprietary GPU driver. On Windows, use the latest Game Ready/Studio driver from the GPU vendor. After pulling a model and making a test prompt, watch the Ollama logs. If the run mentions the GPU and performance is high (tokens per second are significantly better than CPU), acceleration is working.

If you suspect CPU fallback, update drivers, make sure your GPU has enough VRAM for the chosen model size, and try a smaller variant (for example, 8B instead of 13B). On laptops with hybrid graphics, set the app/GPU preferences so Ollama can use the discrete GPU.

Step 3 — Pull a Model and Test Locally

Pull a model using the Ollama CLI. Popular, high-quality choices include Llama 3 and Mistral. The first run downloads and prepares weights; subsequent runs start instantly.

# Examples (pick one)
ollama pull llama3:8b
ollama pull llama3.1:8b
ollama pull mistral:7b

Now run a quick prompt:

ollama run llama3:8b
# At the prompt, type:
# What are three creative use cases for local AI at home?

If responses are slow or you see out-of-memory errors, switch to a smaller model or close GPU-intensive applications.

Step 4 — Deploy Open WebUI with Docker

Open WebUI adds a polished browser interface, prompt library, chat history, and extensions like RAG (retrieve and ground answers in your documents). We will connect it to your host’s Ollama instance. The following Docker Compose works on Linux and Windows. It uses host.docker.internal to reach the host-based Ollama API and maps persistent storage for Open WebUI data.

mkdir -p ~/openwebui && cd ~/openwebui
cat > docker-compose.yml <<'YAML'
services:
  openwebui:
    image: ghcr.io/open-webui/open-webui:latest
    container_name: open-webui
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://host.docker.internal:11434
    extra_hosts:
      - "host.docker.internal:host-gateway"
    volumes:
      - open-webui-data:/app/backend/data
    restart: unless-stopped
volumes:
  open-webui-data:
YAML

docker compose up -d

Open your browser and visit http://localhost:3000. Create your account when prompted, pick your default model (for example, llama3:8b), and send a test message. If the UI cannot connect, ensure the Ollama service is running and that your firewall allows local connections to port 11434.

Optional — Run Both Ollama and Open WebUI in Docker

If you prefer everything containerized, you can run Ollama and Open WebUI in the same Compose file. This is convenient on servers. GPU pass-through in Docker requires recent drivers and, on Linux, the NVIDIA Container Toolkit. When in doubt, keep Ollama native and only containerize Open WebUI, as shown above.

Security, Updates, and Backups

Do not expose ports 11434 or 3000 directly to the internet. If you need remote access, place Open WebUI behind a reverse proxy (Nginx, Caddy, or Traefik) with HTTPS and strong authentication, or publish it through a zero-trust tunnel. Inside Open WebUI, enable authentication and limit registration to trusted users. Keep Docker images current by pulling the latest tags and recreating containers. On Ubuntu, the Ollama installer provides updates via its repository; on Windows, check for updates in the app or Winget. Back up ~/.ollama (models and configs) and your open-webui-data volume to preserve chat history and settings.

Troubleshooting

If Open WebUI says “Cannot connect to Ollama,” verify the API at http://127.0.0.1:11434/api/version and confirm your Compose file includes extra_hosts with host-gateway on Linux. On Windows with Docker Desktop, host.docker.internal works out of the box. If GPU acceleration is missing, update drivers, reboot, and try a smaller model. When Docker containers fail to start, check logs with docker logs open-webui and make sure ports 3000 and 11434 are not in use by other applications.

What You Can Do Next

With Ollama and Open WebUI running, you can add multiple models, create custom system prompts, and enable RAG by uploading PDFs or notes so the model answers with context from your documents. You can also script batch prompts via the Ollama HTTP API, integrate with automation tools, or point a browser extension to your local endpoint to replace cloud calls. The stack is private, fast, and easy to maintain—ideal for personal knowledge work or secure team deployments.

Overview

This tutorial shows how to self-host Open WebUI with Ollama on Ubuntu using Docker Compose. You will get a clean, repeatable setup that runs on CPU or GPU, stores model data persistently, and can be upgraded with a single command. Open WebUI provides a modern interface, while Ollama runs local large language models such as Llama 3, Mistral, Phi-3, and CodeLlama.

Prerequisites

You will need a fresh Ubuntu 22.04 or 24.04 server (cloud VM or local machine), a user with sudo rights, and at least 8 GB of RAM. If you plan to use a GPU, an NVIDIA card is recommended. Open ports 3000 (Web UI) and 11434 (Ollama API) on your firewall or security group.

1) Install Docker and Compose

Install the official Docker Engine and the Compose plugin on Ubuntu. Log out and back in (or run newgrp) after adding your user to the docker group.

sudo apt update
sudo apt install -y ca-certificates curl gnupg

sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | \
  sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg

echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
  https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo $VERSION_CODENAME) stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

sudo usermod -aG docker $USER
newgrp docker
docker --version
docker compose version

2) Optional: Enable NVIDIA GPU Support

If you have an NVIDIA GPU, install the NVIDIA driver and the NVIDIA Container Toolkit. This lets Ollama use your GPU for faster inference.

sudo apt install -y ubuntu-drivers-common
sudo ubuntu-drivers autoinstall
sudo reboot

After the reboot, install the container toolkit and configure Docker:

distribution=$(. /etc/os-release; echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
  sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit.gpg

curl -fsSL https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt update
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Verify GPU access with Docker:

docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi

3) Create the Docker Compose Stack

Create a project directory and a Compose file that defines two services: Ollama (LLM runtime) and Open WebUI (frontend). The volumes preserve your models and settings across restarts.

mkdir -p ~/openwebui-ollama
cd ~/openwebui-ollama
nano docker-compose.yml

Paste the following content and save:

version: "3.9"

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    ports:
      - "11434:11434"
    volumes:
      - ollama:/root/.ollama
    environment:
      - OLLAMA_KEEP_ALIVE=24h
    # Uncomment the next line if you have GPU configured:
    # gpus: all

  open-webui:
    image: ghcr.io/open-webui/open-webui:latest
    container_name: open-webui
    depends_on:
      - ollama
    restart: unless-stopped
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
    volumes:
      - open-webui:/app/backend/data

volumes:
  ollama:
  open-webui:

4) Start the Services and Download a Model

Bring the stack online. The first start will download container images.

docker compose up -d
docker compose ps

Pull a small model to test. You can add more models later.

docker exec -it ollama ollama pull llama3.2:3b
# Other options: mistral:7b, phi3:mini, qwen2.5:7b

Open your browser to http://SERVER-IP:3000. Create your account in Open WebUI. In the model selector, choose the model you pulled and send a prompt to verify everything works.

5) Persist Data and Backups

Docker volumes keep your models and UI data under /var/lib/docker/volumes. To back them up, stop the stack and archive the data directories. This ensures quick recovery after an OS reinstall or server migration.

docker compose down
sudo tar -czf ollama_data.tgz -C /var/lib/docker/volumes \
  $(docker volume ls -q | grep "_ollama$")/_data

sudo tar -czf openwebui_data.tgz -C /var/lib/docker/volumes \
  $(docker volume ls -q | grep "_open-webui$")/_data

docker compose up -d

6) Secure and Publish (Optional)

If you expose the service on the internet, put it behind a reverse proxy with HTTPS (Caddy, Nginx, or Traefik) and set strong authentication in Open WebUI. Use a DNS name, issue a TLS certificate (Let’s Encrypt), and restrict access with IP allowlists or an identity provider. For small teams, consider running it only on a private network or VPN.

7) Update and Maintenance

Update to the newest images regularly. This pulls security updates, new UI features, and performance improvements.

cd ~/openwebui-ollama
docker compose pull
docker compose up -d

To update models to the latest quantizations or fixes, re-pull them in Ollama. You can remove old ones you no longer need.

docker exec -it ollama ollama pull mistral:7b
docker exec -it ollama ollama list
docker exec -it ollama ollama rm modelname:tag

Troubleshooting

Port already in use: Change the host port mappings in docker-compose.yml (for example, 3001:8080 or 11435:11434) and restart.

GPU not detected: Verify nvidia-smi works on the host and in a test container. Ensure the gpus: all line is uncommented and Docker was restarted after installing the NVIDIA Toolkit.

Slow or failed model pulls: Models can be large. Check disk space (df -h), network speed, and try a smaller model first. You can also mirror models by pre-downloading on another machine and copying the volume data.

Permission errors: Ensure your user is in the docker group (id) and you have logged out/in.

What You Achieved

You now have a production-friendly, self-hosted AI chat stack powered by Open WebUI and Ollama. With Docker Compose, you can start, stop, back up, and upgrade the entire setup with a couple of commands. Add or swap models as your use cases evolve—coding assistants, knowledge chat, or creative writing—while keeping your data local and under your control.

Local large language models are now practical on a single PC. In this tutorial, you will deploy Ollama (model runtime) and Open WebUI (a friendly chat interface) using Docker on Windows or Linux. We will enable NVIDIA GPU acceleration, persist models on disk, and cover secure access and troubleshooting. By the end, you will be chatting with a local LLM like llama3.1 in your browser, no cloud required.

What You Will Need

- A 64-bit PC with at least 16 GB RAM. For GPU acceleration, an NVIDIA GPU with 8 GB+ VRAM is recommended.
- Docker Engine or Docker Desktop (Compose v2 included).
- Free disk space (15–30 GB per model is common).
- Optional but recommended: NVIDIA GPU drivers and CUDA runtime for Docker.

Step 1: Prepare Your System (GPU Optional)

Linux (Ubuntu/Debian)
1) Install Docker Engine and the Compose plugin from the official Docker repo.
2) Install NVIDIA GPU drivers from your distro or NVIDIA site.
3) Install the NVIDIA Container Toolkit:
sudo apt-get install -y nvidia-container-toolkit
Then configure and restart Docker:
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
Verify GPU visibility in containers:
docker run --rm --gpus all nvidia/cuda:12.3.2-base-ubuntu22.04 nvidia-smi

Windows 10/11
1) Install the latest NVIDIA GPU driver (Studio or Game Ready).
2) Install Docker Desktop and enable WSL 2 backend during setup.
3) In Docker Desktop > Settings > Resources > WSL integration, enable your default distro.
4) Ensure GPU is exposed to containers. If you run docker run --rm --gpus all nvidia/cuda:12.3.2-base-ubuntu22.04 nvidia-smi and see your GPU, you are ready.

Step 2: Create a Docker Compose File

We will run two containers: ollama (the LLM runtime API) and open-webui (the web front-end). The services will share a network and persistent volumes. Create a folder like ollama-openwebui and a file compose.yaml with the following content:

services:
  ollama:
   image: ollama/ollama:latest
   container_name: ollama
   restart: unless-stopped
   ports:
    - "11434:11434"
   volumes:
    - ollama_data:/root/.ollama
   environment:
    - OLLAMA_KEEP_ALIVE=24h
   deploy:
    resources:
     reservations:
      devices:
       - capabilities: ["gpu"]

  openwebui:
   image: ghcr.io/open-webui/open-webui:latest
   container_name: open-webui
   restart: unless-stopped
   depends_on:
    - ollama
   ports:
    - "3000:8080"
   environment:
    - OLLAMA_BASE_URL=http://ollama:11434
    - WEBUI_AUTH=True
    - DEFAULT_MODELS=llama3.1:8b
   volumes:
    - openwebui_data:/app/backend/data

volumes:
  ollama_data:
  openwebui_data:

Notes:
- The deploy.resources.reservations.devices section hints Compose to request GPU. On Linux, also start with --gpus all if you run containers manually.
- Ports: Ollama API is 11434, Open WebUI is exposed on 3000 (mapped to container 8080).

Step 3: Start the Stack

In the folder containing compose.yaml, run:
docker compose up -d
Wait for both containers to start. You can watch logs with:
docker compose logs -f

Step 4: Pull a Model and Run Your First Chat

Open a terminal and pull a model into Ollama. For a good balance of quality and speed, try Meta’s 8B model:
docker exec -it ollama ollama pull llama3.1:8b
You can test from the CLI:
docker exec -it ollama ollama run llama3.1:8b "Write a haiku about local AI."
If the response appears, the model is working.

Now open your browser and visit http://localhost:3000. Create an admin account (since we set WEBUI_AUTH=True). In Settings > Models, you should see llama3.1:8b. Create a new chat and start prompting.

GPU Acceleration Checks

- If you have an NVIDIA GPU, Ollama should automatically use it. Confirm via logs: docker logs ollama (look for CUDA initialization).
- If you do not have a GPU, Ollama will use CPU. Expect slower generation but it will work.

Useful Options and Performance Tips

- Try smaller variants for low VRAM: llama3.2:3b or phi3:mini.
- You can pin models to GPU RAM by enabling sufficient numa/gpu memory; if VRAM is low, Ollama will offload layers to system RAM.
- To pre-download a model at startup, set DEFAULT_MODELS in the Open WebUI service as shown.
- For multilingual or coding tasks, add models like qwen2.5:7b or codestral.

Security and Remote Access

- Keep WEBUI_AUTH=True to require sign-in. You can also set OPENWEBUI_ADMIN_EMAIL and OPENWEBUI_ADMIN_PASSWORD as environment variables for unattended setups.
- If exposing Open WebUI to the internet, place it behind a reverse proxy (Nginx, Caddy, or Traefik) with HTTPS and strong passwords.
- The Ollama API on port 11434 should remain private unless you need remote access; firewall it if required.

Troubleshooting

- GPU not detected: On Linux, reinstall nvidia-container-toolkit and verify nvidia-smi works both on the host and in a container. On Windows, ensure WSL 2 is enabled and Docker Desktop is up to date.
- “No space left on device”: Increase disk space or prune unused model blobs: docker exec -it ollama ollama rm <model>. You can also clear unused images with docker system prune (caution).
- Slow or out-of-memory: Use a smaller model, reduce context length in Open WebUI, close other GPU-intensive apps, or increase swap on Linux.
- Port in use: Change the published ports in compose.yaml (e.g., "3001:8080") and redeploy.

Updating and Maintenance

To update to the latest versions, run:
docker compose pull
docker compose up -d
Your models are safe in the ollama_data volume, and your chat history lives in openwebui_data. Always back up these volumes before major upgrades.

What’s Next

You now have a privacy-friendly, GPU-accelerated local AI stack. Explore function calling, RAG connectors in Open WebUI, or run multiple models side by side. With Docker and Ollama, swapping models and keeping performance high is only a pull away.