LifeBytes Journal

If you want a fast, private, and cost-effective AI assistant without sending data to third parties, you can self-host one with Ollama and Open WebUI. Ollama runs large language models locally, while Open WebUI gives you a friendly chat interface with features like chat history, prompt templates, and model management. This guide shows how to deploy both using Docker, with optional GPU acceleration for NVIDIA or AMD.

Why this stack

Ollama simplifies running modern models such as Llama 3.1, Mistral, Phi, and more with a single command. Open WebUI connects to Ollama and adds a browser-based chat app, multiple users, and extras like RAG, files, and tools. Docker keeps everything consistent, easy to update, and portable across servers and clouds.

Prerequisites

- A 64-bit Linux host (Ubuntu 22.04/24.04 recommended), macOS, or Windows with WSL2. For production, a Linux VM or server is ideal.
- Docker Engine 24+ and Docker Compose plugin.
- 16 GB RAM minimum (24–32 GB recommended for 8B models; bigger models need more).
- 25–50 GB free disk space per model.
- Optional GPU:
  • NVIDIA: recent driver + nvidia-container-toolkit.
  • AMD: ROCm-capable GPU and kernel/drivers.

Step 1 — Install Docker and (optional) drivers

On Ubuntu, install Docker quickly:

curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER
newgrp docker
docker --version
docker compose version

If you have an NVIDIA GPU, install drivers and container toolkit, then restart Docker:

sudo apt update
sudo apt install -y nvidia-driver-535
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
nvidia-smi

Step 2 — Create a Docker Compose file

Create a project folder, then a docker-compose.yml that runs Ollama and Open WebUI. This setup persists models and app data in Docker volumes and exposes ports 11434 (Ollama) and 3000 (WebUI).

mkdir -p ~/ai-chat && cd ~/ai-chat
cat > docker-compose.yml << 'YAML'
version: "3.8"

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    ports:
      - "11434:11434"
    volumes:
      - ollama:/root/.ollama
    environment:
      - OLLAMA_KEEP_ALIVE=24h
    # For NVIDIA GPU support, uncomment the next line (requires nvidia-container-toolkit)
    # gpus: all

  open-webui:
    image: ghcr.io/open-webui/open-webui:latest
    container_name: open-webui
    restart: unless-stopped
    depends_on:
      - ollama
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
      - WEBUI_SECRET_KEY=change_this_long_random_string
      - ENABLE_SIGNUP=true
      - DEFAULT_MODELS=llama3.1:8b-instruct
    ports:
      - "3000:8080"
    volumes:
      - openwebui:/app/backend/data

volumes:
  ollama:
  openwebui:
YAML

Step 3 — Launch the stack

Start both services in the background:

docker compose up -d
docker compose ps

Open a browser and visit http://SERVER_IP:3000. On first visit, create an admin account. In Settings, confirm the Ollama endpoint shows http://ollama:11434 and the default model list includes llama3.1:8b-instruct.

Step 4 — Pull a model

You can pull models in the WebUI, or via CLI inside the Ollama container:

docker exec -it ollama ollama pull llama3.1:8b-instruct

After the download, start chatting in Open WebUI. If the model is large or your server is low on RAM, start with a smaller one like mistral:7b-instruct or phi3:mini.

Optional — Enable NVIDIA GPU acceleration

If nvidia-smi works on the host and you installed nvidia-container-toolkit, uncomment gpus: all for the ollama service in docker-compose.yml and redeploy:

docker compose down
sed -n '1,200p' docker-compose.yml
docker compose up -d
docker logs -f ollama

When a model runs, Ollama should log CUDA usage. You can also watch GPU load with nvidia-smi.

Optional — Enable AMD GPU (ROCm)

For AMD GPUs supported by ROCm, use the ROCm image and pass GPU devices into the container. Replace the ollama service with:

  ollama:
    image: ollama/ollama:rocm
    container_name: ollama
    restart: unless-stopped
    ports:
      - "11434:11434"
    volumes:
      - ollama:/root/.ollama
    devices:
      - /dev/kfd
      - /dev/dri
    group_add:
      - video

Then redeploy with docker compose up -d. If you see ROCm capability errors, verify your kernel/driver versions and that your user belongs to the video group.

Secure and expose your WebUI

For public access, put a reverse proxy in front with HTTPS. Caddy makes this easy:

your-domain.example {
  reverse_proxy 127.0.0.1:3000
}

Point DNS to your server, install Caddy, and it will fetch certificates automatically. In Open WebUI, set strong passwords, disable open signup if you do not need it (ENABLE_SIGNUP=false), and consider enabling rate limits at the proxy.

Backups and updates

Your important data lives in two volumes: ollama (models) and openwebui (app data, history). To back them up:

docker compose stop
docker run --rm -v ollama:/src -v $PWD:/backup alpine tar czf /backup/ollama-vol.tgz -C /src .
docker run --rm -v openwebui:/src -v $PWD:/backup alpine tar czf /backup/openwebui-vol.tgz -C /src .
docker compose start

To update images and get the latest features:

docker compose pull
docker compose up -d

Models remain unless you explicitly remove the ollama volume.

Troubleshooting

- Open WebUI cannot connect to Ollama: ensure OLLAMA_BASE_URL points to http://ollama:11434 and both containers share the same Docker network (default in Compose).
- CUDA driver not found: confirm nvidia-smi works on the host; re-run nvidia-ctk; restart Docker; ensure gpus: all is enabled.
- AMD permissions error: check /dev/kfd and /dev/dri are present; add group_add: video; ensure your kernel/ROCm version supports your GPU.
- Out of memory or slow responses: choose a smaller model, or reduce threads and context in the model settings; increase swap as a temporary measure.
- No space left on device: models are large; prune unused images and models with docker image prune and ollama list / ollama rm.

Uninstall cleanly

Stop and remove containers and volumes (this also deletes downloaded models and chat data):

cd ~/ai-chat
docker compose down -v

You now have a private AI chatbot that runs entirely on your hardware. Expand it with more models, plug in document retrieval, or publish it behind a secure HTTPS domain for your team.

Running a fast, private AI chatbot on your own computer or server is easier than ever. In this guide, you will install Ollama (a lightweight local LLM runtime) and Open WebUI (a modern web interface) on Ubuntu 24.04. You will be able to chat with models like Llama 3 or Mistral without sending data to the cloud, and with optional GPU acceleration if you have an NVIDIA card.

What you will need: an Ubuntu 22.04/24.04 machine (VM, bare metal, or WSL), at least 8 GB RAM (16 GB recommended for larger models), 15–30 GB free disk space for models, Internet access, and optional NVIDIA GPU drivers for acceleration.

Why Ollama + Open WebUI?

Ollama manages local large language models (LLMs) with simple commands and sensible defaults. Open WebUI gives you a clean, chat-style interface with features like prompt history, file uploads (for some models), and model switching. Together, they are a simple, reliable stack for a self-hosted AI experience.

1) Update Ubuntu and install basics

First, refresh your package list and install required tools:

sudo apt update && sudo apt upgrade -y
sudo apt install -y curl ca-certificates

2) Install Ollama and start the service

Ollama provides an installer script for Linux. Run the following to install and start the service:

curl -fsSL https://ollama.com/install.sh | sh
sudo systemctl enable --now ollama

Verify that the Ollama API is listening on port 11434:

ss -tulpn | grep 11434

3) Pull a model (Llama 3 as an example)

Ollama hosts a registry of optimized models. Pull a popular general-purpose model such as Llama 3 8B:

ollama pull llama3:8b

After the download completes, you can test it quickly:

ollama run llama3:8b

Type a prompt and press Enter. Press Ctrl+C to exit.

4) Install Docker and run Open WebUI

Open WebUI is easiest to deploy with Docker. Install Docker using the official convenience script, then start the container:

curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER # log out/in after this

Run Open WebUI and connect it to your local Ollama service:

docker run -d --name open-webui \
-p 3000:8080 \
-e OLLAMA_BASE_URL=http://127.0.0.1:11434 \
-v open-webui:/app/backend/data \
--restart unless-stopped \
ghcr.io/open-webui/open-webui:latest

Open your browser and go to http://SERVER_IP:3000. On first access you will create an admin account. Then choose your default model (e.g., llama3:8b) from the interface and start chatting.

5) Enable GPU acceleration (optional, NVIDIA)

If your machine has an NVIDIA GPU, install the official driver from Ubuntu’s Additional Drivers or with sudo apt install nvidia-driver-XXX (replace XXX with a recommended version). Reboot and verify with nvidia-smi. Ollama will auto-detect CUDA and use your GPU for supported models, delivering much faster responses. You do not need GPU pass-through to Docker for this setup because Ollama runs on the host.

6) Secure and harden your deployment

Local-only binding: If you are on a public server, avoid exposing the UI directly. Bind Open WebUI to localhost and place a reverse proxy with HTTPS in front:

docker rm -f open-webui
docker run -d --name open-webui \
-p 127.0.0.1:3000:8080 \
-e OLLAMA_BASE_URL=http://127.0.0.1:11434 \
-v open-webui:/app/backend/data \
--restart unless-stopped \
ghcr.io/open-webui/open-webui:latest

Reverse proxy tip: Use any TLS-capable proxy (Nginx, Caddy, Traefik). For example, with Caddy you can map your domain to localhost:3000 and get automatic HTTPS. Protect access using password auth or your proxy’s single sign-on.

7) Daily use and model management

Switch models in the Open WebUI sidebar or pull additional ones via Ollama. Useful commands:

# list local models
ollama list

# pull a different model
ollama pull mistral:7b

# remove unused models to free space
ollama rm model_name

When you click “New Chat” in Open WebUI, you can choose the model and adjust temperature, system prompt, and other parameters. For tasks like coding or reasoning, try llama3.1 or mistral-nemo variants if available for your hardware.

8) Updating the stack

Keep components fresh to get speed and quality improvements:

# update Ollama binary
curl -fsSL https://ollama.com/install.sh | sh
sudo systemctl restart ollama

# update Open WebUI container
docker pull ghcr.io/open-webui/open-webui:latest
docker stop open-webui && docker rm open-webui
docker run ... (same command as before)

9) Troubleshooting common issues

Port conflicts: If 11434 or 3000 is already in use, pick a different host port (for example, -p 8081:8080 for Open WebUI). Check usage with ss -tulpn.

Insufficient VRAM or RAM: Large models may fail to load. Try a smaller variant (e.g., llama3:8b instead of 70b), or use quantized builds where available.

No GPU detected: Ensure the NVIDIA driver is installed and loaded (nvidia-smi works). Reboot after driver installation. Ollama falls back to CPU if no GPU is available.

Docker permissions: If you see “permission denied,” log out and back in after adding your user to the docker group, or run commands with sudo.

Disk space: Models can be large. Use ollama list and ollama rm to remove what you do not need. Check usage with df -h.

10) What’s next?

Explore prompt templates, create system prompts for repeatable tasks, and try specialized models for coding, document Q&A, or SQL. You can also connect Open WebUI to external tools, set up team access behind your company SSO, or run multiple instances for different workloads. With Ollama and Open WebUI, you have a fast, private, and extensible foundation for local generative AI on Ubuntu.

Summary: This hands-on guide shows you how to self-host a privacy-friendly AI chat using Open WebUI and Ollama on Ubuntu 24.04. You’ll install Ollama (CPU or NVIDIA GPU), pull a modern open-source model (like Llama 3), and run Open WebUI as a container so you can chat in your browser. The steps are simple, repeatable, and production-friendly.

Why self-host a ChatGPT alternative?

Self-hosting gives you control, privacy, and predictable costs. Models run locally, so prompts and outputs stay on your server. With a capable CPU or an NVIDIA GPU, you can achieve low-latency responses and customize the setup to match your workflow.

What you need

- Ubuntu 24.04 LTS with a non-root user that can run sudo.
- Internet access and at least 16 GB of RAM for 7–8B models (more is better); an NVIDIA GPU speeds things up.
- Open ports: 11434 (Ollama, local only), 8080 (Open WebUI).
- Basic terminal familiarity.

Step 1 — Prepare Ubuntu 24.04

sudo apt update && sudo apt -y upgrade
sudo apt install -y curl git ufw

Enable the firewall and allow SSH and the Open WebUI port:

sudo ufw allow OpenSSH
sudo ufw allow 8080/tcp
sudo ufw enable

Step 2 — Install Ollama (CPU or NVIDIA GPU)

Ollama is a lightweight runtime for local LLMs with an OpenAI-compatible API. Install it with the official script and enable the system service:

curl -fsSL https://ollama.com/install.sh | sh
sudo systemctl enable --now ollama
ollama --version

By default, Ollama listens on 127.0.0.1:11434 (local only), which is what we want for a secure setup.

GPU acceleration (optional but recommended)

If you have an NVIDIA GPU, install the recommended driver so Ollama can use CUDA automatically:

sudo ubuntu-drivers install
sudo reboot
nvidia-smi

Ollama will use the GPU if the driver and CUDA libraries are present. During generation, watch nvidia-smi to confirm usage. If you do not have a GPU, Ollama still runs well on CPU-only, just expect slower responses.

Step 3 — Pull a model and test locally

Pull a modern, general-purpose model. Llama 3 8B is a great starting point:

ollama pull llama3:8b

Run a quick prompt from the terminal to verify generation works:

ollama run llama3:8b "Write a haiku about Ubuntu servers."

Tip: You can pull other models such as mistral:7b, phi3:latest, or code-focused models depending on your use case.

Step 4 — Install Open WebUI (Docker)

Open WebUI provides a clean, fast chat interface that can talk to Ollama. The easiest way to run it is via Docker. If Docker is not installed yet, set it up:

sudo apt-get install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu noble stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io

Run Open WebUI on the host network so it can reach Ollama at 127.0.0.1:

sudo docker run -d --name open-webui --restart=always --network host -v open-webui:/app/backend/data -e OLLAMA_BASE_URL=http://127.0.0.1:11434 ghcr.io/open-webui/open-webui:latest

If you prefer not to use --network host, publish a port and set OLLAMA_BASE_URL to http://host.docker.internal:11434 (on recent Docker versions) or to your host’s IP.

Step 5 — First run and basic security

Open your browser and visit http://SERVER_IP:8080. Create the first admin user. In Settings → Security, enable authentication and disable public sign-ups to prevent unauthorized access. Keep Ollama bound to localhost and avoid exposing port 11434 to the internet.

In the Models section, you can pull models directly from the UI or reuse models previously pulled with the Ollama CLI. Start a new chat and select your model (e.g., llama3:8b). You now have a private AI chat running on your own hardware.

Optional — TLS with a reverse proxy

For secure remote access, place Open WebUI behind a reverse proxy like Nginx or Caddy with a Let’s Encrypt certificate. Point the proxy to 127.0.0.1:8080, enable HTTPS, and restrict access with authentication. If you use Cloudflare, consider an Origin Certificate and firewall rules for extra protection.

Troubleshooting

Port in use: If 8080 is taken, change Open WebUI’s bind port or stop the conflicting service. Check with sudo ss -ltnp | grep 8080.
GPU not used: Ensure nvidia-smi works, use a recent driver (e.g., 535+), and watch GPU utilization during generation. Restart Ollama after driver changes: sudo systemctl restart ollama.
Model not found: Pull the model again (ollama pull model:tag) and verify spelling. Some models have multiple quantizations; pick one that fits your RAM/VRAM.
Slow responses: Use a smaller or more aggressively quantized model, lower context window, or run on GPU for better throughput.

Next steps

- Add embeddings and RAG with document loaders inside Open WebUI for private knowledge search.
- Tune concurrency: OLLAMA_NUM_PARALLEL and model context size to match your CPU/GPU and memory budget.
- Back up Open WebUI’s volume (open-webui) and the Ollama models directory for quick disaster recovery.

You now have a fast, private, and flexible AI stack on Ubuntu 24.04. With Ollama and Open WebUI, you control your data and costs while enjoying a modern chat experience powered by open models.

Overview

Running a private, fast, and flexible AI chat server on your own computer is easier than ever. In this guide, you will set up Ollama (the local LLM runtime) and Open WebUI (a polished chat interface) on Linux or Windows. You will be able to pull popular models like Llama 3 or Mistral, enable GPU acceleration when available, and access a modern web UI for chatting, prompt management, file uploads, and more. This tutorial focuses on simplicity, security, and reliability, using Docker for the web UI and native installation for Ollama.

What You’ll Need

- A 64-bit machine running Ubuntu 22.04/24.04, Debian 12, or Windows 11/10 (latest updates).
- 16 GB RAM recommended; more helps with larger models.
- Optional GPU for acceleration: NVIDIA/AMD/Intel with recent drivers (Ollama uses CUDA/ROCm/DirectML depending on platform).
- Internet access to download models (they can be several gigabytes).

Step 1 — Install Ollama

On Linux (Ubuntu/Debian):
curl -fsSL https://ollama.com/install.sh | sh
sudo systemctl enable --now ollama
Check that the API is listening on port 11434:
curl http://localhost:11434/api/tags

On Windows:
Install with Winget:
winget install Ollama.Ollama
After install, Ollama runs in the background. Verify the API:
curl http://localhost:11434/api/tags

Tip: If you have a GPU, make sure the latest driver is installed. On Linux (NVIDIA), confirm with nvidia-smi. Ollama will automatically use the GPU when possible and print a GPU line in its logs during the first model load.

Step 2 — Pull a Model and Test Locally

Pull a high-quality, general-purpose model. Two popular choices are Llama 3 and Mistral:

ollama pull llama3
ollama pull mistral

Run a quick prompt to validate generation:

ollama run llama3
When prompted, type: Explain what containers are in simple terms.
Press Ctrl+C to exit.

Step 3 — Install Docker (for Open WebUI)

On Linux (quick installer):
curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER
newgrp docker
Confirm:
docker version

On Windows:
Install Docker Desktop from the official site, enable WSL 2 integration, and confirm it runs correctly.

Step 4 — Deploy Open WebUI

Open WebUI provides a rich chat interface on top of the Ollama API. You will run it in Docker and point it to your local Ollama instance.

Linux (Docker host networking):
docker run -d --name open-webui --restart unless-stopped --network host -e OLLAMA_BASE_URL=http://127.0.0.1:11434 -v open-webui:/app/backend/data ghcr.io/open-webui/open-webui:latest

Windows/macOS (port mapping):
docker run -d --name open-webui --restart unless-stopped -p 3000:8080 -e OLLAMA_BASE_URL=http://host.docker.internal:11434 -v open-webui:/app/backend/data ghcr.io/open-webui/open-webui:latest

Open your browser to http://localhost:3000. On first launch, create an admin account. In Settings, select your default model (e.g., llama3) and adjust UI preferences. The data volume open-webui persists chats and configurations across container restarts.

Step 5 — Basic Security and Remote Access

By default, both Ollama (11434) and Open WebUI (3000) listen on localhost. This is the safest configuration for a single-user machine. If you must access Open WebUI remotely, place it behind a reverse proxy with TLS and a password. For example, using Caddy on a server with a public DNS record:

example.com {
  reverse_proxy 127.0.0.1:3000
}

Caddy will fetch a free Let’s Encrypt certificate automatically. Keep the Ollama API on localhost and expose only the web UI via the proxy.

Step 6 — Verify the Stack

- Test the Ollama API directly:
curl http://localhost:11434/api/tags
- Generate a short reply via API:
curl -s http://localhost:11434/api/generate -d '{"model":"llama3","prompt":"Say hi in one sentence."}'
- Open WebUI at http://localhost:3000 and send a message. Check the model switcher if you pulled multiple models.

Performance Tips

- Prefer smaller or quantized models for limited RAM/VRAM (e.g., 7B Q4_K_M).
- Close other heavy apps to reduce RAM pressure and swapping.
- If you have a strong GPU, pull a larger model (e.g., 13B) for better quality.
- Store models on a fast SSD; Ollama caches models under your user profile (e.g., ~/.ollama on Linux).

Backups and Maintenance

- Models: back up ~/.ollama (Linux) or the Ollama data directory on Windows to avoid re-downloading large files.
- Web UI data: the Docker volume open-webui stores conversations and settings. You can back it up with:
docker run --rm -v open-webui:/data -v "$PWD":/backup alpine sh -c 'cd /data && tar czf /backup/open-webui-data.tgz .'
- Update Ollama occasionally:
curl -fsSL https://ollama.com/install.sh | sh
- Update Open WebUI:
docker pull ghcr.io/open-webui/open-webui:latest && docker restart open-webui

Troubleshooting

- Port in use: If 3000 or 11434 is busy, change the mapping (e.g., -p 8080:8080) or stop the conflicting service.
- Docker cannot reach Ollama on Linux: ensure you used --network host, or alternatively add --add-host=host.docker.internal:host-gateway and set OLLAMA_BASE_URL=http://host.docker.internal:11434.
- GPU not used: update drivers and reboot; on Linux check nvidia-smi; on Windows confirm GPU activity in Task Manager during inference.
- Out-of-memory: switch to a smaller model or a stronger quantization; close other applications.

You’re Done

You now have a private, GPU-ready AI chat server powered by Ollama and Open WebUI. This setup is fast, secure (localhost by default), and extensible. You can add specialized models for coding or RAG, script API calls for automation, and back everything up with a couple of commands. Enjoy building with local AI—no cloud required.