LifeBytes Journal

Overview

This guide shows you how to run a private, local AI chatbot on Ubuntu using Ollama (for running large language models on CPU or GPU) and Open WebUI (a clean web interface). You will be able to chat with models like Llama 3 locally, without sending data to the cloud. The steps work on Ubuntu 22.04 and 24.04, and are suitable for both desktops and servers.

What You Will Set Up

You will install Ollama on the host, pull a model, and run Open WebUI in Docker. Open WebUI will connect to the Ollama API on port 11434. The result is a self-hosted, secure, and fast AI assistant accessible at http://YOUR_SERVER_IP:3000.

Prerequisites

- Ubuntu 22.04 or 24.04 with sudo access
- At least 8 GB RAM (16 GB+ recommended for 7B–8B models; use smaller quantized models on low-RAM systems)
- Optional: NVIDIA or AMD GPU for faster inference
- Internet access to download packages and models

Step 1: Install Ollama

Ollama is a lightweight runtime that serves models locally over an HTTP API (default: 11434). Install it with one command:

curl -fsSL https://ollama.com/install.sh | sh

Verify the installation and service:

ollama --version
systemctl status ollama

If the service is inactive, start it:

sudo systemctl enable --now ollama

Step 2: Pull a Model (Llama 3.1 as an example)

Download a good general-purpose model. Llama 3.1 8B is a solid balance for many machines:

ollama pull llama3.1:8b

Test it in the terminal:

ollama run llama3.1:8b
>>> Write a 1-sentence productivity tip.

Tip: If you run out of memory, pull a quantized variant, for example:

ollama pull llama3.1:8b-instruct-q4_K_M

Step 3: Install Docker Engine

Open WebUI ships a reliable container image. Install Docker from the official repository:

sudo apt update
sudo apt install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" \
| sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker $USER

Log out and in again (or run a new shell) to use Docker without sudo.

Step 4: Run Open WebUI and Connect to Ollama

Start Open WebUI in Docker, mapping port 3000 and pointing it to the Ollama API on the host. The host gateway alias works on modern Docker versions:

docker run -d --name open-webui \
  --restart unless-stopped \
  -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
  -v open-webui:/app/backend/data \
  ghcr.io/open-webui/open-webui:latest

Open your browser at http://YOUR_SERVER_IP:3000, create an admin account on first launch, and select the default model (e.g., llama3.1:8b). You can now chat and manage prompts, files, and settings.

If the container cannot reach the host, an alternative is host networking:

docker run -d --name open-webui \
  --restart unless-stopped \
  --network=host \
  -e OLLAMA_BASE_URL=http://127.0.0.1:11434 \
  -v open-webui:/app/backend/data \
  ghcr.io/open-webui/open-webui:latest

Step 5: Secure Access

- By default, Open WebUI uses account-based sign-in. In Settings > Admin panel, require authentication for all users.
- If you run on a public server, restrict binding to localhost and use an SSH tunnel:

# bind only to localhost:
docker run -d --name open-webui \
  --restart unless-stopped \
  -p 127.0.0.1:3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
  -v open-webui:/app/backend/data \
  ghcr.io/open-webui/open-webui:latest

# from your laptop:
ssh -L 3000:localhost:3000 [email protected]

Step 6: Use Your GPU (Optional)

Ollama will use your GPU if supported drivers are installed. For NVIDIA, install the proprietary driver and CUDA libraries (the standard Ubuntu “Additional Drivers” tool works). For AMD, install ROCm per Ubuntu/AMD documentation. Because Open WebUI talks to Ollama’s API, you do not need GPU support inside the Open WebUI container—only in Ollama on the host.

Step 7: Model Management Tips

- List models: ollama list
- Show model metadata: ollama show llama3.1:8b
- Remove a model: ollama rm MODEL_NAME
- Try alternatives: ollama pull mistral, ollama pull qwen2, or ollama pull phi3:mini for lower memory systems.

Troubleshooting

- Port in use: Change the mapped port, for example -p 4000:8080 and open http://YOUR_SERVER_IP:4000.
- Container cannot reach Ollama: Use --network=host or ensure --add-host=host.docker.internal:host-gateway is present.
- Out-of-memory: Pull a smaller/quantized model (e.g., q4_K_M variants) or close other apps.
- Slow responses: Prefer GPU, reduce context length in Open WebUI settings, or choose a smaller model.

Updating and Maintenance

- Update Ollama: rerun the install script, then restart the service: curl -fsSL https://ollama.com/install.sh | sh && sudo systemctl restart ollama.
- Update Open WebUI:

docker pull ghcr.io/open-webui/open-webui:latest
docker stop open-webui && docker rm open-webui
# run again with the same docker run command used earlier

Conclusion

With Ollama and Open WebUI, you can host a private, fast, and flexible chatbot on your own Ubuntu machine. This setup keeps your data local, supports multiple open models, and can leverage your GPU for speed. Whether you are a helpdesk, a developer, or a power user, this self-hosted stack gives you full control over your AI workflow.

Local large language models (LLMs) are now practical for developers, researchers, and privacy-focused teams. In this step-by-step guide, you will install and run Ollama (LLM runtime) and Open WebUI (a modern chat interface) on Ubuntu 22.04/24.04, with optional NVIDIA GPU acceleration. The setup uses Docker for easy updates, isolation, and backups.

Why this stack?

Ollama makes downloading and running models simple, offering a fast API on your machine. Open WebUI provides a sleek, extensible web app for chatting with multiple models, managing prompts, and moderating access. Together, they create a private, cost-effective alternative to cloud AI services.

Prerequisites

You need an Ubuntu 22.04/24.04 system with internet access and a user with sudo rights. If you have an NVIDIA GPU (recommended), you can enable GPU acceleration for much faster inference. CPU-only works too—skip the GPU steps if you do not have a supported GPU.

Step 1: Update the system

Update packages to ensure you have the latest dependencies and security fixes.

sudo apt update && sudo apt -y upgrade
sudo reboot

Step 2 (Optional): Enable NVIDIA GPU support

Install the latest proprietary NVIDIA driver and the container toolkit so Docker can use your GPU. Reboot when asked.

# Install recommended NVIDIA driver
sudo ubuntu-drivers install
sudo reboot

# Install NVIDIA Container Toolkit
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#' | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list > /dev/null

sudo apt update
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

# Verify GPU visibility
nvidia-smi

Step 3: Install Docker Engine and Docker Compose plugin

Install Docker from the official repository to get the latest stable version. Add your user to the docker group to run Docker without sudo.

sudo apt install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | \
sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

sudo usermod -aG docker $USER
newgrp docker

# Quick test
docker run --rm hello-world

Step 4: Create a Docker Compose file for Ollama + Open WebUI

Create a working directory and define services. The configuration below enables GPU when present; for CPU-only, remove the gpus: all line under the Ollama service.

mkdir -p ~/local-llm && cd ~/local-llm
nano docker-compose.yml

services:
  ollama:
    container_name: ollama
    image: ollama/ollama:latest
    restart: unless-stopped
    ports:
      - "11434:11434"
    volumes:
      - ollama:/root/.ollama
    # Comment the next line if you are CPU-only
    gpus: all
    environment:
      - OLLAMA_KEEP_ALIVE=24h

  open-webui:
    container_name: open-webui
    image: ghcr.io/open-webui/open-webui:latest
    depends_on:
      - ollama
    restart: unless-stopped
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
      - WEBUI_SECRET_KEY=change_this_long_random_secret
    volumes:
      - openwebui:/app/backend/data

volumes:
  ollama:
  openwebui:

Step 5: Start the stack

Bring the services up in the background, then check their status. The Open WebUI will be available at http://SERVER_IP:3000 and the Ollama API at http://SERVER_IP:11434.

docker compose up -d
docker compose ps

Step 6: Pull and run a model

Use Ollama to pull an LLM. You can pick models like llama3.2, mistral, or a coding model. The first pull downloads model weights, which can be several GB.

# Pull a general-purpose model
docker exec -it ollama ollama pull llama3.2

# Test it via CLI
docker exec -it ollama ollama run llama3.2 "Write a two-sentence summary of Ubuntu."

# Or use the API
curl http://localhost:11434/api/generate -d '{"model":"llama3.2","prompt":"Hello!"}'

Open your browser to http://SERVER_IP:3000, select the model from the dropdown, and start chatting. In Settings, you can change default models, system prompts, and appearance.

Step 7: Secure access

If exposing the WebUI beyond your LAN, add authentication. In Open WebUI, create an admin user at first login, then disable new signups in Settings. For internet exposure, place a reverse proxy (Nginx, Caddy, or Traefik) with HTTPS (Let’s Encrypt) in front of port 3000.

Step 8: Update and backup

To update, pull the latest images and recreate containers without losing data stored in volumes. To back up, save the volumes before upgrades.

# Update images
docker compose pull
docker compose up -d

# Backup volumes (example)
docker run --rm -v local-llm_ollama:/data -v "$PWD":/backup \
  busybox tar czf /backup/ollama-vol.tgz /data

docker run --rm -v local-llm_openwebui:/data -v "$PWD":/backup \
  busybox tar czf /backup/openwebui-vol.tgz /data

Troubleshooting

GPU not used: Run docker run --rm --gpus all nvidia/cuda:12.3.2-base-ubuntu22.04 nvidia-smi. If it fails, recheck the driver and NVIDIA Container Toolkit steps. Ensure the gpus: all line is present and Docker was restarted.

Permission denied with Docker: You may need to log out and back in after adding your user to the docker group, or run newgrp docker.

Port conflicts: Change the left side of port mappings in docker-compose.yml (e.g., use "8081:8080" for WebUI) and restart.

Slow or failed model pull: Verify disk space and retry. Large models require several GB of free space in the ollama volume.

Uninstall (optional)

To stop and remove everything, run:

cd ~/local-llm
docker compose down
docker volume rm local-llm_ollama local-llm_openwebui

Wrap-up

You have a modern local AI stack: Ollama for fast model serving and Open WebUI for a friendly, multi-model chat interface. With Docker, updates are quick and backups are simple. Add your favorite models, tune system prompts, and integrate the Ollama API into your apps—all without sending data to the cloud.

Overview

This step-by-step guide shows how to install Ollama and connect it to Open WebUI on Ubuntu 24.04. With this setup, you can run modern large language models like Llama 3 locally, use your NVIDIA GPU for acceleration, and chat through a clean web interface—no cloud required. The process includes installing system dependencies, enabling GPU support, running Open WebUI in Docker, pulling models, and basic troubleshooting. The language is simple, and every command is tested on Ubuntu 24.04.

Prerequisites

Before you start, make sure you have: (1) Ubuntu 24.04 with sudo access, (2) a modern NVIDIA GPU and driver support (optional but recommended), (3) at least 16 GB of RAM for medium models, and (4) stable internet access to download models and containers.

1) Update Ubuntu and install essentials

Begin by updating your packages and installing the tools we will use. If prompted, confirm with Y:

sudo apt update && sudo apt upgrade -y
sudo apt install -y curl ca-certificates gnupg ufw git

2) Install NVIDIA drivers (for GPU acceleration)

Ollama uses your GPU automatically when the correct NVIDIA driver is present. If you do not have a GPU, you can still run models on the CPU (slower). To enable GPU acceleration on NVIDIA cards:

sudo ubuntu-drivers autoinstall
sudo reboot

After reboot, verify the driver:

nvidia-smi

You should see your GPU listed. If you prefer manual control, install a specific driver from “Additional Drivers” in Ubuntu.

3) Install Ollama

Ollama is a lightweight server that manages models locally and exposes an HTTP API on port 11434. Install it with the official script:

curl -fsSL https://ollama.com/install.sh | sh

Enable and verify the system service:

sudo systemctl enable ollama
sudo systemctl start ollama
systemctl status ollama

If you see it active and running, Ollama is ready at http://localhost:11434.

4) Pull a model and test locally

Pull a modern, efficient model. Llama 3.1 8B is a good starting point (adjust model to your hardware):

ollama pull llama3.1:8b

Run a quick chat in the terminal to verify GPU usage:

ollama run llama3.1:8b

If your GPU is recognized, the first generation will warm up, and subsequent responses should be fast. You can also try other models like mistral, phi-3, or neural-chat.

5) Install Docker and run Open WebUI

Open WebUI provides a clean browser interface for chatting with local models. Install Docker from Ubuntu’s repo for simplicity:

sudo apt install -y docker.io docker-compose-plugin
sudo systemctl enable --now docker
sudo usermod -aG docker $USER
# Log out and back in to apply docker group membership (or run a new shell).

Start Open WebUI and point it to the host Ollama API:

docker run -d --name open-webui \
  -p 3000:8080 \
  -v open-webui:/app/backend/data \
  --add-host=host.docker.internal:host-gateway \
  -e OLLAMA_API_BASE_URL=http://host.docker.internal:11434 \
  ghcr.io/open-webui/open-webui:latest

Open a browser and go to http://YOUR_SERVER_IP:3000 to access Open WebUI. On first run, create an admin user. In Settings > Connections, confirm the Ollama endpoint is http://host.docker.internal:11434.

6) Secure basic network access

If UFW is enabled, allow the Open WebUI port:

sudo ufw allow 3000/tcp
sudo ufw status

For internet exposure, place Open WebUI behind a reverse proxy (Nginx/Caddy) with HTTPS. If you only use it on your LAN, keep it on port 3000 and block external access at your router or firewall.

7) Daily use tips

- To list models: ollama list. To remove one: ollama rm MODEL.
- To update Ollama when a new version is released: rerun the install script, then sudo systemctl restart ollama.
- For faster chat, choose 7B–8B models or quantized variants (like Q4_K_M). Larger models need more VRAM and RAM.

Troubleshooting

No compatible GPU found: Check nvidia-smi. If it fails, reinstall drivers with ubuntu-drivers autoinstall and reboot. Ensure Secure Boot is disabled or properly configured for NVIDIA modules.

Open WebUI cannot reach Ollama: Confirm the container can resolve the host gateway. We used --add-host=host.docker.internal:host-gateway. Also verify the env OLLAMA_API_BASE_URL and that the Ollama service is active: systemctl status ollama.

Slow generations on CPU: Use smaller models (e.g., 3–8B) or quantized versions. GPU acceleration is the biggest speed boost; ensure drivers are correct.

Ports already in use: If 3000 or 11434 is used, change the exposed port for Open WebUI (-p 4000:8080 for example) and update firewall rules.

Check logs: Ollama logs: journalctl -u ollama -f. Open WebUI logs: docker logs -f open-webui.

Optional: Reverse proxy with Nginx (HTTPS)

For public access with TLS, install Nginx and Certbot, then map a domain to your server and issue a Let’s Encrypt certificate. Point Nginx to the Open WebUI container on 3000. Keep strong passwords and consider IP allowlists or SSO for security.

What you get

You now have a private, GPU-accelerated local AI stack: Ollama runs models efficiently on your Ubuntu host, and Open WebUI gives you a modern chat interface. This setup is ideal for development, research, and privacy-focused workflows without sending your data to external clouds.