LifeBytes Journal

Overview

This step-by-step guide shows you how to self-host Ollama with Open WebUI on Ubuntu 22.04/24.04 and use your NVIDIA GPU for fast, private large language model (LLM) inference. You will install the correct NVIDIA drivers, Docker, and NVIDIA Container Toolkit, then deploy Ollama and Open WebUI with Docker Compose. The tutorial also covers updating, backing up models, and troubleshooting common errors such as GPU visibility and port conflicts.

Prerequisites

Before you begin, make sure you have: (1) Ubuntu 22.04 or 24.04 with sudo access, (2) an NVIDIA GPU with at least 6 GB VRAM for medium models (smaller models can work with less), (3) a stable internet connection, and (4) at least 20 GB free disk space for images and model files.

Step 1: Install NVIDIA Driver and Verify CUDA

Use Ubuntu’s built-in tool to install a matching proprietary driver. If Secure Boot is enabled, you may need to enroll a Machine Owner Key (MOK) during installation to load the NVIDIA kernel module.

sudo apt update
sudo ubuntu-drivers install
sudo reboot

After the reboot, confirm the driver is active:

nvidia-smi

You should see a table with your GPU and driver version. If you get an error, check Secure Boot (disable it or enroll the NVIDIA module), then repeat the install.

Step 2: Install Docker Engine and NVIDIA Container Toolkit

Install Docker from the official repository and add your user to the docker group so you can run containers without sudo.

sudo apt-get install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker $USER
newgrp docker

Add NVIDIA Container Toolkit so containers can use your GPU:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit.gpg
distribution=$(. /etc/os-release; echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/stable/$distribution/libnvidia-container.list | \
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit.gpg] https://#g' | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Verify Docker can see your GPU:

docker run --rm --gpus all nvidia/cuda:12.5.0-base-ubuntu22.04 nvidia-smi

Step 3: Deploy Ollama and Open WebUI with Docker Compose

We will bind both services to localhost for safety. You can put a reverse proxy in front later for remote access.

mkdir -p ~/ollama-stack && cd ~/ollama-stack
nano compose.yaml

Paste the following compose file (save and exit):

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    ports:
      - "127.0.0.1:11434:11434"
    environment:
      - OLLAMA_HOST=0.0.0.0
      - OLLAMA_NUM_PARALLEL=1
    volumes:
      - ollama:/root/.ollama
    gpus: all

  open-webui:
    image: ghcr.io/open-webui/open-webui:latest
    container_name: open-webui
    restart: unless-stopped
    depends_on:
      - ollama
    environment:
      - OLLAMA_API_BASE=http://ollama:11434
    ports:
      - "127.0.0.1:3000:8080"
    volumes:
      - openwebui:/app/backend/data

volumes:
  ollama:
  openwebui:

Start the stack:

docker compose up -d

Pull a model into Ollama (example: a small, fast model):

docker exec -it ollama ollama pull llama3.2:3b

Quick API test:

curl http://127.0.0.1:11434/api/generate \
  -H "Content-Type: application/json" \
  -d '{"model":"llama3.2:3b","prompt":"Say hello in one sentence."}'

Open your browser at http://127.0.0.1:3000 and select Ollama as the provider. Choose the model you pulled and start chatting.

Step 4: Updates and Backups

To update Ollama and Open WebUI to the latest images while keeping your models and data, run:

cd ~/ollama-stack
docker compose pull
docker compose up -d

Back up volumes (models and WebUI data) with a simple tar archive:

docker stop open-webui ollama
docker run --rm -v ollama:/data -v "$PWD":/backup alpine \
  sh -c 'tar czf /backup/ollama-vol.tar.gz -C /data .'
docker run --rm -v openwebui:/data -v "$PWD":/backup alpine \
  sh -c 'tar czf /backup/openwebui-vol.tar.gz -C /data .'
docker start ollama open-webui

Troubleshooting

No CUDA-capable device detected: Ensure the NVIDIA driver is loaded (nvidia-smi works on the host). If Secure Boot is on, enroll the MOK or disable Secure Boot. Confirm the container sees the GPU with the CUDA test image. Re-run: sudo nvidia-ctk runtime configure --runtime=docker and restart Docker.

Compose error: unknown field "gpus": Your Docker Compose is outdated. Update Docker or use: docker run --gpus all ... Alternatively, in compose.yaml, remove gpus: all and start Ollama with: docker run -d --gpus all -p 127.0.0.1:11434:11434 -v ollama:/root/.ollama --name ollama ollama/ollama:latest

Port already in use: Change the host ports in compose.yaml (for example, 127.0.0.1:11435:11434 and 127.0.0.1:3001:8080) and re-run docker compose up -d.

Out-of-memory or slow responses: Choose a smaller or more quantized model (e.g., llama3.2:1b or a Q4 version if available). Limit parallel requests with OLLAMA_NUM_PARALLEL=1. Ensure you have adequate swap configured on the host for large models.

Security Tips

Keep services bound to 127.0.0.1 and place a reverse proxy with TLS in front (Caddy, Traefik, or Nginx) if you need remote access. For Open WebUI, enable authentication in its settings. Restrict firewall rules to only allow your reverse proxy and management IPs. Regularly update images and prune unused layers with docker system prune -af.

Clean Uninstall

To remove the stack and its volumes (this deletes downloaded models and chat data), run:

cd ~/ollama-stack
docker compose down -v

Conclusion

You have a fully private, GPU-accelerated local AI setup with Ollama and Open WebUI running on Ubuntu. This stack is easy to update, simple to back up, and flexible: you can try multiple models, script against the API, or place it behind a secure reverse proxy for team access. With one machine and an NVIDIA GPU, you now own your LLM workflow end to end.

Overview

This tutorial shows how to deploy a local Large Language Model (LLM) stack on Ubuntu using Docker, with hardware acceleration for NVIDIA or AMD GPUs. We will combine Ollama (model runtime and manager) with Open WebUI (a fast, modern web interface) so you can chat with models like Llama 3.1 or Mistral on your own machine. The steps apply to Ubuntu 22.04/24.04, and are suitable for homelabs and small teams.

Prerequisites

- Ubuntu server or desktop with internet access

- A recent CPU; for GPU acceleration: an NVIDIA GPU with recent drivers, or an AMD GPU with ROCm support

- Sudo privileges and ports 11434 (Ollama) and 3000 (Open WebUI) available

Install Docker Engine

If Docker is not installed, use the official repository to get the latest stable version and the Compose plugin.

sudo apt update
sudo apt install -y ca-certificates curl gnupg lsb-release
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker $USER
newgrp docker
docker --version
docker compose version

Enable GPU Acceleration (NVIDIA)

Install the NVIDIA Container Toolkit so containers can access the GPU. Ensure the proprietary GPU driver is installed (e.g., 535+). Then run:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
nvidia-smi

If nvidia-smi works on the host, the GPU will be available inside the containers when requested.

Enable GPU Acceleration (AMD ROCm)

AMD support relies on ROCm. On supported GPUs and kernels, install ROCm drivers (refer to AMD documentation for your GPU). Start with:

sudo apt update
# Example meta-package (adjust to your distro and GPU generation)
sudo apt install -y rocm-hip-runtime5.7
/opt/rocm/bin/rocminfo

For Docker, we will pass the ROCm devices into Ollama’s container. Note that model availability and performance vary by GPU generation.

Create the Docker Compose file

We will run two services: ollama and open-webui. Create a project directory and a Compose file:

mkdir -p ~/ollama-openwebui
cd ~/ollama-openwebui
nano docker-compose.yml

Paste the following Compose configuration. Choose ONE of the GPU sections (NVIDIA or AMD). If you don’t have a GPU, omit the device configurations to run on CPU.

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    ports:
      - "11434:11434"
    volumes:
      - ollama:/root/.ollama
    # NVIDIA GPU (uncomment for NVIDIA)
    # deploy:
    #   resources:
    #     reservations:
    #       devices:
    #         - driver: nvidia
    #           count: all
    #           capabilities: [gpu]
    # AMD ROCm (uncomment for AMD)
    # devices:
    #   - "/dev/kfd:/dev/kfd"
    #   - "/dev/dri:/dev/dri"
    # environment:
    #   - HSA_OVERRIDE_GFX_VERSION=11.0.0

  open-webui:
    image: ghcr.io/open-webui/open-webui:latest
    container_name: open-webui
    restart: unless-stopped
    depends_on:
      - ollama
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
    volumes:
      - openwebui:/app/backend/data

volumes:
  ollama:
  openwebui:

Start the stack

Bring the services up in the background, then confirm they’re healthy.

docker compose up -d
docker compose ps

Open your browser and visit http://SERVER_IP:3000. The first login creates an admin account. Open WebUI will auto-connect to Ollama.

Download a model in Ollama

You can pull a model via the Open WebUI interface or the CLI. For example, to pull Llama 3.1 and test it:

docker exec -it ollama ollama pull llama3.1
docker exec -it ollama ollama run llama3.1

In Open WebUI, select the model from the top bar and start chatting. If GPU is configured correctly, inference will run on the GPU.

Securing access

By default, Open WebUI is exposed on port 3000 without TLS. For internet access, put it behind a reverse proxy like Nginx or Caddy with HTTPS, or use a VPN (e.g., Tailscale/WireGuard). On Ubuntu, restrict the firewall to your network:

sudo ufw allow from 192.168.0.0/24 to any port 3000 proto tcp
sudo ufw allow from 192.168.0.0/24 to any port 11434 proto tcp

Updating and backups

To update, pull the latest images and recreate containers without losing data (volumes keep models and UI data):

docker compose pull
docker compose up -d

For backups, snapshot the Docker volumes or copy them to external storage. On a single host, you can export and re-import volumes with standard tar workflows.

Troubleshooting

- GPU not detected in container (NVIDIA): ensure the NVIDIA driver matches the toolkit; run docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi. If it fails, recheck the toolkit setup and restart Docker.

- GPU not detected (AMD): verify rocminfo and clinfo on the host. Make sure /dev/kfd and /dev/dri are mapped and the user has permissions. Some older GPUs are unsupported by modern ROCm.

- Slow inference: use a smaller model (e.g., 7B), increase context/kv-caching wisely, and confirm the container is using the GPU. Consider enabling hugepages and ensuring adequate VRAM.

- Port conflicts: change the mapped ports in docker-compose.yml or stop services occupying them.

Cleanup

To stop the stack, run docker compose down. To remove images and volumes too (irreversible), run docker compose down --volumes --rmi all.

You now have a private, GPU-accelerated LLM environment running Ollama with Open WebUI on Ubuntu. This setup is flexible, easy to upgrade, and ideal for secure, local AI experimentation and productivity.

Overview

This step-by-step guide shows how to deploy MinIO with Docker and secure it behind Caddy for automatic HTTPS. MinIO is a high-performance, S3-compatible object storage that you can self-host for backups, logs, media, and AI datasets. We will use Docker Compose on Ubuntu 24.04, set up Caddy as a reverse proxy with free Let’s Encrypt TLS, create a bucket, and test access using the AWS CLI. The result is a production-ready, S3-compatible endpoint at your own domain.

Prerequisites

You will need: (1) a clean Ubuntu 24.04 server with a public IP, (2) a domain with two DNS A records pointing to your server (for example, minio.example.com and console.example.com), (3) Docker and Docker Compose installed, and (4) port 80/443 open in your firewall and cloud security group. Replace example.com with your domain throughout this tutorial.

Step 1: Open firewall and prepare project

Ensure inbound HTTP/HTTPS traffic is allowed so Caddy can obtain and renew TLS certificates. On Ubuntu with UFW:

sudo ufw allow 80,443/tcp
sudo ufw reload
mkdir -p ~/minio-caddy && cd ~/minio-caddy

Step 2: Create Docker Compose file

We will run MinIO and Caddy on the same Docker network. Caddy will request certificates from Let’s Encrypt automatically and reverse proxy to MinIO’s API and web console.

cat > docker-compose.yml <<'YAML'
version: "3.8"
services:
  minio:
    image: minio/minio:latest
    command: server /data --console-address ":9001" --address ":9000"
    environment:
      - MINIO_ROOT_USER=admin
      - MINIO_ROOT_PASSWORD=ChangeMe-StrongSecret123!
    volumes:
      - minio_data:/data
    restart: unless-stopped
    networks:
      - edge

  caddy:
    image: caddy:2
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
    depends_on:
      - minio
    restart: unless-stopped
    networks:
      - edge

volumes:
  minio_data:

networks:
  edge:
    driver: bridge
YAML

Step 3: Create Caddyfile for automatic HTTPS

This configuration terminates TLS, enables compression, and proxies API and console to MinIO. Make sure both hostnames have valid DNS A records pointing to your server’s public IP before starting.

cat > Caddyfile <<'CADDY'
minio.example.com {
  encode zstd gzip
  reverse_proxy minio:9000
}

console.example.com {
  encode zstd gzip
  reverse_proxy minio:9001
}
CADDY

Step 4: Start the stack

Bring everything up in the background. Caddy will automatically request TLS certificates from Let’s Encrypt on the first request and keep them renewed.

docker compose up -d
docker compose logs -f caddy

Once ready, visit https://console.example.com to access the MinIO console. Log in with the root credentials you set (admin / ChangeMe-StrongSecret123!). For security, change this password after your first login and create dedicated users for apps instead of sharing root.

Step 5: Create a bucket and access keys with MinIO Client (mc)

Use the MinIO Client to create a bucket and a non-root user with read/write access. Running mc in Docker avoids installing additional packages on the host.

# Add the MinIO endpoint alias (uses HTTPS through Caddy)
docker run --rm -it minio/mc \
  alias set myminio https://minio.example.com admin 'ChangeMe-StrongSecret123!'

# Create a bucket
docker run --rm -it minio/mc mb myminio/my-bucket

# Create an app user (access key) and attach readwrite policy
docker run --rm -it minio/mc \
  admin user add myminio appuser 'Another-StrongSecret456!'

docker run --rm -it minio/mc \
  admin policy attach myminio readwrite --user appuser

Step 6: Test with AWS CLI

MinIO is S3-compatible, so existing tools work by pointing to your endpoint. The AWS CLI is a convenient way to confirm access.

# Install AWS CLI if missing (Ubuntu)
sudo apt-get update && sudo apt-get install -y awscli

# Export temporary credentials for testing
export AWS_ACCESS_KEY_ID=appuser
export AWS_SECRET_ACCESS_KEY='Another-StrongSecret456!'

# List buckets on MinIO via your HTTPS endpoint
aws s3 ls --endpoint-url https://minio.example.com

# Upload a file to the new bucket
echo "hello from minio" > test.txt
aws s3 cp test.txt s3://my-bucket/ --endpoint-url https://minio.example.com

Maintenance and hardening tips

- Change the root password after initial setup and keep it offline. Create per-application users with the least privileges required. Rotate secrets regularly.

- Back up the MinIO data volume and, if critical, replicate to another MinIO cluster or cloud S3 using lifecycle policies or tools like rclone. Test restores.

- Keep Docker images up to date: run “docker compose pull && docker compose up -d” during a maintenance window. MinIO releases frequent fixes and performance updates.

- Monitor health and logs: “docker compose logs -f minio” and “docker compose logs -f caddy”. Use MinIO Console dashboards to watch capacity and performance.

Troubleshooting

- Certificate errors: confirm DNS is correct and that port 80 is open. Let’s Encrypt requires HTTP-01 reachability for the first certificate. Check “docker compose logs -f caddy”.

- 502/Bad Gateway: ensure “minio” container is running and healthy, and verify the Caddyfile hostnames match your browser URL. Restart with “docker compose restart”.

- Permission or upload failures: verify bucket policies or user permissions via the MinIO Console, and confirm your AWS CLI command uses “--endpoint-url”.

You now have a secure, S3-compatible object storage endpoint running on your own infrastructure with automatic HTTPS. Use it for application artifacts, backups, and large datasets, and manage everything from an easy web console and standard S3 tooling.