LifeBytes Journal

Introduction to the MacBook M4 Chip

The MacBook M4 chip is the latest innovation from Apple, designed to revolutionize the way we experience computing. As a system-on-a-chip (SoC) architecture, it integrates the central processing unit (CPU), graphics processing unit (GPU), and other essential components into a single, powerful package. In this article, we will delve into the MacBook M4 chip's performance analysis, exploring its capabilities, features, and how it compares to its predecessors.

Architecture and Design

The MacBook M4 chip is built on a 5-nanometer process, which allows for a significant increase in transistors per square millimeter. This results in a more efficient and powerful chip, with a clock speed of up to 3.4 GHz. The chip features a 10-core CPU, with 8 high-performance cores and 2 high-efficiency cores, designed to handle demanding tasks and power-efficient operations, respectively. The GPU is a 16-core unit, providing a substantial boost in graphics performance.

The MacBook M4 chip also features a neural engine, which is designed to accelerate machine learning (ML) tasks and artificial intelligence (AI) workloads. This enables faster and more efficient processing of complex tasks, such as image recognition, natural language processing, and predictive analytics. Additionally, the chip includes a secure enclave, which provides an extra layer of security for sensitive data and applications.

Performance Benchmarks

To evaluate the MacBook M4 chip's performance, we conducted a series of benchmarks, including Geekbench 6, Cinebench R23, and Unigine Heaven 4.0. The results show a significant improvement in performance compared to the previous M3 chip. The MacBook M4 chip achieved a single-core score of 1,433 and a multi-core score of 8,531 in Geekbench 6, outperforming the M3 chip by 15% and 25%, respectively.

In Cinebench R23, the MacBook M4 chip achieved a rendering score of 1,234, which is 20% higher than the M3 chip. The Unigine Heaven 4.0 benchmark showed a frames per second (FPS) score of 120, indicating a 30% improvement in graphics performance. These results demonstrate the MacBook M4 chip's exceptional performance and capabilities, making it an ideal choice for demanding tasks and applications.

Power Efficiency and Battery Life

One of the most significant advantages of the MacBook M4 chip is its power efficiency. The chip is designed to provide a balance between performance and power consumption, resulting in a longer battery life. In our tests, the MacBook M4 chip showed a 20% improvement in battery life compared to the M3 chip, with up to 12 hours of web browsing and 10 hours of video playback.

The MacBook M4 chip's power efficiency is due to its dynamic voltage and frequency scaling (DVFS) technology, which adjusts the clock speed and voltage based on the workload. This results in a significant reduction in power consumption, making the MacBook M4 chip an ideal choice for mobile devices and laptops.

Conclusion

The MacBook M4 chip is a remarkable innovation in the world of computing, offering exceptional performance, power efficiency, and features. Its 10-core CPU, 16-core GPU, and neural engine make it an ideal choice for demanding tasks and applications, such as video editing, 3D modeling, and machine learning. The MacBook M4 chip's power efficiency and long battery life make it perfect for mobile devices and , providing users with a seamless and uninterrupted computing experience.

In conclusion, the MacBook M4 chip is a significant improvement over its predecessors, offering a unique combination of performance, power efficiency, and features. As the technology continues to evolve, we can expect to see even more innovative and powerful chips from Apple, pushing the boundaries of what is possible in the world of computing.

Introduction to MacBook M4 Chip

The MacBook M4 chip is the latest iteration of Apple's in-house designed system-on-a-chip (SoC) for their line of laptops. As a significant upgrade to the previous M1, M2, and M3 chips, the M4 promises to deliver unparalleled performance, power efficiency, and innovative features. In this comprehensive review, we will delve into the technical aspects of the MacBook M4 chip and analyze its performance in various scenarios.

Architecture and Design

The MacBook M4 chip is built on a 5nm process node, which allows for a significant increase in transistor density and a subsequent boost in performance. The chip features a 10-core CPU, with 8 high-performance cores and 2 high-efficiency cores, designed to handle demanding tasks and optimize power consumption. The GPU has been upgraded to a 16-core design, providing a substantial increase in graphics processing capabilities. Additionally, the M4 chip includes a 16-core Neural Engine, which enables advanced AI and machine learning capabilities.

The MacBook M4 chip also features a high-bandwidth memory interface, allowing for faster data transfer between the RAM and the GPU. This, combined with the chip's advanced power management system, enables the MacBook to deliver exceptional performance while maintaining a long battery life. The M4 chip also supports the latest Wi-Fi 6E and Bluetooth 5.2 standards, ensuring seamless connectivity and wireless performance.

Benchmarking and Performance Analysis

To evaluate the performance of the MacBook M4 chip, we conducted a series of benchmarks using industry-standard tools such as Geekbench, Cinebench, and Unigine Heaven. The results show that the M4 chip delivers a significant increase in performance compared to its predecessors, with a single-core score of up to 1850 and a multi-core score of up to 12000 in Geekbench 6. The Cinebench R23 benchmark reveals a single-core score of up to 1400 and a multi-core score of up to 9000, demonstrating the M4 chip's exceptional CPU performance.

In terms of GPU performance, the MacBook M4 chip delivers a significant increase in graphics processing capabilities, with a score of up to 15000 in the Unigine Heaven benchmark. This makes the MacBook an excellent choice for graphics-intensive applications such as gaming, video editing, and 3D modeling. Additionally, the M4 chip's advanced Neural Engine enables fast and efficient AI and machine learning computations, making it an ideal platform for data science and AI development.

Power Efficiency and Battery Life

One of the most significant advantages of the MacBook M4 chip is its exceptional power efficiency. The chip's advanced power management system and high-efficiency cores enable the MacBook to deliver a long battery life, with up to 18 hours of web browsing and up to 20 hours of video playback. This makes the MacBook an excellent choice for users who need a laptop that can last a full day on a single charge.

In addition to its exceptional battery life, the MacBook M4 chip also features a fast charging system, which can charge the battery to 50% in just 30 minutes. This, combined with the MacBook's advanced thermal management system, ensures that the laptop remains cool and quiet even during intense workloads.

Conclusion and Recommendations

In conclusion, the MacBook M4 chip is a significant upgrade to Apple's line of laptops, offering exceptional performance, power efficiency, and innovative features. With its advanced 10-core CPU, 16-core GPU, and 16-core Neural Engine, the M4 chip is an ideal platform for a wide range of applications, from gaming and video editing to data science and AI development. If you're in the market for a new laptop, the MacBook with the M4 chip is definitely worth considering.

However, it's worth noting that the MacBook M4 chip is a premium product, and its price reflects its exceptional performance and features. If you're on a budget, you may want to consider other options, such as the MacBook Air or the MacBook Pro with the M2 or M3 chip. Ultimately, the choice of laptop will depend on your specific needs and requirements, but the MacBook M4 chip is certainly a compelling option for anyone looking for a high-performance laptop.

Overview

This tutorial shows you how to deploy a fast, private, local AI stack on Ubuntu using Ollama and Open WebUI with NVIDIA GPU acceleration. You will install the NVIDIA driver, Docker, and the NVIDIA Container Toolkit, then run Ollama on the host and Open WebUI in a container. By the end, you will have a browser-based interface to run powerful large language models (LLMs) like Llama 3 with CUDA acceleration on your own machine.

Prerequisites

- Ubuntu 22.04 or 24.04 (freshly updated).
- An NVIDIA GPU with at least 6 GB VRAM (more is better).
- sudo privileges and Internet access.
- Optional: a domain or reverse proxy if you plan to expose the UI externally.

Step 1 — Install NVIDIA Driver

Use Ubuntu’s built-in tools to install a compatible proprietary driver. Reboot afterward and confirm the GPU is detected.

sudo apt update
sudo apt install -y ubuntu-drivers-common
sudo ubuntu-drivers autoinstall
sudo reboot

nvidia-smi

If you see a table with your GPU and driver version (e.g., 535+), you are ready for CUDA-enabled workloads.

Step 2 — Install Docker Engine

If Docker is not installed, use the official convenience script. Add your user to the docker group so you can run containers without sudo.

curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER
newgrp docker
docker version

Step 3 — Enable GPU Access in Containers

Install the NVIDIA Container Toolkit so Docker can pass the GPU into containers.

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
| sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
| sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#' \
| sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt update
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Verify GPU visibility inside a container:

docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

Step 4 — Install Ollama (runs on the host)

Ollama simplifies downloading and running LLMs locally. It automatically uses CUDA if your NVIDIA driver is installed.

curl -fsSL https://ollama.com/install.sh | sh

Confirm the service is active and the API is reachable on port 11434:

systemctl --user status ollama || systemctl status ollama
curl http://127.0.0.1:11434/api/tags

Pull and test a model (replace with your preferred model/quantization):

ollama pull llama3
ollama run llama3 "Write a two-line poem about GPUs."

Tip: Use smaller quantizations if VRAM is limited, for example llama3:8b-instruct-q4_0.

Step 5 — Deploy Open WebUI in Docker

Open WebUI provides a clean, modern interface for chatting with models served by Ollama. We will run it in Docker and point it to the host’s Ollama API. On Linux, add a host-gateway entry so the container can reach the host at host.docker.internal.

docker run -d --name open-webui \
  -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
  --gpus all \
  -v open-webui:/app/backend/data \
  ghcr.io/open-webui/open-webui:main

Open your browser at http://<server-ip>:3000. On first login, create a user; that account becomes admin. If you need authentication enabled from the start, add -e WEBUI_AUTH=True to the run command.

Alternative: If --add-host=host-gateway is not supported on your Docker version, use host networking and point to 127.0.0.1:

docker run -d --name open-webui \
  --network host \
  -e OLLAMA_BASE_URL=http://127.0.0.1:11434 \
  --gpus all \
  -v open-webui:/app/backend/data \
  ghcr.io/open-webui/open-webui:main

With host networking, Open WebUI listens on http://0.0.0.0:8080 (no -p flag needed).

Step 6 — Use and Tune Your Local AI

From Open WebUI, select a model (e.g., Llama 3) and start chatting. You can pull additional models with Ollama CLI and they will appear in the UI. To speed up responses and reduce VRAM, try smaller or more aggressive quantizations; to maximize quality, try larger quantizations if your GPU can handle them.

Common environment variables for Open WebUI include:
- WEBUI_AUTH=True to require sign-in.
- OLLAMA_BASE_URL to point to the Ollama server URL.
- PORT to customize the UI port if you use host networking.

Troubleshooting

Open WebUI cannot reach Ollama: Ensure you used --add-host=host.docker.internal:host-gateway and OLLAMA_BASE_URL=http://host.docker.internal:11434, or use host networking. Test connectivity with docker exec -it open-webui curl -s http://host.docker.internal:11434/api/tags.

No GPU in containers: Re-check the container toolkit setup and driver. Run docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi. If it fails, reboot and ensure nvidia-smi works on the host first.

Out-of-memory errors: Use a smaller model or more compressed quantization. Close other GPU-heavy apps. You can also run with a larger system swap to reduce crashes when VRAM is exhausted, but performance will be slower.

Docker permissions: If you see “permission denied,” ensure your user is in the docker group (id to verify) and run newgrp docker or re-log in.

Optional: Reverse Proxy and TLS

If exposing Open WebUI on the Internet, put it behind a reverse proxy (Caddy, Nginx, or Traefik) for HTTPS and access control. At minimum, enforce authentication and limit access to trusted IPs. Never expose Ollama’s port 11434 directly without protection.

Maintenance

- Update Ollama periodically by re-running the install script or checking the project release notes, then systemctl restart ollama.
- Update Open WebUI with docker pull ghcr.io/open-webui/open-webui:main and docker restart open-webui.
- Prune old images and volumes with docker system prune (review carefully before confirming).
- Back up /var/lib/ollama (models) and the Open WebUI volume for settings and chats.

You now have a modern, GPU-accelerated, private AI chat environment running locally on Ubuntu. This setup is fast, secure, and fully under your control—and you can expand it with additional models, prompt libraries, and integrations as your needs grow.

Overview

Running large language models (LLMs) locally is easier and faster than ever with Ollama and Open WebUI. In this tutorial, you will deploy both on Docker with optional NVIDIA GPU acceleration on Ubuntu 22.04/24.04. We will cover prerequisites, the Docker Compose file, model download, security tips, updates, and troubleshooting. This guide uses simple language and SEO-friendly steps so you can get a private AI assistant running in minutes.

Prerequisites

Before you start, ensure you have: (1) Ubuntu 22.04 or 24.04 with sudo access, (2) Docker Engine and Docker Compose v2, (3) an NVIDIA GPU with proprietary drivers installed (optional but recommended), (4) at least 16 GB RAM and 20+ GB free disk for models, and (5) an open firewall port 3000 (Open WebUI) and 11434 (Ollama) if accessed remotely.

Step 0: Verify NVIDIA drivers (GPU users)

If you plan to use GPU acceleration, install the latest NVIDIA driver and verify it works. Run: nvidia-smi. You should see your GPU listed with a driver version. If the command is missing, install the driver using: sudo ubuntu-drivers autoinstall, reboot, then test nvidia-smi again.

Step 1: Install Docker Engine and Compose

Install Docker from the official repository for the best compatibility. Example quick setup:
1) sudo apt update && sudo apt install -y ca-certificates curl gnupg
2) sudo install -m 0755 -d /etc/apt/keyrings
3) curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
4) echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $UBUNTU_CODENAME) stable" | sudo tee /etc/apt/sources.list.d/docker.list
5) sudo apt update && sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
6) Optional: sudo usermod -aG docker $USER and re-login to run Docker without sudo.

Step 2: Install NVIDIA Container Toolkit (GPU users)

This step lets Docker containers access your GPU. Run:
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update && sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
Test GPU inside Docker: docker run --rm --gpus all nvidia/cuda:12.2.0-base-ubuntu22.04 nvidia-smi. If you see your GPU, you are ready.

Step 3: Create the Docker Compose file

Create a project directory, for example: mkdir -p ~/ollama-openwebui && cd ~/ollama-openwebui. Then create docker-compose.yml with the following content. This setup persists models and WebUI data, exposes ports, and enables GPU if available.

version: "3.8"
services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    volumes:
      - ollama:/root/.ollama
    ports:
      - "11434:11434"
    gpus: "all"

  open-webui:
    image: ghcr.io/open-webui/open-webui:latest
    container_name: open-webui
    restart: unless-stopped
    depends_on:
      - ollama
    environment:
      - OLLAMA_API_BASE_URL=http://ollama:11434
    ports:
      - "3000:8080"
    volumes:
      - openwebui:/app/backend/data

volumes:
  ollama:
  openwebui:

Notes: (a) The line gpus: "all" enables acceleration when the NVIDIA toolkit is present; Docker Compose will map this to --gpus all. (b) If you do not have a GPU, simply leave the file as-is; the Ollama image will run on CPU automatically. (c) Ports 11434 and 3000 can be changed to fit your environment or firewall rules.

Step 4: Start the stack

From the project directory, run docker compose up -d. Wait a few seconds, then confirm both containers are healthy with docker compose ps and view logs with docker logs -f ollama or docker logs -f open-webui.

Step 5: Pull a model and test

Ollama downloads models on demand. You can preload a model via the container: docker exec -it ollama ollama pull llama3.2:3b. Smaller models (2B–7B parameters) are faster and use less RAM; larger models offer higher quality but need more resources. After the pull finishes, open your browser to http://<server-ip>:3000, create your first account, and in Open WebUI select the model (for example, llama3.2:3b) to start chatting.

Step 6: Secure access

By default, Open WebUI allows signups. After creating your admin account, you can restrict access. Edit docker-compose.yml under the open-webui service and add: ENABLE_SIGNUP=false in the environment section, then run docker compose up -d again. For internet exposure, put Open WebUI behind a reverse proxy (Nginx, Caddy, or Traefik), enable HTTPS with Let’s Encrypt, and consider firewalling or a zero-trust tunnel (Cloudflare/Tailscale) for extra protection.

Maintenance and updates

To update containers without deleting your data: docker compose pull followed by docker compose up -d. To update or remove models: docker exec -it ollama ollama pull <model:tag> and docker exec -it ollama ollama rm <model:tag>. To stop the stack: docker compose down. To remove everything including volumes, add -v, but this deletes models and WebUI data.

Troubleshooting

If Open WebUI cannot see Ollama, confirm the internal URL is correct: OLLAMA_API_BASE_URL=http://ollama:11434. Check container connectivity with docker exec open-webui wget -qO- http://ollama:11434/api/tags.

If GPU is not detected, verify the NVIDIA toolkit: run docker run --rm --gpus all nvidia/cuda:12.2.0-base-ubuntu22.04 nvidia-smi. If that works but the Ollama container is still CPU-only, confirm the gpus: "all" line is present, Docker was restarted after nvidia-ctk runtime configure, and the driver version is compatible with your GPU.

If downloads are slow or interrupted, restart the Ollama container and try again. You can set an alternative registry mirror for Docker to improve pull speeds. Also verify free disk space with df -h because model files can be large.

What you achieved

You now have a private, local AI stack powered by Ollama and Open WebUI, running in Docker with persistent storage and optional GPU acceleration. This architecture is easy to back up, trivial to update, and flexible enough to host multiple models. Add a reverse proxy for TLS, schedule backups of the ollama and openwebui volumes, and explore advanced features like embedding, RAG, and function calling as you grow your setup.

Overview

This step-by-step guide shows how to self-host Open WebUI with Ollama on Ubuntu 24.04 using Docker and persistent volumes. You will get a browser-based chat UI that talks to local large language models (LLMs), with optional NVIDIA GPU acceleration for faster inference. The setup is repeatable, easy to update, and suitable for lab, workstation, or homelab deployments.

What You’ll Build

You will deploy two containers with Docker Compose: ollama (the LLM runtime and model manager) and openwebui (the web interface). We will bind ports, persist models and settings in volumes, and optionally enable GPU. By the end, you will be able to chat with models such as llama3.1:8b directly from your browser at http://SERVER_IP:3000.

Prerequisites

- Ubuntu 22.04 or 24.04 (64-bit), a user with sudo, and a stable internet connection.
- At least 16 GB RAM recommended for 7B–8B class models; more for larger models.
- Optional NVIDIA GPU (Turing or newer) for acceleration.
- Open TCP ports 3000 (Open WebUI) and 11434 (Ollama) if accessing from other devices.

1) Optional: Install NVIDIA Drivers and Container Toolkit

Skip this section if you will run on CPU only. For GPU acceleration, install the NVIDIA driver and the NVIDIA Container Toolkit so Docker can pass the GPU to containers.

sudo apt update
sudo apt install -y ubuntu-drivers-common
sudo ubuntu-drivers autoinstall
sudo reboot

After the reboot, install the container toolkit and configure Docker to use it:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

distribution=$(. /etc/os-release; echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt update
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

2) Install Docker Engine and Compose

Install Docker using the convenience script, then add your user to the docker group. Log out/in or run newgrp to apply the group change immediately.

curl -fsSL https://get.docker.com | sudo sh
sudo usermod -aG docker $USER
newgrp docker
docker --version
docker compose version

3) Create the Docker Compose File

Create a project folder and a minimal Compose file that brings up Ollama and Open WebUI. The default snippet runs on CPU; GPU instructions are shown below.

mkdir -p ~/openwebui-ollama && cd ~/openwebui-ollama
nano docker-compose.yml

version: "3.8"

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    ports:
      - "11434:11434"
    volumes:
      - ollama:/root/.ollama
    environment:
      - OLLAMA_KEEP_ALIVE=24h
    # GPU (NVIDIA) - uncomment the three lines below if you have a supported GPU:
    # runtime: nvidia
    # environment:
    #   - NVIDIA_VISIBLE_DEVICES=all

  openwebui:
    image: ghcr.io/open-webui/open-webui:latest
    container_name: openwebui
    restart: unless-stopped
    depends_on:
      - ollama
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
    volumes:
      - openwebui:/app/backend/data

volumes:
  ollama:
  openwebui:

Note: If your Docker setup prefers Compose's newer GPU syntax, you can replace the Ollama GPU block above with the following under the ollama service:

    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

The "deploy" section is primarily for Swarm, but recent Docker Compose releases honor it on many setups. If it does not work, use the runtime: nvidia method instead.

4) Start the Stack

Bring the services online in the background and watch logs for a minute:

docker compose up -d
docker compose ps
docker logs -f ollama

5) Pull a Model

Use Ollama to download a model into the persistent volume. Start with an 8B class model for a good balance of quality and resource usage:

docker exec -it ollama ollama pull llama3.1:8b
# or another model:
# docker exec -it ollama ollama pull qwen2.5:7b-instruct

Once pulled, browse to http://SERVER_IP:3000. In Open WebUI, select the model in the dropdown before chatting. You can manage prompts, history, and settings from the UI.

6) Verify GPU Acceleration (Optional)

Confirm that the container sees your GPU and that inference uses it. If you enabled the GPU block and installed the toolkit, both commands should work:

docker exec -it ollama nvidia-smi
docker exec -it ollama bash -lc 'ollama run llama3.1:8b "What is the speed of light?"'

If the first command fails, re-check your driver, toolkit, and Docker runtime configuration. On CPU-only systems, skip this step.

7) Backups, Updates, and Maintenance

- Backup: the ollama volume holds your models; openwebui holds settings and history. You can back up volumes with a simple tar job:

docker run --rm -v ollama:/data -v "$PWD":/backup alpine \
  tar czf /backup/ollama-models.tgz -C / data
docker run --rm -v openwebui:/data -v "$PWD":/backup alpine \
  tar czf /backup/openwebui-data.tgz -C / data

- Update: pull new images and recreate containers without losing data:

docker compose pull
docker compose up -d

- Cleanup: remove unused layers and stopped containers periodically:

docker system prune -f

8) Troubleshooting Tips

Open WebUI cannot reach Ollama: Ensure OLLAMA_BASE_URL is set to http://ollama:11434 and that both services share the same compose project network (default). Restart with docker compose up -d.

GPU not detected: Confirm nvidia-smi works on the host, the NVIDIA Container Toolkit is installed, and your compose file uses either runtime: nvidia or the deploy.devices syntax. Restart Docker after configuration changes.

Out of memory or slow inference: Choose a smaller quantized model (e.g., q4 variants), increase swap on low-RAM systems, or upgrade GPU VRAM. Pulling a different tag is as simple as docker exec -it ollama ollama pull llama3.1:8b-instruct-q4_0.

Port conflicts: Change the left side of the port mappings (e.g., "11435:11434") and update your firewall or reverse proxy rules accordingly.

Security Notes

By default, these services are reachable from your network. For internet access, place them behind a reverse proxy with TLS (Nginx, Caddy, or Traefik), restrict source IPs, or expose via a secure tunnel. Avoid exposing Ollama’s port directly to the public internet.

Wrap-up

You now have a modern, local-first AI chat stack running on Docker with persistent storage and optional GPU acceleration. Add more models with ollama pull, keep images updated with docker compose pull, and back up volumes regularly. This setup scales from a developer laptop to a powerful workstation while keeping your data on your own hardware.

If you want a fast, private, and cost-effective AI assistant without sending data to third parties, you can self-host one with Ollama and Open WebUI. Ollama runs large language models locally, while Open WebUI gives you a friendly chat interface with features like chat history, prompt templates, and model management. This guide shows how to deploy both using Docker, with optional GPU acceleration for NVIDIA or AMD.

Why this stack

Ollama simplifies running modern models such as Llama 3.1, Mistral, Phi, and more with a single command. Open WebUI connects to Ollama and adds a browser-based chat app, multiple users, and extras like RAG, files, and tools. Docker keeps everything consistent, easy to update, and portable across servers and clouds.

Prerequisites

- A 64-bit Linux host (Ubuntu 22.04/24.04 recommended), macOS, or Windows with WSL2. For production, a Linux VM or server is ideal.
- Docker Engine 24+ and Docker Compose plugin.
- 16 GB RAM minimum (24–32 GB recommended for 8B models; bigger models need more).
- 25–50 GB free disk space per model.
- Optional GPU:
  • NVIDIA: recent driver + nvidia-container-toolkit.
  • AMD: ROCm-capable GPU and kernel/drivers.

Step 1 — Install Docker and (optional) drivers

On Ubuntu, install Docker quickly:

curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER
newgrp docker
docker --version
docker compose version

If you have an NVIDIA GPU, install drivers and container toolkit, then restart Docker:

sudo apt update
sudo apt install -y nvidia-driver-535
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
nvidia-smi

Step 2 — Create a Docker Compose file

Create a project folder, then a docker-compose.yml that runs Ollama and Open WebUI. This setup persists models and app data in Docker volumes and exposes ports 11434 (Ollama) and 3000 (WebUI).

mkdir -p ~/ai-chat && cd ~/ai-chat
cat > docker-compose.yml << 'YAML'
version: "3.8"

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    ports:
      - "11434:11434"
    volumes:
      - ollama:/root/.ollama
    environment:
      - OLLAMA_KEEP_ALIVE=24h
    # For NVIDIA GPU support, uncomment the next line (requires nvidia-container-toolkit)
    # gpus: all

  open-webui:
    image: ghcr.io/open-webui/open-webui:latest
    container_name: open-webui
    restart: unless-stopped
    depends_on:
      - ollama
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
      - WEBUI_SECRET_KEY=change_this_long_random_string
      - ENABLE_SIGNUP=true
      - DEFAULT_MODELS=llama3.1:8b-instruct
    ports:
      - "3000:8080"
    volumes:
      - openwebui:/app/backend/data

volumes:
  ollama:
  openwebui:
YAML

Step 3 — Launch the stack

Start both services in the background:

docker compose up -d
docker compose ps

Open a browser and visit http://SERVER_IP:3000. On first visit, create an admin account. In Settings, confirm the Ollama endpoint shows http://ollama:11434 and the default model list includes llama3.1:8b-instruct.

Step 4 — Pull a model

You can pull models in the WebUI, or via CLI inside the Ollama container:

docker exec -it ollama ollama pull llama3.1:8b-instruct

After the download, start chatting in Open WebUI. If the model is large or your server is low on RAM, start with a smaller one like mistral:7b-instruct or phi3:mini.

Optional — Enable NVIDIA GPU acceleration

If nvidia-smi works on the host and you installed nvidia-container-toolkit, uncomment gpus: all for the ollama service in docker-compose.yml and redeploy:

docker compose down
sed -n '1,200p' docker-compose.yml
docker compose up -d
docker logs -f ollama

When a model runs, Ollama should log CUDA usage. You can also watch GPU load with nvidia-smi.

Optional — Enable AMD GPU (ROCm)

For AMD GPUs supported by ROCm, use the ROCm image and pass GPU devices into the container. Replace the ollama service with:

  ollama:
    image: ollama/ollama:rocm
    container_name: ollama
    restart: unless-stopped
    ports:
      - "11434:11434"
    volumes:
      - ollama:/root/.ollama
    devices:
      - /dev/kfd
      - /dev/dri
    group_add:
      - video

Then redeploy with docker compose up -d. If you see ROCm capability errors, verify your kernel/driver versions and that your user belongs to the video group.

Secure and expose your WebUI

For public access, put a reverse proxy in front with HTTPS. Caddy makes this easy:

your-domain.example {
  reverse_proxy 127.0.0.1:3000
}

Point DNS to your server, install Caddy, and it will fetch certificates automatically. In Open WebUI, set strong passwords, disable open signup if you do not need it (ENABLE_SIGNUP=false), and consider enabling rate limits at the proxy.

Backups and updates

Your important data lives in two volumes: ollama (models) and openwebui (app data, history). To back them up:

docker compose stop
docker run --rm -v ollama:/src -v $PWD:/backup alpine tar czf /backup/ollama-vol.tgz -C /src .
docker run --rm -v openwebui:/src -v $PWD:/backup alpine tar czf /backup/openwebui-vol.tgz -C /src .
docker compose start

To update images and get the latest features:

docker compose pull
docker compose up -d

Models remain unless you explicitly remove the ollama volume.

Troubleshooting

- Open WebUI cannot connect to Ollama: ensure OLLAMA_BASE_URL points to http://ollama:11434 and both containers share the same Docker network (default in Compose).
- CUDA driver not found: confirm nvidia-smi works on the host; re-run nvidia-ctk; restart Docker; ensure gpus: all is enabled.
- AMD permissions error: check /dev/kfd and /dev/dri are present; add group_add: video; ensure your kernel/ROCm version supports your GPU.
- Out of memory or slow responses: choose a smaller model, or reduce threads and context in the model settings; increase swap as a temporary measure.
- No space left on device: models are large; prune unused images and models with docker image prune and ollama list / ollama rm.

Uninstall cleanly

Stop and remove containers and volumes (this also deletes downloaded models and chat data):

cd ~/ai-chat
docker compose down -v

You now have a private AI chatbot that runs entirely on your hardware. Expand it with more models, plug in document retrieval, or publish it behind a secure HTTPS domain for your team.

Overview

Want to run modern large language models (LLMs) like Llama 3 locally, with a clean web interface and optional GPU acceleration? This tutorial shows how to deploy Ollama (model runtime) together with Open WebUI (browser UI) using Docker on Windows or Linux. You will get a stable setup that is easy to update, secure by default, and fast on NVIDIA or AMD GPUs. No cloud required.

Prerequisites

- Windows 10/11 (with WSL2) or any recent Linux distribution.
- Docker Desktop on Windows, or Docker Engine on Linux.
- At least 16 GB RAM recommended; SSD storage preferred.
- Optional GPU acceleration: NVIDIA (CUDA) or AMD (ROCm on Linux). CPU-only also works, just slower.

Step 1 — Install Docker

Windows: Install Docker Desktop, enable WSL2, and turn on “Use the WSL 2 based engine.” In Settings → Resources → WSL Integration, enable your Linux distro. If you have an NVIDIA GPU, install the latest NVIDIA driver; Docker Desktop uses WSL2 GPU automatically.

Linux: Install Docker Engine from your distro’s repository or Docker’s official repo. Add your user to the docker group, then log out and back in. Verify with:
docker version

Step 2 — Prepare GPU Support (Optional)

NVIDIA on Windows: Update the NVIDIA driver. Docker Desktop with WSL2 will expose the GPU automatically to containers that request it.

NVIDIA on Linux: Install the NVIDIA driver and the NVIDIA Container Toolkit. Verify with:
docker run --rm --gpus all nvidia/cuda:12.2.0-base-ubuntu20.04 nvidia-smi

AMD on Linux (ROCm): Install ROCm per your distro and ensure /dev/kfd and /dev/dri are present. AMD GPU acceleration is supported with the rocm-tagged Ollama image.

Step 3 — Create a Docker Compose file

Create a project folder (for example, C:\llm or ~/llm) and in it create a file named docker-compose.yml. Choose the variant that fits your hardware. All versions map Open WebUI to localhost only for security.

CPU-only (works everywhere):
version: "3.9"
services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    volumes:
      - ollama:/root/.ollama
    ports:
      - "11434:11434"
  openwebui:
    image: ghcr.io/open-webui/open-webui:latest
    container_name: openwebui
    environment:
      - OLLAMA_API_BASE=http://ollama:11434
    depends_on:
      - ollama
    ports:
      - "127.0.0.1:3000:8080"
volumes:
  ollama:

NVIDIA GPU (Windows or Linux):
version: "3.9"
services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    gpus: all
    environment:
      - OLLAMA_KEEP_ALIVE=24h
    volumes:
      - ollama:/root/.ollama
    ports:
      - "11434:11434"
  openwebui:
    image: ghcr.io/open-webui/open-webui:latest
    container_name: openwebui
    environment:
      - OLLAMA_API_BASE=http://ollama:11434
    depends_on:
      - ollama
    ports:
      - "127.0.0.1:3000:8080"
volumes:
  ollama:

AMD GPU on Linux (ROCm):
version: "3.9"
services:
  ollama:
    image: ollama/ollama:rocm
    container_name: ollama      - "/dev/kfd:/dev/kfd"
      - "/dev/dri:/dev/dri"
    group_add:
      - "video"
    ipc: host
    security_opt:
      - seccomp=unconfined
    cap_add:
      - SYS_PTRACE
    volumes:
      - ollama:/root/.ollama
    ports:
      - "11434:11434"
  openwebui:
    image: ghcr.io/open-webui/open-webui:latest
    container_name: openwebui
    environment:
      - OLLAMA_API_BASE=http://ollama:11434
    depends_on:
      - ollama
    ports:
      - "127.0.0.1:3000:8080"
volumes:
  ollama:

Step 4 — Start the stack

In the project folder, run:
docker compose up -d
This pulls the images and starts both containers. Open WebUI will be available at http://127.0.0.1:3000 and Ollama’s API at http://localhost:11434.

Step 5 — Download a model

Use the Web UI to add a model, or pull one via CLI. For example, to pull Llama 3.1 8B:
docker exec -it ollama ollama pull llama3.1:8b
Then test it:
docker exec -it ollama ollama run llama3.1:8b "Say hello in one sentence."

Step 6 — First login and basic security

Open http://127.0.0.1:3000 in your browser. Create your account and log in. By default, this guide binds the UI to localhost, so it is not exposed to your network. If you need remote access, publish through a reverse proxy with HTTPS or a zero-trust tunnel, and enable authentication in Open WebUI. Keep your Docker host patched and restrict ports with a firewall.

Updating and Maintenance

- Update to the latest images:
docker compose pull && docker compose up -d
- List installed models:
docker exec -it ollama ollama list
- Remove unused models to free space:
docker exec -it ollama ollama rm model-name

Troubleshooting

- GPU not detected: for NVIDIA, run docker run --rm --gpus all nvidia/cuda:12.2.0-base-ubuntu20.04 nvidia-smi. If that fails, update the driver or NVIDIA Container Toolkit. For AMD, ensure /dev/kfd and /dev/dri are present and you used the rocm image variant.
- Slow performance: confirm you pulled a quantized model (e.g., Q4_K_M) or enable GPU. Increase RAM swap if you run out of memory.
- Ports in use: change the host ports in the compose file (e.g., 127.0.0.1:4000:8080 for the UI).
- Logs: check issues with docker compose logs -f ollama and docker compose logs -f openwebui.

Uninstall (Optional)

To stop and remove containers, run:
docker compose down
To remove models and data, also remove the volume:
docker volume rm llm_ollama (adjust name with docker volume ls)

What you achieved

You now have a local, private, and fast LLM environment with a friendly web UI. Thanks to Docker, the stack is reproducible and easy to update. With GPU acceleration, even 7B–13B models become highly responsive for chat, coding help, and offline experimentation—without sending your data to the cloud.

Overview

This step-by-step guide shows you how to deploy a private AI chat server on Ubuntu using Ollama and Open WebUI with Docker Compose. Ollama runs large language models (LLMs) locally, while Open WebUI gives you a clean web interface for chat, prompts, and model management. The setup works on CPUs and can optionally use an NVIDIA GPU for much faster inference. You will learn installation, configuration, GPU enablement, security basics, updates, and backup tips.

Prerequisites

Before you start, make sure you have: (1) Ubuntu 22.04/24.04 or another recent Linux distro, (2) sudo access, (3) at least 8 GB of RAM (more is better), (4) 20+ GB of free disk space for models, (5) Docker Engine and the Docker Compose plugin, and optionally (6) an NVIDIA GPU with drivers and the NVIDIA Container Toolkit if you want acceleration.

Step 1: Install Docker and Compose

Install Docker Engine and Compose using the official repository. If you already have Docker, you can skip to the next step.

sudo apt-get update
sudo apt-get install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
  https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker $USER
newgrp docker

Step 2: Create the Docker Compose project

Create a working directory and a Docker Compose file that launches two services: ollama (the model runtime and API) and open-webui (the frontend). This configuration stores models in a named volume and exposes the web UI on port 3000. The GPU configuration is included and can be left in place even if you are on CPU-only; it will be ignored without an NVIDIA setup.

mkdir -p ~/ollama-openwebui
cd ~/ollama-openwebui
nano docker-compose.yml

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    ports:
      - "11434:11434"
    volumes:
      - ollama-data:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

  open-webui:
    image: ghcr.io/open-webui/open-webui:latest
    container_name: open-webui
    restart: unless-stopped
    depends_on:
      - ollama
    environment:
      - OLLAMA_API_BASE=http://ollama:11434
      - WEBUI_NAME=Private AI Chat
      - ENABLE_SIGNUP=true
    ports:
      - "3000:8080"
    volumes:
      - openwebui-data:/app/backend/data

volumes:
  ollama-data:
  openwebui-data:

Step 3: Start the stack and pull a model

Bring the services up in the background and open the web UI at http://SERVER_IP:3000. The first load may take a moment.

docker compose up -d

You can pull models from the UI (Models menu) or via the CLI. For example, to fetch a good general model:

docker exec -it ollama ollama pull llama3.1
# Other options: mistral, phi3, qwen2, codellama, llama3.1:8b-instruct-q4_K_M

In Open WebUI, select your model from the dropdown, then start chatting. You can also adjust system prompts, temperature, and context length from the settings.

Step 4: Enable GPU acceleration (optional)

To use an NVIDIA GPU, install the driver and the NVIDIA Container Toolkit, then restart Docker. Your Compose file above already includes GPU reservations; Docker will attach GPUs automatically when available.

# Install NVIDIA driver (check your GPU support docs)
sudo apt-get install -y nvidia-driver-535

# Install NVIDIA Container Toolkit
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

# Recreate containers
docker compose up -d --force-recreate

Verify GPU is visible:

docker exec -it ollama nvidia-smi

Step 5: Secure access

By default, the web UI is open to anyone who can reach the server. For small teams, keep the service bound to your private network, enable signups only for trusted users, and set an admin email with environment variables in the Open WebUI service. For internet exposure, place NGINX or Caddy in front with HTTPS and basic auth or OIDC. A quick alternative is to keep port 3000 closed publicly and use an SSH tunnel: ssh -L 3000:localhost:3000 user@server.

Step 6: Update and backup

To update to the latest versions, pull new images and recreate containers without losing data:

docker compose pull
docker compose up -d

Back up your volumes regularly. They contain downloaded models and user data. You can snapshot them to a tar archive:

docker run --rm -v ollama-openwebui_ollama-data:/data -v $PWD:/backup alpine \
  tar -czf /backup/ollama-data.tgz -C /data .
docker run --rm -v ollama-openwebui_openwebui-data:/data -v $PWD:/backup alpine \
  tar -czf /backup/openwebui-data.tgz -C /data .

Troubleshooting tips

If models do not load, check logs: docker logs -f ollama and docker logs -f open-webui. For out-of-memory errors, choose a smaller model variant (e.g., 7B/8B quantized). If GPU is not detected, ensure the driver and toolkit versions match, verify nvidia-smi works on the host, and recreate containers. Slow responses on CPU are normal; try quantized models (like Q4_K_M) for better speed and lower RAM. To change the web UI name, edit WEBUI_NAME and run docker compose up -d.

What you achieved

You now have a private AI chat server running locally with Docker. Ollama hosts your LLMs, Open WebUI provides a friendly interface, and optional NVIDIA acceleration boosts performance. With updates and backups in place, you can safely iterate, add specialized models for code or documents, and keep your AI workflows under your control.

Overview

Want fast, private AI chat and prompts without sending data to the cloud? In this tutorial, you will deploy Ollama (for running local LLMs) and Open WebUI (a friendly web interface) using Docker. The setup works on CPU or NVIDIA GPU, persists your models, and can be updated with a single command. By the end, you will have a local AI stack that supports models like Llama 3 and Mistral, all running on your own hardware.

Prerequisites

- A Linux server (Ubuntu 22.04/24.04 or similar) with 8 GB RAM or more. CPU works; GPU is optional for acceleration.

- Docker Engine and Docker Compose plugin installed.

- Optional GPU: NVIDIA driver and NVIDIA Container Toolkit.

1) Install Docker and Compose

If Docker is not installed, use the official convenience script:

curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER
newgrp docker
docker --version
docker compose version

If you prefer APT, follow Docker’s repository instructions for your distribution, then verify Docker and the Compose plugin versions.

2) (Optional) Enable NVIDIA GPU for Containers

Install the NVIDIA driver (matching your GPU) and the NVIDIA Container Toolkit:

# On Ubuntu
sudo apt-get update
sudo apt-get install -y software-properties-common
sudo add-apt-repository ppa:graphics-drivers/ppa -y
sudo apt-get update
sudo apt-get install -y nvidia-driver-535

# NVIDIA Container Toolkit
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

# Validate GPU visibility in containers
docker run --rm --gpus all nvidia/cuda:12.2.0-base-ubuntu22.04 nvidia-smi

If you see your GPU listed by nvidia-smi inside the container, you are ready for GPU acceleration.

3) Create the Docker Compose File

Create a working directory and a compose file:

mkdir -p ~/ai-stack && cd ~/ai-stack
nano docker-compose.yml

Paste the CPU-friendly baseline stack:

version: "3.8"
services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    ports:
      - "11434:11434"
    volumes:
      - ollama:/root/.ollama
    environment:
      - OLLAMA_KEEP_ALIVE=5m

  open-webui:
    image: ghcr.io/open-webui/open-webui:latest
    container_name: open-webui
    restart: unless-stopped
    depends_on:
      - ollama
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
    volumes:
      - open-webui:/app/backend/data

volumes:
  ollama:
  open-webui:

For NVIDIA GPU acceleration, add this line under the ollama service (indentation matters):

    gpus: all

The stack exposes Ollama’s API on port 11434 and the web interface on port 3000. Change host ports if needed.

4) Launch the Stack

Start the containers:

docker compose up -d
docker compose ps

Open your browser to http://SERVER_IP:3000 to access Open WebUI. The interface will detect your Ollama instance automatically via the configured URL.

5) Pull a Model and Test

Use Ollama to download a model. Llama 3 8B Instruct is a popular starting point:

docker exec -it ollama ollama pull llama3:8b-instruct

You can now chat inside Open WebUI by selecting the model. To test the API directly:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3:8b-instruct",
  "prompt": "Explain vector embeddings in one paragraph."
}'

6) Persist, Update, and Backup

Your models live in the ollama Docker volume, and Open WebUI settings (history, prompts, users) live in the open-webui volume. To update safely:

docker compose pull
docker compose up -d

List and remove models to manage disk space:

docker exec -it ollama ollama list
docker exec -it ollama ollama rm llama3:8b-instruct

Quick volume backups can be made with a temporary container:

# Backup Ollama models
docker run --rm -v ollama:/data -v "$PWD":/backup busybox tar czf /backup/ollama-backup.tgz -C /data .

# Backup Open WebUI data
docker run --rm -v open-webui:/data -v "$PWD":/backup busybox tar czf /backup/openwebui-backup.tgz -C /data .

7) Secure Access

By default, the web UI is reachable on your server’s IP and port 3000. For a safer setup, bind Open WebUI to localhost and place it behind a TLS reverse proxy. Edit the port mapping to 127.0.0.1:3000:8080 and use a reverse proxy like Caddy or Nginx with a domain and HTTPS.

Example Caddyfile (auto TLS):

ai.example.com {
  reverse_proxy 127.0.0.1:3000
}

Run Caddy with Docker:

docker run -d --name caddy \
  -p 80:80 -p 443:443 \
  -v $PWD/Caddyfile:/etc/caddy/Caddyfile \
  -v caddy_data:/data \
  caddy:latest

Open WebUI includes auth and user management; create an admin on first login and restrict sign-ups in settings if you do not want public registration.

Troubleshooting

- Port conflicts: If 3000 or 11434 are in use, change the host ports in docker-compose.yml and restart.

- GPU not detected: Ensure docker run --rm --gpus all nvidia/cuda:12.2.0-base-ubuntu22.04 nvidia-smi works. Verify drivers and container toolkit, then add gpus: all to the Ollama service.

- Slow downloads: Models are large. Use a reliable connection and enough disk space; consider a local cache or mirror if multiple hosts will pull the same model.

- Memory errors: Choose smaller models (e.g., 7B/8B) or reduce context length in the UI. On CPU-only systems, expect slower generation; try quantized variants when available.

What You Achieved

You now have a modern local AI stack running Ollama and Open WebUI on Docker, with optional GPU acceleration, persistent storage, and HTTPS-ready reverse proxy integration. This setup is easy to update, secure, and extend—perfect for private experimentation, helpdesk assistants, coding copilots, or offline R&D.

Overview

Running large language models locally is easier than ever thanks to Ollama and Open WebUI. This tutorial shows you how to deploy both on Docker with optional NVIDIA GPU acceleration on Ubuntu 22.04/24.04. You will get a clean, repeatable setup, persistent storage, and a modern web interface to chat with models like Llama 3.1, Gemma, and Phi-3. If you do not have a GPU, you can still run smaller models on CPU.

What You Will Build

We will launch two containers: one for Ollama (the model runtime and API) and one for Open WebUI (the interface). They will be connected on a Docker network, with volumes for persistence. When done, you can open your browser, select a model, and start chatting locally—no cloud required.

Prerequisites

- Ubuntu 22.04 or 24.04, sudo access, and a stable internet connection.

- Docker installed (we will cover a quick install).

- Optional NVIDIA GPU with drivers for acceleration. CPU-only instructions are included.

1) Install Docker (if not installed)

Install the latest Docker CE from the official repo for better compatibility and security updates.

sudo apt update
sudo apt install -y ca-certificates curl gnupg lsb-release
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
  https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list >/dev/null
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker $USER
newgrp docker

2) Enable NVIDIA GPU Support (optional but recommended)

If you have an NVIDIA GPU, install the proprietary driver and NVIDIA Container Toolkit so Docker can access the GPU.

# Install recommended NVIDIA driver
sudo apt update
sudo apt install -y ubuntu-drivers-common
sudo ubuntu-drivers autoinstall
sudo reboot

After reboot, install the NVIDIA Container Toolkit and integrate with Docker.

sudo apt update
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Verify GPUs are visible to Docker:

docker run --rm --gpus all nvidia/cuda:12.3.2-base-ubuntu22.04 nvidia-smi

3) Create a Network and Volumes

We will keep Ollama models and WebUI data persistent across container restarts.

docker network create ai-net
docker volume create ollama-data
docker volume create openwebui-data

4) Run the Ollama Container

Start Ollama with GPU acceleration if available. The container exposes port 11434 for the API.

# GPU-enabled
docker run -d --name ollama --restart unless-stopped \
  --gpus all \
  -p 11434:11434 \
  -v ollama-data:/root/.ollama \
  --network ai-net \
  ollama/ollama:latest

# CPU-only (no GPU flag)
# docker run -d --name ollama --restart unless-stopped \
#   -p 11434:11434 \
#   -v ollama-data:/root/.ollama \
#   --network ai-net \
#   ollama/ollama:latest

5) Pull a Model

Use the Ollama CLI inside the container to download a model. Start with a balanced choice like Llama 3.1 8B or pick a smaller one if you are on CPU.

# Enter the container and pull a model
docker exec -it ollama bash -lc "ollama pull llama3.1:8b"

# Alternative smaller models:
# docker exec -it ollama bash -lc "ollama pull phi3:mini"
# docker exec -it ollama bash -lc "ollama pull mistral:7b"

6) Run Open WebUI

Open WebUI connects to the Ollama API. We will publish it on port 3000 and persist its configuration.

docker run -d --name openwebui --restart unless-stopped \
  -p 3000:8080 \
  -v openwebui-data:/app/backend/data \
  -e OLLAMA_BASE_URL=http://ollama:11434 \
  --network ai-net \
  ghcr.io/open-webui/open-webui:latest

Open your browser to http://localhost:3000, create an admin account, and pick your default model (e.g., llama3.1:8b). You can switch models anytime in the interface.

7) Test the Setup

Send a quick API test to confirm Ollama is serving responses:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.1:8b",
  "prompt": "In one sentence, explain what a container is."
}'

If you get a streamed JSON response with text tokens, the backend is working. In Open WebUI, start a new chat and ask a question to validate the full stack.

8) Performance Tips

- Prefer GPU for 7B–14B models; CPU can be slow or memory-constrained. Smaller models like Phi-3 Mini run decently on modern CPUs.

- Add “q4_0” or “q5_1” quantized variants if available to reduce VRAM/RAM usage. Example: llama3.1:8b-instruct-q4_0.

- Limit GPU layers or context length if you see out-of-memory errors. In Open WebUI, lower max tokens and system prompt size.

9) Troubleshooting

Docker can’t see the GPU: Ensure the NVIDIA driver is installed, run nvidia-smi on the host, and confirm nvidia-ctk runtime configure was applied. Restart Docker and try the CUDA test container again.

Model download is slow or fails: Check DNS and firewall, or use a different mirror via environment variables if your network requires a proxy. You can also prefetch models on a faster connection and copy the volume.

Open WebUI cannot reach Ollama: Confirm both containers are on the ai-net network and OLLAMA_BASE_URL points to http://ollama:11434. Check logs with docker logs openwebui and docker logs ollama.

Out of memory: Choose a smaller model, use a stronger quantization, or on GPU, close other VRAM-heavy apps. For CPU, add swap if RAM is limited.

10) Security and Maintenance

Do not expose ports 11434 or 3000 publicly without authentication and TLS. If you must access remotely, restrict with a reverse proxy, HTTPS, and basic auth or OAuth. Keep images updated:

docker pull ollama/ollama:latest
docker pull ghcr.io/open-webui/open-webui:latest
docker stop openwebui ollama
docker rm openwebui ollama
# re-run the docker run commands from above (volumes preserve your data)

Wrap-Up

You now have a robust, GPU-capable local AI stack: Ollama serving models and Open WebUI providing a polished chat interface. Because everything runs in Docker with volumes, you can upgrade, backup, and migrate with minimal friction. Experiment with different models, tune prompts, and enjoy private, offline AI on your own hardware.