LifeBytes Journal

Why run a local AI assistant?

Cloud AI tools are convenient, but a local setup can be faster for repeated tasks, cheaper over time, and more private for sensitive notes, logs, or internal documentation. Running an AI model locally is also a great way to learn modern AI tooling without committing to a paid API. In this guide, you will install Ollama on Linux, download a model, test it from the terminal, and optionally add a lightweight web interface for a more comfortable chat experience.

Prerequisites

You need a Linux machine (Ubuntu/Debian/Fedora/Arch all work), at least 8 GB RAM for smaller models, and preferably a modern CPU. A GPU is helpful but not required for many models. You will also need curl and basic terminal access with sudo privileges.

Step 1: Install Ollama

Ollama provides a simple installer for Linux. Open a terminal and run:

curl -fsSL https://ollama.com/install.sh | sh

After installation, verify the service is available:

ollama --version

On most systems, Ollama runs as a background service. If you want to check its status on a systemd-based distribution, use:

systemctl status ollama

Step 2: Download a model (and understand what you are pulling)

With Ollama, you download models using the pull command. A good starting point for general chat is a smaller, responsive model. For example:

ollama pull llama3.1

If disk space or RAM is limited, consider smaller variants (often labeled with fewer parameters). If you want code-focused answers, try a coding model such as:

ollama pull codellama

Model size matters. Larger models typically produce better results but require more RAM and may run slower. If performance feels sluggish, choose a smaller model rather than assuming something is broken.

Step 3: Chat with the model from the terminal

To start a chat session:

ollama run llama3.1

You can now type prompts and get replies immediately. This is perfect for quick tasks like generating a bash one-liner, summarizing a local change log, or drafting troubleshooting steps.

For scripting, you can also pass a prompt directly:

ollama run llama3.1 "Write a systemd unit that restarts a service on failure."

Step 4: Enable remote access safely (optional but common)

By default, many local AI setups listen only on localhost for safety. If you want to use Ollama from another machine on your LAN, you need to bind it carefully and protect it with firewall rules. First, check what address Ollama is listening on:

ss -tulpen | grep 11434

If you decide to expose it, do it on a trusted network only, and restrict access to specific IPs. On Ubuntu with UFW, for example, you can allow a single workstation:

sudo ufw allow from 192.168.1.50 to any port 11434

Avoid opening the port to the public internet. A local AI endpoint without authentication is not something you want exposed.

Step 5: Add a simple web UI (Open WebUI)

Terminal chat is efficient, but a web interface makes long conversations easier and adds quality-of-life features. One popular option is Open WebUI, which can connect to Ollama. The easiest deployment is with Docker. If Docker is not installed, install it from your distribution’s official docs first.

Run Open WebUI as a container:

docker run -d --name open-webui -p 3000:8080 -e OLLAMA_BASE_URL=http://host.docker.internal:11434 -v open-webui:/app/backend/data --restart unless-stopped ghcr.io/open-webui/open-webui:main

On some Linux hosts, host.docker.internal may not resolve by default. If that happens, use your host’s LAN IP (for example, http://192.168.1.10:11434) or add the Docker host gateway option depending on your Docker version. Once the container is running, open:

http://localhost:3000

Complete the initial setup in the browser, then select the Ollama model you downloaded. You should be able to chat immediately through the UI while Ollama continues doing the inference locally.

Troubleshooting tips (the issues people actually hit)

Model is slow or the system becomes unresponsive: Use a smaller model, close memory-heavy apps, or move to a machine with more RAM. Local AI is RAM-hungry, and swapping to disk will kill performance.

Ollama service is not running: Restart it with sudo systemctl restart ollama and check logs using journalctl -u ollama --no-pager -n 100.

Web UI cannot connect to Ollama: Confirm Ollama is reachable at http://127.0.0.1:11434 from the host, then adjust the Open WebUI environment variable OLLAMA_BASE_URL to point to the correct address.

Next steps

Once your local assistant is working, you can create repeatable prompts for helpdesk replies, generate configuration templates, or summarize technical notes without sending data to a third party. For better results, experiment with different models and keep your prompts specific. Local AI gets impressive quickly when you give it clear context and constraints.

Running an AI assistant locally is no longer a niche experiment. With modern open models and lightweight serving tools, you can build a private, ChatGPT-style interface on your own Linux machine—no API keys, no data leaving your network, and full control over updates. In this tutorial, you will install Ollama (for downloading and serving LLMs) and Open WebUI (a clean web interface) using Docker. The result is a fast, self-hosted AI chat you can use for drafting, troubleshooting, and internal knowledge work.

What You’ll Build

By the end, you will have: (1) Ollama running as a local model server, (2) Open WebUI running in a container, and (3) a browser-based chat UI available on your LAN or localhost. This setup works well on Ubuntu Server, Debian, and most modern Linux distributions.

Prerequisites

Hardware: At least 8 GB RAM is recommended for smaller models. For better results, use 16 GB or more. A GPU helps but is not required for CPU-only usage.

Software: A Linux system with sudo access, Docker installed, and basic command-line familiarity. If you don’t have Docker yet, install it via your distribution’s official Docker instructions.

Step 1: Install and Start Ollama

Ollama provides a simple way to download and run large language models locally. Install it with the official script:

Command:

curl -fsSL https://ollama.com/install.sh | sh

After installation, start and enable the service (on most systemd-based systems):

sudo systemctl enable --now ollama

Confirm it is active:

systemctl status ollama

Step 2: Download a Model

Now you’ll pull a model. Choose one that matches your hardware. For a balanced option on many systems, try Llama 3 (size availability depends on what Ollama offers at the moment). Pulling a model can take time because it downloads several GB.

ollama pull llama3

Test it directly in the terminal:

ollama run llama3

Type a short prompt like “Explain RAID 1 in simple terms” to confirm it responds. Exit the session when done.

Step 3: Run Open WebUI in Docker

Open WebUI provides a user-friendly interface that feels similar to popular AI chat apps. It can connect to Ollama running on the host. Start by creating a persistent volume for WebUI data:

docker volume create open-webui

Then run the container. The key setting is the environment variable that tells WebUI where to find Ollama:

docker run -d \
--name open-webui \
-p 3000:8080 \
-e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
-v open-webui:/app/backend/data \
--restart unless-stopped \
ghcr.io/open-webui/open-webui:main

On some Linux hosts, host.docker.internal may not resolve by default. If Open WebUI can’t connect to Ollama, rerun the container with an extra host mapping:

docker rm -f open-webui

docker run -d \
--name open-webui \
-p 3000:8080 \
--add-host=host.docker.internal:host-gateway \
-e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
-v open-webui:/app/backend/data \
--restart unless-stopped \
ghcr.io/open-webui/open-webui:main

Step 4: Open the Web Interface

In your browser, open:

http://localhost:3000

If you’re accessing it from another computer on the same network, replace localhost with your Linux server’s IP address. The first time you open Open WebUI, you will create an admin account. After login, look for the model selection and choose the model you pulled (for example, llama3).

Step 5: Basic Security Hardening

A local AI chat can contain sensitive data, so treat it like an internal app. If this is only for you, bind to localhost by using Docker’s loopback mapping:

-p 127.0.0.1:3000:8080

If you need LAN access, consider placing it behind a reverse proxy (Nginx or Caddy) with HTTPS and authentication. Also make sure your firewall only allows trusted networks to reach port 3000.

Troubleshooting Tips

WebUI shows no models: Confirm Ollama is running and reachable. Try curl http://127.0.0.1:11434 on the host, then check the container logs with docker logs open-webui.

Slow responses: Use a smaller model, close other memory-heavy services, or run on a machine with more RAM. CPU-only inference is usable, but performance varies widely by hardware.

Connection errors from container to host: Use the --add-host=host.docker.internal:host-gateway option shown earlier, and keep the OLLAMA_BASE_URL pointing to http://host.docker.internal:11434.

Next Steps

Once your private AI assistant is working, you can expand it with additional models for different tasks, create separate chats for projects, and experiment with system prompts for consistent tone and formatting. The big advantage of this setup is control: you decide what runs, what gets stored, and how it’s exposed—without relying on external services.

Why run a private AI chatbot?

If you like the convenience of ChatGPT-style assistants but need better privacy, lower latency on your local network, or predictable costs, a self-hosted setup is a strong option. With Ollama you can run modern large language models (LLMs) locally, and with Open WebUI you get a clean web interface for chatting, managing models, and organizing prompts. In this tutorial you will deploy both on an Ubuntu server using Docker, so the install is repeatable and easy to maintain.

What you will build

By the end, you will have:

1) Ollama running as a service (the model runtime)
2) Open WebUI running in Docker (the chat UI)
3) Persistent storage for models and chat data
4) Optional GPU support notes if your server has NVIDIA

Prerequisites

Use an Ubuntu 22.04/24.04 server (VM or bare metal). A modern CPU and at least 8 GB RAM is workable for smaller models; 16–32 GB is more comfortable. You also need a user with sudo rights, outbound internet access to pull images/models, and Docker installed. If you plan to expose the UI beyond your LAN, put it behind a reverse proxy with TLS.

Step 1: Install Docker and Docker Compose

First, install Docker from Ubuntu’s repository (simple and reliable for most homelab and SMB setups):

Commands:
sudo apt update
sudo apt install -y docker.io docker-compose-plugin
sudo systemctl enable --now docker

Add your user to the docker group so you can run Docker without sudo (log out/in after this):

Command:
sudo usermod -aG docker $USER

Step 2: Create folders for persistent data

Persistent volumes are important because LLM files can be large and you do not want to re-download models after every container update. Create a working directory:

Commands:
mkdir -p ~/ai-stack/ollama
mkdir -p ~/ai-stack/openwebui
cd ~/ai-stack

Step 3: Create a Docker Compose file

Create a file named docker-compose.yml in ~/ai-stack. This setup runs Ollama and Open WebUI on the same Docker network. Ollama will listen on port 11434 internally; Open WebUI will be published on port 3000.

docker-compose.yml:

Copy and paste:
version: "3.8"

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    volumes:
      - ./ollama:/root/.ollama
    ports:
      - "11434:11434"

  openwebui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: openwebui
    restart: unless-stopped
    depends_on:
      - ollama
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
    volumes:
      - ./openwebui:/app/backend/data
    ports:
      - "3000:8080"

Step 4: Start the services

Bring the stack up in detached mode:

Command:
docker compose up -d

Verify containers are running:

Command:
docker ps

Step 5: Open the Web UI and pull a model

In a browser, open:

http://YOUR_SERVER_IP:3000

Open WebUI will ask you to create an admin account on first run. After login, you can download models through the interface, or you can pull models from the server side using Ollama.

To pull a popular small model (good for testing), run:

Command:
docker exec -it ollama ollama pull llama3.2

Once the model is downloaded, refresh Open WebUI and select the model for chat. If you want a lighter footprint, try smaller parameter models; if you need better answers, larger models require more RAM/VRAM.

Step 6: Basic troubleshooting (the common issues)

Open WebUI loads but shows no models: Confirm the environment variable points to Ollama. Run docker logs openwebui and make sure it can reach http://ollama:11434. Also verify Ollama is healthy with curl http://localhost:11434 on the host.

Model downloads are slow or fail: Check disk space (df -h) and DNS connectivity. LLM downloads can be multiple gigabytes, so a nearly full disk will cause strange errors.

High CPU and slow replies: This is normal on CPU-only servers with larger models. Use a smaller model, reduce concurrent users, or add GPU acceleration.

Optional: NVIDIA GPU acceleration notes

If you have an NVIDIA GPU, install the NVIDIA driver and the NVIDIA Container Toolkit so Docker containers can access the GPU. Then adjust the Ollama service to request GPU resources (exact configuration depends on your Docker and driver versions). GPU support can dramatically improve response time and allow you to run larger models smoothly.

Step 7: Keep it secure and maintainable

For a safer deployment, do not expose port 3000 directly to the internet. Put Open WebUI behind Nginx or Caddy with HTTPS and authentication. For updates, pull new images and recreate containers:

Commands:
cd ~/ai-stack
docker compose pull
docker compose up -d

Because you used persistent volumes, your downloaded models and chat database stay intact across updates.

Wrap-up

Running Ollama with Open WebUI on Ubuntu gives you a practical private AI chatbot you can use for internal documentation, code explanations, drafting emails, and brainstorming without sending prompts to a third-party cloud service. Start with a smaller model to confirm everything works, then scale up based on your hardware and the quality you need.

Running a private AI assistant locally is becoming a practical option for developers and IT teams who want faster responses, lower cloud costs, and better control over sensitive code. In this tutorial, you will set up a self-hosted AI “code helper” on a Linux server using Ollama (for running large language models locally) and Open WebUI (a clean web interface). The result is a browser-based assistant you can use for code reviews, script generation, troubleshooting, and documentation drafts—without sending prompts to external services.

What You’ll Build

You will deploy two components: Ollama, which downloads and serves models via a local API, and Open WebUI, which connects to Ollama and provides a chat UI with conversation history. This guide uses Docker to keep the installation clean and easy to update.

Prerequisites

Before you start, prepare a Linux machine (Ubuntu 22.04/24.04, Debian 12, or similar) with at least 8 GB RAM (16 GB is better for larger models) and 20+ GB free disk. A GPU is optional, but a modern CPU works fine for smaller models. You also need Docker and Docker Compose (or the Docker Compose plugin).

Step 1: Install Docker (Ubuntu/Debian)

If Docker is not installed, run the commands below. On other distributions, use the official Docker documentation for your package manager.

Commands:

sudo apt update
sudo apt install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Optional but recommended: allow your user to run Docker without sudo.

sudo usermod -aG docker $USER

Log out and back in after changing group membership.

Step 2: Create a Project Directory

Create a dedicated folder for your deployment so configuration and volumes stay organized.

mkdir -p ~/private-ai && cd ~/private-ai

Step 3: Create a Docker Compose File

Create a file named docker-compose.yml with the content below. It starts Ollama and Open WebUI, stores model data on disk, and makes the web UI available on port 3000.

cat > docker-compose.yml <<'EOF'
services:
ollama:
image: ollama/ollama:latest
container_name: ollama
restart: unless-stopped
ports:
- "11434:11434"
volumes:
- ollama:/root/.ollama

open-webui:
image: ghcr.io/open-webui/open-webui:main
container_name: open-webui
restart: unless-stopped
ports:
- "3000:8080"
environment:
- OLLAMA_BASE_URL=http://ollama:11434
volumes:
- openwebui:/app/backend/data
depends_on:
- ollama

volumes:
ollama:
openwebui:
EOF

Step 4: Start the Services

Bring the stack up in the background and confirm both containers are healthy.

docker compose up -d
docker ps

Now open your browser and go to http://YOUR_SERVER_IP:3000. The first user you create in Open WebUI typically becomes the admin, depending on the version.

Step 5: Download a Model with Ollama

Ollama pulls models on demand. For a lightweight code-focused start, try a smaller model first. Run the command below to download and test a model from inside the Ollama container.

docker exec -it ollama ollama pull codellama:7b
docker exec -it ollama ollama run codellama:7b

If you prefer a general assistant model, you can also try:

docker exec -it ollama ollama pull llama3.1:8b

Once pulled, go back to Open WebUI, start a new chat, and select the model. Your prompts will be processed locally on your server.

Step 6: Basic Hardening and Access Tips

If this server is not strictly internal, place Open WebUI behind a reverse proxy such as Nginx or Caddy and enable HTTPS. At a minimum, restrict access with a firewall so only your office IP/VPN can reach port 3000. On Ubuntu with UFW, you can allow only your admin workstation and block the rest.

sudo ufw allow from YOUR_IP to any port 3000 proto tcp
sudo ufw enable

Troubleshooting Common Problems

Open WebUI can’t see Ollama models: confirm the environment variable OLLAMA_BASE_URL points to http://ollama:11434 (container-to-container), and verify Ollama is listening: docker logs ollama.

Slow responses: smaller models respond faster on CPU. Also check system load and RAM usage. If the machine is swapping heavily, upgrade RAM or choose a smaller model.

Disk usage grows quickly: model files are large. Keep an eye on volumes and remove unused models with docker exec -it ollama ollama list and docker exec -it ollama ollama rm MODELNAME.

Conclusion

With Ollama and Open WebUI, you can run a capable private AI code assistant on your own Linux server in under an hour. This setup is ideal for testing prompts safely, speeding up daily scripting tasks, and keeping sensitive code and logs under your control. Once it’s running, you can experiment with different models, tighten access via HTTPS and VPN, and even dedicate a GPU host later for faster generation.

Running an AI assistant locally is one of the most practical “advanced” upgrades you can make to a Linux workstation or home lab. You get faster iteration, more privacy, and you avoid sending sensitive text to a third-party cloud service. In this tutorial, you’ll install Ollama (a lightweight local LLM runtime) and Open WebUI (a clean web interface) on Ubuntu, then secure access and confirm everything is working.

This setup is great for a personal helpdesk bot, drafting and summarizing documents, generating scripts, or building an internal knowledge tool. It works on CPU-only systems, but performance improves significantly if you have a modern GPU. The steps below focus on a reliable, repeatable install that you can maintain like any other server service.

Prerequisites

Before you start, make sure you have: Ubuntu 22.04/24.04 (or a compatible Debian-based distro), a user with sudo permissions, at least 8 GB RAM (16 GB recommended for larger models), and roughly 15–30 GB of free disk space depending on which models you download.

Step 1: Update the system

Open a terminal and update your packages to avoid dependency issues:

sudo apt update && sudo apt -y upgrade

Step 2: Install Ollama

Ollama provides a simple command-line experience for downloading and running models. Install it using the official installer:

curl -fsSL https://ollama.com/install.sh | sudo sh

After installation, confirm the service is running:

systemctl status ollama

If it’s not active, start and enable it:

sudo systemctl enable --now ollama

Step 3: Download and test a model

Now pull a model. A good starting point is a smaller “general chat” model to verify your environment first:

ollama pull llama3.1

Then run a quick test:

ollama run llama3.1

Type a prompt like: “Write a bash one-liner to list the 10 largest files in a directory.” If you get a response, the core engine is working.

Step 4: Install Docker (recommended for Open WebUI)

Open WebUI is easiest to deploy in a container. Install Docker using Ubuntu packages:

sudo apt install -y docker.io

Enable the Docker service:

sudo systemctl enable --now docker

Optional but convenient: allow your user to run Docker without sudo (log out and back in after this):

sudo usermod -aG docker $USER

Step 5: Run Open WebUI and connect it to Ollama

Run the container and map it to a local port (3000). We’ll also mount a persistent volume so settings and chat history survive reboots:

docker run -d --name open-webui --restart unless-stopped -p 3000:8080 -e OLLAMA_BASE_URL=http://host.docker.internal:11434 -v open-webui:/app/backend/data ghcr.io/open-webui/open-webui:main

On Linux, host.docker.internal may not work on older Docker builds. If you open the web UI and it cannot reach Ollama, rerun the container using host networking instead:

docker rm -f open-webui

docker run -d --name open-webui --restart unless-stopped --network=host -e OLLAMA_BASE_URL=http://127.0.0.1:11434 -v open-webui:/app/backend/data ghcr.io/open-webui/open-webui:main

Now open your browser and go to http://localhost:3000 (or the server IP with port 3000). Create the first admin account. In most cases, Open WebUI will automatically detect the Ollama endpoint once the environment variable is set.

Step 6: Basic security hardening (don’t skip this)

If this is only for your local machine, binding to localhost is usually enough. If you plan to access it from other devices, you should secure it properly. At a minimum, configure the firewall to allow only trusted networks.

With UFW, you can allow port 3000 only from your LAN (example uses 192.168.1.0/24):

sudo ufw allow from 192.168.1.0/24 to any port 3000 proto tcp

sudo ufw enable

For a more professional setup, place Open WebUI behind Nginx with HTTPS (Let’s Encrypt) and optionally basic auth or SSO. That way, you’re not exposing a plain HTTP admin login to the network.

Step 7: Troubleshooting common issues

Open WebUI can’t see any models: confirm Ollama is running and reachable. Test locally with curl http://127.0.0.1:11434. If the container can’t reach the host, switch to --network=host as shown above.

Model downloads are slow or fail: try again later or switch networks. Large models are multi-GB downloads. Ensure you have enough disk space under /usr/share/ollama (or your configured storage path).

High CPU/RAM usage: use a smaller model, reduce parallel usage, or move the server to a machine with more memory. For older hardware, smaller models typically feel much more responsive.

Final check

At this point you have a private AI assistant running entirely on your own Ubuntu system. Ollama handles the model runtime, and Open WebUI provides an easy interface for chat, prompt testing, and daily use. Once you’re comfortable with the basics, you can explore model choices, system prompts for a “helpdesk” personality, and integrating local documents for internal Q&A workflows.

Why a private RAG chatbot?

If your team needs an internal chatbot that can answer questions from company documents, you’ve probably looked at cloud AI services. The problem is compliance: sending sensitive data outside your network can be a deal-breaker. A practical alternative is a private RAG setup (Retrieval-Augmented Generation), where a local language model generates answers while a local index retrieves relevant text from your own files. In this tutorial, you’ll build a private RAG chatbot on a Linux server using Ollama (local LLM runtime) and Open WebUI (a friendly chat interface), then connect your documents to it.

What you will build

You will deploy two services with Docker: Ollama to run a model locally, and Open WebUI to provide a web-based chat UI and document ingestion features. This approach is ideal for homelabs, IT departments, and helpdesk teams who want AI-assisted answers without exposing internal knowledge to third parties.

Prerequisites

You need a Linux server or VM (Ubuntu/Debian recommended) with at least 8 GB RAM for smaller models; 16 GB+ is better. Disk space depends on the model (expect several GB). You also need Docker and Docker Compose. If you have an NVIDIA GPU, you can accelerate inference, but this guide works on CPU as well.

Step 1: Install Docker and Docker Compose

On Ubuntu, install Docker with the official packages, then enable the service:

Commands:
sudo apt update
sudo apt install -y docker.io docker-compose-plugin
sudo systemctl enable --now docker
sudo usermod -aG docker $USER

Log out and back in so your user can run Docker without sudo. Verify with:

Command:
docker version

Step 2: Create a Docker Compose file

Create a working directory and a compose file. This setup stores model files and WebUI data in persistent volumes so upgrades won’t wipe your configuration.

Commands:
mkdir -p ~/private-rag
cd ~/private-rag
nano docker-compose.yml

Paste the following content:

docker-compose.yml
version: "3.9"
services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    ports:
      - "11434:11434"
    volumes:
      - ollama:/root/.ollama

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    restart: unless-stopped
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
    volumes:
      - openwebui:/app/backend/data
    depends_on:
      - ollama

volumes:
  ollama:
  openwebui:

Step 3: Start the services

Bring the stack online:

Command:
docker compose up -d

Check that both containers are healthy:

Command:
docker ps

Open WebUI in your browser at http://YOUR_SERVER_IP:3000. The first account you create becomes the admin by default, so choose a strong password.

Step 4: Pull a model with Ollama

You can pull models directly inside the Ollama container. A good starting point for many servers is a smaller, fast instruct model.

Command:
docker exec -it ollama ollama pull llama3.1:8b

If your server has less RAM, try a smaller model. If you have more resources, you can experiment with larger variants for better reasoning. After pulling, confirm it’s available:

Command:
docker exec -it ollama ollama list

Step 5: Connect Open WebUI to the local model

In Open WebUI, go to the model selection menu and choose the model you pulled (for example, llama3.1:8b). Start a basic chat to confirm responses are generated locally. If it errors, verify that Open WebUI can reach Ollama on the internal Docker network and that the environment variable OLLAMA_BASE_URL matches the compose file.

Step 6: Enable RAG by adding your documents

To turn a general chatbot into a “knows our docs” assistant, ingest your content. In Open WebUI, find the section for Documents or Knowledge (wording may vary by version). Upload text-heavy sources such as internal runbooks, SOPs, FAQs, or exported wiki pages. For best retrieval results, prefer clean text formats like TXT, MD, PDF (machine-readable), and avoid scans without OCR.

After upload, Open WebUI will index the content so it can retrieve relevant chunks during chat. Test with a question that can only be answered from your document set, such as “What is our VPN reset procedure?” The response should cite or clearly reflect your internal wording. If the answer seems generic, add more targeted documents or refine your question.

Step 7: Secure access (quick hardening)

A private AI system can still leak data if it’s publicly exposed. First, bind access to trusted networks using a firewall (UFW on Ubuntu is simple) and consider putting Open WebUI behind a reverse proxy with HTTPS. Also, keep the service updated:

Commands:
cd ~/private-rag
docker compose pull
docker compose up -d

Finally, treat uploaded documents as sensitive: only allow authenticated users, and review what content is ingested. A RAG chatbot is powerful precisely because it can surface internal text quickly.

Troubleshooting tips

Model is slow: Use a smaller model, add RAM, or use GPU acceleration. Also reduce concurrent users.
WebUI can’t see the model: Confirm Ollama is running and reachable on port 11434 inside Docker, and that the model is listed in ollama list.
RAG answers are inaccurate: Upload more relevant documents, remove outdated versions, and prefer clean text sources. Retrieval quality depends heavily on document quality.

Next steps

Once your private RAG chatbot works, you can expand it by creating separate knowledge collections for different departments, adding a reverse proxy for SSO-like access control, or running multiple models for different tasks (fast model for chat, larger model for complex reasoning). This setup gives you a modern AI assistant while keeping your data inside your own environment.

Running a large language model (LLM) locally is one of the fastest ways to get private, low-latency AI assistance without sending your prompts to a third-party cloud. In this tutorial, you will set up Ollama (a lightweight LLM runtime) and Open WebUI (a clean web interface) on Linux. The result is a self-hosted AI chat you can use for drafting, coding help, log analysis, and knowledge base searching—while keeping data on your own machine.

What You Need

Hardware: A modern CPU system works, but more RAM helps a lot. For small models (like 7B), aim for 8–16 GB RAM. For smoother performance or larger models, 32 GB+ is recommended. If you have an NVIDIA GPU, you can accelerate generation, but this guide focuses on a reliable CPU-first setup.

Software: A recent Linux distribution (Ubuntu/Debian/Fedora), terminal access, and either Docker (recommended for Open WebUI) or Python knowledge if you prefer manual setups.

Step 1: Install Ollama

Ollama makes local model management simple: you download a model once and then run it with a single command. To install Ollama, open a terminal and run:

curl -fsSL https://ollama.com/install.sh | sh

After installation, verify it works:

ollama --version

On most systems, Ollama starts as a service automatically. If you need to start it manually, you can run:

ollama serve

Step 2: Pull a Model (Example: Llama 3.1)

Now download a model. A good starting point is a modern 7B or 8B model. Pull it with:

ollama pull llama3.1

Once it finishes, test a quick prompt directly in the terminal:

ollama run llama3.1

Type a message (for example, “Summarize the difference between TCP and UDP”) and press Enter. If you get a response, the local model runtime is working.

Step 3: Install Docker (for Open WebUI)

Open WebUI is easiest to run in a container. If Docker is not installed, on Ubuntu/Debian you can do:

sudo apt update
sudo apt install -y docker.io
sudo systemctl enable --now docker

Optional but recommended: allow your user to run Docker without sudo:

sudo usermod -aG docker $USER

Log out and back in for the group change to apply.

Step 4: Run Open WebUI and Connect It to Ollama

Start Open WebUI with Docker. This command creates persistent storage and publishes the web interface on port 3000:

docker run -d --name open-webui -p 3000:8080 -v open-webui:/app/backend/data --restart unless-stopped ghcr.io/open-webui/open-webui:main

Next, ensure Open WebUI can reach Ollama. If Open WebUI does not automatically detect it, the most common fix is to point it to the Ollama API endpoint. Ollama listens on http://localhost:11434 by default. Depending on your Docker networking setup, “localhost” inside the container is not the host machine.

A practical approach is to run Open WebUI using host networking (Linux only). Stop the existing container and re-run:

docker rm -f open-webui
docker run -d --name open-webui --network=host -v open-webui:/app/backend/data --restart unless-stopped ghcr.io/open-webui/open-webui:main

Now open your browser and go to:

http://localhost:3000

Create an admin account when prompted. In the Open WebUI settings, you should see Ollama as an available provider. Select the model you pulled (for example, llama3.1) and start chatting.

Step 5: Improve Performance and Reliability

Choose the right model size: If responses feel slow, try a smaller model. Ollama supports many options; you can keep multiple models and switch depending on the task. Smaller models are great for quick drafts, command explanations, and lightweight Q&A.

Keep your data private: Local LLMs are only “private” if you avoid sending data out through plugins or external integrations. Treat the WebUI like any internal tool: secure access, avoid exposing it to the public internet, and consider a reverse proxy with authentication if you need remote access.

Troubleshoot connectivity: If Open WebUI can’t see Ollama, confirm the Ollama service is running and listening on port 11434:

ss -tulpn | grep 11434

If you prefer not to use host networking, you can configure Ollama to bind to an address reachable from Docker and then point Open WebUI to that address. The exact method depends on your distro and firewall rules, so host networking is the fastest baseline to validate your setup.

Next Steps (Useful Ideas)

Once your local AI chat is stable, you can level it up: create model presets for different writing styles, connect it to internal documentation, or use it for structured tasks like generating incident summaries from sanitized logs. The biggest advantage of this setup is control—you decide what runs, where it runs, and what data it can access.

With Ollama and Open WebUI, a private LLM workstation is no longer a weekend project. It’s a practical tool you can deploy in minutes and refine over time.

Why host a local AI assistant?

If you write scripts, manage servers, or handle helpdesk tickets, an AI assistant can speed up routine work like summarizing logs, drafting commands, or explaining configuration files. The problem is that many cloud tools send your prompts and snippets to third-party services. A local setup keeps sensitive data on your own machine, works offline, and can be tuned for your workflow.

In this tutorial, you will install Ollama (a lightweight local LLM runtime) and Open WebUI (a web interface similar to popular chat tools) on Linux. The result is a private AI assistant you can access from your browser on your LAN.

What you need

Hardware: A modern 64-bit Linux system. For acceptable performance, aim for 16 GB RAM or more. A GPU helps but is not required for basic use. Lighter models can run on CPU-only machines, including small servers.

Software: A recent Linux distribution (Ubuntu/Debian/Fedora are all fine), Docker for Open WebUI, and basic terminal access with sudo.

Step 1: Install Ollama

Ollama runs the model locally and exposes an API that other tools (like Open WebUI) can call. Install it using the official script:

Command:

curl -fsSL https://ollama.com/install.sh | sh

After installation, check that the service is working:

ollama --version

On many distros, Ollama runs as a service. If you need to confirm it is active:

systemctl status ollama

Step 2: Pull a model and test it

Next, download a model. If you are CPU-only or want fast responses, start with a smaller model. For general coding help, you can also try code-focused models once the basics work.

Example (general model):

ollama pull llama3.1

Run a quick prompt to confirm everything works:

ollama run llama3.1

Type a question like “Explain what journald does on Linux” and confirm you get a response. Exit with /bye or Ctrl+C depending on your shell behavior.

Step 3: Install Docker (if not installed)

Open WebUI is easiest to deploy with Docker. On Ubuntu/Debian, you can install Docker like this:

sudo apt update

sudo apt install -y docker.io

sudo systemctl enable --now docker

Optional but recommended: allow your user to run Docker without sudo (log out and back in after this):

sudo usermod -aG docker $USER

Step 4: Run Open WebUI and connect it to Ollama

Open WebUI will provide a clean browser interface and conversation history. The key is pointing it at Ollama’s API endpoint.

First, make sure Ollama is listening locally. By default it is typically available at http://127.0.0.1:11434. Now start Open WebUI in Docker:

docker run -d --name open-webui --restart unless-stopped -p 3000:8080 -e OLLAMA_BASE_URL=http://host.docker.internal:11434 -v open-webui:/app/backend/data ghcr.io/open-webui/open-webui:main

On Linux, host.docker.internal may not be available depending on your Docker version. If the UI cannot connect, rerun the container using host networking instead:

docker rm -f open-webui

docker run -d --name open-webui --restart unless-stopped --network=host -e OLLAMA_BASE_URL=http://127.0.0.1:11434 -v open-webui:/app/backend/data ghcr.io/open-webui/open-webui:main

Now open your browser and visit:

http://localhost:3000

Create the first admin user when prompted. Once logged in, you should see available Ollama models. If you do not, go to settings and verify the Ollama base URL.

Step 5: Enable LAN access (optional and safer if restricted)

If you want to access the assistant from another device on your network, bind the service to a reachable interface and restrict it with firewall rules. For Open WebUI using Docker with port publishing, ensure your firewall only allows trusted subnets to connect to port 3000.

For example, on Ubuntu with UFW you can allow only your local subnet (adjust the CIDR):

sudo ufw allow from 192.168.1.0/24 to any port 3000 proto tcp

Avoid exposing the service directly to the internet. If you need remote access, put it behind a VPN (WireGuard is a good choice) or a reverse proxy with authentication.

Troubleshooting tips

Open WebUI shows “cannot reach Ollama”: Confirm Ollama is running with systemctl status ollama. Then check connectivity from the container. If you are using port mapping, the simplest fix on Linux is often --network=host.

Model downloads are slow or fail: Verify DNS and outbound access. Large models can be tens of gigabytes. If disk space is tight, remove unused models with ollama list and ollama rm <model>.

Responses are too slow: Try a smaller model, reduce context size in settings, and close other memory-heavy applications. CPU-only systems benefit from lightweight models and shorter prompts.

Next steps: make it useful for real admin work

Once the UI is running, build a few saved prompts for your daily tasks: “Summarize this syslog excerpt,” “Write a Bash one-liner to find large files,” or “Draft a polite helpdesk reply.” Because the assistant is local, you can safely paste internal error messages, configuration snippets, or playbook fragments without sending them to a third party.

With Ollama and Open WebUI, you get a practical self-hosted AI assistant that fits nicely into a Linux admin toolbox: fast to deploy, easy to maintain, and private by design.

Overview

If you want an AI assistant for internal documentation, troubleshooting, or drafting replies without sending company data to a third-party cloud, a self-hosted setup is a strong option. In this tutorial, you will deploy a private AI stack on an Ubuntu Server using Docker: Ollama (to run large language models locally) and Open WebUI (a clean web interface for chatting, prompts, and basic management). This approach is practical for homelabs and small teams, and it keeps your prompts and conversation history inside your own network.

What You Will Build

By the end, you will have two containers running: one for Ollama (the model runtime/API) and one for Open WebUI (the front-end). You will also configure persistent storage, pull a model, and confirm everything works from a browser. The steps below are written for Ubuntu Server 22.04/24.04, but will work on most modern Ubuntu releases.

Prerequisites

You need an Ubuntu Server with at least 8 GB RAM (16 GB recommended), 20+ GB free disk, and a modern CPU. A GPU helps performance but is not required for a functional deployment. You also need root or sudo access and a working network connection. If this is a server on a LAN, decide which port you will expose for the web interface (we will use 3000).

Step 1: Install Docker and Docker Compose

First, install Docker using the official repository packages. This ensures you get up-to-date components and fewer compatibility issues with Compose.

Run:

sudo apt update
sudo apt install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin

Optionally allow your user to run Docker without sudo (log out and back in afterward):

sudo usermod -aG docker $USER

Step 2: Create a Project Folder and Compose File

Create a directory to keep your deployment clean and manageable. Then create a docker-compose.yml file that defines both services and persistent volumes.

mkdir -p ~/ai-stack
cd ~/ai-stack
nano docker-compose.yml

Paste the following Compose configuration:

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    ports:
      - "11434:11434"
    volumes:
      - ollama:/root/.ollama

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    restart: unless-stopped
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
    volumes:
      - openwebui:/app/backend/data
    depends_on:
      - ollama

volumes:
  ollama:
  openwebui:

Step 3: Start the Stack

Bring up the containers in the background and confirm they are running.

docker compose up -d
docker compose ps

If you see both services with a “running” state, the base deployment is complete.

Step 4: Pull a Model with Ollama

Now download a model. The best choice depends on your RAM and use case. For many servers, a smaller model is a safe starting point. The command below pulls a popular lightweight model.

docker exec -it ollama ollama pull llama3.2

You can list installed models anytime:

docker exec -it ollama ollama list

Step 5: Log In to Open WebUI and Connect to Ollama

Open a browser and go to http://SERVER-IP:3000. On first launch, Open WebUI asks you to create an admin account. After login, the interface should automatically detect Ollama through the internal Docker network using the OLLAMA_BASE_URL you configured.

Start a new chat, select the model you pulled (for example llama3.2), and send a test prompt such as “Write a short troubleshooting checklist for DNS issues.” If the response appears, your private AI assistant is working end-to-end.

Step 6: Basic Hardening and Practical Tips

Firewall: If this is a public-facing server, do not expose it directly without protection. At minimum, allow only your LAN or VPN subnet to reach port 3000. With UFW, you can restrict access instead of opening the port to everyone.

Reverse proxy: For production use, place Open WebUI behind Nginx or Caddy with HTTPS and authentication. This also makes it easier to use a friendly hostname.

Backups: Your important data lives in Docker volumes. Back up the Open WebUI volume (chat history, settings) and the Ollama volume (models) according to your retention needs.

Updates: Refresh images regularly to get security fixes and new features:

docker compose pull
docker compose up -d

Troubleshooting

Open WebUI loads but no models appear: Verify Ollama is reachable from the Open WebUI container. Check logs with docker logs open-webui and confirm OLLAMA_BASE_URL=http://ollama:11434 is correct.

Model downloads are slow: Large model pulls can take time. Ensure your server has stable internet and enough free disk. You can also choose smaller models to start.

High RAM usage or slow responses: Use a smaller model, reduce concurrent users, or run the service on hardware with more memory. Local AI is resource-intensive by design, and tuning is part of a realistic deployment.

Conclusion

Running Ollama and Open WebUI on Ubuntu Server gives you a private, self-hosted AI assistant that you can control, secure, and integrate into your workflow. Once the base stack is stable, you can expand it with HTTPS, SSO, logging, and routine backups. The key advantage is simple: your prompts and internal context stay on your infrastructure while still giving your team an easy web-based AI experience.

This step-by-step guide shows you how to run a fast, private, and local AI chatbot on Ubuntu 22.04 or 24.04 using Ollama and Open WebUI. You will install the Ollama runtime, pull a modern large language model, and add a clean chat interface via Open WebUI in Docker. Optional steps cover NVIDIA GPU acceleration, API usage, and persistence. The result is a secure, offline-friendly setup suitable for helpdesk, coding assistance, or knowledge base querying without sending data to the cloud.

Why Ollama + Open WebUI

Ollama makes it simple to run and manage open-source LLMs locally (Llama 3.x, Mistral, Phi, Qwen, and more). Open WebUI adds a user-friendly, browser-based chat interface with conversation history, prompt templates, and multi-model support. Together they form a robust, low-maintenance local AI stack for Linux desktops and servers.

Prerequisites

- Ubuntu 22.04 or 24.04 with a non-root sudo user.
- At least 8 GB RAM (16 GB recommended for larger models).
- Optional NVIDIA GPU for acceleration (T4/RTX/RTX A-series, etc.).
- Internet access to download models and containers.

Step 1 — Install Ollama

1) Update packages:
sudo apt update && sudo apt install -y curl ca-certificates
2) Install Ollama:
curl -fsSL https://ollama.com/install.sh | sh
3) Enable as a service:
sudo systemctl enable --now ollama
4) Verify the API is up:
curl http://localhost:11434/api/tags
If you see JSON, Ollama is running correctly.

Step 2 — Pull and test a model

Pull a compact, capable model first to validate your setup:
ollama pull llama3.2
Run an interactive test:
ollama run llama3.2
Type a prompt, then press Ctrl+C to exit. You can later try larger models (for example, ollama pull mistral or ollama pull llama3.1), but start small to confirm everything works.

Step 3 — Install Docker Engine

1) Add Docker’s repo key and source:
sudo apt-get update && sudo apt-get install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(. /etc/os-release; echo $UBUNTU_CODENAME) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
2) Install Docker and the Compose plugin:
sudo apt-get update && sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
3) Add your user to the docker group and refresh your shell:
sudo usermod -aG docker $USER
newgrp docker

Step 4 — Run Open WebUI connected to Ollama

Start Open WebUI and point it to the Ollama API on the host. The --add-host flag maps host.docker.internal to your host’s gateway so the container can reach http://localhost:11434 on the host:

docker run -d --name open-webui --restart=unless-stopped -p 3000:8080 \
--add-host=host.docker.internal:host-gateway \
-e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
-v open-webui:/app/backend/data \
ghcr.io/open-webui/open-webui:main

Open your browser to http://SERVER_IP:3000 (or http://localhost:3000). Create the first admin account, choose a model (for example, llama3.2), and start chatting.

Step 5 — Enable NVIDIA GPU acceleration (optional)

1) Install the latest NVIDIA driver for your GPU using Ubuntu’s Additional Drivers or apt. Reboot if prompted.
2) Install the NVIDIA Container Toolkit so Docker can access the GPU:
distribution=$(. /etc/os-release; echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#' | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update && sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
3) Recreate Open WebUI with GPU access:
docker rm -f open-webui
docker run -d --name open-webui --restart=unless-stopped --gpus all -p 3000:8080 \
--add-host=host.docker.internal:host-gateway \
-e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
-v open-webui:/app/backend/data \
ghcr.io/open-webui/open-webui:main
4) Ollama will also use the GPU automatically when a compatible model is loaded. You can confirm GPU use with nvidia-smi during inference.

Step 6 — Use the Ollama HTTP API

You can script local inference via HTTP without the UI. Example generation request:
curl http://localhost:11434/api/generate -d '{"model":"llama3.2","prompt":"Write a haiku about backups."}'
Chat format with memory:
curl http://localhost:11434/api/chat -d '{"model":"llama3.2","messages":[{"role":"user","content":"Explain DNS in one sentence."}]}'

Step 7 — Persistence, autostart, and updates

- Ollama models are stored under ~/.ollama/models. Back up this directory to avoid re-downloading models.
- The Open WebUI container uses a named volume (open-webui) for its data, which persists across restarts.
- Ollama is already set to start at boot (systemctl enable ollama). The WebUI container uses --restart=unless-stopped so it will auto-start after a reboot.
- Update Ollama: curl -fsSL https://ollama.com/install.sh | sh
- Update Open WebUI: docker pull ghcr.io/open-webui/open-webui:main && docker restart open-webui

Troubleshooting

- Open WebUI cannot connect to Ollama: ensure you used --add-host=host.docker.internal:host-gateway and that curl http://localhost:11434/api/tags works on the host.
- Port already in use: change -p 3000:8080 to a different host port like -p 3333:8080.
- Out of memory or slow responses: try a smaller model (for example, llama3.2 or phi3). Close other apps or add swap. For CPU-only hosts, expect slower performance on large models.
- GPU not used: verify drivers, nvidia-smi, and that the container runs with --gpus all. Pull a GPU-optimized model variant if available.

What you can do next

- Connect knowledge bases or documents using Open WebUI’s RAG features to power local search over PDFs and wikis.
- Add multiple models and switch per chat, benchmarking speed and quality.
- Put Nginx or Caddy in front of :3000 for HTTPS and trusted network access.
- Automate prompts with shell scripts or Python by calling the local Ollama API.

You now have a private, local AI assistant on Ubuntu with a clean web interface, GPU-ready acceleration, and a stable upgrade path—all without sending your data to third-party services.

Overview

Running large language models locally is easier than ever. In this guide, you will deploy a private, GPU-accelerated AI stack with Ollama and Open WebUI on Ubuntu 24.04 using Docker. Ollama handles model downloads and inference, while Open WebUI provides a friendly chat interface, prompt library, RAG features, and multi-user management. By the end, you will have a fast, on-prem AI assistant accessible via a browser, without sending data to third parties.

Prerequisites

System: Ubuntu 24.04 (Noble), an NVIDIA GPU (e.g., RTX 3060+), NVIDIA driver 535+ (or latest), at least 16 GB RAM, and reliable internet. You should have sudo access. This tutorial uses Docker; no prior Kubernetes skills required.

Step 1 — Install NVIDIA Driver and Reboot

If you have not installed the proprietary driver, do it now and reboot:

sudo ubuntu-drivers autoinstall
sudo reboot

Step 2 — Install Docker Engine

Add Docker’s official repository and install the engine:

sudo apt update
sudo apt install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu noble stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Optional: allow your user to run Docker without sudo and start a new shell:

sudo usermod -aG docker $USER
newgrp docker

Step 3 — Enable GPU inside Containers

Install NVIDIA Container Toolkit so Docker can use your GPU:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Verify GPU access from Docker:

docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

Step 4 — Create Network and Volumes

Create an isolated Docker network and persistent volumes for data:

docker network create ai
docker volume create ollama
docker volume create openwebui

Step 5 — Run Ollama (GPU-accelerated)

Start the Ollama server and keep it local-only on port 11434. The volume stores models and caches:

docker run -d --name ollama --gpus all --restart unless-stopped --network ai -p 127.0.0.1:11434:11434 -v ollama:/root/.ollama ollama/ollama:latest

Pull a starter model (choose one that fits your GPU memory):

docker exec -it ollama ollama pull llama3.1:8b
docker exec -it ollama ollama pull qwen2.5:7b-instruct

Step 6 — Run Open WebUI

Open WebUI will use Ollama via the internal Docker network. Expose the web interface on port 3000:

docker run -d --name open-webui --restart unless-stopped --network ai -e OLLAMA_BASE_URL=http://ollama:11434 -p 3000:8080 -v openwebui:/app/backend/data ghcr.io/open-webui/open-webui:latest

Open a browser and visit http://SERVER_IP:3000/. Create the first admin user. In the model selector, choose the model you pulled (e.g., llama3.1:8b) and start chatting.

Step 7 — Test the API

You can also use the local API directly. From the host:

curl http://127.0.0.1:11434/api/generate -d '{"model":"llama3.1:8b","prompt":"Say hello in one sentence."}'

Developers can point tools like LangChain or LlamaIndex at http://127.0.0.1:11434 for private inference.

Optimization Tips

If you see out-of-memory errors, switch to smaller or more aggressively quantized models (e.g., q4_k_m) when pulling: ollama pull llama3.1:8b-instruct-q4_K_M. Keep prompts concise and lower context length in Open WebUI settings. On multi-GPU systems, Ollama auto-detects devices; you can fine-tune behavior with environment variables like OLLAMA_NUM_GPU and OLLAMA_NUM_GPU_LAYERS if needed. Always use persistent volumes to avoid re-downloading models after updates.

Security Hardening

By binding Ollama to 127.0.0.1, the API is not exposed externally. Expose Open WebUI only to trusted networks. If you must publish it on the internet, use a reverse proxy with TLS and authentication. For example, with UFW, allow only your LAN:

sudo ufw allow from 192.168.0.0/16 to any port 3000 proto tcp

For Nginx or Caddy, enable HTTPS and basic auth or OIDC. Never expose port 11434 directly without protection.

Troubleshooting

GPU not detected in containers: Re-run sudo nvidia-ctk runtime configure --runtime=docker, then sudo systemctl restart docker. Confirm host drivers with nvidia-smi and container access with the CUDA test image.

Permission errors with Docker: Add your user to the docker group (sudo usermod -aG docker $USER) and re-login.

Slow responses or model crashes: Try a smaller/quantized model, reduce context window, and verify VRAM usage. Ensure swap is enabled for stability when RAM is tight.

Update containers: docker pull ollama/ollama:latest and docker pull ghcr.io/open-webui/open-webui:latest, then docker restart your containers.

Cleanup (Optional)

To stop and remove everything:

docker rm -f open-webui ollama
docker volume rm openwebui ollama
docker network rm ai

What You Achieved

You now have a modern, private AI stack with GPU acceleration on Ubuntu 24.04. Ollama simplifies model management and inference, while Open WebUI offers a polished interface ready for daily use, prototyping, and team collaboration. With careful model choice, proper security, and regular updates, this setup can replace many cloud-based assistants—keeping your data on your hardware.

Local large language models are now practical on a single server. In this step-by-step guide, you will deploy a private AI chatbot by running Ollama (for models) and Open WebUI (for the user interface) on Ubuntu using Docker. We will enable GPU acceleration with NVIDIA so responses are fast and efficient. By the end, you will have a persistent setup that survives reboots and is easy to update.

Overview

Ollama is a lightweight runtime that downloads and serves popular open-source models like Llama 3. Open WebUI is a web app that connects to Ollama and provides a clean chat interface, prompt templates, conversation history, and model management. We will run both components in Docker containers on the same Docker network and map persistent volumes for data. Optional GPU acceleration uses the NVIDIA Container Toolkit.

Prerequisites

- Ubuntu Server or Desktop (22.04 or 24.04 recommended)
- An NVIDIA GPU with proprietary drivers installed (verify with nvidia-smi) if you want GPU acceleration; CPU-only also works
- Sudo access and outbound internet connectivity
- Basic command line familiarity

1) Install Docker Engine and Compose

Install the official Docker packages and add your user to the docker group for passwordless usage.

sudo apt-get update
sudo apt-get install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker $USER
newgrp docker

2) Enable NVIDIA GPUs in Docker (optional but recommended)

If you have an NVIDIA GPU and drivers are installed, add the NVIDIA Container Toolkit so Docker can access the GPU. Verify drivers first with nvidia-smi. Then install the toolkit and restart Docker.

distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Test compute visibility by running a CUDA-enabled container (optional):

docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

3) Create a Docker network and persistent volumes

We will create an isolated network for both containers and define persistent volumes so model files and WebUI data survive restarts.

docker network create ollama-net
docker volume create ollama
docker volume create open-webui

4) Run Ollama (model server)

Start the Ollama container. If you have a GPU, include --gpus all. The port 11434 is the Ollama API.

# GPU-enabled
docker run -d --name ollama --gpus all --restart unless-stopped \
  -p 11434:11434 \
  -v ollama:/root/.ollama \
  --network ollama-net \
  ollama/ollama:latest

# CPU-only (if you do not have an NVIDIA GPU)
# docker run -d --name ollama --restart unless-stopped \
#   -p 11434:11434 \
#   -v ollama:/root/.ollama \
#   --network ollama-net \
#   ollama/ollama:latest

Pull a model and do a quick test inside the container. Llama 3 and Qwen are great starting options.

docker exec -it ollama ollama pull llama3.1:8b
docker exec -it ollama ollama run llama3.1:8b "Write a two-line poem about local AI."

5) Run Open WebUI (front-end)

Open WebUI connects to the Ollama API. On first launch it creates an admin account when you sign in. We will point it at Ollama via the internal Docker network name.

docker run -d --name open-webui --restart unless-stopped \
  -p 3000:8080 \
  -e OLLAMA_BASE_URL=http://ollama:11434 \
  -v open-webui:/app/backend/data \
  --network ollama-net \
  ghcr.io/open-webui/open-webui:latest

Open a browser to http://<server-ip>:3000. Create your account, select the model you pulled (for example, llama3.1:8b), and start chatting. You can pull additional models anytime using docker exec -it ollama ollama pull qwen2.5:7b and select them in Open WebUI.

6) Optional: Use Docker Compose instead of docker run

If you prefer a single file, create docker-compose.yml in an empty folder. The gpus: all key enables GPU acceleration when the NVIDIA toolkit is installed.

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    ports:
      - "11434:11434"
    volumes:
      - ollama:/root/.ollama
    networks:
      - ollama-net
    deploy:
      resources:
        reservations:
          devices:
            - capabilities: ["gpu"]
    # Alternatively for Compose v2+:
    # gpus: all

  open-webui:
    image: ghcr.io/open-webui/open-webui:latest
    container_name: open-webui
    restart: unless-stopped
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
    volumes:
      - open-webui:/app/backend/data
    networks:
      - ollama-net

volumes:
  ollama:
  open-webui:

networks:
  ollama-net:
    external: true

Then run:

docker network create ollama-net
docker compose up -d

7) Updating and maintenance

To update images, pull the latest versions and recreate the containers. Your data and models remain in volumes.

docker pull ollama/ollama:latest
docker pull ghcr.io/open-webui/open-webui:latest
docker stop open-webui ollama && docker rm open-webui ollama
# re-run the docker run commands (or docker compose up -d)

To see logs for troubleshooting, run docker logs -f ollama and docker logs -f open-webui. If the WebUI cannot see models, ensure the environment variable OLLAMA_BASE_URL points to http://ollama:11434 and both containers share the same Docker network.

Troubleshooting tips

GPU not detected: Confirm the NVIDIA driver works on the host (nvidia-smi), the NVIDIA Container Toolkit is installed, and the container uses --gpus all. If using Compose, ensure gpus: all or the device reservation is defined.

Ports already in use: Change host ports in the run commands (for example, map Open WebUI to -p 8080:8080 instead of 3000).

Slow downloads or storage limits: Models are large. Consider attaching a larger Docker volume or moving /var/lib/docker to a disk with more space. You can also choose smaller models (7B) or quantized variants.

HTTPS and access control: Put Open WebUI behind a reverse proxy such as Nginx or Caddy with HTTPS and firewall rules. For internet exposure, add authentication, rate limits, and consider a VPN or zero-trust tunnel.

What you built

You now have a local, private AI chatbot with GPU acceleration on Ubuntu using Docker. Ollama handles model serving, while Open WebUI gives you a friendly interface with history, prompts, and multi-model management. This setup is repeatable, easy to update, and keeps your data on your own hardware.

Overview

If you want a fast, private, and low-cost way to chat with large language models on your own machine, pairing Ollama with Open WebUI inside Docker is a great setup. Ollama handles model downloads and inference (CPU or NVIDIA GPU), while Open WebUI provides a clean, modern chat interface in your browser. This guide shows how to deploy both with Docker Compose on Ubuntu 22.04/24.04 and enable GPU acceleration for significant speedups.

By the end, you will have a persistent, self-hosted AI chat running at http://localhost:3000, with models managed by Ollama at http://localhost:11434. The instructions also include CPU-only notes, backup tips, and troubleshooting for common pitfalls.

Prerequisites

- Ubuntu 22.04 or 24.04 with sudo access. Windows and macOS work with Docker too, but this tutorial focuses on Ubuntu.
- For GPU acceleration: an NVIDIA GPU with a recent driver (typically 525+). CPU-only also works, just slower.
- Docker Engine and Docker Compose plugin (we will install them below).
- At least 16 GB RAM for 7B–8B models; more is better for larger models.
- Open ports: 11434 (Ollama API) and 3000 (Open WebUI).

Step 1 — Install Docker Engine and Compose

Commands:
sudo apt-get update
sudo apt-get install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker $USER && newgrp docker

Verify Docker works: docker run --rm hello-world. Verify Compose works: docker compose version.

Step 2 — Install NVIDIA Driver (GPU users)

If you already have a recent NVIDIA driver, you can skip this step. Otherwise, install the recommended driver and reboot:

sudo ubuntu-drivers autoinstall
sudo reboot

After reboot, confirm the GPU is visible: nvidia-smi. You should see your GPU model and driver version.

Step 3 — Enable GPU inside Docker

Install the NVIDIA Container Toolkit so Docker containers can access your GPU:

distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit

Configure Docker to use the NVIDIA runtime by default:

sudo mkdir -p /etc/docker
cat <<'EOF' | sudo tee /etc/docker/daemon.json
{
"default-runtime": "nvidia",
"runtimes": {
"nvidia": {
"path": "nvidia-container-runtime",
"runtimeArgs": []
}
}
}
EOF
sudo systemctl restart docker

Test GPU access in containers: docker run --rm --gpus all nvidia/cuda:12.3.2-base-ubuntu22.04 nvidia-smi. If you see the usual output, you are set.

CPU-only? Skip Step 3 and the GPU test. The rest works the same, just remove the NVIDIA-specific line from the compose file noted below.

Step 4 — Create a Docker Compose file

Create a new folder and the compose file:

mkdir -p ~/ai-stack && cd ~/ai-stack
nano docker-compose.yml

Paste the following content, then save:

version: "3.9"
services:
ollama:
image: ollama/ollama:latest
container_name: ollama
restart: unless-stopped
ports:
- "11434:11434"
environment:
- OLLAMA_KEEP_ALIVE=30m
volumes:
- ollama:/root/.ollama
runtime: nvidia

open-webui:
image: ghcr.io/open-webui/open-webui:main
container_name: open-webui
restart: unless-stopped
depends_on:
- ollama
environment:
- OLLAMA_API_BASE_URL=http://ollama:11434
ports:
- "3000:8080"
volumes:
- openwebui:/app/backend/data

volumes:
ollama:
openwebui:

Note: If you are running CPU-only, delete the line runtime: nvidia and keep everything else.

Step 5 — Launch the stack and pull a model

docker compose up -d

Wait a few seconds and confirm both containers are healthy: docker ps. Next, pull a model into Ollama. Good starters are llama3.1:8b, mistral:7b, or a small qwen2:7b.

docker exec -it ollama ollama pull llama3.1:8b

List installed models with: docker exec -it ollama ollama list.

Step 6 — Chat in Open WebUI

Open http://localhost:3000 in your browser. Open WebUI should auto-detect Ollama via the environment variable, but you can also set the API in Settings > Connections to http://ollama:11434 (inside Docker) or http://localhost:11434 (host access). Create a new chat, choose your model (for example, llama3.1:8b), and start chatting locally.

Backups, Updates, and Performance Tips

Persistence: Your models and chats are stored in the named volumes ollama and openwebui. Back them up with docker run --rm -v ollama:/data -v $(pwd):/backup alpine tar czf /backup/ollama.tgz -C / data (and similarly for openwebui).

Updates: Pull fresh images and recreate: docker compose pull && docker compose up -d. Ollama keeps your models; no need to re-download.

Performance: Prefer GPU for best speed. If RAM/VRAM is tight, choose smaller or quantized models (e.g., llama3.1:8b-q4_K_M). Set OLLAMA_KEEP_ALIVE to keep models warm between requests.

Remote access: If exposing over the internet, place Open WebUI behind a reverse proxy (Nginx, Caddy, or Traefik) and enable authentication and TLS. Never expose Ollama directly without controls.

Troubleshooting

GPU not detected: Check nvidia-smi works on the host. Then run docker run --rm --gpus all nvidia/cuda:12.3.2-base-ubuntu22.04 nvidia-smi. If that fails, revisit Step 3 and confirm /etc/docker/daemon.json is correct and Docker was restarted.

Open WebUI cannot reach Ollama: Ensure both containers are up. Verify docker logs open-webui and confirm OLLAMA_API_BASE_URL is http://ollama:11434. From the host, curl http://localhost:11434/api/tags should list installed models.

Downloads are slow: Models can be several GB. Use a wired connection or pre-fetch models off-peak. You can also copy existing models into the ollama volume if you have them from another machine.

Port conflicts: If ports 11434 or 3000 are in use, change them in the compose file (left side of the colon) and recreate the stack.

What You Achieved

You now have a self-hosted AI chat stack running locally with Docker. Ollama manages lightweight, high-quality models, and Open WebUI provides a comfortable chat experience. With GPU acceleration, responses are significantly faster, and your data never leaves your machine. Extend this setup with a reverse proxy, add more models, or integrate the Ollama API into your own apps for a powerful private AI workstation.

Overview

This step-by-step guide shows how to deploy Ollama (for running local LLMs) together with Open WebUI (a modern, browser-based interface) using Docker on Ubuntu. We will enable NVIDIA GPU acceleration for faster inference, set up persistent storage, and cover useful operational tips. By the end, you will have a local, privacy-friendly AI stack that you can run offline and manage easily.

Prerequisites

- Ubuntu 22.04 or 24.04 with sudo access
- An NVIDIA GPU with recent drivers (535+ recommended)
- At least 16 GB RAM for larger models; ensure enough disk space for model files
- Internet access to pull Docker images and models

1) Install Docker and Docker Compose

Install Docker from the official repository so you get the latest engine and the Compose plugin:

sudo apt update && sudo apt install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker $USER && newgrp docker

2) Install NVIDIA Drivers and Container Toolkit

Make sure the proprietary NVIDIA driver is installed and working (verify with nvidia-smi). Then install the NVIDIA Container Toolkit so Docker can access the GPU:

distribution=$(. /etc/os-release;echo $ID$VERSION_ID) && \
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update && sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Test GPU visibility inside containers:

docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

3) Create the Docker Compose file

Create a project directory and a docker-compose.yml with two services: ollama and open-webui. This configuration exposes Open WebUI on port 8080 and mounts persistent volumes for both services.

mkdir -p ~/ollama-stack && cd ~/ollama-stack
nano docker-compose.yml

services:
ollama:
image: ollama/ollama:latest
container_name: ollama
restart: unless-stopped
ports:
- "11434:11434"
volumes:
- ollama-data:/root/.ollama
deploy:
resources:
reservations:
devices:
- capabilities: ["gpu"]
environment:
- OLLAMA_MAX_LOADED_MODELS=2
runtime: nvidia

open-webui:
image: ghcr.io/open-webui/open-webui:latest
container_name: open-webui
depends_on:
- ollama
restart: unless-stopped
ports:
- "8080:8080"
environment:
- OLLAMA_BASE_URL=http://ollama:11434
- WEBUI_NAME=Local LLM Console
volumes:
- openwebui-data:/app/backend/data

volumes:
ollama-data:
openwebui-data:

4) Launch the stack

Start both containers with Docker Compose:

docker compose up -d

Open WebUI will be available at http://<your-server-ip>:8080. The first visit prompts you to create an admin user.

5) Pull and run a model

You can pull models using the Ollama CLI within the container. For example, to fetch and run a popular 7B model:

docker exec -it ollama ollama pull llama3
docker exec -it ollama ollama run llama3

In Open WebUI, select the same model name (e.g., llama3) from the model dropdown. You can add system prompts, adjust temperature, or manage multiple models.

6) Persist, update, and back up

- Persistent data: All models live in the ollama-data volume, and Open WebUI settings live in openwebui-data.
- Update images: docker compose pull && docker compose up -d
- Back up volumes: docker run --rm -v ollama-data:/data -v $(pwd):/backup busybox tar czf /backup/ollama-data.tgz /data

7) Optional: restrict access and add HTTPS

For a single-user setup, it is safer to bind Open WebUI to localhost and use an SSH tunnel. Change the Open WebUI service port mapping from "8080:8080" to "127.0.0.1:8080:8080" and restart. Then access it with ssh -L 8080:localhost:8080 user@server.

If you need public access with HTTPS, place a reverse proxy (e.g., Caddy or Nginx) in front. With Caddy, a minimal site block looks like:

ai.example.com {
reverse_proxy 127.0.0.1:8080
}

Point DNS to your server and Caddy will fetch certificates automatically. Add HTTP auth or an allowlist if the instance is exposed to the internet.

8) Troubleshooting

Docker container cannot see the GPU: Confirm the host runs nvidia-smi successfully. Reinstall the NVIDIA Container Toolkit, run sudo nvidia-ctk runtime configure --runtime=docker, then sudo systemctl restart docker. Also ensure your compose file sets runtime: nvidia or uses --gpus all if running directly with docker run.

Out of memory or slow performance: Start with smaller models (e.g., 3B–7B). For better throughput, set num_ctx and num_gpu options when creating models with Ollama, and close other GPU-heavy apps.

Port conflicts: Change host port mappings in the compose file, e.g., "8081:8080" for Open WebUI or "11435:11434" for Ollama, and redeploy.

9) Notes for AMD/Apple users

This tutorial targets NVIDIA on Linux. For AMD GPUs on Linux, investigate ROCm builds of Ollama and ensure your GPU is supported by ROCm. On Apple Silicon, you can run both services natively or with Docker Desktop; GPU acceleration leverages Apple’s Metal backend automatically in the native build.

Wrap-up

You’ve deployed a modern local AI stack with Ollama and Open WebUI using Docker and enabled NVIDIA GPU acceleration on Ubuntu. With persistent storage, easy upgrades, and optional HTTPS, this setup is production-friendly for personal research, prototyping, and helpdesk automations. Experiment with different models, tweak performance flags, and keep your system updated for the best results.