LifeBytes Journal

Why run a local AI assistant?

Cloud AI tools are convenient, but a local setup can be faster for repeated tasks, cheaper over time, and more private for sensitive notes, logs, or internal documentation. Running an AI model locally is also a great way to learn modern AI tooling without committing to a paid API. In this guide, you will install Ollama on Linux, download a model, test it from the terminal, and optionally add a lightweight web interface for a more comfortable chat experience.

Prerequisites

You need a Linux machine (Ubuntu/Debian/Fedora/Arch all work), at least 8 GB RAM for smaller models, and preferably a modern CPU. A GPU is helpful but not required for many models. You will also need curl and basic terminal access with sudo privileges.

Step 1: Install Ollama

Ollama provides a simple installer for Linux. Open a terminal and run:

curl -fsSL https://ollama.com/install.sh | sh

After installation, verify the service is available:

ollama --version

On most systems, Ollama runs as a background service. If you want to check its status on a systemd-based distribution, use:

systemctl status ollama

Step 2: Download a model (and understand what you are pulling)

With Ollama, you download models using the pull command. A good starting point for general chat is a smaller, responsive model. For example:

ollama pull llama3.1

If disk space or RAM is limited, consider smaller variants (often labeled with fewer parameters). If you want code-focused answers, try a coding model such as:

ollama pull codellama

Model size matters. Larger models typically produce better results but require more RAM and may run slower. If performance feels sluggish, choose a smaller model rather than assuming something is broken.

Step 3: Chat with the model from the terminal

To start a chat session:

ollama run llama3.1

You can now type prompts and get replies immediately. This is perfect for quick tasks like generating a bash one-liner, summarizing a local change log, or drafting troubleshooting steps.

For scripting, you can also pass a prompt directly:

ollama run llama3.1 "Write a systemd unit that restarts a service on failure."

Step 4: Enable remote access safely (optional but common)

By default, many local AI setups listen only on localhost for safety. If you want to use Ollama from another machine on your LAN, you need to bind it carefully and protect it with firewall rules. First, check what address Ollama is listening on:

ss -tulpen | grep 11434

If you decide to expose it, do it on a trusted network only, and restrict access to specific IPs. On Ubuntu with UFW, for example, you can allow a single workstation:

sudo ufw allow from 192.168.1.50 to any port 11434

Avoid opening the port to the public internet. A local AI endpoint without authentication is not something you want exposed.

Step 5: Add a simple web UI (Open WebUI)

Terminal chat is efficient, but a web interface makes long conversations easier and adds quality-of-life features. One popular option is Open WebUI, which can connect to Ollama. The easiest deployment is with Docker. If Docker is not installed, install it from your distribution’s official docs first.

Run Open WebUI as a container:

docker run -d --name open-webui -p 3000:8080 -e OLLAMA_BASE_URL=http://host.docker.internal:11434 -v open-webui:/app/backend/data --restart unless-stopped ghcr.io/open-webui/open-webui:main

On some Linux hosts, host.docker.internal may not resolve by default. If that happens, use your host’s LAN IP (for example, http://192.168.1.10:11434) or add the Docker host gateway option depending on your Docker version. Once the container is running, open:

http://localhost:3000

Complete the initial setup in the browser, then select the Ollama model you downloaded. You should be able to chat immediately through the UI while Ollama continues doing the inference locally.

Troubleshooting tips (the issues people actually hit)

Model is slow or the system becomes unresponsive: Use a smaller model, close memory-heavy apps, or move to a machine with more RAM. Local AI is RAM-hungry, and swapping to disk will kill performance.

Ollama service is not running: Restart it with sudo systemctl restart ollama and check logs using journalctl -u ollama --no-pager -n 100.

Web UI cannot connect to Ollama: Confirm Ollama is reachable at http://127.0.0.1:11434 from the host, then adjust the Open WebUI environment variable OLLAMA_BASE_URL to point to the correct address.

Next steps

Once your local assistant is working, you can create repeatable prompts for helpdesk replies, generate configuration templates, or summarize technical notes without sending data to a third party. For better results, experiment with different models and keep your prompts specific. Local AI gets impressive quickly when you give it clear context and constraints.

Running an AI assistant locally is no longer a science project. With modern open-source tools, you can host a private chatbot on your own Linux machine, keep sensitive data off third-party servers, and still get fast, high-quality responses. In this tutorial, you’ll install Ollama (a lightweight local LLM runtime) and Open WebUI (a clean web interface) to create a self-hosted AI assistant you can access from your browser.

This guide targets Ubuntu/Debian-based systems, but the same approach works on many other Linux distributions with small adjustments. The setup is great for IT documentation drafting, code review, internal knowledge-base Q&A, and quick command-line help—without sending prompts to the cloud.

What You’ll Build

By the end, you will have:

1) Ollama installed and running as a local service
2) A model downloaded and ready to use (for example, Llama 3.x class models)
3) Open WebUI running in Docker, connected to Ollama
4) Optional remote access for your LAN with basic safety notes

Prerequisites

Before you start, make sure you have:

• A Linux server or workstation (8 GB RAM minimum; 16 GB+ recommended)
• At least 15–30 GB free disk space (model files can be large)
• Sudo access
• Docker installed (for Open WebUI)

Step 1: Install Ollama on Linux

Ollama provides a simple way to download and run local models. Install it with the official script:

Command:

curl -fsSL https://ollama.com/install.sh | sh

After installation, confirm the service is working:

ollama --version

If your system uses systemd, Ollama typically runs as a service. You can also test it by listing models (it will likely be empty at first):

ollama list

Step 2: Pull a Model and Test It

Now download a model. A common starting point is a Llama-family instruct model. Pull it using:

ollama pull llama3

Once the download completes, run a quick interactive test:

ollama run llama3

Type a short question (for example, “Explain systemd targets in simple terms”) and confirm you get a response. If this works, your local AI runtime is ready.

Step 3: Install Docker (If Needed)

If Docker is not installed yet, install it on Ubuntu/Debian with:

sudo apt update && sudo apt install -y docker.io

Enable and start Docker:

sudo systemctl enable --now docker

Optional but useful: allow your user to run Docker without sudo (log out and back in after this):

sudo usermod -aG docker $USER

Step 4: Run Open WebUI and Connect It to Ollama

Open WebUI provides a friendly ChatGPT-like interface and supports Ollama as a backend. Start it with Docker:

docker run -d --name open-webui \
-p 3000:8080 \
-e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
-v open-webui:/app/backend/data \
--restart unless-stopped \
ghcr.io/open-webui/open-webui:main

On Linux, host.docker.internal may not be available by default on older Docker versions. If your WebUI can’t connect, rerun the container using the host network mode:

docker rm -f open-webui
docker run -d --name open-webui \
--network=host \
-e OLLAMA_BASE_URL=http://127.0.0.1:11434 \
-v open-webui:/app/backend/data \
--restart unless-stopped \
ghcr.io/open-webui/open-webui:main

Open your browser and go to:

http://localhost:3000

Create the admin account when prompted. After login, you should see your Ollama model available in the model selector. Start a chat and confirm it responds.

Step 5: Make It Usable on Your Local Network (Optional)

If you want to access the WebUI from another device on your LAN, ensure the server firewall allows TCP port 3000. On Ubuntu with UFW:

sudo ufw allow 3000/tcp

Then browse to:

http://YOUR_SERVER_IP:3000

Security note: Don’t expose this directly to the internet without authentication and TLS. If you need remote access, put it behind a VPN (WireGuard is a solid choice) or a reverse proxy with HTTPS.

Troubleshooting Tips

WebUI loads but no models appear: Verify Ollama is running and reachable. On the host, test: curl http://127.0.0.1:11434. If Docker networking is the issue, use the --network=host method.

Slow responses: Try a smaller model, close heavy applications, or run on a machine with more RAM/CPU. Local LLM performance is mostly hardware-dependent.

Disk fills up quickly: Models can consume many gigabytes. Remove unused models with: ollama list then ollama rm MODELNAME.

Wrap-Up

With Ollama and Open WebUI, you can run a capable private AI assistant on Linux in under an hour. It’s an excellent setup for IT pros, developers, and small teams who want AI features without cloud costs or privacy concerns. Once it’s working, you can experiment with different models, create prompt presets, and build a local workflow that feels like a modern AI platform—fully under your control.

Running an AI assistant locally is no longer a niche experiment. With modern open models and lightweight serving tools, you can build a private, ChatGPT-style interface on your own Linux machine—no API keys, no data leaving your network, and full control over updates. In this tutorial, you will install Ollama (for downloading and serving LLMs) and Open WebUI (a clean web interface) using Docker. The result is a fast, self-hosted AI chat you can use for drafting, troubleshooting, and internal knowledge work.

What You’ll Build

By the end, you will have: (1) Ollama running as a local model server, (2) Open WebUI running in a container, and (3) a browser-based chat UI available on your LAN or localhost. This setup works well on Ubuntu Server, Debian, and most modern Linux distributions.

Prerequisites

Hardware: At least 8 GB RAM is recommended for smaller models. For better results, use 16 GB or more. A GPU helps but is not required for CPU-only usage.

Software: A Linux system with sudo access, Docker installed, and basic command-line familiarity. If you don’t have Docker yet, install it via your distribution’s official Docker instructions.

Step 1: Install and Start Ollama

Ollama provides a simple way to download and run large language models locally. Install it with the official script:

Command:

curl -fsSL https://ollama.com/install.sh | sh

After installation, start and enable the service (on most systemd-based systems):

sudo systemctl enable --now ollama

Confirm it is active:

systemctl status ollama

Step 2: Download a Model

Now you’ll pull a model. Choose one that matches your hardware. For a balanced option on many systems, try Llama 3 (size availability depends on what Ollama offers at the moment). Pulling a model can take time because it downloads several GB.

ollama pull llama3

Test it directly in the terminal:

ollama run llama3

Type a short prompt like “Explain RAID 1 in simple terms” to confirm it responds. Exit the session when done.

Step 3: Run Open WebUI in Docker

Open WebUI provides a user-friendly interface that feels similar to popular AI chat apps. It can connect to Ollama running on the host. Start by creating a persistent volume for WebUI data:

docker volume create open-webui

Then run the container. The key setting is the environment variable that tells WebUI where to find Ollama:

docker run -d \
--name open-webui \
-p 3000:8080 \
-e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
-v open-webui:/app/backend/data \
--restart unless-stopped \
ghcr.io/open-webui/open-webui:main

On some Linux hosts, host.docker.internal may not resolve by default. If Open WebUI can’t connect to Ollama, rerun the container with an extra host mapping:

docker rm -f open-webui

docker run -d \
--name open-webui \
-p 3000:8080 \
--add-host=host.docker.internal:host-gateway \
-e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
-v open-webui:/app/backend/data \
--restart unless-stopped \
ghcr.io/open-webui/open-webui:main

Step 4: Open the Web Interface

In your browser, open:

http://localhost:3000

If you’re accessing it from another computer on the same network, replace localhost with your Linux server’s IP address. The first time you open Open WebUI, you will create an admin account. After login, look for the model selection and choose the model you pulled (for example, llama3).

Step 5: Basic Security Hardening

A local AI chat can contain sensitive data, so treat it like an internal app. If this is only for you, bind to localhost by using Docker’s loopback mapping:

-p 127.0.0.1:3000:8080

If you need LAN access, consider placing it behind a reverse proxy (Nginx or Caddy) with HTTPS and authentication. Also make sure your firewall only allows trusted networks to reach port 3000.

Troubleshooting Tips

WebUI shows no models: Confirm Ollama is running and reachable. Try curl http://127.0.0.1:11434 on the host, then check the container logs with docker logs open-webui.

Slow responses: Use a smaller model, close other memory-heavy services, or run on a machine with more RAM. CPU-only inference is usable, but performance varies widely by hardware.

Connection errors from container to host: Use the --add-host=host.docker.internal:host-gateway option shown earlier, and keep the OLLAMA_BASE_URL pointing to http://host.docker.internal:11434.

Next Steps

Once your private AI assistant is working, you can expand it with additional models for different tasks, create separate chats for projects, and experiment with system prompts for consistent tone and formatting. The big advantage of this setup is control: you decide what runs, what gets stored, and how it’s exposed—without relying on external services.

Why run a private AI chatbot?

If you like the convenience of ChatGPT-style assistants but need better privacy, lower latency on your local network, or predictable costs, a self-hosted setup is a strong option. With Ollama you can run modern large language models (LLMs) locally, and with Open WebUI you get a clean web interface for chatting, managing models, and organizing prompts. In this tutorial you will deploy both on an Ubuntu server using Docker, so the install is repeatable and easy to maintain.

What you will build

By the end, you will have:

1) Ollama running as a service (the model runtime)
2) Open WebUI running in Docker (the chat UI)
3) Persistent storage for models and chat data
4) Optional GPU support notes if your server has NVIDIA

Prerequisites

Use an Ubuntu 22.04/24.04 server (VM or bare metal). A modern CPU and at least 8 GB RAM is workable for smaller models; 16–32 GB is more comfortable. You also need a user with sudo rights, outbound internet access to pull images/models, and Docker installed. If you plan to expose the UI beyond your LAN, put it behind a reverse proxy with TLS.

Step 1: Install Docker and Docker Compose

First, install Docker from Ubuntu’s repository (simple and reliable for most homelab and SMB setups):

Commands:
sudo apt update
sudo apt install -y docker.io docker-compose-plugin
sudo systemctl enable --now docker

Add your user to the docker group so you can run Docker without sudo (log out/in after this):

Command:
sudo usermod -aG docker $USER

Step 2: Create folders for persistent data

Persistent volumes are important because LLM files can be large and you do not want to re-download models after every container update. Create a working directory:

Commands:
mkdir -p ~/ai-stack/ollama
mkdir -p ~/ai-stack/openwebui
cd ~/ai-stack

Step 3: Create a Docker Compose file

Create a file named docker-compose.yml in ~/ai-stack. This setup runs Ollama and Open WebUI on the same Docker network. Ollama will listen on port 11434 internally; Open WebUI will be published on port 3000.

docker-compose.yml:

Copy and paste:
version: "3.8"

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    volumes:
      - ./ollama:/root/.ollama
    ports:
      - "11434:11434"

  openwebui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: openwebui
    restart: unless-stopped
    depends_on:
      - ollama
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
    volumes:
      - ./openwebui:/app/backend/data
    ports:
      - "3000:8080"

Step 4: Start the services

Bring the stack up in detached mode:

Command:
docker compose up -d

Verify containers are running:

Command:
docker ps

Step 5: Open the Web UI and pull a model

In a browser, open:

http://YOUR_SERVER_IP:3000

Open WebUI will ask you to create an admin account on first run. After login, you can download models through the interface, or you can pull models from the server side using Ollama.

To pull a popular small model (good for testing), run:

Command:
docker exec -it ollama ollama pull llama3.2

Once the model is downloaded, refresh Open WebUI and select the model for chat. If you want a lighter footprint, try smaller parameter models; if you need better answers, larger models require more RAM/VRAM.

Step 6: Basic troubleshooting (the common issues)

Open WebUI loads but shows no models: Confirm the environment variable points to Ollama. Run docker logs openwebui and make sure it can reach http://ollama:11434. Also verify Ollama is healthy with curl http://localhost:11434 on the host.

Model downloads are slow or fail: Check disk space (df -h) and DNS connectivity. LLM downloads can be multiple gigabytes, so a nearly full disk will cause strange errors.

High CPU and slow replies: This is normal on CPU-only servers with larger models. Use a smaller model, reduce concurrent users, or add GPU acceleration.

Optional: NVIDIA GPU acceleration notes

If you have an NVIDIA GPU, install the NVIDIA driver and the NVIDIA Container Toolkit so Docker containers can access the GPU. Then adjust the Ollama service to request GPU resources (exact configuration depends on your Docker and driver versions). GPU support can dramatically improve response time and allow you to run larger models smoothly.

Step 7: Keep it secure and maintainable

For a safer deployment, do not expose port 3000 directly to the internet. Put Open WebUI behind Nginx or Caddy with HTTPS and authentication. For updates, pull new images and recreate containers:

Commands:
cd ~/ai-stack
docker compose pull
docker compose up -d

Because you used persistent volumes, your downloaded models and chat database stay intact across updates.

Wrap-up

Running Ollama with Open WebUI on Ubuntu gives you a practical private AI chatbot you can use for internal documentation, code explanations, drafting emails, and brainstorming without sending prompts to a third-party cloud service. Start with a smaller model to confirm everything works, then scale up based on your hardware and the quality you need.

Running a private AI assistant locally is becoming a practical option for developers and IT teams who want faster responses, lower cloud costs, and better control over sensitive code. In this tutorial, you will set up a self-hosted AI “code helper” on a Linux server using Ollama (for running large language models locally) and Open WebUI (a clean web interface). The result is a browser-based assistant you can use for code reviews, script generation, troubleshooting, and documentation drafts—without sending prompts to external services.

What You’ll Build

You will deploy two components: Ollama, which downloads and serves models via a local API, and Open WebUI, which connects to Ollama and provides a chat UI with conversation history. This guide uses Docker to keep the installation clean and easy to update.

Prerequisites

Before you start, prepare a Linux machine (Ubuntu 22.04/24.04, Debian 12, or similar) with at least 8 GB RAM (16 GB is better for larger models) and 20+ GB free disk. A GPU is optional, but a modern CPU works fine for smaller models. You also need Docker and Docker Compose (or the Docker Compose plugin).

Step 1: Install Docker (Ubuntu/Debian)

If Docker is not installed, run the commands below. On other distributions, use the official Docker documentation for your package manager.

Commands:

sudo apt update
sudo apt install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Optional but recommended: allow your user to run Docker without sudo.

sudo usermod -aG docker $USER

Log out and back in after changing group membership.

Step 2: Create a Project Directory

Create a dedicated folder for your deployment so configuration and volumes stay organized.

mkdir -p ~/private-ai && cd ~/private-ai

Step 3: Create a Docker Compose File

Create a file named docker-compose.yml with the content below. It starts Ollama and Open WebUI, stores model data on disk, and makes the web UI available on port 3000.

cat > docker-compose.yml <<'EOF'
services:
ollama:
image: ollama/ollama:latest
container_name: ollama
restart: unless-stopped
ports:
- "11434:11434"
volumes:
- ollama:/root/.ollama

open-webui:
image: ghcr.io/open-webui/open-webui:main
container_name: open-webui
restart: unless-stopped
ports:
- "3000:8080"
environment:
- OLLAMA_BASE_URL=http://ollama:11434
volumes:
- openwebui:/app/backend/data
depends_on:
- ollama

volumes:
ollama:
openwebui:
EOF

Step 4: Start the Services

Bring the stack up in the background and confirm both containers are healthy.

docker compose up -d
docker ps

Now open your browser and go to http://YOUR_SERVER_IP:3000. The first user you create in Open WebUI typically becomes the admin, depending on the version.

Step 5: Download a Model with Ollama

Ollama pulls models on demand. For a lightweight code-focused start, try a smaller model first. Run the command below to download and test a model from inside the Ollama container.

docker exec -it ollama ollama pull codellama:7b
docker exec -it ollama ollama run codellama:7b

If you prefer a general assistant model, you can also try:

docker exec -it ollama ollama pull llama3.1:8b

Once pulled, go back to Open WebUI, start a new chat, and select the model. Your prompts will be processed locally on your server.

Step 6: Basic Hardening and Access Tips

If this server is not strictly internal, place Open WebUI behind a reverse proxy such as Nginx or Caddy and enable HTTPS. At a minimum, restrict access with a firewall so only your office IP/VPN can reach port 3000. On Ubuntu with UFW, you can allow only your admin workstation and block the rest.

sudo ufw allow from YOUR_IP to any port 3000 proto tcp
sudo ufw enable

Troubleshooting Common Problems

Open WebUI can’t see Ollama models: confirm the environment variable OLLAMA_BASE_URL points to http://ollama:11434 (container-to-container), and verify Ollama is listening: docker logs ollama.

Slow responses: smaller models respond faster on CPU. Also check system load and RAM usage. If the machine is swapping heavily, upgrade RAM or choose a smaller model.

Disk usage grows quickly: model files are large. Keep an eye on volumes and remove unused models with docker exec -it ollama ollama list and docker exec -it ollama ollama rm MODELNAME.

Conclusion

With Ollama and Open WebUI, you can run a capable private AI code assistant on your own Linux server in under an hour. This setup is ideal for testing prompts safely, speeding up daily scripting tasks, and keeping sensitive code and logs under your control. Once it’s running, you can experiment with different models, tighten access via HTTPS and VPN, and even dedicate a GPU host later for faster generation.

Why run AI locally?

If you work in IT, helpdesk, development, or sysadmin roles, you probably paste logs, configs, or customer data into tools to get quick answers. The problem is that cloud AI services can be expensive, limited, or simply not allowed in regulated environments. Running a local AI stack on your own Linux box gives you privacy, predictable performance, and the ability to keep everything inside your network.

In this tutorial you will install Ollama (a lightweight local LLM runner) and Open WebUI (a clean web interface) on Ubuntu. You will end up with a browser-based “ChatGPT-style” experience that talks to models running on your machine. This setup works on CPU-only systems and can also use an NVIDIA GPU if you have one.

What you will build

You will install Ollama as a service, then run Open WebUI in Docker and connect it to Ollama. After that, you will download a model (for example, Llama or Mistral variants), confirm it responds, and finally secure and persist the environment for daily use.

Requirements

You need an Ubuntu system (22.04 or newer is ideal), at least 8 GB RAM (16 GB+ recommended), and enough free disk space (models can range from a few GB to tens of GB). For the web UI portion you should have Docker installed. If you want GPU acceleration, you will also need a compatible NVIDIA driver and the NVIDIA Container Toolkit.

Step 1: Update Ubuntu and install Docker

Start by updating packages and installing Docker. If Docker is already installed, you can skip the installation step and just confirm it works.

Commands:

sudo apt update && sudo apt -y upgrade
sudo apt -y install ca-certificates curl gnupg
sudo apt -y install docker.io
sudo systemctl enable --now docker
sudo usermod -aG docker $USER

Log out and log back in so your user can run Docker without sudo. Verify:

docker run --rm hello-world

Step 2: Install Ollama

Ollama is simple to install and runs as a background service. It exposes an API on your machine, which the web interface will use.

Commands:

curl -fsSL https://ollama.com/install.sh | sh

Confirm the service is running:

systemctl status ollama --no-pager

Step 3: Pull a model and test Ollama from the terminal

Now download a model. A good starting point is a smaller model that runs well on CPU. If you have more RAM and want better responses, choose a larger one. The exact names can change over time, but these examples are commonly available.

Commands:

ollama pull llama3.1
ollama run llama3.1

When prompted, ask something practical like: “Explain what this Nginx error means and how to fix it.” If you get a sensible answer, Ollama is working.

Step 4: Run Open WebUI with Docker

Open WebUI provides a friendly interface, chat history, and a simple model selector. We will run it as a container and point it at the Ollama API.

Commands:

docker volume create open-webui
docker run -d --name open-webui --restart unless-stopped \
-p 3000:8080 \
-e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
-v open-webui:/app/backend/data \
ghcr.io/open-webui/open-webui:main

On Linux, host.docker.internal may not resolve on some setups. If the WebUI cannot connect to Ollama, rerun the container using the host network mode instead:

docker rm -f open-webui
docker run -d --name open-webui --restart unless-stopped \
--network=host \
-v open-webui:/app/backend/data \
ghcr.io/open-webui/open-webui:main

Open your browser and visit http://localhost:3000 (or your server IP if remote). Create an admin account when prompted. You should see your Ollama models listed in the UI.

Step 5: Optional GPU acceleration (NVIDIA)

If you have an NVIDIA GPU, install the correct driver first, then add the NVIDIA Container Toolkit if you plan to run GPU-enabled containers. Ollama itself can use the GPU on the host if the drivers are correctly installed. Confirm your GPU is visible:

nvidia-smi

If nvidia-smi works, test performance by running a model and watching GPU utilization in another terminal. If you see GPU usage increase during generation, your local AI is accelerated.

Step 6: Basic hardening and useful tips

If this server is on a network, avoid exposing the WebUI to the entire internet. Place it behind a reverse proxy with authentication (for example, Nginx with basic auth) or restrict access at the firewall. Also remember that models can store chat history in the WebUI volume, so treat the data directory like sensitive application data and back it up appropriately.

A few practical tips: keep an eye on disk usage as you try different models, standardize on one or two “default” models for your team, and document prompts for your most common workflows (log analysis, PowerShell troubleshooting, ticket replies, and postmortems). Local AI is at its best when it is part of a repeatable process, not just a toy.

Conclusion

With Ollama and Open WebUI, you can run a capable local AI assistant on Ubuntu in under an hour, using either CPU-only hardware or an NVIDIA GPU for faster output. The result is a private, controllable tool that can help with troubleshooting, scripting ideas, documentation drafts, and day-to-day IT tasks—without sending your data to external services.

Running an AI assistant locally is one of the most practical “advanced” upgrades you can make to a Linux workstation or home lab. You get faster iteration, more privacy, and you avoid sending sensitive text to a third-party cloud service. In this tutorial, you’ll install Ollama (a lightweight local LLM runtime) and Open WebUI (a clean web interface) on Ubuntu, then secure access and confirm everything is working.

This setup is great for a personal helpdesk bot, drafting and summarizing documents, generating scripts, or building an internal knowledge tool. It works on CPU-only systems, but performance improves significantly if you have a modern GPU. The steps below focus on a reliable, repeatable install that you can maintain like any other server service.

Prerequisites

Before you start, make sure you have: Ubuntu 22.04/24.04 (or a compatible Debian-based distro), a user with sudo permissions, at least 8 GB RAM (16 GB recommended for larger models), and roughly 15–30 GB of free disk space depending on which models you download.

Step 1: Update the system

Open a terminal and update your packages to avoid dependency issues:

sudo apt update && sudo apt -y upgrade

Step 2: Install Ollama

Ollama provides a simple command-line experience for downloading and running models. Install it using the official installer:

curl -fsSL https://ollama.com/install.sh | sudo sh

After installation, confirm the service is running:

systemctl status ollama

If it’s not active, start and enable it:

sudo systemctl enable --now ollama

Step 3: Download and test a model

Now pull a model. A good starting point is a smaller “general chat” model to verify your environment first:

ollama pull llama3.1

Then run a quick test:

ollama run llama3.1

Type a prompt like: “Write a bash one-liner to list the 10 largest files in a directory.” If you get a response, the core engine is working.

Step 4: Install Docker (recommended for Open WebUI)

Open WebUI is easiest to deploy in a container. Install Docker using Ubuntu packages:

sudo apt install -y docker.io

Enable the Docker service:

sudo systemctl enable --now docker

Optional but convenient: allow your user to run Docker without sudo (log out and back in after this):

sudo usermod -aG docker $USER

Step 5: Run Open WebUI and connect it to Ollama

Run the container and map it to a local port (3000). We’ll also mount a persistent volume so settings and chat history survive reboots:

docker run -d --name open-webui --restart unless-stopped -p 3000:8080 -e OLLAMA_BASE_URL=http://host.docker.internal:11434 -v open-webui:/app/backend/data ghcr.io/open-webui/open-webui:main

On Linux, host.docker.internal may not work on older Docker builds. If you open the web UI and it cannot reach Ollama, rerun the container using host networking instead:

docker rm -f open-webui

docker run -d --name open-webui --restart unless-stopped --network=host -e OLLAMA_BASE_URL=http://127.0.0.1:11434 -v open-webui:/app/backend/data ghcr.io/open-webui/open-webui:main

Now open your browser and go to http://localhost:3000 (or the server IP with port 3000). Create the first admin account. In most cases, Open WebUI will automatically detect the Ollama endpoint once the environment variable is set.

Step 6: Basic security hardening (don’t skip this)

If this is only for your local machine, binding to localhost is usually enough. If you plan to access it from other devices, you should secure it properly. At a minimum, configure the firewall to allow only trusted networks.

With UFW, you can allow port 3000 only from your LAN (example uses 192.168.1.0/24):

sudo ufw allow from 192.168.1.0/24 to any port 3000 proto tcp

sudo ufw enable

For a more professional setup, place Open WebUI behind Nginx with HTTPS (Let’s Encrypt) and optionally basic auth or SSO. That way, you’re not exposing a plain HTTP admin login to the network.

Step 7: Troubleshooting common issues

Open WebUI can’t see any models: confirm Ollama is running and reachable. Test locally with curl http://127.0.0.1:11434. If the container can’t reach the host, switch to --network=host as shown above.

Model downloads are slow or fail: try again later or switch networks. Large models are multi-GB downloads. Ensure you have enough disk space under /usr/share/ollama (or your configured storage path).

High CPU/RAM usage: use a smaller model, reduce parallel usage, or move the server to a machine with more memory. For older hardware, smaller models typically feel much more responsive.

Final check

At this point you have a private AI assistant running entirely on your own Ubuntu system. Ollama handles the model runtime, and Open WebUI provides an easy interface for chat, prompt testing, and daily use. Once you’re comfortable with the basics, you can explore model choices, system prompts for a “helpdesk” personality, and integrating local documents for internal Q&A workflows.

Running an AI chatbot locally is no longer a research project reserved for labs. With today’s lightweight LLM runtimes, you can host a private assistant on your own Linux machine and keep your data off third-party servers. This tutorial shows how to install Ollama (a simple local LLM runner) and Open WebUI (a clean web interface) using Docker, then load a model and start chatting from your browser.

What you will build

By the end of this guide, you will have a local web-based AI chat interface reachable from your LAN (or just your own PC). You’ll be able to pull models on demand, start conversations, and keep everything on your own storage. The setup works well for home labs, internal IT tools, offline environments, and privacy-focused workflows.

Prerequisites

Recommended system: 64-bit Linux (Ubuntu/Debian/Fedora work fine), at least 8 GB RAM (16 GB is better), and plenty of disk space (models can take several GB). A GPU is optional; CPU-only is still usable with smaller models. You also need admin access (sudo) and an internet connection for the initial downloads.

Step 1: Install Docker

If Docker is not installed, install it using your distro’s package manager or the official Docker repository. On Ubuntu/Debian, you can use:

sudo apt update
sudo apt install -y docker.io
sudo systemctl enable --now docker

To run Docker without typing sudo every time, add your user to the docker group (log out and back in after):

sudo usermod -aG docker $USER

Step 2: Create a working folder and Docker network

A dedicated folder keeps configuration tidy. Create it anywhere you like:

mkdir -p ~/local-ai
cd ~/local-ai

Create a Docker network so containers can reliably talk to each other by name:

docker network create localai

Step 3: Start Ollama (LLM runtime)

Ollama exposes an API that other apps (like WebUI) can use. Start it with a persistent volume so models survive reboots:

docker run -d --name ollama
--network localai
-p 11434:11434
-v ollama:/root/.ollama
ollama/ollama:latest

Verify it is running:

docker ps

Step 4: Pull a model and test from the command line

Now pull a model inside the Ollama container. For a balanced first run, try a smaller model if your RAM is limited. Example:

docker exec -it ollama ollama pull llama3.2

Test a quick prompt:

docker exec -it ollama ollama run llama3.2 "Write a short checklist for patching a Linux server safely."

If you get a response, the core runtime is working.

Step 5: Start Open WebUI (browser chat interface)

Open WebUI provides a friendly UI similar to popular online chat tools, but it stays in your environment. Start it and point it to Ollama using the container name:

docker run -d --name open-webui
--network localai
-p 3000:8080
-e OLLAMA_BASE_URL=http://ollama:11434
-v open-webui:/app/backend/data
ghcr.io/open-webui/open-webui:main

Open your browser and go to:

http://localhost:3000

If you’re accessing from another PC on the network, replace localhost with the Linux server’s IP (for example, http://192.168.1.50:3000).

Step 6: Select the model and start chatting

In Open WebUI, look for the model selector. If your Ollama container already pulled llama3.2, it should appear automatically. Choose it, start a new chat, and try an IT-focused prompt such as “Explain the difference between RAID1 and RAID10 with practical examples.”

Troubleshooting tips (common issues)

WebUI loads but no models appear: Confirm the environment variable is correct and that both containers share the same Docker network. Run docker logs open-webui and look for connection errors to http://ollama:11434.

Slow responses or timeouts: Use a smaller model, close other heavy workloads, and verify you have enough RAM. On CPU-only systems, large models can feel sluggish.

Cannot access from another computer: Make sure port 3000 is allowed through your firewall (UFW, firewalld, or your cloud security group). Also verify Open WebUI is bound via Docker’s port mapping.

Optional: Make it easier with Docker Compose

Once you’re happy with the setup, consider moving these commands into a Docker Compose file for simpler restarts and upgrades. The key idea remains the same: one container runs Ollama on port 11434, another runs Open WebUI on port 3000, and a shared network connects them.

With this local AI stack, you can experiment safely, build internal tools, and keep sensitive prompts under your control. As you get comfortable, try different models, tune prompts for helpdesk automation, or connect the WebUI to documentation snippets for faster internal answers.

Why a private RAG chatbot?

If your team needs an internal chatbot that can answer questions from company documents, you’ve probably looked at cloud AI services. The problem is compliance: sending sensitive data outside your network can be a deal-breaker. A practical alternative is a private RAG setup (Retrieval-Augmented Generation), where a local language model generates answers while a local index retrieves relevant text from your own files. In this tutorial, you’ll build a private RAG chatbot on a Linux server using Ollama (local LLM runtime) and Open WebUI (a friendly chat interface), then connect your documents to it.

What you will build

You will deploy two services with Docker: Ollama to run a model locally, and Open WebUI to provide a web-based chat UI and document ingestion features. This approach is ideal for homelabs, IT departments, and helpdesk teams who want AI-assisted answers without exposing internal knowledge to third parties.

Prerequisites

You need a Linux server or VM (Ubuntu/Debian recommended) with at least 8 GB RAM for smaller models; 16 GB+ is better. Disk space depends on the model (expect several GB). You also need Docker and Docker Compose. If you have an NVIDIA GPU, you can accelerate inference, but this guide works on CPU as well.

Step 1: Install Docker and Docker Compose

On Ubuntu, install Docker with the official packages, then enable the service:

Commands:
sudo apt update
sudo apt install -y docker.io docker-compose-plugin
sudo systemctl enable --now docker
sudo usermod -aG docker $USER

Log out and back in so your user can run Docker without sudo. Verify with:

Command:
docker version

Step 2: Create a Docker Compose file

Create a working directory and a compose file. This setup stores model files and WebUI data in persistent volumes so upgrades won’t wipe your configuration.

Commands:
mkdir -p ~/private-rag
cd ~/private-rag
nano docker-compose.yml

Paste the following content:

docker-compose.yml
version: "3.9"
services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    ports:
      - "11434:11434"
    volumes:
      - ollama:/root/.ollama

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    restart: unless-stopped
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
    volumes:
      - openwebui:/app/backend/data
    depends_on:
      - ollama

volumes:
  ollama:
  openwebui:

Step 3: Start the services

Bring the stack online:

Command:
docker compose up -d

Check that both containers are healthy:

Command:
docker ps

Open WebUI in your browser at http://YOUR_SERVER_IP:3000. The first account you create becomes the admin by default, so choose a strong password.

Step 4: Pull a model with Ollama

You can pull models directly inside the Ollama container. A good starting point for many servers is a smaller, fast instruct model.

Command:
docker exec -it ollama ollama pull llama3.1:8b

If your server has less RAM, try a smaller model. If you have more resources, you can experiment with larger variants for better reasoning. After pulling, confirm it’s available:

Command:
docker exec -it ollama ollama list

Step 5: Connect Open WebUI to the local model

In Open WebUI, go to the model selection menu and choose the model you pulled (for example, llama3.1:8b). Start a basic chat to confirm responses are generated locally. If it errors, verify that Open WebUI can reach Ollama on the internal Docker network and that the environment variable OLLAMA_BASE_URL matches the compose file.

Step 6: Enable RAG by adding your documents

To turn a general chatbot into a “knows our docs” assistant, ingest your content. In Open WebUI, find the section for Documents or Knowledge (wording may vary by version). Upload text-heavy sources such as internal runbooks, SOPs, FAQs, or exported wiki pages. For best retrieval results, prefer clean text formats like TXT, MD, PDF (machine-readable), and avoid scans without OCR.

After upload, Open WebUI will index the content so it can retrieve relevant chunks during chat. Test with a question that can only be answered from your document set, such as “What is our VPN reset procedure?” The response should cite or clearly reflect your internal wording. If the answer seems generic, add more targeted documents or refine your question.

Step 7: Secure access (quick hardening)

A private AI system can still leak data if it’s publicly exposed. First, bind access to trusted networks using a firewall (UFW on Ubuntu is simple) and consider putting Open WebUI behind a reverse proxy with HTTPS. Also, keep the service updated:

Commands:
cd ~/private-rag
docker compose pull
docker compose up -d

Finally, treat uploaded documents as sensitive: only allow authenticated users, and review what content is ingested. A RAG chatbot is powerful precisely because it can surface internal text quickly.

Troubleshooting tips

Model is slow: Use a smaller model, add RAM, or use GPU acceleration. Also reduce concurrent users.
WebUI can’t see the model: Confirm Ollama is running and reachable on port 11434 inside Docker, and that the model is listed in ollama list.
RAG answers are inaccurate: Upload more relevant documents, remove outdated versions, and prefer clean text sources. Retrieval quality depends heavily on document quality.

Next steps

Once your private RAG chatbot works, you can expand it by creating separate knowledge collections for different departments, adding a reverse proxy for SSO-like access control, or running multiple models for different tasks (fast model for chat, larger model for complex reasoning). This setup gives you a modern AI assistant while keeping your data inside your own environment.

Running an AI assistant locally is quickly becoming a practical option for developers and IT teams who want faster responses, offline access, and better control over sensitive code. In this tutorial, you will set up a private, local “ChatGPT-like” interface on a Linux server or workstation using Ollama (to run large language models) and Open WebUI (a web interface you can access from a browser). The result is a self-hosted AI assistant you can use for coding help, troubleshooting, and documentation drafts—without sending prompts to a third-party cloud.

This guide focuses on modern Linux distributions (Ubuntu/Debian-based commands are shown). The same approach works on many other distros with minor package differences. You’ll also learn basic hardening steps so the UI is not accidentally exposed to the internet.

Prerequisites

Hardware: A machine with at least 8 GB RAM is workable for smaller models, but 16 GB+ is recommended. A GPU is optional; many models run on CPU, just slower.

Software: Linux with sudo access, and either Docker (recommended) or Python for Open WebUI. You’ll also want an SSH session if you’re setting this up on a server.

Step 1: Install Ollama

Ollama is a lightweight runtime that downloads and runs models locally. Install it with the official script:

Command:

curl -fsSL https://ollama.com/install.sh | sh

After installation, verify it’s working:

ollama --version

Now pull a model. For a good balance of speed and capability, try a smaller modern model first:

ollama pull llama3.1

Test a quick prompt in the terminal:

ollama run llama3.1

Type a question, press Enter, and confirm you get a response. Exit with /bye or Ctrl+C depending on your session.

Step 2: Install Docker (Recommended)

Open WebUI can be installed in several ways, but Docker keeps it clean and easy to upgrade. Install Docker if you don’t already have it:

sudo apt update
sudo apt install -y docker.io
sudo systemctl enable --now docker

Optional but useful: allow your user to run Docker without sudo (log out and back in afterward):

sudo usermod -aG docker $USER

Step 3: Run Open WebUI and Connect It to Ollama

Open WebUI will provide a browser-based chat interface. Start it with Docker. The easiest approach is to map the container port to your host and point it at the Ollama API.

First, confirm Ollama is running. On many systems it runs as a service automatically after installation. You can check:

sudo systemctl status ollama

Now run Open WebUI:

docker run -d --name open-webui \
-p 3000:8080 \
-e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
-v open-webui:/app/backend/data \
--restart unless-stopped \
ghcr.io/open-webui/open-webui:main

If your Docker setup doesn’t support host.docker.internal on Linux, use the host network or the server IP instead. A common workaround is:

-e OLLAMA_BASE_URL=http://172.17.0.1:11434

Then open your browser to:

http://localhost:3000

Create the admin account on first run. After login, you should see available Ollama models. If you already pulled llama3.1, it should appear in the model list or be selectable.

Step 4: Make It Safe (Local Network Access Without Public Exposure)

By default, mapping -p 3000:8080 may expose the UI on all interfaces. If this is a server, you typically want LAN-only access. A simple method is to bind to a specific interface or localhost. For local-only access:

docker rm -f open-webui
docker run -d --name open-webui \
-p 127.0.0.1:3000:8080 \
-e OLLAMA_BASE_URL=http://127.0.0.1:11434 \
-v open-webui:/app/backend/data \
--restart unless-stopped \
ghcr.io/open-webui/open-webui:main

If you need access from another PC, consider using an SSH tunnel instead of opening a firewall port:

ssh -L 3000:127.0.0.1:3000 user@your-server

Then browse to http://localhost:3000 on your local machine.

Step 5: Common Troubleshooting

Model not showing up: Make sure the model is installed with ollama list. If it isn’t listed, run ollama pull <model>.

Open WebUI can’t connect to Ollama: Confirm Ollama listens on port 11434 and is reachable from the container. Check logs with docker logs open-webui. If needed, try the Docker bridge gateway IP (172.17.0.1) as the base URL.

Slow responses: Use a smaller model, close other memory-heavy apps, or run on a machine with more RAM. CPU-only inference is normal but slower.

Final Notes

With Ollama and Open WebUI, you get a practical local AI assistant that can help write scripts, explain logs, draft runbooks, and speed up troubleshooting—while keeping prompts on your own hardware. Once it’s running, experiment with different models and create reusable “system prompts” for tasks like helpdesk triage, Linux administration, or code review.

Running a large language model (LLM) locally is one of the fastest ways to get private, low-latency AI assistance without sending your prompts to a third-party cloud. In this tutorial, you will set up Ollama (a lightweight LLM runtime) and Open WebUI (a clean web interface) on Linux. The result is a self-hosted AI chat you can use for drafting, coding help, log analysis, and knowledge base searching—while keeping data on your own machine.

What You Need

Hardware: A modern CPU system works, but more RAM helps a lot. For small models (like 7B), aim for 8–16 GB RAM. For smoother performance or larger models, 32 GB+ is recommended. If you have an NVIDIA GPU, you can accelerate generation, but this guide focuses on a reliable CPU-first setup.

Software: A recent Linux distribution (Ubuntu/Debian/Fedora), terminal access, and either Docker (recommended for Open WebUI) or Python knowledge if you prefer manual setups.

Step 1: Install Ollama

Ollama makes local model management simple: you download a model once and then run it with a single command. To install Ollama, open a terminal and run:

curl -fsSL https://ollama.com/install.sh | sh

After installation, verify it works:

ollama --version

On most systems, Ollama starts as a service automatically. If you need to start it manually, you can run:

ollama serve

Step 2: Pull a Model (Example: Llama 3.1)

Now download a model. A good starting point is a modern 7B or 8B model. Pull it with:

ollama pull llama3.1

Once it finishes, test a quick prompt directly in the terminal:

ollama run llama3.1

Type a message (for example, “Summarize the difference between TCP and UDP”) and press Enter. If you get a response, the local model runtime is working.

Step 3: Install Docker (for Open WebUI)

Open WebUI is easiest to run in a container. If Docker is not installed, on Ubuntu/Debian you can do:

sudo apt update
sudo apt install -y docker.io
sudo systemctl enable --now docker

Optional but recommended: allow your user to run Docker without sudo:

sudo usermod -aG docker $USER

Log out and back in for the group change to apply.

Step 4: Run Open WebUI and Connect It to Ollama

Start Open WebUI with Docker. This command creates persistent storage and publishes the web interface on port 3000:

docker run -d --name open-webui -p 3000:8080 -v open-webui:/app/backend/data --restart unless-stopped ghcr.io/open-webui/open-webui:main

Next, ensure Open WebUI can reach Ollama. If Open WebUI does not automatically detect it, the most common fix is to point it to the Ollama API endpoint. Ollama listens on http://localhost:11434 by default. Depending on your Docker networking setup, “localhost” inside the container is not the host machine.

A practical approach is to run Open WebUI using host networking (Linux only). Stop the existing container and re-run:

docker rm -f open-webui
docker run -d --name open-webui --network=host -v open-webui:/app/backend/data --restart unless-stopped ghcr.io/open-webui/open-webui:main

Now open your browser and go to:

http://localhost:3000

Create an admin account when prompted. In the Open WebUI settings, you should see Ollama as an available provider. Select the model you pulled (for example, llama3.1) and start chatting.

Step 5: Improve Performance and Reliability

Choose the right model size: If responses feel slow, try a smaller model. Ollama supports many options; you can keep multiple models and switch depending on the task. Smaller models are great for quick drafts, command explanations, and lightweight Q&A.

Keep your data private: Local LLMs are only “private” if you avoid sending data out through plugins or external integrations. Treat the WebUI like any internal tool: secure access, avoid exposing it to the public internet, and consider a reverse proxy with authentication if you need remote access.

Troubleshoot connectivity: If Open WebUI can’t see Ollama, confirm the Ollama service is running and listening on port 11434:

ss -tulpn | grep 11434

If you prefer not to use host networking, you can configure Ollama to bind to an address reachable from Docker and then point Open WebUI to that address. The exact method depends on your distro and firewall rules, so host networking is the fastest baseline to validate your setup.

Next Steps (Useful Ideas)

Once your local AI chat is stable, you can level it up: create model presets for different writing styles, connect it to internal documentation, or use it for structured tasks like generating incident summaries from sanitized logs. The biggest advantage of this setup is control—you decide what runs, where it runs, and what data it can access.

With Ollama and Open WebUI, a private LLM workstation is no longer a weekend project. It’s a practical tool you can deploy in minutes and refine over time.

Why host a local AI assistant?

If you write scripts, manage servers, or handle helpdesk tickets, an AI assistant can speed up routine work like summarizing logs, drafting commands, or explaining configuration files. The problem is that many cloud tools send your prompts and snippets to third-party services. A local setup keeps sensitive data on your own machine, works offline, and can be tuned for your workflow.

In this tutorial, you will install Ollama (a lightweight local LLM runtime) and Open WebUI (a web interface similar to popular chat tools) on Linux. The result is a private AI assistant you can access from your browser on your LAN.

What you need

Hardware: A modern 64-bit Linux system. For acceptable performance, aim for 16 GB RAM or more. A GPU helps but is not required for basic use. Lighter models can run on CPU-only machines, including small servers.

Software: A recent Linux distribution (Ubuntu/Debian/Fedora are all fine), Docker for Open WebUI, and basic terminal access with sudo.

Step 1: Install Ollama

Ollama runs the model locally and exposes an API that other tools (like Open WebUI) can call. Install it using the official script:

Command:

curl -fsSL https://ollama.com/install.sh | sh

After installation, check that the service is working:

ollama --version

On many distros, Ollama runs as a service. If you need to confirm it is active:

systemctl status ollama

Step 2: Pull a model and test it

Next, download a model. If you are CPU-only or want fast responses, start with a smaller model. For general coding help, you can also try code-focused models once the basics work.

Example (general model):

ollama pull llama3.1

Run a quick prompt to confirm everything works:

ollama run llama3.1

Type a question like “Explain what journald does on Linux” and confirm you get a response. Exit with /bye or Ctrl+C depending on your shell behavior.

Step 3: Install Docker (if not installed)

Open WebUI is easiest to deploy with Docker. On Ubuntu/Debian, you can install Docker like this:

sudo apt update

sudo apt install -y docker.io

sudo systemctl enable --now docker

Optional but recommended: allow your user to run Docker without sudo (log out and back in after this):

sudo usermod -aG docker $USER

Step 4: Run Open WebUI and connect it to Ollama

Open WebUI will provide a clean browser interface and conversation history. The key is pointing it at Ollama’s API endpoint.

First, make sure Ollama is listening locally. By default it is typically available at http://127.0.0.1:11434. Now start Open WebUI in Docker:

docker run -d --name open-webui --restart unless-stopped -p 3000:8080 -e OLLAMA_BASE_URL=http://host.docker.internal:11434 -v open-webui:/app/backend/data ghcr.io/open-webui/open-webui:main

On Linux, host.docker.internal may not be available depending on your Docker version. If the UI cannot connect, rerun the container using host networking instead:

docker rm -f open-webui

docker run -d --name open-webui --restart unless-stopped --network=host -e OLLAMA_BASE_URL=http://127.0.0.1:11434 -v open-webui:/app/backend/data ghcr.io/open-webui/open-webui:main

Now open your browser and visit:

http://localhost:3000

Create the first admin user when prompted. Once logged in, you should see available Ollama models. If you do not, go to settings and verify the Ollama base URL.

Step 5: Enable LAN access (optional and safer if restricted)

If you want to access the assistant from another device on your network, bind the service to a reachable interface and restrict it with firewall rules. For Open WebUI using Docker with port publishing, ensure your firewall only allows trusted subnets to connect to port 3000.

For example, on Ubuntu with UFW you can allow only your local subnet (adjust the CIDR):

sudo ufw allow from 192.168.1.0/24 to any port 3000 proto tcp

Avoid exposing the service directly to the internet. If you need remote access, put it behind a VPN (WireGuard is a good choice) or a reverse proxy with authentication.

Troubleshooting tips

Open WebUI shows “cannot reach Ollama”: Confirm Ollama is running with systemctl status ollama. Then check connectivity from the container. If you are using port mapping, the simplest fix on Linux is often --network=host.

Model downloads are slow or fail: Verify DNS and outbound access. Large models can be tens of gigabytes. If disk space is tight, remove unused models with ollama list and ollama rm <model>.

Responses are too slow: Try a smaller model, reduce context size in settings, and close other memory-heavy applications. CPU-only systems benefit from lightweight models and shorter prompts.

Next steps: make it useful for real admin work

Once the UI is running, build a few saved prompts for your daily tasks: “Summarize this syslog excerpt,” “Write a Bash one-liner to find large files,” or “Draft a polite helpdesk reply.” Because the assistant is local, you can safely paste internal error messages, configuration snippets, or playbook fragments without sending them to a third party.

With Ollama and Open WebUI, you get a practical self-hosted AI assistant that fits nicely into a Linux admin toolbox: fast to deploy, easy to maintain, and private by design.

Why a local RAG chatbot?

If you work in IT, you probably have internal documents that never belong in a public cloud: runbooks, SOPs, incident postmortems, customer notes, firewall rules, or server inventories. A local chatbot can answer questions from those files without uploading anything outside your network. The modern approach is RAG (Retrieval-Augmented Generation): the system searches your documents for relevant passages and then asks the language model to respond using that context.

In this tutorial you will deploy a practical, self-hosted setup on Linux using Ollama (to run local LLMs) and Open WebUI (a friendly web interface). You will end with a browser-based chat that can be extended with document ingestion features and can run fully offline.

What you will build

Ollama will run the language model on your Linux host. Open WebUI will provide the web UI and manage connections to Ollama. This combination is popular because it’s simple to update, works well with Docker, and supports a “private by default” workflow.

Prerequisites

You need a Linux server or workstation (Ubuntu/Debian/Fedora are fine). Recommended: 16 GB RAM or more, and SSD storage. A GPU is optional; CPU-only works, but responses will be slower. You also need root or sudo access and an internet connection for the initial downloads (you can later run offline).

Step 1: Install Ollama

On most Linux distributions, the quickest method is the official install script. Run the following:

Command:
curl -fsSL https://ollama.com/install.sh | sh

After installation, verify that the service is running:

ollama --version
systemctl status ollama

If your firewall is strict, note that Ollama typically listens on 127.0.0.1:11434 by default (local-only). That’s good for security. You can keep it that way when Open WebUI runs on the same machine.

Step 2: Pull a model with Ollama

Choose a model that matches your hardware. A solid general-purpose starting point is a smaller Llama-family model. Pull a model like this:

ollama pull llama3.1

Then test it quickly:

ollama run llama3.1

Type a short prompt (for example: “Summarize the purpose of RAG in one paragraph.”) and confirm you get a response. Exit with /bye.

Step 3: Install Docker (if needed)

Open WebUI is commonly deployed with Docker. If Docker is not installed, install it using your distro’s recommended method. On Ubuntu, this is typically:

sudo apt update
sudo apt install -y docker.io
sudo systemctl enable --now docker

To avoid running Docker commands with sudo, you can add your user to the docker group (log out and back in afterward):

sudo usermod -aG docker $USER

Step 4: Run Open WebUI connected to Ollama

Start Open WebUI as a container and point it to Ollama. If Ollama is running on the same host, the container can reach it using host networking (simple on Linux):

docker run -d --name open-webui --restart=unless-stopped --network=host -e OLLAMA_BASE_URL=http://127.0.0.1:11434 -v open-webui:/app/backend/data ghcr.io/open-webui/open-webui:main

Open your browser and go to:

http://localhost:8080

Create the first admin account when prompted. Once logged in, confirm that your Ollama model appears in the model list. If it doesn’t, re-check that Ollama is running and that the URL is correct.

Step 5: Basic security hardening (recommended)

If this is more than a lab setup, don’t expose port 8080 directly to the internet. Instead, put it behind a reverse proxy (Nginx/Traefik/Caddy) with HTTPS and authentication. At minimum, restrict access to your LAN via firewall rules. If multiple users will access it, create separate accounts and disable anonymous access in the UI settings.

Step 6: Add documents for RAG (practical approach)

RAG requires two pieces: (1) a place to store your documents and (2) an index/search layer that can retrieve relevant chunks. Many teams start with a controlled folder of PDFs/Markdown/TXT and progressively add ingestion and indexing tools as needs grow.

A simple, safe workflow is:

1) Put sanitized internal docs in a dedicated directory (example: /srv/knowledgebase).
2) Convert “messy” formats to text where possible (Markdown and text files work best).
3) In Open WebUI, look for knowledge or document features (often called “Knowledge,” “Documents,” or “RAG” depending on version) and import your files.

If you do not see document ingestion in your build, treat this deployment as the base LLM layer and add a dedicated RAG service later (for example, a vector database plus an ingestion pipeline). The key advantage is that you already have the model hosting and UI stable and local.

Troubleshooting common issues

Open WebUI can’t see Ollama models: Verify Ollama is running (systemctl status ollama) and confirm the base URL. If you didn’t use --network=host, use Docker’s host gateway options or run both services in the same Docker network and reference Ollama by container name.

Slow responses: Try a smaller model, close other heavy workloads, and ensure you have enough RAM. CPU-only inference is normal but slower. If you have a supported GPU, check Ollama’s documentation for acceleration support on your platform.

High disk usage: Models can be several GB each. Remove unused models with ollama rm <model> and keep only what you use.

Next steps

Once the local chatbot is working, you can improve accuracy and trust by tightening your knowledge base: keep documents current, remove duplicates, and structure key procedures in Markdown. If you expand into a full RAG stack, define clear ingestion rules and access controls so the chatbot only retrieves what each user is allowed to see.

Overview

If you want an AI assistant for internal documentation, troubleshooting, or drafting replies without sending company data to a third-party cloud, a self-hosted setup is a strong option. In this tutorial, you will deploy a private AI stack on an Ubuntu Server using Docker: Ollama (to run large language models locally) and Open WebUI (a clean web interface for chatting, prompts, and basic management). This approach is practical for homelabs and small teams, and it keeps your prompts and conversation history inside your own network.

What You Will Build

By the end, you will have two containers running: one for Ollama (the model runtime/API) and one for Open WebUI (the front-end). You will also configure persistent storage, pull a model, and confirm everything works from a browser. The steps below are written for Ubuntu Server 22.04/24.04, but will work on most modern Ubuntu releases.

Prerequisites

You need an Ubuntu Server with at least 8 GB RAM (16 GB recommended), 20+ GB free disk, and a modern CPU. A GPU helps performance but is not required for a functional deployment. You also need root or sudo access and a working network connection. If this is a server on a LAN, decide which port you will expose for the web interface (we will use 3000).

Step 1: Install Docker and Docker Compose

First, install Docker using the official repository packages. This ensures you get up-to-date components and fewer compatibility issues with Compose.

Run:

sudo apt update
sudo apt install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin

Optionally allow your user to run Docker without sudo (log out and back in afterward):

sudo usermod -aG docker $USER

Step 2: Create a Project Folder and Compose File

Create a directory to keep your deployment clean and manageable. Then create a docker-compose.yml file that defines both services and persistent volumes.

mkdir -p ~/ai-stack
cd ~/ai-stack
nano docker-compose.yml

Paste the following Compose configuration:

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    ports:
      - "11434:11434"
    volumes:
      - ollama:/root/.ollama

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    restart: unless-stopped
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
    volumes:
      - openwebui:/app/backend/data
    depends_on:
      - ollama

volumes:
  ollama:
  openwebui:

Step 3: Start the Stack

Bring up the containers in the background and confirm they are running.

docker compose up -d
docker compose ps

If you see both services with a “running” state, the base deployment is complete.

Step 4: Pull a Model with Ollama

Now download a model. The best choice depends on your RAM and use case. For many servers, a smaller model is a safe starting point. The command below pulls a popular lightweight model.

docker exec -it ollama ollama pull llama3.2

You can list installed models anytime:

docker exec -it ollama ollama list

Step 5: Log In to Open WebUI and Connect to Ollama

Open a browser and go to http://SERVER-IP:3000. On first launch, Open WebUI asks you to create an admin account. After login, the interface should automatically detect Ollama through the internal Docker network using the OLLAMA_BASE_URL you configured.

Start a new chat, select the model you pulled (for example llama3.2), and send a test prompt such as “Write a short troubleshooting checklist for DNS issues.” If the response appears, your private AI assistant is working end-to-end.

Step 6: Basic Hardening and Practical Tips

Firewall: If this is a public-facing server, do not expose it directly without protection. At minimum, allow only your LAN or VPN subnet to reach port 3000. With UFW, you can restrict access instead of opening the port to everyone.

Reverse proxy: For production use, place Open WebUI behind Nginx or Caddy with HTTPS and authentication. This also makes it easier to use a friendly hostname.

Backups: Your important data lives in Docker volumes. Back up the Open WebUI volume (chat history, settings) and the Ollama volume (models) according to your retention needs.

Updates: Refresh images regularly to get security fixes and new features:

docker compose pull
docker compose up -d

Troubleshooting

Open WebUI loads but no models appear: Verify Ollama is reachable from the Open WebUI container. Check logs with docker logs open-webui and confirm OLLAMA_BASE_URL=http://ollama:11434 is correct.

Model downloads are slow: Large model pulls can take time. Ensure your server has stable internet and enough free disk. You can also choose smaller models to start.

High RAM usage or slow responses: Use a smaller model, reduce concurrent users, or run the service on hardware with more memory. Local AI is resource-intensive by design, and tuning is part of a realistic deployment.

Conclusion

Running Ollama and Open WebUI on Ubuntu Server gives you a private, self-hosted AI assistant that you can control, secure, and integrate into your workflow. Once the base stack is stable, you can expand it with HTTPS, SSO, logging, and routine backups. The key advantage is simple: your prompts and internal context stay on your infrastructure while still giving your team an easy web-based AI experience.

Running an AI chatbot locally is no longer a “lab-only” project. With today’s lightweight models and tools like Ollama and Open WebUI, you can build a private, fast, and surprisingly capable assistant on a Windows PC—without sending prompts to third-party cloud services. This tutorial walks you through a practical setup that works well for IT notes, scripting help, documentation drafts, and troubleshooting ideas, all while keeping your data on your own machine.

This guide focuses on an up-to-date approach: Ollama provides an easy local model runtime, and Open WebUI gives you a clean web interface with chat history, model selection, and basic admin options. The result feels like a polished “ChatGPT-style” experience, but running on your own hardware.

Prerequisites

Before you start, confirm you have the following:

1) A Windows 10/11 system (64-bit). 2) At least 16 GB RAM recommended (8 GB can work with smaller models). 3) Enough disk space for models (5–20 GB depending on what you install). 4) Optional but helpful: an NVIDIA GPU for faster inference. CPU-only still works—just slower.

Step 1: Install Ollama on Windows

Ollama is the engine that downloads and runs the model files locally. Install Ollama from its official site and complete the installer. After installation, Ollama runs a local service and exposes an API on your machine.

To verify it’s working, open PowerShell and run a quick model test. First, pull a model and run it:

Command:

ollama run llama3.1

If the model downloads and you see a prompt where you can type, Ollama is functioning. Type something simple like “Explain DNS in one paragraph” and confirm you get a response.

Step 2: Choose a Model That Fits Your Hardware

Local AI is all about picking a model that matches your PC. As a rule, smaller models load faster and use less RAM, while larger models can be more accurate but require better hardware.

Here are practical starting points you can try with Ollama:

• llama3.1: good general assistant for many tasks.
• mistral: fast and solid for summaries and troubleshooting.
• phi3: lightweight option for lower-end machines.

To download a model without launching it immediately, you can use:

ollama pull mistral

Step 3: Install Open WebUI (Web Interface)

Ollama is powerful, but the default terminal chat is not ideal for daily use. Open WebUI adds a browser-based interface so you can manage chats, switch models, and work comfortably.

The simplest method on Windows is to run Open WebUI using Docker Desktop. Install Docker Desktop, enable WSL 2 integration if prompted, then open PowerShell and run:

docker run -d --name open-webui -p 3000:8080 -e OLLAMA_BASE_URL=http://host.docker.internal:11434 ghcr.io/open-webui/open-webui:main

This command downloads the latest Open WebUI image and connects it to Ollama running on your host. After it starts, open your browser and go to:

http://localhost:3000

Create the first admin user when prompted. Once logged in, Open WebUI should automatically detect your Ollama models. If you don’t see them, check the Ollama service is running and confirm the URL points to http://host.docker.internal:11434.

Step 4: Test a Chat and Tune the Basics

In Open WebUI, select a model (for example, llama3.1) and start a new conversation. A good first test prompt is something specific:

Write a PowerShell script that checks free disk space on C: and warns if it is below 15%.

If you get a usable script, your pipeline is working end-to-end: browser UI → Open WebUI → Ollama → local model → response back to your browser.

If answers feel slow, try a smaller model, close memory-heavy apps, and keep your prompt concise. On CPU-only systems, switching to a lighter model often makes a bigger difference than any other tweak.

Step 5: Common Problems and Fixes

Open WebUI can’t connect to Ollama: Confirm Ollama is running and listening locally. Restart the Ollama service, then restart the Open WebUI container:
docker restart open-webui

Models don’t appear in the UI: Pull a model first using Ollama (for example, ollama pull llama3.1), then refresh Open WebUI.

Performance is poor: Use a smaller model like phi3 or mistral. Also ensure Windows power mode is not set to battery saver, and keep adequate free RAM available.

Disk fills up quickly: Local models are large. Remove models you don’t use with:
ollama rm <modelname>

Why This Setup Is Worth It

A local chatbot won’t replace every cloud AI feature, but it shines in day-to-day technical work: drafting SOPs, generating scripts, summarizing logs you can’t upload, and brainstorming troubleshooting steps while keeping everything on your own PC. Once you have Ollama and Open WebUI running, adding new models is a one-command task, and the browser interface makes it feel like a real tool—not a demo.

If you want to go further later, you can explore model fine-tuning, retrieval-augmented generation (RAG) with your internal documents, or running the same stack on a small home server. For now, this Windows setup is a clean, practical starting point for private AI that you control.