LifeBytes Journal

Why a private RAG chatbot?

If your team needs an internal chatbot that can answer questions from company documents, you’ve probably looked at cloud AI services. The problem is compliance: sending sensitive data outside your network can be a deal-breaker. A practical alternative is a private RAG setup (Retrieval-Augmented Generation), where a local language model generates answers while a local index retrieves relevant text from your own files. In this tutorial, you’ll build a private RAG chatbot on a Linux server using Ollama (local LLM runtime) and Open WebUI (a friendly chat interface), then connect your documents to it.

What you will build

You will deploy two services with Docker: Ollama to run a model locally, and Open WebUI to provide a web-based chat UI and document ingestion features. This approach is ideal for homelabs, IT departments, and helpdesk teams who want AI-assisted answers without exposing internal knowledge to third parties.

Prerequisites

You need a Linux server or VM (Ubuntu/Debian recommended) with at least 8 GB RAM for smaller models; 16 GB+ is better. Disk space depends on the model (expect several GB). You also need Docker and Docker Compose. If you have an NVIDIA GPU, you can accelerate inference, but this guide works on CPU as well.

Step 1: Install Docker and Docker Compose

On Ubuntu, install Docker with the official packages, then enable the service:

Commands:
sudo apt update
sudo apt install -y docker.io docker-compose-plugin
sudo systemctl enable --now docker
sudo usermod -aG docker $USER

Log out and back in so your user can run Docker without sudo. Verify with:

Command:
docker version

Step 2: Create a Docker Compose file

Create a working directory and a compose file. This setup stores model files and WebUI data in persistent volumes so upgrades won’t wipe your configuration.

Commands:
mkdir -p ~/private-rag
cd ~/private-rag
nano docker-compose.yml

Paste the following content:

docker-compose.yml
version: "3.9"
services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    ports:
      - "11434:11434"
    volumes:
      - ollama:/root/.ollama

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    restart: unless-stopped
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
    volumes:
      - openwebui:/app/backend/data
    depends_on:
      - ollama

volumes:
  ollama:
  openwebui:

Step 3: Start the services

Bring the stack online:

Command:
docker compose up -d

Check that both containers are healthy:

Command:
docker ps

Open WebUI in your browser at http://YOUR_SERVER_IP:3000. The first account you create becomes the admin by default, so choose a strong password.

Step 4: Pull a model with Ollama

You can pull models directly inside the Ollama container. A good starting point for many servers is a smaller, fast instruct model.

Command:
docker exec -it ollama ollama pull llama3.1:8b

If your server has less RAM, try a smaller model. If you have more resources, you can experiment with larger variants for better reasoning. After pulling, confirm it’s available:

Command:
docker exec -it ollama ollama list

Step 5: Connect Open WebUI to the local model

In Open WebUI, go to the model selection menu and choose the model you pulled (for example, llama3.1:8b). Start a basic chat to confirm responses are generated locally. If it errors, verify that Open WebUI can reach Ollama on the internal Docker network and that the environment variable OLLAMA_BASE_URL matches the compose file.

Step 6: Enable RAG by adding your documents

To turn a general chatbot into a “knows our docs” assistant, ingest your content. In Open WebUI, find the section for Documents or Knowledge (wording may vary by version). Upload text-heavy sources such as internal runbooks, SOPs, FAQs, or exported wiki pages. For best retrieval results, prefer clean text formats like TXT, MD, PDF (machine-readable), and avoid scans without OCR.

After upload, Open WebUI will index the content so it can retrieve relevant chunks during chat. Test with a question that can only be answered from your document set, such as “What is our VPN reset procedure?” The response should cite or clearly reflect your internal wording. If the answer seems generic, add more targeted documents or refine your question.

Step 7: Secure access (quick hardening)

A private AI system can still leak data if it’s publicly exposed. First, bind access to trusted networks using a firewall (UFW on Ubuntu is simple) and consider putting Open WebUI behind a reverse proxy with HTTPS. Also, keep the service updated:

Commands:
cd ~/private-rag
docker compose pull
docker compose up -d

Finally, treat uploaded documents as sensitive: only allow authenticated users, and review what content is ingested. A RAG chatbot is powerful precisely because it can surface internal text quickly.

Troubleshooting tips

Model is slow: Use a smaller model, add RAM, or use GPU acceleration. Also reduce concurrent users.
WebUI can’t see the model: Confirm Ollama is running and reachable on port 11434 inside Docker, and that the model is listed in ollama list.
RAG answers are inaccurate: Upload more relevant documents, remove outdated versions, and prefer clean text sources. Retrieval quality depends heavily on document quality.

Next steps

Once your private RAG chatbot works, you can expand it by creating separate knowledge collections for different departments, adding a reverse proxy for SSO-like access control, or running multiple models for different tasks (fast model for chat, larger model for complex reasoning). This setup gives you a modern AI assistant while keeping your data inside your own environment.

Running an AI assistant locally is quickly becoming a practical option for developers and IT teams who want faster responses, offline access, and better control over sensitive code. In this tutorial, you will set up a private, local “ChatGPT-like” interface on a Linux server or workstation using Ollama (to run large language models) and Open WebUI (a web interface you can access from a browser). The result is a self-hosted AI assistant you can use for coding help, troubleshooting, and documentation drafts—without sending prompts to a third-party cloud.

This guide focuses on modern Linux distributions (Ubuntu/Debian-based commands are shown). The same approach works on many other distros with minor package differences. You’ll also learn basic hardening steps so the UI is not accidentally exposed to the internet.

Prerequisites

Hardware: A machine with at least 8 GB RAM is workable for smaller models, but 16 GB+ is recommended. A GPU is optional; many models run on CPU, just slower.

Software: Linux with sudo access, and either Docker (recommended) or Python for Open WebUI. You’ll also want an SSH session if you’re setting this up on a server.

Step 1: Install Ollama

Ollama is a lightweight runtime that downloads and runs models locally. Install it with the official script:

Command:

curl -fsSL https://ollama.com/install.sh | sh

After installation, verify it’s working:

ollama --version

Now pull a model. For a good balance of speed and capability, try a smaller modern model first:

ollama pull llama3.1

Test a quick prompt in the terminal:

ollama run llama3.1

Type a question, press Enter, and confirm you get a response. Exit with /bye or Ctrl+C depending on your session.

Step 2: Install Docker (Recommended)

Open WebUI can be installed in several ways, but Docker keeps it clean and easy to upgrade. Install Docker if you don’t already have it:

sudo apt update
sudo apt install -y docker.io
sudo systemctl enable --now docker

Optional but useful: allow your user to run Docker without sudo (log out and back in afterward):

sudo usermod -aG docker $USER

Step 3: Run Open WebUI and Connect It to Ollama

Open WebUI will provide a browser-based chat interface. Start it with Docker. The easiest approach is to map the container port to your host and point it at the Ollama API.

First, confirm Ollama is running. On many systems it runs as a service automatically after installation. You can check:

sudo systemctl status ollama

Now run Open WebUI:

docker run -d --name open-webui \
-p 3000:8080 \
-e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
-v open-webui:/app/backend/data \
--restart unless-stopped \
ghcr.io/open-webui/open-webui:main

If your Docker setup doesn’t support host.docker.internal on Linux, use the host network or the server IP instead. A common workaround is:

-e OLLAMA_BASE_URL=http://172.17.0.1:11434

Then open your browser to:

http://localhost:3000

Create the admin account on first run. After login, you should see available Ollama models. If you already pulled llama3.1, it should appear in the model list or be selectable.

Step 4: Make It Safe (Local Network Access Without Public Exposure)

By default, mapping -p 3000:8080 may expose the UI on all interfaces. If this is a server, you typically want LAN-only access. A simple method is to bind to a specific interface or localhost. For local-only access:

docker rm -f open-webui
docker run -d --name open-webui \
-p 127.0.0.1:3000:8080 \
-e OLLAMA_BASE_URL=http://127.0.0.1:11434 \
-v open-webui:/app/backend/data \
--restart unless-stopped \
ghcr.io/open-webui/open-webui:main

If you need access from another PC, consider using an SSH tunnel instead of opening a firewall port:

ssh -L 3000:127.0.0.1:3000 user@your-server

Then browse to http://localhost:3000 on your local machine.

Step 5: Common Troubleshooting

Model not showing up: Make sure the model is installed with ollama list. If it isn’t listed, run ollama pull <model>.

Open WebUI can’t connect to Ollama: Confirm Ollama listens on port 11434 and is reachable from the container. Check logs with docker logs open-webui. If needed, try the Docker bridge gateway IP (172.17.0.1) as the base URL.

Slow responses: Use a smaller model, close other memory-heavy apps, or run on a machine with more RAM. CPU-only inference is normal but slower.

Final Notes

With Ollama and Open WebUI, you get a practical local AI assistant that can help write scripts, explain logs, draft runbooks, and speed up troubleshooting—while keeping prompts on your own hardware. Once it’s running, experiment with different models and create reusable “system prompts” for tasks like helpdesk triage, Linux administration, or code review.

Why a local RAG chatbot?

If you work in IT, you probably have internal documents that never belong in a public cloud: runbooks, SOPs, incident postmortems, customer notes, firewall rules, or server inventories. A local chatbot can answer questions from those files without uploading anything outside your network. The modern approach is RAG (Retrieval-Augmented Generation): the system searches your documents for relevant passages and then asks the language model to respond using that context.

In this tutorial you will deploy a practical, self-hosted setup on Linux using Ollama (to run local LLMs) and Open WebUI (a friendly web interface). You will end with a browser-based chat that can be extended with document ingestion features and can run fully offline.

What you will build

Ollama will run the language model on your Linux host. Open WebUI will provide the web UI and manage connections to Ollama. This combination is popular because it’s simple to update, works well with Docker, and supports a “private by default” workflow.

Prerequisites

You need a Linux server or workstation (Ubuntu/Debian/Fedora are fine). Recommended: 16 GB RAM or more, and SSD storage. A GPU is optional; CPU-only works, but responses will be slower. You also need root or sudo access and an internet connection for the initial downloads (you can later run offline).

Step 1: Install Ollama

On most Linux distributions, the quickest method is the official install script. Run the following:

Command:
curl -fsSL https://ollama.com/install.sh | sh

After installation, verify that the service is running:

ollama --version
systemctl status ollama

If your firewall is strict, note that Ollama typically listens on 127.0.0.1:11434 by default (local-only). That’s good for security. You can keep it that way when Open WebUI runs on the same machine.

Step 2: Pull a model with Ollama

Choose a model that matches your hardware. A solid general-purpose starting point is a smaller Llama-family model. Pull a model like this:

ollama pull llama3.1

Then test it quickly:

ollama run llama3.1

Type a short prompt (for example: “Summarize the purpose of RAG in one paragraph.”) and confirm you get a response. Exit with /bye.

Step 3: Install Docker (if needed)

Open WebUI is commonly deployed with Docker. If Docker is not installed, install it using your distro’s recommended method. On Ubuntu, this is typically:

sudo apt update
sudo apt install -y docker.io
sudo systemctl enable --now docker

To avoid running Docker commands with sudo, you can add your user to the docker group (log out and back in afterward):

sudo usermod -aG docker $USER

Step 4: Run Open WebUI connected to Ollama

Start Open WebUI as a container and point it to Ollama. If Ollama is running on the same host, the container can reach it using host networking (simple on Linux):

docker run -d --name open-webui --restart=unless-stopped --network=host -e OLLAMA_BASE_URL=http://127.0.0.1:11434 -v open-webui:/app/backend/data ghcr.io/open-webui/open-webui:main

Open your browser and go to:

http://localhost:8080

Create the first admin account when prompted. Once logged in, confirm that your Ollama model appears in the model list. If it doesn’t, re-check that Ollama is running and that the URL is correct.

Step 5: Basic security hardening (recommended)

If this is more than a lab setup, don’t expose port 8080 directly to the internet. Instead, put it behind a reverse proxy (Nginx/Traefik/Caddy) with HTTPS and authentication. At minimum, restrict access to your LAN via firewall rules. If multiple users will access it, create separate accounts and disable anonymous access in the UI settings.

Step 6: Add documents for RAG (practical approach)

RAG requires two pieces: (1) a place to store your documents and (2) an index/search layer that can retrieve relevant chunks. Many teams start with a controlled folder of PDFs/Markdown/TXT and progressively add ingestion and indexing tools as needs grow.

A simple, safe workflow is:

1) Put sanitized internal docs in a dedicated directory (example: /srv/knowledgebase).
2) Convert “messy” formats to text where possible (Markdown and text files work best).
3) In Open WebUI, look for knowledge or document features (often called “Knowledge,” “Documents,” or “RAG” depending on version) and import your files.

If you do not see document ingestion in your build, treat this deployment as the base LLM layer and add a dedicated RAG service later (for example, a vector database plus an ingestion pipeline). The key advantage is that you already have the model hosting and UI stable and local.

Troubleshooting common issues

Open WebUI can’t see Ollama models: Verify Ollama is running (systemctl status ollama) and confirm the base URL. If you didn’t use --network=host, use Docker’s host gateway options or run both services in the same Docker network and reference Ollama by container name.

Slow responses: Try a smaller model, close other heavy workloads, and ensure you have enough RAM. CPU-only inference is normal but slower. If you have a supported GPU, check Ollama’s documentation for acceleration support on your platform.

High disk usage: Models can be several GB each. Remove unused models with ollama rm <model> and keep only what you use.

Next steps

Once the local chatbot is working, you can improve accuracy and trust by tightening your knowledge base: keep documents current, remove duplicates, and structure key procedures in Markdown. If you expand into a full RAG stack, define clear ingestion rules and access controls so the chatbot only retrieves what each user is allowed to see.

Why self-hosted runners are worth it

If your builds are slow, your workflow uses special tools, or you need access to an internal network, a self-hosted GitHub Actions runner can be a game-changer. Instead of relying on GitHub-hosted runners (which are shared and time-limited), you run jobs on your own Linux machine. This tutorial shows how to set up a runner on Ubuntu Server with a clean, secure approach: a dedicated user, a systemd service, and basic hardening tips.

What you need before you start

You’ll need: (1) an Ubuntu Server 22.04/24.04 machine (VM or bare metal), (2) outbound internet access to GitHub, (3) a GitHub repository or organization where you can register runners, and (4) sudo privileges on the Linux host. For best results, use a separate machine or VM for CI jobs—treat it as disposable infrastructure.

Step 1: Update the server and install prerequisites

First, patch the system and install common dependencies used by build pipelines. Run the following commands:

sudo apt update && sudo apt -y upgrade

sudo apt -y install curl tar git ca-certificates

If your workflows build containers, install Docker later (and consider isolating it). For now, keep the base runner simple.

Step 2: Create a dedicated runner user

Avoid running CI as your personal account or as root. Create a dedicated user and a working directory:

sudo adduser --disabled-password --gecos "" actions

sudo mkdir -p /opt/actions-runner

sudo chown -R actions:actions /opt/actions-runner

This helps with least privilege and keeps runner files in a predictable location for maintenance.

Step 3: Download the GitHub Actions runner

Switch to the runner user and download the latest Linux x64 runner package. You can find the current version on GitHub’s official runner releases page, but the process is always the same:

sudo -iu actions

cd /opt/actions-runner

curl -o actions-runner-linux-x64.tar.gz -L https://github.com/actions/runner/releases/latest/download/actions-runner-linux-x64-2.0.0.tar.gz

tar xzf actions-runner-linux-x64.tar.gz

Note: the filename in the URL can change as new versions are released. If you get a 404 error, open the releases page and copy the exact download link for Linux x64.

Step 4: Register the runner with your repo or organization

In GitHub, go to your repository: Settings > Actions > Runners > New self-hosted runner. Choose Linux, and GitHub will display a short set of commands including a registration token.

Back on your server (still as the actions user), run the configuration script using the URL and token GitHub provides:

./config.sh --url https://github.com/OWNER/REPO --token YOUR_TOKEN

When prompted, set a clear runner name (for example, ubuntu-ci-01) and add labels that match your use case (like linux, self-hosted, docker, gpu). Labels let you target specific runners in workflows.

Step 5: Install the runner as a systemd service

Running the runner in a terminal works, but it’s not reliable. Install it as a service so it starts on boot and restarts on failure:

sudo ./svc.sh install

sudo ./svc.sh start

Then verify status:

sudo ./svc.sh status

Within a minute, the runner should show as Idle in GitHub under Actions runners.

Step 6: Update a workflow to use your self-hosted runner

In your workflow YAML, set runs-on to include self-hosted plus any labels you assigned:

runs-on: [self-hosted, linux]

If you want to guarantee the job lands on a specific capability (like Docker), use a custom label such as docker and specify it in runs-on.

Security and hardening tips (don’t skip these)

A self-hosted runner executes code from your repository, including pull requests if you allow it. Treat it like a production entry point. Use these practical safeguards: (1) run on a dedicated VM, (2) restrict which branches and events can use the runner, (3) avoid running untrusted fork PRs on self-hosted runners, and (4) keep the OS patched.

Also consider network segmentation: if the runner can reach internal services, use firewall rules so it can only access what it truly needs. If you install Docker, be careful with granting the runner user access to the Docker socket—Docker can effectively become root on the host. For higher-risk environments, run builds inside isolated containers or ephemeral VMs.

Troubleshooting common problems

Runner is offline: check service status (sudo ./svc.sh status) and logs with journalctl -u actions.runner.* -n 200 --no-pager. Reboot-safe service configuration usually fixes “works in terminal, fails after reboot” issues.

Token expired: registration tokens are short-lived. Generate a new token in GitHub and re-run ./config.sh after removing the old configuration (./config.sh remove).

Jobs stuck waiting: your workflow’s runs-on labels must match the runner labels exactly. If the workflow requires [self-hosted, linux, docker] but your runner is only labeled linux, GitHub will keep the job queued.

Conclusion

With a self-hosted GitHub Actions runner on Linux, you can speed up CI/CD, use specialized build tools, and keep deployment workflows closer to your infrastructure. The key is to set it up cleanly (dedicated user and systemd service) and to treat security as part of the installation—not an afterthought.

Overview

This tutorial shows how to deploy Traefik v3 as a modern reverse proxy with Docker Compose and automatic HTTPS certificates using Let’s Encrypt via the Cloudflare DNS-01 challenge. The DNS-01 method works even behind NAT, on residential ISPs, and with Cloudflare’s orange cloud (proxy) enabled. By the end, you will have a secure, production-ready reverse proxy and a sample container accessible over HTTPS.

Prerequisites

- A Linux server (Ubuntu 22.04/24.04 or similar) with Docker Engine and Docker Compose v2 installed.
- A domain managed by Cloudflare (nameservers pointing to Cloudflare).
- A Cloudflare API token with Zone.DNS:Edit and Zone:Read permissions for the zone you’ll use.
- Basic terminal access and a user with Docker privileges.

Step 1 — Verify Docker and Compose

Confirm your Docker setup is ready. Run: docker --version and docker compose version. If Compose v2 is not present, install the latest Docker Engine from the official repository. On Ubuntu, ensure the docker group exists and your user is a member: sudo usermod -aG docker $USER then re-log.

Step 2 — Create a Dedicated Docker Network

Create an external network so Traefik can share it with your app containers: docker network create proxy. Using a dedicated network helps isolate traffic and makes adding new services predictable.

Step 3 — Prepare Folders and Secrets

Make a working directory for Traefik and create a place to store ACME data (certificates):

mkdir -p ~/traefik/letsencrypt
cd ~/traefik
touch ./letsencrypt/acme.json
chmod 600 ./letsencrypt/acme.json

Create an .env file to hold your Cloudflare token securely. This file will be read by Docker Compose:

echo "CF_DNS_API_TOKEN=<paste_your_cloudflare_api_token>" > .env

The API token should include Zone.DNS:Edit and Zone.Zone:Read permissions for the domain. Restrict the token to the specific zone for better security.

Step 4 — Create docker-compose.yml

Create a docker-compose.yml file in ~/traefik with the following content. Replace [email protected] with your email and whoami.example.com with a real subdomain in your zone.

services:
  traefik:
    image: traefik:v3.1
    command:
     - --providers.docker=true
     - --providers.docker.exposedbydefault=false
     - --entrypoints.web.address=:80
     - --entrypoints.websecure.address=:443
     - --entrypoints.web.http.redirections.entrypoint.to=websecure
     - --entrypoints.web.http.redirections.entrypoint.scheme=https
     - [email protected]
     - --certificatesresolvers.letsencrypt.acme.storage=/letsencrypt/acme.json
     - --certificatesresolvers.letsencrypt.acme.dnschallenge=true
     - --certificatesresolvers.letsencrypt.acme.dnschallenge.provider=cloudflare
     - --api.dashboard=true
    ports:
     - "80:80"
     - "443:443"
    environment:
     - CF_DNS_API_TOKEN=${CF_DNS_API_TOKEN}
    volumes:
     - /var/run/docker.sock:/var/run/docker.sock:ro
     - ./letsencrypt:/letsencrypt
    networks:
     - proxy

  whoami:
    image: traefik/whoami:v1.10
    labels:
     - "traefik.enable=true"
     - "traefik.http.routers.whoami.rule=Host(`whoami.example.com`)"
     - "traefik.http.routers.whoami.entrypoints=websecure"
     - "traefik.http.routers.whoami.tls.certresolver=letsencrypt"
    networks:
     - proxy

networks:
  proxy:
    external: true

Create a DNS A/AAAA record for whoami.example.com pointing to your server’s public IP in Cloudflare. The orange cloud (proxy) can be ON or OFF; DNS-01 works either way.

Step 5 — Launch and Test

Start the stack from the ~/traefik directory: docker compose up -d. Traefik will request a certificate from Let’s Encrypt using the Cloudflare DNS-01 challenge. Check logs with docker compose logs -f traefik to confirm issuance (look for “Server responded with a certificate”).

Open https://whoami.example.com in your browser. You should see the whoami test service showing headers and IP details over HTTPS.

Optional: Secure the Traefik Dashboard

The dashboard is enabled but not published by default in this setup. To expose it safely, add labels to a new service or to Traefik itself using a distinct host like traefik.example.com, require basic auth middleware, and keep it behind TLS. Always disable --api.insecure=true in production.

Troubleshooting Tips

- If certificates do not issue, verify the API token has Zone.DNS:Edit and is scoped to the correct zone. Also confirm the .env is loaded and the environment variable name matches.
- If you see rate-limit errors, you may have requested too many certificates; wait and try again or use the Let’s Encrypt staging endpoint during testing (--certificatesresolvers.letsencrypt.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory).
- Ensure your firewall allows TCP 80 and 443 inbound to the server.

Maintaining and Adding Services

To add a new app behind Traefik, place it on the proxy network and attach labels for rule, entrypoint, and TLS resolver. Example labels: traefik.enable=true, traefik.http.routers.app.rule=Host(`app.example.com`), traefik.http.routers.app.entrypoints=websecure, and traefik.http.routers.app.tls.certresolver=letsencrypt. Restart only the new service; Traefik hot-reloads routes automatically.

Conclusion

With Traefik v3, Docker Compose, and Cloudflare’s DNS-01 challenge, you can ship secure containers with automatic HTTPS and minimal friction. This setup scales cleanly, keeps certificates current, and works in challenging network environments. Add your apps with labels, keep tokens scoped and secret, and enjoy a tidy, TLS-by-default container platform.