LifeBytes Journal

Why run AI locally?

If you work in IT, helpdesk, development, or sysadmin roles, you probably paste logs, configs, or customer data into tools to get quick answers. The problem is that cloud AI services can be expensive, limited, or simply not allowed in regulated environments. Running a local AI stack on your own Linux box gives you privacy, predictable performance, and the ability to keep everything inside your network.

In this tutorial you will install Ollama (a lightweight local LLM runner) and Open WebUI (a clean web interface) on Ubuntu. You will end up with a browser-based “ChatGPT-style” experience that talks to models running on your machine. This setup works on CPU-only systems and can also use an NVIDIA GPU if you have one.

What you will build

You will install Ollama as a service, then run Open WebUI in Docker and connect it to Ollama. After that, you will download a model (for example, Llama or Mistral variants), confirm it responds, and finally secure and persist the environment for daily use.

Requirements

You need an Ubuntu system (22.04 or newer is ideal), at least 8 GB RAM (16 GB+ recommended), and enough free disk space (models can range from a few GB to tens of GB). For the web UI portion you should have Docker installed. If you want GPU acceleration, you will also need a compatible NVIDIA driver and the NVIDIA Container Toolkit.

Step 1: Update Ubuntu and install Docker

Start by updating packages and installing Docker. If Docker is already installed, you can skip the installation step and just confirm it works.

Commands:

sudo apt update && sudo apt -y upgrade
sudo apt -y install ca-certificates curl gnupg
sudo apt -y install docker.io
sudo systemctl enable --now docker
sudo usermod -aG docker $USER

Log out and log back in so your user can run Docker without sudo. Verify:

docker run --rm hello-world

Step 2: Install Ollama

Ollama is simple to install and runs as a background service. It exposes an API on your machine, which the web interface will use.

Commands:

curl -fsSL https://ollama.com/install.sh | sh

Confirm the service is running:

systemctl status ollama --no-pager

Step 3: Pull a model and test Ollama from the terminal

Now download a model. A good starting point is a smaller model that runs well on CPU. If you have more RAM and want better responses, choose a larger one. The exact names can change over time, but these examples are commonly available.

Commands:

ollama pull llama3.1
ollama run llama3.1

When prompted, ask something practical like: “Explain what this Nginx error means and how to fix it.” If you get a sensible answer, Ollama is working.

Step 4: Run Open WebUI with Docker

Open WebUI provides a friendly interface, chat history, and a simple model selector. We will run it as a container and point it at the Ollama API.

Commands:

docker volume create open-webui
docker run -d --name open-webui --restart unless-stopped \
-p 3000:8080 \
-e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
-v open-webui:/app/backend/data \
ghcr.io/open-webui/open-webui:main

On Linux, host.docker.internal may not resolve on some setups. If the WebUI cannot connect to Ollama, rerun the container using the host network mode instead:

docker rm -f open-webui
docker run -d --name open-webui --restart unless-stopped \
--network=host \
-v open-webui:/app/backend/data \
ghcr.io/open-webui/open-webui:main

Open your browser and visit http://localhost:3000 (or your server IP if remote). Create an admin account when prompted. You should see your Ollama models listed in the UI.

Step 5: Optional GPU acceleration (NVIDIA)

If you have an NVIDIA GPU, install the correct driver first, then add the NVIDIA Container Toolkit if you plan to run GPU-enabled containers. Ollama itself can use the GPU on the host if the drivers are correctly installed. Confirm your GPU is visible:

nvidia-smi

If nvidia-smi works, test performance by running a model and watching GPU utilization in another terminal. If you see GPU usage increase during generation, your local AI is accelerated.

Step 6: Basic hardening and useful tips

If this server is on a network, avoid exposing the WebUI to the entire internet. Place it behind a reverse proxy with authentication (for example, Nginx with basic auth) or restrict access at the firewall. Also remember that models can store chat history in the WebUI volume, so treat the data directory like sensitive application data and back it up appropriately.

A few practical tips: keep an eye on disk usage as you try different models, standardize on one or two “default” models for your team, and document prompts for your most common workflows (log analysis, PowerShell troubleshooting, ticket replies, and postmortems). Local AI is at its best when it is part of a repeatable process, not just a toy.

Conclusion

With Ollama and Open WebUI, you can run a capable local AI assistant on Ubuntu in under an hour, using either CPU-only hardware or an NVIDIA GPU for faster output. The result is a private, controllable tool that can help with troubleshooting, scripting ideas, documentation drafts, and day-to-day IT tasks—without sending your data to external services.

Running an AI assistant locally is quickly becoming a practical option for developers and IT teams who want faster responses, offline access, and better control over sensitive code. In this tutorial, you will set up a private, local “ChatGPT-like” interface on a Linux server or workstation using Ollama (to run large language models) and Open WebUI (a web interface you can access from a browser). The result is a self-hosted AI assistant you can use for coding help, troubleshooting, and documentation drafts—without sending prompts to a third-party cloud.

This guide focuses on modern Linux distributions (Ubuntu/Debian-based commands are shown). The same approach works on many other distros with minor package differences. You’ll also learn basic hardening steps so the UI is not accidentally exposed to the internet.

Prerequisites

Hardware: A machine with at least 8 GB RAM is workable for smaller models, but 16 GB+ is recommended. A GPU is optional; many models run on CPU, just slower.

Software: Linux with sudo access, and either Docker (recommended) or Python for Open WebUI. You’ll also want an SSH session if you’re setting this up on a server.

Step 1: Install Ollama

Ollama is a lightweight runtime that downloads and runs models locally. Install it with the official script:

Command:

curl -fsSL https://ollama.com/install.sh | sh

After installation, verify it’s working:

ollama --version

Now pull a model. For a good balance of speed and capability, try a smaller modern model first:

ollama pull llama3.1

Test a quick prompt in the terminal:

ollama run llama3.1

Type a question, press Enter, and confirm you get a response. Exit with /bye or Ctrl+C depending on your session.

Step 2: Install Docker (Recommended)

Open WebUI can be installed in several ways, but Docker keeps it clean and easy to upgrade. Install Docker if you don’t already have it:

sudo apt update
sudo apt install -y docker.io
sudo systemctl enable --now docker

Optional but useful: allow your user to run Docker without sudo (log out and back in afterward):

sudo usermod -aG docker $USER

Step 3: Run Open WebUI and Connect It to Ollama

Open WebUI will provide a browser-based chat interface. Start it with Docker. The easiest approach is to map the container port to your host and point it at the Ollama API.

First, confirm Ollama is running. On many systems it runs as a service automatically after installation. You can check:

sudo systemctl status ollama

Now run Open WebUI:

docker run -d --name open-webui \
-p 3000:8080 \
-e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
-v open-webui:/app/backend/data \
--restart unless-stopped \
ghcr.io/open-webui/open-webui:main

If your Docker setup doesn’t support host.docker.internal on Linux, use the host network or the server IP instead. A common workaround is:

-e OLLAMA_BASE_URL=http://172.17.0.1:11434

Then open your browser to:

http://localhost:3000

Create the admin account on first run. After login, you should see available Ollama models. If you already pulled llama3.1, it should appear in the model list or be selectable.

Step 4: Make It Safe (Local Network Access Without Public Exposure)

By default, mapping -p 3000:8080 may expose the UI on all interfaces. If this is a server, you typically want LAN-only access. A simple method is to bind to a specific interface or localhost. For local-only access:

docker rm -f open-webui
docker run -d --name open-webui \
-p 127.0.0.1:3000:8080 \
-e OLLAMA_BASE_URL=http://127.0.0.1:11434 \
-v open-webui:/app/backend/data \
--restart unless-stopped \
ghcr.io/open-webui/open-webui:main

If you need access from another PC, consider using an SSH tunnel instead of opening a firewall port:

ssh -L 3000:127.0.0.1:3000 user@your-server

Then browse to http://localhost:3000 on your local machine.

Step 5: Common Troubleshooting

Model not showing up: Make sure the model is installed with ollama list. If it isn’t listed, run ollama pull <model>.

Open WebUI can’t connect to Ollama: Confirm Ollama listens on port 11434 and is reachable from the container. Check logs with docker logs open-webui. If needed, try the Docker bridge gateway IP (172.17.0.1) as the base URL.

Slow responses: Use a smaller model, close other memory-heavy apps, or run on a machine with more RAM. CPU-only inference is normal but slower.

Final Notes

With Ollama and Open WebUI, you get a practical local AI assistant that can help write scripts, explain logs, draft runbooks, and speed up troubleshooting—while keeping prompts on your own hardware. Once it’s running, experiment with different models and create reusable “system prompts” for tasks like helpdesk triage, Linux administration, or code review.

Why run a local AI assistant?

If you write code or manage infrastructure, an AI assistant can speed up tasks like generating scripts, explaining logs, reviewing configs, and drafting documentation. The problem is that many tools send prompts and code to a cloud service. In regulated environments, that is not always allowed. Running an AI model locally gives you better privacy, predictable costs, and the option to keep everything inside your LAN.

In this tutorial, you will install Ollama (a lightweight local model runtime) and Open WebUI (a clean browser interface) on Linux using Docker. The result is a self-hosted, ChatGPT-like web app that talks to your local models.

What you need

Requirements: a modern Linux server or workstation (Ubuntu/Debian/Fedora are fine), at least 8 GB RAM (16 GB+ recommended), and enough disk space for models (10–30 GB is common). CPU-only works, but a supported GPU can improve speed significantly. You also need a user with sudo privileges and outbound internet access to download containers and models.

Step 1: Install Docker (and Docker Compose)

If Docker is already installed, you can skip this section. On Ubuntu/Debian, one of the simplest approaches is using the official repository packages. Install Docker Engine and enable the service. If your distribution provides Docker Compose as a plugin, you can use docker compose (with a space) instead of the older docker-compose binary.

After installation, verify it works by running a test container. If you want to avoid using sudo for every Docker command, add your user to the docker group and log out/in once.

Step 2: Create a project folder

Create a dedicated directory for your stack so it is easy to manage and back up later. For example:

mkdir -p ~/local-ai && cd ~/local-ai

Step 3: Create a Docker Compose file for Ollama and Open WebUI

Create a file named docker-compose.yml in the folder. This setup uses persistent volumes so that downloaded models and WebUI data survive reboots and container upgrades.

nano docker-compose.yml

Paste the following:

<pre> version: "3.9" services: ollama: image: ollama/ollama:latest container_name: ollama restart: unless-stopped ports: - "11434:11434" volumes: - ollama:/root/.ollama open-webui: image: ghcr.io/open-webui/open-webui:main container_name: open-webui restart: unless-stopped depends_on: - ollama ports: - "3000:8080" environment: - OLLAMA_BASE_URL=http://ollama:11434 volumes: - openwebui:/app/backend/data volumes: ollama: openwebui: </pre>

Step 4: Start the services

From the same directory, start the stack:

docker compose up -d

Check that both containers are running:

docker ps

If something fails, view logs:

docker logs -f ollama

docker logs -f open-webui

Step 5: Download a model into Ollama

Ollama pulls models on demand. A practical starting point is a small-to-mid model that fits your RAM. For example, you can pull a general-purpose model like:

docker exec -it ollama ollama pull llama3.2

You can list installed models at any time:

docker exec -it ollama ollama list

If you prefer a coding-focused model, search Ollama’s library and choose one that matches your hardware. The key is to start small, confirm everything works, then scale up to larger models.

Step 6: Open WebUI in your browser

Open your browser and go to:

http://localhost:3000

If you installed this on a server, replace localhost with the server’s IP or hostname (for example, http://10.0.0.20:3000). The first time you load the page, you will create an admin account. After login, Open WebUI should detect Ollama automatically via the OLLAMA_BASE_URL setting.

Step 7: Basic usage tips (for real work)

To make the assistant useful for sysadmin and helpdesk tasks, be specific and provide context. Instead of “Fix this,” try prompts like: “Explain what this Nginx config does and suggest safer defaults” or “Write a bash script that checks disk usage and emails an alert”. When you paste logs, remove secrets and tokens, even if you are staying local.

If responses are slow, use a smaller model, reduce parallel users, or move the stack to a machine with more RAM. Local models are sensitive to memory pressure; swapping to disk can make them feel unusable.

Troubleshooting common problems

Open WebUI loads but shows no models: confirm Ollama is reachable from the WebUI container. The environment variable should be OLLAMA_BASE_URL=http://ollama:11434. Also confirm the model is actually pulled with ollama list.

Port already in use: if another service uses port 3000, change the mapping to something else, for example "8085:8080", then restart with docker compose up -d.

Downloads are slow or failing: this is usually DNS, proxy, or firewall related. Test connectivity from the host, and check whether your environment requires an HTTP proxy for container traffic.

Next steps: make it production-friendly

For a lab setup, HTTP on a LAN is fine. For teams, place Open WebUI behind a reverse proxy like Nginx or Caddy, add TLS, and restrict access with SSO or at least strong passwords. Finally, back up Docker volumes so you do not lose your settings and conversation history when you migrate.

Why Syncthing is a Smart Alternative to Cloud Sync

If you want Dropbox-style file synchronization without handing your data to a third-party cloud, Syncthing is one of the most practical tools available today. It is open-source, uses strong encryption, and syncs files directly between your devices. That makes it ideal for IT pros, homelab users, and small teams that need fast and private file replication across Windows and Linux systems.

This tutorial walks you through a modern, stable setup: installing Syncthing on Windows and Linux, pairing devices securely, setting up reliable folder sync, and applying best-practice tweaks for performance and safety. You will end up with a “set it and forget it” file synchronization system that works on your LAN and also remotely.

What You Need Before You Start

Before configuring anything, prepare the basics. You need at least two devices (for example, a Windows 11 workstation and an Ubuntu server), a stable network connection, and permission to install software. If your devices will sync over the internet (not just on the same LAN), you should also have access to your router/firewall settings for optional port forwarding.

Syncthing does not require a central server. Each device runs the same software and participates equally. The only thing you must protect carefully is the device pairing process, because that determines which machines are trusted to access your data.

Step 1: Install Syncthing on Windows

On Windows, the cleanest approach is to use the official Syncthing for Windows package. Download it from the official site and extract it into a dedicated folder such as C:\Syncthing. Launch syncthing.exe once to initialize the configuration and open the web interface.

To make Syncthing reliable, configure it to run automatically. A common method is to install it as a background startup task using Windows Task Scheduler. Create a task that runs at user logon (or at system startup if appropriate), points to syncthing.exe, and uses the “Run whether user is logged on or not” option for always-on syncing.

Step 2: Install Syncthing on Linux (Systemd Service)

On modern Linux distributions, installing Syncthing from your package manager is straightforward. After installation, enable it as a user service so it restarts automatically after reboots. This gives you a robust “daemon-like” setup without needing a desktop session.

Once enabled, the Syncthing web UI typically binds to 127.0.0.1:8384 by default. If you are managing a headless server, use SSH port forwarding to access it securely from your workstation rather than exposing the UI publicly.

Step 3: Secure the Web Interface (Do This Early)

Open the Syncthing web interface and go to settings. Set a strong GUI username and password. Even if you only plan to use it on your LAN, credentials prevent accidental access and reduce risk if a port is ever opened incorrectly.

If you must access the GUI from another machine, avoid binding it to all interfaces unless you have a clear firewall rule and a trusted network. In many environments, SSH tunneling is the safest and simplest choice.

Step 4: Pair Your Devices (Trusted Device Setup)

Each Syncthing node has a unique Device ID. To connect two systems, add one device to the other using this ID. In the web UI, choose “Add Remote Device,” paste the Device ID, and give it a recognizable name like Win-Workstation or Ubuntu-NAS.

When the second device receives the pairing request, accept it. At this point, the devices can communicate securely. Syncthing uses encrypted transport and validates identities using those Device IDs, which is why you should only exchange IDs over a trusted channel (not in public chat logs).

Step 5: Create and Share a Sync Folder

Now create a folder on the first device. Click “Add Folder,” set a clear folder label (for example, Projects), and select a folder path such as D:\Projects on Windows or /srv/sync/projects on Linux.

When adding the folder, choose which remote device(s) should receive it. On the receiving device, Syncthing will prompt you to accept the shared folder and choose a local path. Be careful here: selecting the wrong directory can cause files to sync into an unexpected location and create confusion later.

Step 6: Choose the Right Folder Type (Send/Receive vs One-Way)

Syncthing offers multiple folder types. For most two-way collaboration, use Send & Receive. If you want a one-way replica (for example, a workstation pushing data to a Linux backup box), configure the source as Send Only and the target as Receive Only. This prevents accidental deletions or edits on the backup side from syncing back and damaging your primary copy.

For important data, one-way replication is often safer. It behaves like a continuous mirroring job, but still gives you Syncthing’s speed, versioning options, and cross-platform support.

Step 7: Improve Reliability with Versioning and Ignore Rules

If you want protection against accidental deletion or ransomware-like mass changes, enable File Versioning in the folder settings. A common choice is “Staggered File Versioning,” which keeps older versions for longer periods. This is not a full backup solution, but it can save you when a file is overwritten or removed by mistake.

Also consider ignore patterns. You can exclude temporary files, caches, and build outputs that don’t belong in a sync workflow. Ignoring unnecessary files reduces CPU load, database size, and sync churn.

Step 8: Network and Firewall Tips (LAN and Remote Sync)

On a typical LAN, Syncthing works with no firewall changes because it can discover peers automatically. For remote syncing, it can still work using Syncthing’s relay system, but performance is usually better with direct connections.

If you control both ends and want consistent direct connectivity, you can forward Syncthing’s default listening port 22000/TCP to the internal device. Keep security in mind: only forward what you must, and ensure the GUI port is not exposed. In business environments, a VPN is often the cleaner solution for remote syncing.

Quick Troubleshooting Checklist

If syncing is not happening, start with simple checks. Confirm both devices show “Connected” in the web UI, verify you accepted the folder share on the target device, and make sure the folder paths actually exist and have correct permissions. On Linux, permission issues are a frequent cause of “out of sync” behavior.

If devices are “Disconnected,” check local firewalls, confirm both systems have correct time settings, and try disabling and re-enabling the connection. You can also review Syncthing logs in the web UI to identify port conflicts, rejected connections, or permission errors.

Final Notes: Sync is Not a Backup

Syncthing is excellent for real-time file replication, but it is not a complete backup strategy by itself. If you need disaster recovery, pair this setup with periodic offline backups or immutable snapshots. A strong combo is Syncthing for fast syncing plus a separate backup tool for long-term retention.

With the steps above, you now have a secure, cloud-free synchronization system that runs on Windows and Linux, survives reboots, and can be tuned for two-way collaboration or one-way protected replication.

Why run a local AI assistant?

Cloud AI tools are convenient, but they are not always the best fit for technical teams. Running an AI model locally can help when you need to keep data on-premises, reduce recurring API costs, work offline, or experiment with custom prompts without sending logs to third-party services. In this tutorial, you will set up Ollama on Ubuntu and run a modern large language model locally. You will also expose a small HTTP endpoint so your helpdesk tools, scripts, or internal apps can query the model in a controlled way.

Prerequisites

You will need an Ubuntu machine (22.04 or newer is ideal), at least 8 GB RAM (16 GB recommended), and enough disk space for models (many are 4–10 GB each). CPU-only is fine for testing; a supported NVIDIA GPU will improve speed significantly. You should also have sudo access and basic terminal familiarity.

Step 1: Update Ubuntu and install basic tools

Start by updating packages and installing common utilities. This keeps the system clean and avoids dependency issues during installation.

Run:

sudo apt update && sudo apt -y upgrade
sudo apt -y install curl ca-certificates jq

Step 2: Install Ollama

Ollama provides a simple way to download and run models locally with a consistent CLI and a built-in service. Install it using the official script.

Run:

curl -fsSL https://ollama.com/install.sh | sudo sh

After installation, confirm that the service is available and the CLI responds.

ollama --version

Step 3 (Optional): Enable NVIDIA GPU acceleration

If you have an NVIDIA GPU, install the recommended driver and verify it is working. GPU support can drastically reduce response time for larger prompts.

Run:

sudo ubuntu-drivers autoinstall
reboot

After reboot:

nvidia-smi

If you see GPU details, you are ready. If not, confirm Secure Boot settings and driver installation. Ollama typically detects available acceleration automatically.

Step 4: Download and run a model

Now pull a model. A common starting point is a lightweight instruct model that performs well on general tasks. Example:

ollama pull llama3.2

Run an interactive session:

ollama run llama3.2

Ask something practical, such as: “Draft a troubleshooting checklist for DNS resolution issues on Ubuntu.” If you get coherent output, the base setup is complete.

Step 5: Use the local HTTP API

Ollama includes an API endpoint on the local machine. This makes it easy to integrate with scripts, ticketing systems, or internal tools. From the same server, test a generation request.

Run:

curl http://localhost:11434/api/generate -d '{
"model": "llama3.2",
"prompt": "Write a 6-step incident response note for a failed backup job.",
"stream": false
}' | jq -r '.response'

If you see a clear response, your local model is ready for automation.

Step 6: Create a simple “chat gateway” service (safe internal use)

For internal teams, it is often helpful to provide a tiny wrapper service that your tools can call without exposing the full Ollama interface. The example below uses Python and forwards requests to Ollama while keeping the endpoint minimal.

Install Python tools:

sudo apt -y install python3 python3-venv

Create a small app:

mkdir -p ~/ollama-gateway && cd ~/ollama-gateway
python3 -m venv .venv
. .venv/bin/activate
pip install flask requests

Create app.py with the following content:

from flask import Flask, request, jsonify
import requests

app = Flask(__name__)

@app.post("/chat")
def chat():
data = request.get_json(force=True)
prompt = data.get("prompt", "").strip()
if not prompt:
return jsonify({"error": "Missing prompt"}), 400

payload = {
"model": data.get("model", "llama3.2"),
"prompt": prompt,
"stream": False
}
r = requests.post("http://127.0.0.1:11434/api/generate", json=payload, timeout=120)
r.raise_for_status()
return jsonify({"response": r.json().get("response", "")})

if __name__ == "__main__":
app.run(host="127.0.0.1", port=8088)

Run it:

python app.py

Test in another terminal:

curl -s http://127.0.0.1:8088/chat -H "Content-Type: application/json" -d '{"prompt":"Summarize the key steps to fix a full disk on Linux."}' | jq

Step 7: Secure and operationalize the setup

Keep the API bound to 127.0.0.1 unless you have a clear need to expose it. If you must allow LAN access, put it behind a reverse proxy with authentication and strict firewall rules. Also consider separating prompts from sensitive data: even local tools can leak secrets via logs or copied outputs. For reliability, you can convert the gateway into a systemd service later, but even a basic local-only endpoint is useful for automation and experimentation.

Common troubleshooting tips

Slow responses: use a smaller model, reduce prompt length, or enable GPU acceleration. Out of memory: close other apps, add swap (temporary fix), or choose a smaller model variant. Port issues: confirm Ollama listens on 11434 locally and that your gateway points to 127.0.0.1. Model not found: run ollama list and verify the model name matches.

What you can build next

With a local model running, you can create internal knowledge assistants for helpdesk teams, draft incident updates, summarize logs, or generate standard operating procedures. The biggest win is control: you decide where prompts go, how access works, and what gets stored. Once this is stable, try adding model-specific system prompts for consistent tone, or integrate the gateway into a chatbot UI used by your team.

Running an AI assistant locally is no longer just a hobby project. With today’s efficient open models and lightweight runtimes, you can build a private “ChatGPT-style” environment on a Linux machine without sending prompts to any cloud service. This tutorial shows how to install Ollama (for downloading and serving models) and Open WebUI (a clean web interface) on Ubuntu/Debian-based systems. The result is a fast local chatbot you can use for troubleshooting, documentation drafts, code explanations, and more—while keeping your data on your own hardware.

What You Will Build

By the end of this guide, you will have: (1) Ollama installed and running as a service, (2) at least one model pulled and tested from the terminal, and (3) Open WebUI running in Docker and connected to Ollama. This setup works well on a modern CPU, and it can be accelerated if you have a compatible GPU, but GPU support is optional for getting started.

Prerequisites

You need a Linux server or workstation (Ubuntu 22.04/24.04 or Debian 12 is ideal), a user with sudo access, and at least 8 GB RAM for smaller models (16 GB+ is recommended for smoother performance). You also need enough disk space for model files; many popular models take several gigabytes each.

Step 1: Update the System

First, update packages and reboot if your kernel or core libraries are upgraded:

Commands:

sudo apt update && sudo apt -y upgrade
sudo reboot

Step 2: Install Ollama

Ollama provides a simple way to download and run large language models locally. It also exposes an HTTP API on your machine, which Open WebUI can talk to.

Install Ollama:

curl -fsSL https://ollama.com/install.sh | sh

After installation, confirm the service is running:

systemctl status ollama --no-pager

If it is not active, start and enable it:

sudo systemctl enable --now ollama

Step 3: Pull a Model and Test It

Next, download a model. For many users, a good starting point is a smaller model that runs comfortably on CPU. Choose one that matches your hardware and use case.

Example (pull and run a model):

ollama pull llama3.2
ollama run llama3.2

Type a prompt such as: “Explain how DNS caching works in Linux.” If you get a sensible response, Ollama is working.

To see what models you have installed:

ollama list

Step 4: Install Docker (for Open WebUI)

Open WebUI is easiest to deploy in a container. If Docker is not installed, add it using the official Ubuntu/Debian packages:

sudo apt -y install docker.io
sudo systemctl enable --now docker

Optional but convenient: allow your user to run Docker without sudo (log out and back in after this):

sudo usermod -aG docker $USER

Step 5: Run Open WebUI and Connect It to Ollama

Ollama listens locally (typically on port 11434). Open WebUI will run on port 3000. The key is to give the container access to the host’s Ollama API. The most reliable approach on Linux is to use host networking.

Run Open WebUI:

docker run -d --name open-webui --restart unless-stopped --network=host -e OLLAMA_BASE_URL=http://127.0.0.1:11434 -v open-webui:/app/backend/data ghcr.io/open-webui/open-webui:main

Now open your browser and go to:

http://YOUR_SERVER_IP:3000

Create the admin account when prompted. After login, Open WebUI should detect Ollama. If it does not, check the Ollama URL in settings and confirm Ollama is running.

Step 6: Basic Security and Network Tips

If this system is reachable over a network, treat it like any internal web service. At minimum, restrict access to port 3000 with a firewall or run it behind a reverse proxy with TLS. On Ubuntu, you can use UFW to allow only your LAN subnet or a specific IP.

Example (allow only a trusted subnet):

sudo ufw allow from 192.168.1.0/24 to any port 3000 proto tcp
sudo ufw enable

If you are deploying for multiple users, consider placing Open WebUI behind Nginx with HTTPS and basic authentication or SSO, depending on your environment.

Troubleshooting Checklist

Open WebUI loads but shows no models: Make sure you successfully ran ollama pull and that the container can reach http://127.0.0.1:11434. Using --network=host usually fixes connectivity issues on Linux.

Slow responses: Try a smaller model, close other memory-heavy apps, or upgrade RAM. Local AI performance is heavily tied to available memory bandwidth and CPU speed when running without GPU.

Ollama service not running: Check logs with journalctl -u ollama -n 100 --no-pager and verify you have enough free disk space for the model cache.

Conclusion

With Ollama and Open WebUI, you can run a practical AI assistant entirely on your own Linux system. This approach is ideal for homelabs, IT teams, and privacy-focused users who want modern AI capabilities without exposing internal prompts or data to external providers. Once the basics are working, you can experiment with different models, create custom system prompts for your helpdesk workflow, and even integrate the Ollama API into scripts and internal tools.

Why a local AI assistant is worth it

Cloud AI tools are convenient, but they also raise real concerns: sensitive prompts, internal documents, compliance rules, and recurring subscription costs. A local AI assistant can solve many of these issues by keeping your data on your own machine while still delivering fast, useful responses. In this tutorial, you will set up a private AI assistant on Linux using Ollama (for running local LLMs) and Open WebUI (a clean web interface that feels like a chat product). The result is a browser-based AI assistant available on your LAN, with no external API keys.

What you will build

By the end of this guide, you will have: (1) Ollama installed and running as a service, (2) at least one model downloaded and tested from the command line, and (3) Open WebUI running in Docker and connected to Ollama. This setup works well for homelabs, helpdesk teams, and developers who want a private “ChatGPT-style” assistant for drafting emails, explaining logs, generating scripts, and summarizing text.

Prerequisites

You need a modern Linux machine (Ubuntu 22.04/24.04, Debian 12, Fedora, etc.), at least 8 GB RAM (16 GB recommended), and 20+ GB free disk space depending on the model you download. A GPU is optional; many models run on CPU, just slower. You also need curl and Docker (or Docker Engine) for the WebUI portion.

Step 1: Install Ollama

Ollama provides a simple way to download and run LLMs locally. Install it with the official script:

Command: curl -fsSL https://ollama.com/install.sh | sh

After installation, confirm the service is running:

Command: systemctl status ollama

If your distro does not use systemd, you can still run Ollama manually, but most server installs will use systemd by default.

Step 2: Download a model and test it

Next, pull a model. For general-purpose tasks, many people start with a small, fast instruction model. Try one of these based on your hardware: llama3.2 (smaller), qwen2.5 (strong reasoning), or another model you prefer. Example:

Command: ollama pull llama3.2

Now run a quick test in your terminal:

Command: ollama run llama3.2

Type a prompt like: Explain what DNS does in simple terms. If you get a response, Ollama is working correctly.

Step 3: Install Docker (if needed)

Open WebUI is easiest to deploy with Docker. On Ubuntu, you can install Docker Engine from the official repository, but for many lab environments the packaged version is sufficient. If Docker is already installed, verify it:

Command: docker --version

Also ensure your user can run Docker without sudo (optional but convenient). If you add yourself to the docker group, log out and back in for it to take effect.

Step 4: Run Open WebUI and connect it to Ollama

Ollama listens locally on port 11434 by default. Open WebUI will connect to it via an environment variable. Run Open WebUI like this:

Command:

docker run -d --name open-webui --restart unless-stopped -p 3000:8080 -e OLLAMA_BASE_URL=http://host.docker.internal:11434 -v open-webui:/app/backend/data ghcr.io/open-webui/open-webui:main

On many Linux systems, host.docker.internal may not resolve by default. If the WebUI cannot reach Ollama, use Docker’s host networking (simple for trusted LAN setups):

Alternative command:

docker run -d --name open-webui --restart unless-stopped --network=host -e OLLAMA_BASE_URL=http://127.0.0.1:11434 -v open-webui:/app/backend/data ghcr.io/open-webui/open-webui:main

After the container starts, open your browser and visit http://localhost:3000 (or the server IP with the same port). Create an admin account when prompted. Open WebUI should automatically detect the models you pulled with Ollama.

Step 5: Make it accessible on your LAN (optional)

If you want other machines to access the WebUI, ensure your firewall allows inbound TCP on port 3000. On Ubuntu with UFW, for example, allow it explicitly. Keep the service private to your LAN and avoid exposing it to the internet unless you add proper authentication, TLS, and a reverse proxy.

Troubleshooting tips

WebUI shows “No models”: Make sure you pulled a model with ollama pull and that Open WebUI can reach Ollama at http://127.0.0.1:11434. If you used bridge networking, test name resolution and connectivity from inside the container.

Slow responses: CPU-only inference can be slow on larger models. Try a smaller model, close other heavy applications, or consider a machine with more RAM and a supported GPU.

Disk fills up quickly: Models are large. Remove unused ones with ollama rm <model> and keep an eye on your Docker volume usage as well.

Next steps: making it production-friendly

Once the basics work, you can harden the deployment: put Open WebUI behind Nginx with HTTPS, restrict access by IP, and keep your models curated for your team’s use cases (helpdesk knowledge, scripting assistance, log explanation). The biggest win is control: your prompts stay local, your costs are predictable, and your assistant is always available—even when the internet is not.