LifeBytes Journal

Why run AI locally?

If you work in IT, helpdesk, development, or sysadmin roles, you probably paste logs, configs, or customer data into tools to get quick answers. The problem is that cloud AI services can be expensive, limited, or simply not allowed in regulated environments. Running a local AI stack on your own Linux box gives you privacy, predictable performance, and the ability to keep everything inside your network.

In this tutorial you will install Ollama (a lightweight local LLM runner) and Open WebUI (a clean web interface) on Ubuntu. You will end up with a browser-based “ChatGPT-style” experience that talks to models running on your machine. This setup works on CPU-only systems and can also use an NVIDIA GPU if you have one.

What you will build

You will install Ollama as a service, then run Open WebUI in Docker and connect it to Ollama. After that, you will download a model (for example, Llama or Mistral variants), confirm it responds, and finally secure and persist the environment for daily use.

Requirements

You need an Ubuntu system (22.04 or newer is ideal), at least 8 GB RAM (16 GB+ recommended), and enough free disk space (models can range from a few GB to tens of GB). For the web UI portion you should have Docker installed. If you want GPU acceleration, you will also need a compatible NVIDIA driver and the NVIDIA Container Toolkit.

Step 1: Update Ubuntu and install Docker

Start by updating packages and installing Docker. If Docker is already installed, you can skip the installation step and just confirm it works.

Commands:

sudo apt update && sudo apt -y upgrade
sudo apt -y install ca-certificates curl gnupg
sudo apt -y install docker.io
sudo systemctl enable --now docker
sudo usermod -aG docker $USER

Log out and log back in so your user can run Docker without sudo. Verify:

docker run --rm hello-world

Step 2: Install Ollama

Ollama is simple to install and runs as a background service. It exposes an API on your machine, which the web interface will use.

Commands:

curl -fsSL https://ollama.com/install.sh | sh

Confirm the service is running:

systemctl status ollama --no-pager

Step 3: Pull a model and test Ollama from the terminal

Now download a model. A good starting point is a smaller model that runs well on CPU. If you have more RAM and want better responses, choose a larger one. The exact names can change over time, but these examples are commonly available.

Commands:

ollama pull llama3.1
ollama run llama3.1

When prompted, ask something practical like: “Explain what this Nginx error means and how to fix it.” If you get a sensible answer, Ollama is working.

Step 4: Run Open WebUI with Docker

Open WebUI provides a friendly interface, chat history, and a simple model selector. We will run it as a container and point it at the Ollama API.

Commands:

docker volume create open-webui
docker run -d --name open-webui --restart unless-stopped \
-p 3000:8080 \
-e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
-v open-webui:/app/backend/data \
ghcr.io/open-webui/open-webui:main

On Linux, host.docker.internal may not resolve on some setups. If the WebUI cannot connect to Ollama, rerun the container using the host network mode instead:

docker rm -f open-webui
docker run -d --name open-webui --restart unless-stopped \
--network=host \
-v open-webui:/app/backend/data \
ghcr.io/open-webui/open-webui:main

Open your browser and visit http://localhost:3000 (or your server IP if remote). Create an admin account when prompted. You should see your Ollama models listed in the UI.

Step 5: Optional GPU acceleration (NVIDIA)

If you have an NVIDIA GPU, install the correct driver first, then add the NVIDIA Container Toolkit if you plan to run GPU-enabled containers. Ollama itself can use the GPU on the host if the drivers are correctly installed. Confirm your GPU is visible:

nvidia-smi

If nvidia-smi works, test performance by running a model and watching GPU utilization in another terminal. If you see GPU usage increase during generation, your local AI is accelerated.

Step 6: Basic hardening and useful tips

If this server is on a network, avoid exposing the WebUI to the entire internet. Place it behind a reverse proxy with authentication (for example, Nginx with basic auth) or restrict access at the firewall. Also remember that models can store chat history in the WebUI volume, so treat the data directory like sensitive application data and back it up appropriately.

A few practical tips: keep an eye on disk usage as you try different models, standardize on one or two “default” models for your team, and document prompts for your most common workflows (log analysis, PowerShell troubleshooting, ticket replies, and postmortems). Local AI is at its best when it is part of a repeatable process, not just a toy.

Conclusion

With Ollama and Open WebUI, you can run a capable local AI assistant on Ubuntu in under an hour, using either CPU-only hardware or an NVIDIA GPU for faster output. The result is a private, controllable tool that can help with troubleshooting, scripting ideas, documentation drafts, and day-to-day IT tasks—without sending your data to external services.

Why run a local AI assistant?

Cloud AI tools are convenient, but they are not always the best fit for technical teams. Running an AI model locally can help when you need to keep data on-premises, reduce recurring API costs, work offline, or experiment with custom prompts without sending logs to third-party services. In this tutorial, you will set up Ollama on Ubuntu and run a modern large language model locally. You will also expose a small HTTP endpoint so your helpdesk tools, scripts, or internal apps can query the model in a controlled way.

Prerequisites

You will need an Ubuntu machine (22.04 or newer is ideal), at least 8 GB RAM (16 GB recommended), and enough disk space for models (many are 4–10 GB each). CPU-only is fine for testing; a supported NVIDIA GPU will improve speed significantly. You should also have sudo access and basic terminal familiarity.

Step 1: Update Ubuntu and install basic tools

Start by updating packages and installing common utilities. This keeps the system clean and avoids dependency issues during installation.

Run:

sudo apt update && sudo apt -y upgrade
sudo apt -y install curl ca-certificates jq

Step 2: Install Ollama

Ollama provides a simple way to download and run models locally with a consistent CLI and a built-in service. Install it using the official script.

Run:

curl -fsSL https://ollama.com/install.sh | sudo sh

After installation, confirm that the service is available and the CLI responds.

ollama --version

Step 3 (Optional): Enable NVIDIA GPU acceleration

If you have an NVIDIA GPU, install the recommended driver and verify it is working. GPU support can drastically reduce response time for larger prompts.

Run:

sudo ubuntu-drivers autoinstall
reboot

After reboot:

nvidia-smi

If you see GPU details, you are ready. If not, confirm Secure Boot settings and driver installation. Ollama typically detects available acceleration automatically.

Step 4: Download and run a model

Now pull a model. A common starting point is a lightweight instruct model that performs well on general tasks. Example:

ollama pull llama3.2

Run an interactive session:

ollama run llama3.2

Ask something practical, such as: “Draft a troubleshooting checklist for DNS resolution issues on Ubuntu.” If you get coherent output, the base setup is complete.

Step 5: Use the local HTTP API

Ollama includes an API endpoint on the local machine. This makes it easy to integrate with scripts, ticketing systems, or internal tools. From the same server, test a generation request.

Run:

curl http://localhost:11434/api/generate -d '{
"model": "llama3.2",
"prompt": "Write a 6-step incident response note for a failed backup job.",
"stream": false
}' | jq -r '.response'

If you see a clear response, your local model is ready for automation.

Step 6: Create a simple “chat gateway” service (safe internal use)

For internal teams, it is often helpful to provide a tiny wrapper service that your tools can call without exposing the full Ollama interface. The example below uses Python and forwards requests to Ollama while keeping the endpoint minimal.

Install Python tools:

sudo apt -y install python3 python3-venv

Create a small app:

mkdir -p ~/ollama-gateway && cd ~/ollama-gateway
python3 -m venv .venv
. .venv/bin/activate
pip install flask requests

Create app.py with the following content:

from flask import Flask, request, jsonify
import requests

app = Flask(__name__)

@app.post("/chat")
def chat():
data = request.get_json(force=True)
prompt = data.get("prompt", "").strip()
if not prompt:
return jsonify({"error": "Missing prompt"}), 400

payload = {
"model": data.get("model", "llama3.2"),
"prompt": prompt,
"stream": False
}
r = requests.post("http://127.0.0.1:11434/api/generate", json=payload, timeout=120)
r.raise_for_status()
return jsonify({"response": r.json().get("response", "")})

if __name__ == "__main__":
app.run(host="127.0.0.1", port=8088)

Run it:

python app.py

Test in another terminal:

curl -s http://127.0.0.1:8088/chat -H "Content-Type: application/json" -d '{"prompt":"Summarize the key steps to fix a full disk on Linux."}' | jq

Step 7: Secure and operationalize the setup

Keep the API bound to 127.0.0.1 unless you have a clear need to expose it. If you must allow LAN access, put it behind a reverse proxy with authentication and strict firewall rules. Also consider separating prompts from sensitive data: even local tools can leak secrets via logs or copied outputs. For reliability, you can convert the gateway into a systemd service later, but even a basic local-only endpoint is useful for automation and experimentation.

Common troubleshooting tips

Slow responses: use a smaller model, reduce prompt length, or enable GPU acceleration. Out of memory: close other apps, add swap (temporary fix), or choose a smaller model variant. Port issues: confirm Ollama listens on 11434 locally and that your gateway points to 127.0.0.1. Model not found: run ollama list and verify the model name matches.

What you can build next

With a local model running, you can create internal knowledge assistants for helpdesk teams, draft incident updates, summarize logs, or generate standard operating procedures. The biggest win is control: you decide where prompts go, how access works, and what gets stored. Once this is stable, try adding model-specific system prompts for consistent tone, or integrate the gateway into a chatbot UI used by your team.