LifeBytes Journal

Why run a local AI assistant?

Cloud AI tools are convenient, but a local setup can be faster for repeated tasks, cheaper over time, and more private for sensitive notes, logs, or internal documentation. Running an AI model locally is also a great way to learn modern AI tooling without committing to a paid API. In this guide, you will install Ollama on Linux, download a model, test it from the terminal, and optionally add a lightweight web interface for a more comfortable chat experience.

Prerequisites

You need a Linux machine (Ubuntu/Debian/Fedora/Arch all work), at least 8 GB RAM for smaller models, and preferably a modern CPU. A GPU is helpful but not required for many models. You will also need curl and basic terminal access with sudo privileges.

Step 1: Install Ollama

Ollama provides a simple installer for Linux. Open a terminal and run:

curl -fsSL https://ollama.com/install.sh | sh

After installation, verify the service is available:

ollama --version

On most systems, Ollama runs as a background service. If you want to check its status on a systemd-based distribution, use:

systemctl status ollama

Step 2: Download a model (and understand what you are pulling)

With Ollama, you download models using the pull command. A good starting point for general chat is a smaller, responsive model. For example:

ollama pull llama3.1

If disk space or RAM is limited, consider smaller variants (often labeled with fewer parameters). If you want code-focused answers, try a coding model such as:

ollama pull codellama

Model size matters. Larger models typically produce better results but require more RAM and may run slower. If performance feels sluggish, choose a smaller model rather than assuming something is broken.

Step 3: Chat with the model from the terminal

To start a chat session:

ollama run llama3.1

You can now type prompts and get replies immediately. This is perfect for quick tasks like generating a bash one-liner, summarizing a local change log, or drafting troubleshooting steps.

For scripting, you can also pass a prompt directly:

ollama run llama3.1 "Write a systemd unit that restarts a service on failure."

Step 4: Enable remote access safely (optional but common)

By default, many local AI setups listen only on localhost for safety. If you want to use Ollama from another machine on your LAN, you need to bind it carefully and protect it with firewall rules. First, check what address Ollama is listening on:

ss -tulpen | grep 11434

If you decide to expose it, do it on a trusted network only, and restrict access to specific IPs. On Ubuntu with UFW, for example, you can allow a single workstation:

sudo ufw allow from 192.168.1.50 to any port 11434

Avoid opening the port to the public internet. A local AI endpoint without authentication is not something you want exposed.

Step 5: Add a simple web UI (Open WebUI)

Terminal chat is efficient, but a web interface makes long conversations easier and adds quality-of-life features. One popular option is Open WebUI, which can connect to Ollama. The easiest deployment is with Docker. If Docker is not installed, install it from your distribution’s official docs first.

Run Open WebUI as a container:

docker run -d --name open-webui -p 3000:8080 -e OLLAMA_BASE_URL=http://host.docker.internal:11434 -v open-webui:/app/backend/data --restart unless-stopped ghcr.io/open-webui/open-webui:main

On some Linux hosts, host.docker.internal may not resolve by default. If that happens, use your host’s LAN IP (for example, http://192.168.1.10:11434) or add the Docker host gateway option depending on your Docker version. Once the container is running, open:

http://localhost:3000

Complete the initial setup in the browser, then select the Ollama model you downloaded. You should be able to chat immediately through the UI while Ollama continues doing the inference locally.

Troubleshooting tips (the issues people actually hit)

Model is slow or the system becomes unresponsive: Use a smaller model, close memory-heavy apps, or move to a machine with more RAM. Local AI is RAM-hungry, and swapping to disk will kill performance.

Ollama service is not running: Restart it with sudo systemctl restart ollama and check logs using journalctl -u ollama --no-pager -n 100.

Web UI cannot connect to Ollama: Confirm Ollama is reachable at http://127.0.0.1:11434 from the host, then adjust the Open WebUI environment variable OLLAMA_BASE_URL to point to the correct address.

Next steps

Once your local assistant is working, you can create repeatable prompts for helpdesk replies, generate configuration templates, or summarize technical notes without sending data to a third party. For better results, experiment with different models and keep your prompts specific. Local AI gets impressive quickly when you give it clear context and constraints.

Running an AI assistant locally is no longer a science project. With modern open-source tools, you can host a private chatbot on your own Linux machine, keep sensitive data off third-party servers, and still get fast, high-quality responses. In this tutorial, you’ll install Ollama (a lightweight local LLM runtime) and Open WebUI (a clean web interface) to create a self-hosted AI assistant you can access from your browser.

This guide targets Ubuntu/Debian-based systems, but the same approach works on many other Linux distributions with small adjustments. The setup is great for IT documentation drafting, code review, internal knowledge-base Q&A, and quick command-line help—without sending prompts to the cloud.

What You’ll Build

By the end, you will have:

1) Ollama installed and running as a local service
2) A model downloaded and ready to use (for example, Llama 3.x class models)
3) Open WebUI running in Docker, connected to Ollama
4) Optional remote access for your LAN with basic safety notes

Prerequisites

Before you start, make sure you have:

• A Linux server or workstation (8 GB RAM minimum; 16 GB+ recommended)
• At least 15–30 GB free disk space (model files can be large)
• Sudo access
• Docker installed (for Open WebUI)

Step 1: Install Ollama on Linux

Ollama provides a simple way to download and run local models. Install it with the official script:

Command:

curl -fsSL https://ollama.com/install.sh | sh

After installation, confirm the service is working:

ollama --version

If your system uses systemd, Ollama typically runs as a service. You can also test it by listing models (it will likely be empty at first):

ollama list

Step 2: Pull a Model and Test It

Now download a model. A common starting point is a Llama-family instruct model. Pull it using:

ollama pull llama3

Once the download completes, run a quick interactive test:

ollama run llama3

Type a short question (for example, “Explain systemd targets in simple terms”) and confirm you get a response. If this works, your local AI runtime is ready.

Step 3: Install Docker (If Needed)

If Docker is not installed yet, install it on Ubuntu/Debian with:

sudo apt update && sudo apt install -y docker.io

Enable and start Docker:

sudo systemctl enable --now docker

Optional but useful: allow your user to run Docker without sudo (log out and back in after this):

sudo usermod -aG docker $USER

Step 4: Run Open WebUI and Connect It to Ollama

Open WebUI provides a friendly ChatGPT-like interface and supports Ollama as a backend. Start it with Docker:

docker run -d --name open-webui \
-p 3000:8080 \
-e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
-v open-webui:/app/backend/data \
--restart unless-stopped \
ghcr.io/open-webui/open-webui:main

On Linux, host.docker.internal may not be available by default on older Docker versions. If your WebUI can’t connect, rerun the container using the host network mode:

docker rm -f open-webui
docker run -d --name open-webui \
--network=host \
-e OLLAMA_BASE_URL=http://127.0.0.1:11434 \
-v open-webui:/app/backend/data \
--restart unless-stopped \
ghcr.io/open-webui/open-webui:main

Open your browser and go to:

http://localhost:3000

Create the admin account when prompted. After login, you should see your Ollama model available in the model selector. Start a chat and confirm it responds.

Step 5: Make It Usable on Your Local Network (Optional)

If you want to access the WebUI from another device on your LAN, ensure the server firewall allows TCP port 3000. On Ubuntu with UFW:

sudo ufw allow 3000/tcp

Then browse to:

http://YOUR_SERVER_IP:3000

Security note: Don’t expose this directly to the internet without authentication and TLS. If you need remote access, put it behind a VPN (WireGuard is a solid choice) or a reverse proxy with HTTPS.

Troubleshooting Tips

WebUI loads but no models appear: Verify Ollama is running and reachable. On the host, test: curl http://127.0.0.1:11434. If Docker networking is the issue, use the --network=host method.

Slow responses: Try a smaller model, close heavy applications, or run on a machine with more RAM/CPU. Local LLM performance is mostly hardware-dependent.

Disk fills up quickly: Models can consume many gigabytes. Remove unused models with: ollama list then ollama rm MODELNAME.

Wrap-Up

With Ollama and Open WebUI, you can run a capable private AI assistant on Linux in under an hour. It’s an excellent setup for IT pros, developers, and small teams who want AI features without cloud costs or privacy concerns. Once it’s working, you can experiment with different models, create prompt presets, and build a local workflow that feels like a modern AI platform—fully under your control.

Why run AI locally?

If you work in IT, helpdesk, development, or sysadmin roles, you probably paste logs, configs, or customer data into tools to get quick answers. The problem is that cloud AI services can be expensive, limited, or simply not allowed in regulated environments. Running a local AI stack on your own Linux box gives you privacy, predictable performance, and the ability to keep everything inside your network.

In this tutorial you will install Ollama (a lightweight local LLM runner) and Open WebUI (a clean web interface) on Ubuntu. You will end up with a browser-based “ChatGPT-style” experience that talks to models running on your machine. This setup works on CPU-only systems and can also use an NVIDIA GPU if you have one.

What you will build

You will install Ollama as a service, then run Open WebUI in Docker and connect it to Ollama. After that, you will download a model (for example, Llama or Mistral variants), confirm it responds, and finally secure and persist the environment for daily use.

Requirements

You need an Ubuntu system (22.04 or newer is ideal), at least 8 GB RAM (16 GB+ recommended), and enough free disk space (models can range from a few GB to tens of GB). For the web UI portion you should have Docker installed. If you want GPU acceleration, you will also need a compatible NVIDIA driver and the NVIDIA Container Toolkit.

Step 1: Update Ubuntu and install Docker

Start by updating packages and installing Docker. If Docker is already installed, you can skip the installation step and just confirm it works.

Commands:

sudo apt update && sudo apt -y upgrade
sudo apt -y install ca-certificates curl gnupg
sudo apt -y install docker.io
sudo systemctl enable --now docker
sudo usermod -aG docker $USER

Log out and log back in so your user can run Docker without sudo. Verify:

docker run --rm hello-world

Step 2: Install Ollama

Ollama is simple to install and runs as a background service. It exposes an API on your machine, which the web interface will use.

Commands:

curl -fsSL https://ollama.com/install.sh | sh

Confirm the service is running:

systemctl status ollama --no-pager

Step 3: Pull a model and test Ollama from the terminal

Now download a model. A good starting point is a smaller model that runs well on CPU. If you have more RAM and want better responses, choose a larger one. The exact names can change over time, but these examples are commonly available.

Commands:

ollama pull llama3.1
ollama run llama3.1

When prompted, ask something practical like: “Explain what this Nginx error means and how to fix it.” If you get a sensible answer, Ollama is working.

Step 4: Run Open WebUI with Docker

Open WebUI provides a friendly interface, chat history, and a simple model selector. We will run it as a container and point it at the Ollama API.

Commands:

docker volume create open-webui
docker run -d --name open-webui --restart unless-stopped \
-p 3000:8080 \
-e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
-v open-webui:/app/backend/data \
ghcr.io/open-webui/open-webui:main

On Linux, host.docker.internal may not resolve on some setups. If the WebUI cannot connect to Ollama, rerun the container using the host network mode instead:

docker rm -f open-webui
docker run -d --name open-webui --restart unless-stopped \
--network=host \
-v open-webui:/app/backend/data \
ghcr.io/open-webui/open-webui:main

Open your browser and visit http://localhost:3000 (or your server IP if remote). Create an admin account when prompted. You should see your Ollama models listed in the UI.

Step 5: Optional GPU acceleration (NVIDIA)

If you have an NVIDIA GPU, install the correct driver first, then add the NVIDIA Container Toolkit if you plan to run GPU-enabled containers. Ollama itself can use the GPU on the host if the drivers are correctly installed. Confirm your GPU is visible:

nvidia-smi

If nvidia-smi works, test performance by running a model and watching GPU utilization in another terminal. If you see GPU usage increase during generation, your local AI is accelerated.

Step 6: Basic hardening and useful tips

If this server is on a network, avoid exposing the WebUI to the entire internet. Place it behind a reverse proxy with authentication (for example, Nginx with basic auth) or restrict access at the firewall. Also remember that models can store chat history in the WebUI volume, so treat the data directory like sensitive application data and back it up appropriately.

A few practical tips: keep an eye on disk usage as you try different models, standardize on one or two “default” models for your team, and document prompts for your most common workflows (log analysis, PowerShell troubleshooting, ticket replies, and postmortems). Local AI is at its best when it is part of a repeatable process, not just a toy.

Conclusion

With Ollama and Open WebUI, you can run a capable local AI assistant on Ubuntu in under an hour, using either CPU-only hardware or an NVIDIA GPU for faster output. The result is a private, controllable tool that can help with troubleshooting, scripting ideas, documentation drafts, and day-to-day IT tasks—without sending your data to external services.

Running an AI assistant locally is one of the most practical “advanced” upgrades you can make to a Linux workstation or home lab. You get faster iteration, more privacy, and you avoid sending sensitive text to a third-party cloud service. In this tutorial, you’ll install Ollama (a lightweight local LLM runtime) and Open WebUI (a clean web interface) on Ubuntu, then secure access and confirm everything is working.

This setup is great for a personal helpdesk bot, drafting and summarizing documents, generating scripts, or building an internal knowledge tool. It works on CPU-only systems, but performance improves significantly if you have a modern GPU. The steps below focus on a reliable, repeatable install that you can maintain like any other server service.

Prerequisites

Before you start, make sure you have: Ubuntu 22.04/24.04 (or a compatible Debian-based distro), a user with sudo permissions, at least 8 GB RAM (16 GB recommended for larger models), and roughly 15–30 GB of free disk space depending on which models you download.

Step 1: Update the system

Open a terminal and update your packages to avoid dependency issues:

sudo apt update && sudo apt -y upgrade

Step 2: Install Ollama

Ollama provides a simple command-line experience for downloading and running models. Install it using the official installer:

curl -fsSL https://ollama.com/install.sh | sudo sh

After installation, confirm the service is running:

systemctl status ollama

If it’s not active, start and enable it:

sudo systemctl enable --now ollama

Step 3: Download and test a model

Now pull a model. A good starting point is a smaller “general chat” model to verify your environment first:

ollama pull llama3.1

Then run a quick test:

ollama run llama3.1

Type a prompt like: “Write a bash one-liner to list the 10 largest files in a directory.” If you get a response, the core engine is working.

Step 4: Install Docker (recommended for Open WebUI)

Open WebUI is easiest to deploy in a container. Install Docker using Ubuntu packages:

sudo apt install -y docker.io

Enable the Docker service:

sudo systemctl enable --now docker

Optional but convenient: allow your user to run Docker without sudo (log out and back in after this):

sudo usermod -aG docker $USER

Step 5: Run Open WebUI and connect it to Ollama

Run the container and map it to a local port (3000). We’ll also mount a persistent volume so settings and chat history survive reboots:

docker run -d --name open-webui --restart unless-stopped -p 3000:8080 -e OLLAMA_BASE_URL=http://host.docker.internal:11434 -v open-webui:/app/backend/data ghcr.io/open-webui/open-webui:main

On Linux, host.docker.internal may not work on older Docker builds. If you open the web UI and it cannot reach Ollama, rerun the container using host networking instead:

docker rm -f open-webui

docker run -d --name open-webui --restart unless-stopped --network=host -e OLLAMA_BASE_URL=http://127.0.0.1:11434 -v open-webui:/app/backend/data ghcr.io/open-webui/open-webui:main

Now open your browser and go to http://localhost:3000 (or the server IP with port 3000). Create the first admin account. In most cases, Open WebUI will automatically detect the Ollama endpoint once the environment variable is set.

Step 6: Basic security hardening (don’t skip this)

If this is only for your local machine, binding to localhost is usually enough. If you plan to access it from other devices, you should secure it properly. At a minimum, configure the firewall to allow only trusted networks.

With UFW, you can allow port 3000 only from your LAN (example uses 192.168.1.0/24):

sudo ufw allow from 192.168.1.0/24 to any port 3000 proto tcp

sudo ufw enable

For a more professional setup, place Open WebUI behind Nginx with HTTPS (Let’s Encrypt) and optionally basic auth or SSO. That way, you’re not exposing a plain HTTP admin login to the network.

Step 7: Troubleshooting common issues

Open WebUI can’t see any models: confirm Ollama is running and reachable. Test locally with curl http://127.0.0.1:11434. If the container can’t reach the host, switch to --network=host as shown above.

Model downloads are slow or fail: try again later or switch networks. Large models are multi-GB downloads. Ensure you have enough disk space under /usr/share/ollama (or your configured storage path).

High CPU/RAM usage: use a smaller model, reduce parallel usage, or move the server to a machine with more memory. For older hardware, smaller models typically feel much more responsive.

Final check

At this point you have a private AI assistant running entirely on your own Ubuntu system. Ollama handles the model runtime, and Open WebUI provides an easy interface for chat, prompt testing, and daily use. Once you’re comfortable with the basics, you can explore model choices, system prompts for a “helpdesk” personality, and integrating local documents for internal Q&A workflows.

Running an AI chatbot locally is no longer a research project reserved for labs. With today’s lightweight LLM runtimes, you can host a private assistant on your own Linux machine and keep your data off third-party servers. This tutorial shows how to install Ollama (a simple local LLM runner) and Open WebUI (a clean web interface) using Docker, then load a model and start chatting from your browser.

What you will build

By the end of this guide, you will have a local web-based AI chat interface reachable from your LAN (or just your own PC). You’ll be able to pull models on demand, start conversations, and keep everything on your own storage. The setup works well for home labs, internal IT tools, offline environments, and privacy-focused workflows.

Prerequisites

Recommended system: 64-bit Linux (Ubuntu/Debian/Fedora work fine), at least 8 GB RAM (16 GB is better), and plenty of disk space (models can take several GB). A GPU is optional; CPU-only is still usable with smaller models. You also need admin access (sudo) and an internet connection for the initial downloads.

Step 1: Install Docker

If Docker is not installed, install it using your distro’s package manager or the official Docker repository. On Ubuntu/Debian, you can use:

sudo apt update
sudo apt install -y docker.io
sudo systemctl enable --now docker

To run Docker without typing sudo every time, add your user to the docker group (log out and back in after):

sudo usermod -aG docker $USER

Step 2: Create a working folder and Docker network

A dedicated folder keeps configuration tidy. Create it anywhere you like:

mkdir -p ~/local-ai
cd ~/local-ai

Create a Docker network so containers can reliably talk to each other by name:

docker network create localai

Step 3: Start Ollama (LLM runtime)

Ollama exposes an API that other apps (like WebUI) can use. Start it with a persistent volume so models survive reboots:

docker run -d --name ollama
--network localai
-p 11434:11434
-v ollama:/root/.ollama
ollama/ollama:latest

Verify it is running:

docker ps

Step 4: Pull a model and test from the command line

Now pull a model inside the Ollama container. For a balanced first run, try a smaller model if your RAM is limited. Example:

docker exec -it ollama ollama pull llama3.2

Test a quick prompt:

docker exec -it ollama ollama run llama3.2 "Write a short checklist for patching a Linux server safely."

If you get a response, the core runtime is working.

Step 5: Start Open WebUI (browser chat interface)

Open WebUI provides a friendly UI similar to popular online chat tools, but it stays in your environment. Start it and point it to Ollama using the container name:

docker run -d --name open-webui
--network localai
-p 3000:8080
-e OLLAMA_BASE_URL=http://ollama:11434
-v open-webui:/app/backend/data
ghcr.io/open-webui/open-webui:main

Open your browser and go to:

http://localhost:3000

If you’re accessing from another PC on the network, replace localhost with the Linux server’s IP (for example, http://192.168.1.50:3000).

Step 6: Select the model and start chatting

In Open WebUI, look for the model selector. If your Ollama container already pulled llama3.2, it should appear automatically. Choose it, start a new chat, and try an IT-focused prompt such as “Explain the difference between RAID1 and RAID10 with practical examples.”

Troubleshooting tips (common issues)

WebUI loads but no models appear: Confirm the environment variable is correct and that both containers share the same Docker network. Run docker logs open-webui and look for connection errors to http://ollama:11434.

Slow responses or timeouts: Use a smaller model, close other heavy workloads, and verify you have enough RAM. On CPU-only systems, large models can feel sluggish.

Cannot access from another computer: Make sure port 3000 is allowed through your firewall (UFW, firewalld, or your cloud security group). Also verify Open WebUI is bound via Docker’s port mapping.

Optional: Make it easier with Docker Compose

Once you’re happy with the setup, consider moving these commands into a Docker Compose file for simpler restarts and upgrades. The key idea remains the same: one container runs Ollama on port 11434, another runs Open WebUI on port 3000, and a shared network connects them.

With this local AI stack, you can experiment safely, build internal tools, and keep sensitive prompts under your control. As you get comfortable, try different models, tune prompts for helpdesk automation, or connect the WebUI to documentation snippets for faster internal answers.

Running an AI assistant locally is quickly becoming a practical option for developers and IT teams who want faster responses, offline access, and better control over sensitive code. In this tutorial, you will set up a private, local “ChatGPT-like” interface on a Linux server or workstation using Ollama (to run large language models) and Open WebUI (a web interface you can access from a browser). The result is a self-hosted AI assistant you can use for coding help, troubleshooting, and documentation drafts—without sending prompts to a third-party cloud.

This guide focuses on modern Linux distributions (Ubuntu/Debian-based commands are shown). The same approach works on many other distros with minor package differences. You’ll also learn basic hardening steps so the UI is not accidentally exposed to the internet.

Prerequisites

Hardware: A machine with at least 8 GB RAM is workable for smaller models, but 16 GB+ is recommended. A GPU is optional; many models run on CPU, just slower.

Software: Linux with sudo access, and either Docker (recommended) or Python for Open WebUI. You’ll also want an SSH session if you’re setting this up on a server.

Step 1: Install Ollama

Ollama is a lightweight runtime that downloads and runs models locally. Install it with the official script:

Command:

curl -fsSL https://ollama.com/install.sh | sh

After installation, verify it’s working:

ollama --version

Now pull a model. For a good balance of speed and capability, try a smaller modern model first:

ollama pull llama3.1

Test a quick prompt in the terminal:

ollama run llama3.1

Type a question, press Enter, and confirm you get a response. Exit with /bye or Ctrl+C depending on your session.

Step 2: Install Docker (Recommended)

Open WebUI can be installed in several ways, but Docker keeps it clean and easy to upgrade. Install Docker if you don’t already have it:

sudo apt update
sudo apt install -y docker.io
sudo systemctl enable --now docker

Optional but useful: allow your user to run Docker without sudo (log out and back in afterward):

sudo usermod -aG docker $USER

Step 3: Run Open WebUI and Connect It to Ollama

Open WebUI will provide a browser-based chat interface. Start it with Docker. The easiest approach is to map the container port to your host and point it at the Ollama API.

First, confirm Ollama is running. On many systems it runs as a service automatically after installation. You can check:

sudo systemctl status ollama

Now run Open WebUI:

docker run -d --name open-webui \
-p 3000:8080 \
-e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
-v open-webui:/app/backend/data \
--restart unless-stopped \
ghcr.io/open-webui/open-webui:main

If your Docker setup doesn’t support host.docker.internal on Linux, use the host network or the server IP instead. A common workaround is:

-e OLLAMA_BASE_URL=http://172.17.0.1:11434

Then open your browser to:

http://localhost:3000

Create the admin account on first run. After login, you should see available Ollama models. If you already pulled llama3.1, it should appear in the model list or be selectable.

Step 4: Make It Safe (Local Network Access Without Public Exposure)

By default, mapping -p 3000:8080 may expose the UI on all interfaces. If this is a server, you typically want LAN-only access. A simple method is to bind to a specific interface or localhost. For local-only access:

docker rm -f open-webui
docker run -d --name open-webui \
-p 127.0.0.1:3000:8080 \
-e OLLAMA_BASE_URL=http://127.0.0.1:11434 \
-v open-webui:/app/backend/data \
--restart unless-stopped \
ghcr.io/open-webui/open-webui:main

If you need access from another PC, consider using an SSH tunnel instead of opening a firewall port:

ssh -L 3000:127.0.0.1:3000 user@your-server

Then browse to http://localhost:3000 on your local machine.

Step 5: Common Troubleshooting

Model not showing up: Make sure the model is installed with ollama list. If it isn’t listed, run ollama pull <model>.

Open WebUI can’t connect to Ollama: Confirm Ollama listens on port 11434 and is reachable from the container. Check logs with docker logs open-webui. If needed, try the Docker bridge gateway IP (172.17.0.1) as the base URL.

Slow responses: Use a smaller model, close other memory-heavy apps, or run on a machine with more RAM. CPU-only inference is normal but slower.

Final Notes

With Ollama and Open WebUI, you get a practical local AI assistant that can help write scripts, explain logs, draft runbooks, and speed up troubleshooting—while keeping prompts on your own hardware. Once it’s running, experiment with different models and create reusable “system prompts” for tasks like helpdesk triage, Linux administration, or code review.

Why a local RAG chatbot?

If you work in IT, you probably have internal documents that never belong in a public cloud: runbooks, SOPs, incident postmortems, customer notes, firewall rules, or server inventories. A local chatbot can answer questions from those files without uploading anything outside your network. The modern approach is RAG (Retrieval-Augmented Generation): the system searches your documents for relevant passages and then asks the language model to respond using that context.

In this tutorial you will deploy a practical, self-hosted setup on Linux using Ollama (to run local LLMs) and Open WebUI (a friendly web interface). You will end with a browser-based chat that can be extended with document ingestion features and can run fully offline.

What you will build

Ollama will run the language model on your Linux host. Open WebUI will provide the web UI and manage connections to Ollama. This combination is popular because it’s simple to update, works well with Docker, and supports a “private by default” workflow.

Prerequisites

You need a Linux server or workstation (Ubuntu/Debian/Fedora are fine). Recommended: 16 GB RAM or more, and SSD storage. A GPU is optional; CPU-only works, but responses will be slower. You also need root or sudo access and an internet connection for the initial downloads (you can later run offline).

Step 1: Install Ollama

On most Linux distributions, the quickest method is the official install script. Run the following:

Command:
curl -fsSL https://ollama.com/install.sh | sh

After installation, verify that the service is running:

ollama --version
systemctl status ollama

If your firewall is strict, note that Ollama typically listens on 127.0.0.1:11434 by default (local-only). That’s good for security. You can keep it that way when Open WebUI runs on the same machine.

Step 2: Pull a model with Ollama

Choose a model that matches your hardware. A solid general-purpose starting point is a smaller Llama-family model. Pull a model like this:

ollama pull llama3.1

Then test it quickly:

ollama run llama3.1

Type a short prompt (for example: “Summarize the purpose of RAG in one paragraph.”) and confirm you get a response. Exit with /bye.

Step 3: Install Docker (if needed)

Open WebUI is commonly deployed with Docker. If Docker is not installed, install it using your distro’s recommended method. On Ubuntu, this is typically:

sudo apt update
sudo apt install -y docker.io
sudo systemctl enable --now docker

To avoid running Docker commands with sudo, you can add your user to the docker group (log out and back in afterward):

sudo usermod -aG docker $USER

Step 4: Run Open WebUI connected to Ollama

Start Open WebUI as a container and point it to Ollama. If Ollama is running on the same host, the container can reach it using host networking (simple on Linux):

docker run -d --name open-webui --restart=unless-stopped --network=host -e OLLAMA_BASE_URL=http://127.0.0.1:11434 -v open-webui:/app/backend/data ghcr.io/open-webui/open-webui:main

Open your browser and go to:

http://localhost:8080

Create the first admin account when prompted. Once logged in, confirm that your Ollama model appears in the model list. If it doesn’t, re-check that Ollama is running and that the URL is correct.

Step 5: Basic security hardening (recommended)

If this is more than a lab setup, don’t expose port 8080 directly to the internet. Instead, put it behind a reverse proxy (Nginx/Traefik/Caddy) with HTTPS and authentication. At minimum, restrict access to your LAN via firewall rules. If multiple users will access it, create separate accounts and disable anonymous access in the UI settings.

Step 6: Add documents for RAG (practical approach)

RAG requires two pieces: (1) a place to store your documents and (2) an index/search layer that can retrieve relevant chunks. Many teams start with a controlled folder of PDFs/Markdown/TXT and progressively add ingestion and indexing tools as needs grow.

A simple, safe workflow is:

1) Put sanitized internal docs in a dedicated directory (example: /srv/knowledgebase).
2) Convert “messy” formats to text where possible (Markdown and text files work best).
3) In Open WebUI, look for knowledge or document features (often called “Knowledge,” “Documents,” or “RAG” depending on version) and import your files.

If you do not see document ingestion in your build, treat this deployment as the base LLM layer and add a dedicated RAG service later (for example, a vector database plus an ingestion pipeline). The key advantage is that you already have the model hosting and UI stable and local.

Troubleshooting common issues

Open WebUI can’t see Ollama models: Verify Ollama is running (systemctl status ollama) and confirm the base URL. If you didn’t use --network=host, use Docker’s host gateway options or run both services in the same Docker network and reference Ollama by container name.

Slow responses: Try a smaller model, close other heavy workloads, and ensure you have enough RAM. CPU-only inference is normal but slower. If you have a supported GPU, check Ollama’s documentation for acceleration support on your platform.

High disk usage: Models can be several GB each. Remove unused models with ollama rm <model> and keep only what you use.

Next steps

Once the local chatbot is working, you can improve accuracy and trust by tightening your knowledge base: keep documents current, remove duplicates, and structure key procedures in Markdown. If you expand into a full RAG stack, define clear ingestion rules and access controls so the chatbot only retrieves what each user is allowed to see.

Running an AI chatbot locally is no longer a “lab-only” project. With today’s lightweight models and tools like Ollama and Open WebUI, you can build a private, fast, and surprisingly capable assistant on a Windows PC—without sending prompts to third-party cloud services. This tutorial walks you through a practical setup that works well for IT notes, scripting help, documentation drafts, and troubleshooting ideas, all while keeping your data on your own machine.

This guide focuses on an up-to-date approach: Ollama provides an easy local model runtime, and Open WebUI gives you a clean web interface with chat history, model selection, and basic admin options. The result feels like a polished “ChatGPT-style” experience, but running on your own hardware.

Prerequisites

Before you start, confirm you have the following:

1) A Windows 10/11 system (64-bit). 2) At least 16 GB RAM recommended (8 GB can work with smaller models). 3) Enough disk space for models (5–20 GB depending on what you install). 4) Optional but helpful: an NVIDIA GPU for faster inference. CPU-only still works—just slower.

Step 1: Install Ollama on Windows

Ollama is the engine that downloads and runs the model files locally. Install Ollama from its official site and complete the installer. After installation, Ollama runs a local service and exposes an API on your machine.

To verify it’s working, open PowerShell and run a quick model test. First, pull a model and run it:

Command:

ollama run llama3.1

If the model downloads and you see a prompt where you can type, Ollama is functioning. Type something simple like “Explain DNS in one paragraph” and confirm you get a response.

Step 2: Choose a Model That Fits Your Hardware

Local AI is all about picking a model that matches your PC. As a rule, smaller models load faster and use less RAM, while larger models can be more accurate but require better hardware.

Here are practical starting points you can try with Ollama:

• llama3.1: good general assistant for many tasks.
• mistral: fast and solid for summaries and troubleshooting.
• phi3: lightweight option for lower-end machines.

To download a model without launching it immediately, you can use:

ollama pull mistral

Step 3: Install Open WebUI (Web Interface)

Ollama is powerful, but the default terminal chat is not ideal for daily use. Open WebUI adds a browser-based interface so you can manage chats, switch models, and work comfortably.

The simplest method on Windows is to run Open WebUI using Docker Desktop. Install Docker Desktop, enable WSL 2 integration if prompted, then open PowerShell and run:

docker run -d --name open-webui -p 3000:8080 -e OLLAMA_BASE_URL=http://host.docker.internal:11434 ghcr.io/open-webui/open-webui:main

This command downloads the latest Open WebUI image and connects it to Ollama running on your host. After it starts, open your browser and go to:

http://localhost:3000

Create the first admin user when prompted. Once logged in, Open WebUI should automatically detect your Ollama models. If you don’t see them, check the Ollama service is running and confirm the URL points to http://host.docker.internal:11434.

Step 4: Test a Chat and Tune the Basics

In Open WebUI, select a model (for example, llama3.1) and start a new conversation. A good first test prompt is something specific:

Write a PowerShell script that checks free disk space on C: and warns if it is below 15%.

If you get a usable script, your pipeline is working end-to-end: browser UI → Open WebUI → Ollama → local model → response back to your browser.

If answers feel slow, try a smaller model, close memory-heavy apps, and keep your prompt concise. On CPU-only systems, switching to a lighter model often makes a bigger difference than any other tweak.

Step 5: Common Problems and Fixes

Open WebUI can’t connect to Ollama: Confirm Ollama is running and listening locally. Restart the Ollama service, then restart the Open WebUI container:
docker restart open-webui

Models don’t appear in the UI: Pull a model first using Ollama (for example, ollama pull llama3.1), then refresh Open WebUI.

Performance is poor: Use a smaller model like phi3 or mistral. Also ensure Windows power mode is not set to battery saver, and keep adequate free RAM available.

Disk fills up quickly: Local models are large. Remove models you don’t use with:
ollama rm <modelname>

Why This Setup Is Worth It

A local chatbot won’t replace every cloud AI feature, but it shines in day-to-day technical work: drafting SOPs, generating scripts, summarizing logs you can’t upload, and brainstorming troubleshooting steps while keeping everything on your own PC. Once you have Ollama and Open WebUI running, adding new models is a one-command task, and the browser interface makes it feel like a real tool—not a demo.

If you want to go further later, you can explore model fine-tuning, retrieval-augmented generation (RAG) with your internal documents, or running the same stack on a small home server. For now, this Windows setup is a clean, practical starting point for private AI that you control.

Why run a local AI assistant?

If you write code or manage infrastructure, an AI assistant can speed up tasks like generating scripts, explaining logs, reviewing configs, and drafting documentation. The problem is that many tools send prompts and code to a cloud service. In regulated environments, that is not always allowed. Running an AI model locally gives you better privacy, predictable costs, and the option to keep everything inside your LAN.

In this tutorial, you will install Ollama (a lightweight local model runtime) and Open WebUI (a clean browser interface) on Linux using Docker. The result is a self-hosted, ChatGPT-like web app that talks to your local models.

What you need

Requirements: a modern Linux server or workstation (Ubuntu/Debian/Fedora are fine), at least 8 GB RAM (16 GB+ recommended), and enough disk space for models (10–30 GB is common). CPU-only works, but a supported GPU can improve speed significantly. You also need a user with sudo privileges and outbound internet access to download containers and models.

Step 1: Install Docker (and Docker Compose)

If Docker is already installed, you can skip this section. On Ubuntu/Debian, one of the simplest approaches is using the official repository packages. Install Docker Engine and enable the service. If your distribution provides Docker Compose as a plugin, you can use docker compose (with a space) instead of the older docker-compose binary.

After installation, verify it works by running a test container. If you want to avoid using sudo for every Docker command, add your user to the docker group and log out/in once.

Step 2: Create a project folder

Create a dedicated directory for your stack so it is easy to manage and back up later. For example:

mkdir -p ~/local-ai && cd ~/local-ai

Step 3: Create a Docker Compose file for Ollama and Open WebUI

Create a file named docker-compose.yml in the folder. This setup uses persistent volumes so that downloaded models and WebUI data survive reboots and container upgrades.

nano docker-compose.yml

Paste the following:

<pre> version: "3.9" services: ollama: image: ollama/ollama:latest container_name: ollama restart: unless-stopped ports: - "11434:11434" volumes: - ollama:/root/.ollama open-webui: image: ghcr.io/open-webui/open-webui:main container_name: open-webui restart: unless-stopped depends_on: - ollama ports: - "3000:8080" environment: - OLLAMA_BASE_URL=http://ollama:11434 volumes: - openwebui:/app/backend/data volumes: ollama: openwebui: </pre>

Step 4: Start the services

From the same directory, start the stack:

docker compose up -d

Check that both containers are running:

docker ps

If something fails, view logs:

docker logs -f ollama

docker logs -f open-webui

Step 5: Download a model into Ollama

Ollama pulls models on demand. A practical starting point is a small-to-mid model that fits your RAM. For example, you can pull a general-purpose model like:

docker exec -it ollama ollama pull llama3.2

You can list installed models at any time:

docker exec -it ollama ollama list

If you prefer a coding-focused model, search Ollama’s library and choose one that matches your hardware. The key is to start small, confirm everything works, then scale up to larger models.

Step 6: Open WebUI in your browser

Open your browser and go to:

http://localhost:3000

If you installed this on a server, replace localhost with the server’s IP or hostname (for example, http://10.0.0.20:3000). The first time you load the page, you will create an admin account. After login, Open WebUI should detect Ollama automatically via the OLLAMA_BASE_URL setting.

Step 7: Basic usage tips (for real work)

To make the assistant useful for sysadmin and helpdesk tasks, be specific and provide context. Instead of “Fix this,” try prompts like: “Explain what this Nginx config does and suggest safer defaults” or “Write a bash script that checks disk usage and emails an alert”. When you paste logs, remove secrets and tokens, even if you are staying local.

If responses are slow, use a smaller model, reduce parallel users, or move the stack to a machine with more RAM. Local models are sensitive to memory pressure; swapping to disk can make them feel unusable.

Troubleshooting common problems

Open WebUI loads but shows no models: confirm Ollama is reachable from the WebUI container. The environment variable should be OLLAMA_BASE_URL=http://ollama:11434. Also confirm the model is actually pulled with ollama list.

Port already in use: if another service uses port 3000, change the mapping to something else, for example "8085:8080", then restart with docker compose up -d.

Downloads are slow or failing: this is usually DNS, proxy, or firewall related. Test connectivity from the host, and check whether your environment requires an HTTP proxy for container traffic.

Next steps: make it production-friendly

For a lab setup, HTTP on a LAN is fine. For teams, place Open WebUI behind a reverse proxy like Nginx or Caddy, add TLS, and restrict access with SSO or at least strong passwords. Finally, back up Docker volumes so you do not lose your settings and conversation history when you migrate.

Running an AI assistant locally is no longer just a hobby project. With today’s open models and lightweight runtimes, you can build a private “coding helper” that works even when the internet is down, keeps your prompts off third-party servers, and still feels fast enough for daily use. In this tutorial, you’ll install Ollama (a local LLM runtime) and Open WebUI (a clean web interface) on a Linux machine, then connect them and load a practical code-focused model.

This setup is ideal for admins, developers, and helpdesk teams who want quick answers for scripting, log parsing, config explanations, and command-line guidance without sending internal details to a cloud AI provider.

What You’ll Need

Requirements: A modern Linux distro (Ubuntu/Debian/Fedora are all fine), at least 8 GB RAM (16 GB recommended), and 15–30 GB free disk space depending on the model. CPU-only is supported; if you have a compatible GPU, responses may be faster, but it is not required for a functional installation.

Step 1: Install Ollama

Ollama provides a simple way to download and run large language models locally. On most distributions, the fastest method is the official install script. Open a terminal and run:

Command:
curl -fsSL https://ollama.com/install.sh | sh

After installation, confirm the service is available:

Command:
ollama --version

If you’re on a systemd-based distro, Ollama usually starts automatically. If not, start it manually or check the service status:

Commands:
sudo systemctl status ollama
sudo systemctl enable --now ollama

Step 2: Download a Coding-Friendly Model

You can choose different models depending on your hardware. For a balanced setup on a typical workstation, start with a mid-sized instruction model. For coding tasks, many users prefer models tuned for code completion and explanations.

Download a model with:

Example command:
ollama pull deepseek-coder:latest

Then test a quick prompt:

Example command:
ollama run deepseek-coder:latest

Type something like “Explain what a reverse proxy is in simple terms” or “Write a bash script to rotate logs.” If you get a response, the runtime is working.

Step 3: Install Docker (for Open WebUI)

Open WebUI is commonly deployed as a container. If Docker is not installed, install it using your distro’s package manager. On Ubuntu/Debian, this usually works:

Commands (Ubuntu/Debian example):
sudo apt update
sudo apt install -y docker.io
sudo systemctl enable --now docker

To avoid typing sudo for every Docker command, add your user to the docker group (log out and back in afterward):

Command:
sudo usermod -aG docker $USER

Step 4: Run Open WebUI and Connect It to Ollama

Now you’ll start Open WebUI and point it to Ollama’s API. By default, Ollama listens on http://localhost:11434. The container needs to reach the host service. On many Linux systems, you can use the host network mode to keep it simple.

Command:
docker run -d --name open-webui --restart always --network=host -e OLLAMA_BASE_URL=http://127.0.0.1:11434 -v open-webui:/app/backend/data ghcr.io/open-webui/open-webui:main

When it’s running, open your browser and visit:

URL: http://localhost:8080

Create an admin user when prompted. After login, Open WebUI should automatically detect available Ollama models. If you don’t see your model, check the model list from the terminal:

Command:
ollama list

Step 5: Make It Useful for Real Work (Practical Settings)

To get consistent, “helpdesk-ready” answers, create a default system prompt in Open WebUI that matches your environment. For example, set a short instruction like: “You are a Linux and networking assistant. Ask clarifying questions before suggesting risky commands. Provide commands with brief explanations.” This reduces accidental destructive advice and keeps responses focused.

For troubleshooting and scripting, you can also create saved prompts such as:

• “Analyze this error log and list likely root causes in order.”
• “Suggest a safe rollback plan before applying changes.”
• “Convert this one-liner into a readable bash script with comments.”

Common Problems and Fixes

Open WebUI loads, but no models appear: Verify Ollama is running: systemctl status ollama. Then confirm the base URL is correct and reachable. If you didn’t use --network=host, the container may not reach localhost on the host.

Responses are slow: Use a smaller model, close other memory-heavy apps, and consider increasing swap. On limited hardware, a 7B model often feels far more responsive than a 13B+ model.

Disk space disappears quickly: Models are large. Remove unused models with ollama rm MODELNAME, and periodically review ollama list.

Next Steps: Hardening and Remote Access

Once everything works locally, you can place Open WebUI behind a reverse proxy (like Nginx) with HTTPS, restrict access by IP, and enable authentication. If you plan to share it with a small team, consider running it on a dedicated VM and keeping a strict update routine for the container image and the host OS.

With Ollama and Open WebUI, you get a clean, private AI assistant that can help with scripts, configs, and troubleshooting—without turning your internal prompts into someone else’s training data.

Why run a local AI assistant?

Cloud AI tools are convenient, but they are not always the best fit for real work. If you handle private code, customer data, or internal repositories, sending prompts to a third-party service can raise compliance and security questions. A local AI setup gives you more control over your data, works even when the network is slow, and can reduce ongoing costs. In this tutorial, you will install Ollama (a lightweight local LLM runtime) and connect it to Continue (a popular AI coding extension) in Visual Studio Code on Linux.

What you will build

By the end, you will have a working local coding assistant that can explain code, refactor functions, generate tests, and answer questions about your project directly inside VS Code. The assistant will run on your machine and serve requests through a local HTTP endpoint. This guide focuses on a practical, repeatable setup that you can copy to developer laptops or a shared workstation.

Requirements

You need a modern Linux distribution (Ubuntu, Debian, Fedora, or similar), at least 8 GB RAM (16 GB is better), and enough disk space for models (start with 5–10 GB free). A GPU is optional; many models run well on CPU for moderate usage. You also need VS Code installed and permission to run shell commands.

Step 1: Install Ollama

Ollama provides a simple way to download and run LLMs locally. Install it using the official install script. Open a terminal and run:

curl -fsSL https://ollama.com/install.sh | sh

After installation, verify it is available:

ollama --version

On most systems, Ollama also starts a background service automatically. If you want to confirm the service is listening locally, run:

ss -lntp | grep 11434

The default API endpoint is typically http://localhost:11434.

Step 2: Download a model (recommended options)

Next, pull a model that fits your hardware. For a balanced local coding assistant, start with a smaller model and upgrade later. Run one of these commands:

ollama pull qwen2.5-coder:7b

ollama pull codellama:7b

To test it quickly in the terminal:

ollama run qwen2.5-coder:7b

Ask something simple like “Explain what a mutex is in plain English” or paste a small function and request a refactor. If responses are extremely slow, consider switching to a smaller model or closing memory-heavy applications.

Step 3: Install Continue in VS Code

Open VS Code, go to the Extensions view, and search for Continue. Install the extension published as “Continue - AI Code Assistant”. After installation, you will see a Continue panel (usually on the left activity bar) where you can chat and run code-related actions.

Step 4: Configure Continue to use Ollama

Continue reads its configuration from a JSON file. Open the Continue settings/config (the extension provides a link inside its UI, often labeled “Open Config”). In many setups the file is located under your home directory, such as:

~/.continue/config.json

Set the provider to Ollama and specify the model name you downloaded. A typical configuration looks like this:

{
  "models": [
    {
      "title": "Local Qwen Coder",
      "provider": "ollama",
      "model": "qwen2.5-coder:7b"
    }
  ],
  "defaultModel": "Local Qwen Coder"
}

Save the file and reload VS Code. In Continue, select your local model if prompted. From this point, your chat requests and code actions should be handled by Ollama on localhost.

Step 5: Practical workflows you can use immediately

A local model becomes useful when you give it tight, specific tasks. Try these workflows in Continue: Explain a complex function in your repo, Refactor a block of code to reduce duplication, or Generate unit tests for a module. For best results, include constraints such as “keep public function signatures unchanged” or “write tests using pytest and avoid network calls”.

If you work on infrastructure code, ask the assistant to review a systemd unit file, a Kubernetes manifest, or an Nginx config for common mistakes. Since the model runs locally, it is easier to iterate quickly without worrying about token limits or uploading sensitive config snippets.

Troubleshooting tips

Continue cannot connect to Ollama: Make sure Ollama is running and listening on port 11434. Restart it if needed, or check for local firewall rules. Confirm the endpoint is reachable with curl http://localhost:11434/api/tags.

Model name mismatch: The model string in Continue must match what Ollama lists. Run ollama list and copy the exact name (including tags like :7b).

Slow responses: Use a smaller model, reduce parallel applications, or consider enabling GPU acceleration if your system supports it. Also keep prompts focused; sending entire repositories in one request will slow down any local model.

Next steps

Once the basic setup works, you can standardize it for a team by documenting the chosen model and shipping a ready-to-use Continue config. You can also experiment with different local models for different tasks (one optimized for coding, another for general documentation). The most important habit is to keep prompts precise and treat the assistant like a fast helper, not a source of final truth. With that mindset, a local AI coding assistant can become a reliable part of your daily Linux development workflow.

Running a coding-focused AI assistant locally is no longer a “lab-only” experiment. With modern open-source tooling, you can host a private chatbot on your own Linux machine, avoid sending prompts to third-party services, and keep sensitive code snippets inside your network. In this tutorial, you will install Ollama (a lightweight local LLM runtime) and Open WebUI (a clean web interface) using Docker. The result is a fast, browser-based AI assistant you can use for debugging, code review, documentation drafts, and scripting help.

What You Will Build

By the end, you will have a local web app accessible at http://localhost:3000 (or your server’s IP) that talks to an Ollama service running on the same host. You will also learn how to persist data, pull a model, and validate that everything is working. This guide assumes Ubuntu Server or another modern Linux distribution with Docker support.

Prerequisites

Before starting, confirm you have: (1) a Linux server or desktop with at least 8 GB RAM (16 GB+ is better for larger models), (2) Docker and Docker Compose, and (3) enough disk space for model files (often 4–20 GB depending on the model). If you plan to access the UI from another device, ensure your firewall allows inbound TCP 3000.

Step 1: Install Docker and Docker Compose

If Docker is not installed yet on Ubuntu, you can install it with the official repository packages. Run:

sudo apt update
sudo apt install -y docker.io docker-compose-plugin

Enable and start Docker:

sudo systemctl enable --now docker

Optional but recommended: allow your user to run Docker without sudo (log out and back in after):

sudo usermod -aG docker $USER

Step 2: Create a Project Folder

Create a dedicated directory so your configuration and persistent volumes stay organized:

mkdir -p ~/local-ai-webui
cd ~/local-ai-webui

Step 3: Create a Docker Compose File

Create a file named docker-compose.yml and paste the following. This configuration runs Ollama and Open WebUI, and stores data in Docker volumes so updates do not wipe your models or settings:

nano docker-compose.yml

<pre>services: ollama: image: ollama/ollama:latest container_name: ollama restart: unless-stopped ports: - "11434:11434" volumes: - ollama:/root/.ollama open-webui: image: ghcr.io/open-webui/open-webui:main container_name: open-webui restart: unless-stopped ports: - "3000:8080" environment: - OLLAMA_BASE_URL=http://ollama:11434 volumes: - openwebui:/app/backend/data depends_on: - ollama volumes: ollama: openwebui:</pre>

Save and exit. The key line is OLLAMA_BASE_URL, which tells Open WebUI how to reach the Ollama container internally.

Step 4: Start the Services

Launch everything in the background:

docker compose up -d

Check container status:

docker ps

You should see both ollama and open-webui running.

Step 5: Download a Model

Ollama doesn’t ship with a model by default. Pull one that fits your hardware. For a solid balance of speed and quality, many users start with a 7–8B class model. Run:

docker exec -it ollama ollama pull llama3.1:8b

If disk space is tight, choose a smaller model. If you have more RAM and want higher quality, try a larger model, but expect slower responses on CPU-only systems.

Step 6: Open the Web Interface

In your browser, open:

http://localhost:3000

If you’re on a remote server, use:

http://SERVER_IP:3000

On first launch, Open WebUI typically asks you to create an admin account. After logging in, select your downloaded model (for example, llama3.1:8b) and send a test prompt like “Explain this Bash one-liner” or “Refactor this Python function for readability.”

Step 7: Basic Troubleshooting

Open WebUI loads but no models appear: Confirm the model exists inside Ollama with docker exec -it ollama ollama list. If the list is empty, re-run the pull command.

Connection errors to Ollama: Verify the containers are on the same Docker network (Compose does this automatically) and that OLLAMA_BASE_URL points to http://ollama:11434, not localhost.

Slow responses: Local inference is hardware-dependent. Smaller models respond faster. Also check CPU and memory usage with docker stats. If the system is swapping heavily, reduce model size.

Step 8: Updating Safely

To update to newer images without losing data, run:

docker compose pull
docker compose up -d

Because models and settings are stored in volumes, your downloaded models and WebUI configuration should remain intact.

Conclusion

A local AI assistant is a practical upgrade for developers and IT teams who want speed, privacy, and control. With Ollama handling model execution and Open WebUI providing a friendly interface, you can run a capable coding helper on a single Linux host and keep your prompts and snippets in-house. Once this baseline is working, consider placing it behind a reverse proxy, enabling HTTPS, or restricting access to your LAN for a more production-ready setup.

Why run a local AI assistant?

Cloud AI tools are convenient, but they are not always the best fit for technical teams. Running an AI model locally can help when you need to keep data on-premises, reduce recurring API costs, work offline, or experiment with custom prompts without sending logs to third-party services. In this tutorial, you will set up Ollama on Ubuntu and run a modern large language model locally. You will also expose a small HTTP endpoint so your helpdesk tools, scripts, or internal apps can query the model in a controlled way.

Prerequisites

You will need an Ubuntu machine (22.04 or newer is ideal), at least 8 GB RAM (16 GB recommended), and enough disk space for models (many are 4–10 GB each). CPU-only is fine for testing; a supported NVIDIA GPU will improve speed significantly. You should also have sudo access and basic terminal familiarity.

Step 1: Update Ubuntu and install basic tools

Start by updating packages and installing common utilities. This keeps the system clean and avoids dependency issues during installation.

Run:

sudo apt update && sudo apt -y upgrade
sudo apt -y install curl ca-certificates jq

Step 2: Install Ollama

Ollama provides a simple way to download and run models locally with a consistent CLI and a built-in service. Install it using the official script.

Run:

curl -fsSL https://ollama.com/install.sh | sudo sh

After installation, confirm that the service is available and the CLI responds.

ollama --version

Step 3 (Optional): Enable NVIDIA GPU acceleration

If you have an NVIDIA GPU, install the recommended driver and verify it is working. GPU support can drastically reduce response time for larger prompts.

Run:

sudo ubuntu-drivers autoinstall
reboot

After reboot:

nvidia-smi

If you see GPU details, you are ready. If not, confirm Secure Boot settings and driver installation. Ollama typically detects available acceleration automatically.

Step 4: Download and run a model

Now pull a model. A common starting point is a lightweight instruct model that performs well on general tasks. Example:

ollama pull llama3.2

Run an interactive session:

ollama run llama3.2

Ask something practical, such as: “Draft a troubleshooting checklist for DNS resolution issues on Ubuntu.” If you get coherent output, the base setup is complete.

Step 5: Use the local HTTP API

Ollama includes an API endpoint on the local machine. This makes it easy to integrate with scripts, ticketing systems, or internal tools. From the same server, test a generation request.

Run:

curl http://localhost:11434/api/generate -d '{
"model": "llama3.2",
"prompt": "Write a 6-step incident response note for a failed backup job.",
"stream": false
}' | jq -r '.response'

If you see a clear response, your local model is ready for automation.

Step 6: Create a simple “chat gateway” service (safe internal use)

For internal teams, it is often helpful to provide a tiny wrapper service that your tools can call without exposing the full Ollama interface. The example below uses Python and forwards requests to Ollama while keeping the endpoint minimal.

Install Python tools:

sudo apt -y install python3 python3-venv

Create a small app:

mkdir -p ~/ollama-gateway && cd ~/ollama-gateway
python3 -m venv .venv
. .venv/bin/activate
pip install flask requests

Create app.py with the following content:

from flask import Flask, request, jsonify
import requests

app = Flask(__name__)

@app.post("/chat")
def chat():
data = request.get_json(force=True)
prompt = data.get("prompt", "").strip()
if not prompt:
return jsonify({"error": "Missing prompt"}), 400

payload = {
"model": data.get("model", "llama3.2"),
"prompt": prompt,
"stream": False
}
r = requests.post("http://127.0.0.1:11434/api/generate", json=payload, timeout=120)
r.raise_for_status()
return jsonify({"response": r.json().get("response", "")})

if __name__ == "__main__":
app.run(host="127.0.0.1", port=8088)

Run it:

python app.py

Test in another terminal:

curl -s http://127.0.0.1:8088/chat -H "Content-Type: application/json" -d '{"prompt":"Summarize the key steps to fix a full disk on Linux."}' | jq

Step 7: Secure and operationalize the setup

Keep the API bound to 127.0.0.1 unless you have a clear need to expose it. If you must allow LAN access, put it behind a reverse proxy with authentication and strict firewall rules. Also consider separating prompts from sensitive data: even local tools can leak secrets via logs or copied outputs. For reliability, you can convert the gateway into a systemd service later, but even a basic local-only endpoint is useful for automation and experimentation.

Common troubleshooting tips

Slow responses: use a smaller model, reduce prompt length, or enable GPU acceleration. Out of memory: close other apps, add swap (temporary fix), or choose a smaller model variant. Port issues: confirm Ollama listens on 11434 locally and that your gateway points to 127.0.0.1. Model not found: run ollama list and verify the model name matches.

What you can build next

With a local model running, you can create internal knowledge assistants for helpdesk teams, draft incident updates, summarize logs, or generate standard operating procedures. The biggest win is control: you decide where prompts go, how access works, and what gets stored. Once this is stable, try adding model-specific system prompts for consistent tone, or integrate the gateway into a chatbot UI used by your team.

Why Run a Local AI Chatbot?

If you like using ChatGPT-style assistants but you work with sensitive data, cloud tools can be a non-starter. Running a local AI chatbot on your own Linux machine gives you control over privacy, lets you work offline, and can reduce ongoing costs. Thanks to modern lightweight model runners, you can now deploy an AI assistant in minutes without building anything from source.

In this tutorial, you will set up Ollama (a simple local LLM runtime) and Open WebUI (a clean web interface) on a Linux server. The result is a private, browser-based AI chatbot you can access on your LAN.

What You Need

System requirements: A modern Linux distribution (Ubuntu/Debian/RHEL-based), at least 8 GB RAM (16 GB recommended), and 15–30 GB of free disk space depending on the model you choose. A GPU is optional; CPU-only works, but responses may be slower.

Network requirements: If you want to access the chatbot from other devices, ensure you can reach the server over the network and that any firewall rules allow the chosen port.

Step 1: Install Ollama

Ollama is the engine that downloads and runs local AI models. On most Linux systems, the fastest method is the official install script. Open a terminal and run:

curl -fsSL https://ollama.com/install.sh | sh

After installation, confirm the service is working:

ollama --version

If your system uses systemd (most servers do), Ollama typically runs as a service. You can check its status with:

systemctl status ollama

Step 2: Pull a Model and Test It

Next, download a model. For a good balance between quality and speed on typical hardware, many users start with smaller variants. Example:

ollama pull llama3.2

Then run a quick interactive test:

ollama run llama3.2

Type a prompt, press Enter, and confirm you get a response. If the model feels slow, try a smaller one or ensure your server is not memory constrained.

Step 3: Install Open WebUI (Docker Method)

Open WebUI provides the familiar chat interface in your browser. The most reliable way to install it is using Docker, because updates are easy and dependencies stay isolated.

First, install Docker if you don’t already have it. On Ubuntu/Debian, this common approach works (adjust for your distro if needed):

sudo apt update && sudo apt install -y docker.io

sudo systemctl enable --now docker

Now start Open WebUI. The key is to point it to Ollama. If Ollama runs on the same machine, you can use host networking for simplicity:

sudo docker run -d --name open-webui --restart unless-stopped --network=host -v open-webui:/app/backend/data -e OLLAMA_BASE_URL=http://127.0.0.1:11434 ghcr.io/open-webui/open-webui:main

Open WebUI will typically be available on port 8080. From a browser on the server, test:

http://localhost:8080

Step 4: Access It from Another Device (LAN)

To use the chatbot from your laptop or phone on the same network, browse to:

http://SERVER_IP:8080

If it doesn’t load, check firewall rules. On Ubuntu with UFW, you can allow the port like this:

sudo ufw allow 8080/tcp

Also confirm that Docker is running and the container is healthy:

sudo docker ps

Step 5: Add and Switch Models in the Web Interface

Once logged into Open WebUI, you can select available models that Ollama has downloaded. If you want more choices, pull additional models on the server:

ollama pull mistral

ollama pull qwen2.5

Refresh the model list in the UI and switch models depending on your task. Smaller models respond faster; larger models can be better at reasoning and writing, but need more RAM and CPU.

Troubleshooting Tips

Open WebUI loads, but no models appear: Verify the environment variable points to Ollama correctly. If you used host networking, http://127.0.0.1:11434 is usually correct. Also confirm Ollama is listening:

ss -lntp | grep 11434

Model downloads are slow: Try again off-peak, confirm DNS/network stability, and ensure you have enough disk space. Model pulls can be several gigabytes.

Responses are very slow: Check RAM usage with free -h. If the system is swapping, performance will drop sharply. Consider a smaller model or upgrading memory.

Keep It Updated

To update Open WebUI, pull the latest container and recreate it:

sudo docker pull ghcr.io/open-webui/open-webui:main

sudo docker stop open-webui && sudo docker rm open-webui

sudo docker run -d --name open-webui --restart unless-stopped --network=host -v open-webui:/app/backend/data -e OLLAMA_BASE_URL=http://127.0.0.1:11434 ghcr.io/open-webui/open-webui:main

For Ollama, rerun the installer script occasionally or follow your distro’s recommended update path if you installed it through a package manager.

Conclusion

You now have a fully local AI chatbot running on Linux with Ollama and Open WebUI. This setup is practical for internal helpdesk use, drafting documentation, summarizing logs, or experimenting with prompts without sending data to third-party services. From here, you can harden access with a reverse proxy, enable HTTPS, and standardize your model choices for your team.

Running an AI assistant locally is no longer just a hobby project. With today’s efficient open models and lightweight runtimes, you can build a private “ChatGPT-style” environment on a Linux machine without sending prompts to any cloud service. This tutorial shows how to install Ollama (for downloading and serving models) and Open WebUI (a clean web interface) on Ubuntu/Debian-based systems. The result is a fast local chatbot you can use for troubleshooting, documentation drafts, code explanations, and more—while keeping your data on your own hardware.

What You Will Build

By the end of this guide, you will have: (1) Ollama installed and running as a service, (2) at least one model pulled and tested from the terminal, and (3) Open WebUI running in Docker and connected to Ollama. This setup works well on a modern CPU, and it can be accelerated if you have a compatible GPU, but GPU support is optional for getting started.

Prerequisites

You need a Linux server or workstation (Ubuntu 22.04/24.04 or Debian 12 is ideal), a user with sudo access, and at least 8 GB RAM for smaller models (16 GB+ is recommended for smoother performance). You also need enough disk space for model files; many popular models take several gigabytes each.

Step 1: Update the System

First, update packages and reboot if your kernel or core libraries are upgraded:

Commands:

sudo apt update && sudo apt -y upgrade
sudo reboot

Step 2: Install Ollama

Ollama provides a simple way to download and run large language models locally. It also exposes an HTTP API on your machine, which Open WebUI can talk to.

Install Ollama:

curl -fsSL https://ollama.com/install.sh | sh

After installation, confirm the service is running:

systemctl status ollama --no-pager

If it is not active, start and enable it:

sudo systemctl enable --now ollama

Step 3: Pull a Model and Test It

Next, download a model. For many users, a good starting point is a smaller model that runs comfortably on CPU. Choose one that matches your hardware and use case.

Example (pull and run a model):

ollama pull llama3.2
ollama run llama3.2

Type a prompt such as: “Explain how DNS caching works in Linux.” If you get a sensible response, Ollama is working.

To see what models you have installed:

ollama list

Step 4: Install Docker (for Open WebUI)

Open WebUI is easiest to deploy in a container. If Docker is not installed, add it using the official Ubuntu/Debian packages:

sudo apt -y install docker.io
sudo systemctl enable --now docker

Optional but convenient: allow your user to run Docker without sudo (log out and back in after this):

sudo usermod -aG docker $USER

Step 5: Run Open WebUI and Connect It to Ollama

Ollama listens locally (typically on port 11434). Open WebUI will run on port 3000. The key is to give the container access to the host’s Ollama API. The most reliable approach on Linux is to use host networking.

Run Open WebUI:

docker run -d --name open-webui --restart unless-stopped --network=host -e OLLAMA_BASE_URL=http://127.0.0.1:11434 -v open-webui:/app/backend/data ghcr.io/open-webui/open-webui:main

Now open your browser and go to:

http://YOUR_SERVER_IP:3000

Create the admin account when prompted. After login, Open WebUI should detect Ollama. If it does not, check the Ollama URL in settings and confirm Ollama is running.

Step 6: Basic Security and Network Tips

If this system is reachable over a network, treat it like any internal web service. At minimum, restrict access to port 3000 with a firewall or run it behind a reverse proxy with TLS. On Ubuntu, you can use UFW to allow only your LAN subnet or a specific IP.

Example (allow only a trusted subnet):

sudo ufw allow from 192.168.1.0/24 to any port 3000 proto tcp
sudo ufw enable

If you are deploying for multiple users, consider placing Open WebUI behind Nginx with HTTPS and basic authentication or SSO, depending on your environment.

Troubleshooting Checklist

Open WebUI loads but shows no models: Make sure you successfully ran ollama pull and that the container can reach http://127.0.0.1:11434. Using --network=host usually fixes connectivity issues on Linux.

Slow responses: Try a smaller model, close other memory-heavy apps, or upgrade RAM. Local AI performance is heavily tied to available memory bandwidth and CPU speed when running without GPU.

Ollama service not running: Check logs with journalctl -u ollama -n 100 --no-pager and verify you have enough free disk space for the model cache.

Conclusion

With Ollama and Open WebUI, you can run a practical AI assistant entirely on your own Linux system. This approach is ideal for homelabs, IT teams, and privacy-focused users who want modern AI capabilities without exposing internal prompts or data to external providers. Once the basics are working, you can experiment with different models, create custom system prompts for your helpdesk workflow, and even integrate the Ollama API into scripts and internal tools.