Run Local AI with Ollama and Open WebUI on Ubuntu (GPU Optional): A Practical How-To

on

Why run AI locally?

If you work in IT, helpdesk, development, or sysadmin roles, you probably paste logs, configs, or customer data into tools to get quick answers. The problem is that cloud AI services can be expensive, limited, or simply not allowed in regulated environments. Running a local AI stack on your own Linux box gives you privacy, predictable performance, and the ability to keep everything inside your network.

In this tutorial you will install Ollama (a lightweight local LLM runner) and Open WebUI (a clean web interface) on Ubuntu. You will end up with a browser-based “ChatGPT-style” experience that talks to models running on your machine. This setup works on CPU-only systems and can also use an NVIDIA GPU if you have one.

What you will build

You will install Ollama as a service, then run Open WebUI in Docker and connect it to Ollama. After that, you will download a model (for example, Llama or Mistral variants), confirm it responds, and finally secure and persist the environment for daily use.

Requirements

You need an Ubuntu system (22.04 or newer is ideal), at least 8 GB RAM (16 GB+ recommended), and enough free disk space (models can range from a few GB to tens of GB). For the web UI portion you should have Docker installed. If you want GPU acceleration, you will also need a compatible NVIDIA driver and the NVIDIA Container Toolkit.

Step 1: Update Ubuntu and install Docker

Start by updating packages and installing Docker. If Docker is already installed, you can skip the installation step and just confirm it works.

Commands:

sudo apt update && sudo apt -y upgrade
sudo apt -y install ca-certificates curl gnupg
sudo apt -y install docker.io
sudo systemctl enable --now docker
sudo usermod -aG docker $USER

Log out and log back in so your user can run Docker without sudo. Verify:

docker run --rm hello-world

Step 2: Install Ollama

Ollama is simple to install and runs as a background service. It exposes an API on your machine, which the web interface will use.

Commands:

curl -fsSL https://ollama.com/install.sh | sh

Confirm the service is running:

systemctl status ollama --no-pager

Step 3: Pull a model and test Ollama from the terminal

Now download a model. A good starting point is a smaller model that runs well on CPU. If you have more RAM and want better responses, choose a larger one. The exact names can change over time, but these examples are commonly available.

Commands:

ollama pull llama3.1
ollama run llama3.1

When prompted, ask something practical like: “Explain what this Nginx error means and how to fix it.” If you get a sensible answer, Ollama is working.

Step 4: Run Open WebUI with Docker

Open WebUI provides a friendly interface, chat history, and a simple model selector. We will run it as a container and point it at the Ollama API.

Commands:

docker volume create open-webui
docker run -d --name open-webui --restart unless-stopped \
-p 3000:8080 \
-e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
-v open-webui:/app/backend/data \
ghcr.io/open-webui/open-webui:main

On Linux, host.docker.internal may not resolve on some setups. If the WebUI cannot connect to Ollama, rerun the container using the host network mode instead:

docker rm -f open-webui
docker run -d --name open-webui --restart unless-stopped \
--network=host \
-v open-webui:/app/backend/data \
ghcr.io/open-webui/open-webui:main

Open your browser and visit http://localhost:3000 (or your server IP if remote). Create an admin account when prompted. You should see your Ollama models listed in the UI.

Step 5: Optional GPU acceleration (NVIDIA)

If you have an NVIDIA GPU, install the correct driver first, then add the NVIDIA Container Toolkit if you plan to run GPU-enabled containers. Ollama itself can use the GPU on the host if the drivers are correctly installed. Confirm your GPU is visible:

nvidia-smi

If nvidia-smi works, test performance by running a model and watching GPU utilization in another terminal. If you see GPU usage increase during generation, your local AI is accelerated.

Step 6: Basic hardening and useful tips

If this server is on a network, avoid exposing the WebUI to the entire internet. Place it behind a reverse proxy with authentication (for example, Nginx with basic auth) or restrict access at the firewall. Also remember that models can store chat history in the WebUI volume, so treat the data directory like sensitive application data and back it up appropriately.

A few practical tips: keep an eye on disk usage as you try different models, standardize on one or two “default” models for your team, and document prompts for your most common workflows (log analysis, PowerShell troubleshooting, ticket replies, and postmortems). Local AI is at its best when it is part of a repeatable process, not just a toy.

Conclusion

With Ollama and Open WebUI, you can run a capable local AI assistant on Ubuntu in under an hour, using either CPU-only hardware or an NVIDIA GPU for faster output. The result is a private, controllable tool that can help with troubleshooting, scripting ideas, documentation drafts, and day-to-day IT tasks—without sending your data to external services.

Comments

Post a Comment

Subscribe to: Post Comments (Atom)

Popular Posts

Install Ollama and Open WebUI on Ubuntu 24.04 with NVIDIA GPU Acceleration (Step-by-Step)

Overview Running a local large language model (LLM) is easier than ever thanks to Ollama and Open WebUI. Ollama handles model downloads and inference, while Open WebUI gives you a clean, chat-style interface in your browser. In this tutorial, you'll install both on Ubuntu 24.04 (works on 22.04 too), enable NVIDIA GPU acceleration, and deploy them with Docker Compose. The result is a fast, private AI stack you control. What You'll Need - Ubuntu 24.04 or 22.04 (fresh or existing server/desktop). - An NVIDIA GPU with recent drivers (Turing/RTX or newer recommended). - Root or sudo access. - Open ports 3000 (Open WebUI) and 11434 (Ollama) on your firewall if you access remotely. Step 1: Install NVIDIA Drivers and Verify CUDA First, update your system and install the recommended NVIDIA driver. On Ubuntu Desktop you can use Additional Drivers, but the CLI route is reliable: sudo apt update && sudo apt -y upgrade sudo ubuntu-drivers autoinstall sudo reboot After reboo...

Install a Local AI Chatbot on Ubuntu 24.04 with Ollama and Open WebUI (Step-by-Step)

Running a fast, private AI chatbot on your own computer or server is easier than ever. In this guide, you will install Ollama (a lightweight local LLM runtime) and Open WebUI (a modern web interface) on Ubuntu 24.04 . You will be able to chat with models like Llama 3 or Mistral without sending data to the cloud, and with optional GPU acceleration if you have an NVIDIA card. What you will need : an Ubuntu 22.04/24.04 machine (VM, bare metal, or WSL), at least 8 GB RAM (16 GB recommended for larger models), 15–30 GB free disk space for models, Internet access, and optional NVIDIA GPU drivers for acceleration. Why Ollama + Open WebUI? Ollama manages local large language models (LLMs) with simple commands and sensible defaults. Open WebUI gives you a clean, chat-style interface with features like prompt history, file uploads (for some models), and model switching. Together, they are a simple, reliable stack for a self-hosted AI experience. 1) Update Ubuntu and install basics First, ...

Install Ollama + Open WebUI on Ubuntu 24.04 with NVIDIA GPU Acceleration (Step-by-Step)

Overview This guide shows you how to install Ollama and Open WebUI on Ubuntu 24.04 with NVIDIA GPU acceleration. You will get a fast local AI stack that can run modern large language models (LLMs) like Llama 3 on your own hardware, secured and accessible in a web browser. We will cover GPU driver setup, Docker with the NVIDIA Container Toolkit, Ollama installation, Open WebUI deployment, basic security, and troubleshooting. Prerequisites - A machine running Ubuntu 24.04 (fresh or updated). - An NVIDIA GPU with at least 8 GB VRAM recommended for 8B models (more is better). - Internet access and sudo privileges. Step 1: Install the NVIDIA GPU Driver Install the recommended proprietary driver so CUDA becomes available to both Ollama and containers. sudo apt update && sudo apt install -y ubuntu-drivers-common sudo ubuntu-drivers autoinstall sudo reboot After reboot, verify the GPU is recognized: nvidia-smi You should see a driver version and your GPU model. If not, check...

Trending Now

Homogeneous Nucleation on Mars: A Textbook Process Finally Observed in Nature

on
By the end of this piece, you will understand what the Arsia Mons Elongated Cloud is, why it resisted explanation for several years after its first observation, and what the newly published model reveals about a physical process that has never been directly confirmed on any planet's atmosphere. The cloud that broke the model Every morning during Martian spring and summer, a thin, extremely long cloud forms near the Arsia Mons volcano and gradually elongates until it stretches to more than 1,000 miles before dissipating. The cycle then repeats for several months. The formation was first recorded by a probe in 2018, and from that point onward it became a persistent puzzle for atmospheric scientists working with Mars data. The cloud's geometry, its timing, and its seasonal recurrence all pointed to a real and reproducible physical process, yet no existing model could reproduce it. For a technical audience familiar with simulation work, the situation is analogous to a product...