How to Run Local LLMs with GPU: Install Ollama and Open WebUI on Ubuntu or Windows WSL2

on

Overview: This step-by-step guide shows how to run local large language models with GPU acceleration using Ollama and Open WebUI. You will install Ollama to serve models like Llama 3 or Mistral, and deploy Open WebUI as an easy web interface. The tutorial covers Ubuntu 22.04/24.04 and Windows 11 via WSL2, with security, auto-start, and troubleshooting tips.

What you need: A 16 GB RAM system (32 GB recommended), an NVIDIA GPU with recent drivers (8 GB+ VRAM recommended), admin/root access, a stable internet connection, and around 20–30 GB of free disk space for models and containers.

Why this stack? Ollama provides a simple runtime and model manager for local LLMs, while Open WebUI gives a modern, browser-based chat interface, prompt management, RAG integrations, and multi-model switching. Both are lightweight and work on a single machine.

Step 1 — Prepare GPU drivers

Ubuntu (bare metal/VM with GPU passthrough):

sudo apt update && sudo apt install -y ubuntu-drivers-common
sudo ubuntu-drivers autoinstall
sudo reboot

After reboot, verify:

nvidia-smi

If you see your GPU, the driver is good. Ollama downloads the CUDA user-space libs it needs automatically; you only need a working NVIDIA driver.

Windows 11 with WSL2:

Install the latest NVIDIA Game Ready/Studio driver (535+), then update WSL:

wsl --update
wsl --shutdown

Open your Ubuntu WSL distro and confirm:

echo $WSL_DISTRO_NAME

Ollama will use the GPU via WSL automatically if the Windows driver supports it.

Step 2 — Install Docker (for Open WebUI)

Ubuntu:

sudo apt update && sudo apt install -y docker.io
sudo usermod -aG docker $USER
newgrp docker
sudo systemctl enable --now docker

Windows WSL2:

Install Docker Desktop for Windows and enable the WSL2 integration for your Ubuntu distro. Start Docker Desktop before running containers.

Step 3 — Install Ollama

Ubuntu/WSL2:

curl -fsSL https://ollama.com/install.sh | sh

Enable the service on Ubuntu (bare metal):

sudo systemctl enable --now ollama

Test by pulling a small model:

ollama pull llama3.1:8b

Run a quick prompt:

ollama run llama3.1:8b "Write a haiku about GPUs."

If you see a note like “using CUDA,” GPU is active. If not, ensure drivers are correct.

Step 4 — Deploy Open WebUI

Ubuntu (recommended: host networking):

docker run -d --name open-webui --restart unless-stopped --network host \
-e OLLAMA_BASE_URL=http://127.0.0.1:11434 \
-e WEBUI_AUTH=true \
-e [email protected] \
-e DEFAULT_USER_PASSWORD=ChangeMeStrong! \
ghcr.io/open-webui/open-webui:main

Open your browser at http://127.0.0.1:8080 (host network uses the container’s internal port). Log in with the credentials you set.

WSL2 with Docker Desktop (no host networking):

docker run -d --name open-webui --restart unless-stopped -p 3000:8080 \
-e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
-e WEBUI_AUTH=true \
-e [email protected] \
-e DEFAULT_USER_PASSWORD=ChangeMeStrong! \
ghcr.io/open-webui/open-webui:main

Browse to http://localhost:3000. The variable OLLAMA_BASE_URL points Open WebUI to the local Ollama API.

Step 5 — Optimize and manage models

Popular models: llama3.1:8b, mistral-nemo:12b, qwen2.5:7b. You can also choose quantized variants like llama3.1:8b-instruct-q4_K_M for lower VRAM usage.

ollama list
ollama pull mistral-nemo:12b
ollama rm modelname:tag

To limit VRAM, use smaller models or quantized builds. In Open WebUI, select the model per chat. For longer contexts, try a 70B model on a bigger GPU or use smaller context windows to conserve memory.

Step 6 — Secure and persist

Keep Open WebUI bound to localhost if the machine is shared. If you must expose it, use a reverse proxy with TLS (Nginx, Caddy, or Traefik) and keep WEBUI_AUTH=true. On Ubuntu, ensure the firewall blocks unwanted access:

sudo ufw allow 8080/tcp comment 'Open WebUI (local)'
sudo ufw status

Data for Open WebUI is stored in the container’s volume by default. For manual control, mount a volume: -v open-webui:/app/backend/data. Ollama stores models under ~/.ollama; back that up regularly.

Step 7 — Auto-start on boot

Ollama installs a systemd service on Ubuntu. Ensure Docker is enabled (already done) and containers use --restart unless-stopped so Open WebUI comes up after reboots. On Windows, set Docker Desktop to start on login.

Troubleshooting

GPU not used: Update NVIDIA driver, reboot, and confirm with nvidia-smi (Ubuntu) or update WSL and drivers (Windows). Reinstall Ollama if needed. Check that ollama run logs mention CUDA.

Out-of-VRAM: Use a smaller or more aggressively quantized model. Reduce max tokens or context length in Open WebUI settings. Close other GPU apps.

Port conflicts: If 11434 or 8080/3000 are used, change them. For example, -p 3333:8080 for Open WebUI and OLLAMA_HOST=127.0.0.1:11500 ollama serve for Ollama.

Slow downloads: Use a reliable network, or pre-fetch models with ollama pull. You can also host a local model library if bandwidth is limited.

Clean up disk: Remove unused models and images:
ollama list && ollama rm model:tag
docker image prune -f

What’s next

Explore RAG in Open WebUI by attaching local documents, add embeddings, and test function calling or tool integrations. With this setup, you have a fast, private, and GPU-accelerated local AI workstation ready for coding, content, and research.

Read more
Post a Comment
Subscribe to: Posts (Atom)

Popular Posts

Install Ollama and Open WebUI on Ubuntu 24.04 with NVIDIA GPU Acceleration (Step-by-Step)

Overview Running a local large language model (LLM) is easier than ever thanks to Ollama and Open WebUI. Ollama handles model downloads and inference, while Open WebUI gives you a clean, chat-style interface in your browser. In this tutorial, you'll install both on Ubuntu 24.04 (works on 22.04 too), enable NVIDIA GPU acceleration, and deploy them with Docker Compose. The result is a fast, private AI stack you control. What You'll Need - Ubuntu 24.04 or 22.04 (fresh or existing server/desktop). - An NVIDIA GPU with recent drivers (Turing/RTX or newer recommended). - Root or sudo access. - Open ports 3000 (Open WebUI) and 11434 (Ollama) on your firewall if you access remotely. Step 1: Install NVIDIA Drivers and Verify CUDA First, update your system and install the recommended NVIDIA driver. On Ubuntu Desktop you can use Additional Drivers, but the CLI route is reliable: sudo apt update && sudo apt -y upgrade sudo ubuntu-drivers autoinstall sudo reboot After reboo...

Install a Local AI Chatbot on Ubuntu 24.04 with Ollama and Open WebUI (Step-by-Step)

Running a fast, private AI chatbot on your own computer or server is easier than ever. In this guide, you will install Ollama (a lightweight local LLM runtime) and Open WebUI (a modern web interface) on Ubuntu 24.04 . You will be able to chat with models like Llama 3 or Mistral without sending data to the cloud, and with optional GPU acceleration if you have an NVIDIA card. What you will need : an Ubuntu 22.04/24.04 machine (VM, bare metal, or WSL), at least 8 GB RAM (16 GB recommended for larger models), 15–30 GB free disk space for models, Internet access, and optional NVIDIA GPU drivers for acceleration. Why Ollama + Open WebUI? Ollama manages local large language models (LLMs) with simple commands and sensible defaults. Open WebUI gives you a clean, chat-style interface with features like prompt history, file uploads (for some models), and model switching. Together, they are a simple, reliable stack for a self-hosted AI experience. 1) Update Ubuntu and install basics First, ...

Install Ollama + Open WebUI on Ubuntu 24.04 with NVIDIA GPU Acceleration (Step-by-Step)

Overview This guide shows you how to install Ollama and Open WebUI on Ubuntu 24.04 with NVIDIA GPU acceleration. You will get a fast local AI stack that can run modern large language models (LLMs) like Llama 3 on your own hardware, secured and accessible in a web browser. We will cover GPU driver setup, Docker with the NVIDIA Container Toolkit, Ollama installation, Open WebUI deployment, basic security, and troubleshooting. Prerequisites - A machine running Ubuntu 24.04 (fresh or updated). - An NVIDIA GPU with at least 8 GB VRAM recommended for 8B models (more is better). - Internet access and sudo privileges. Step 1: Install the NVIDIA GPU Driver Install the recommended proprietary driver so CUDA becomes available to both Ollama and containers. sudo apt update && sudo apt install -y ubuntu-drivers-common sudo ubuntu-drivers autoinstall sudo reboot After reboot, verify the GPU is recognized: nvidia-smi You should see a driver version and your GPU model. If not, check...

Trending Now

Homogeneous Nucleation on Mars: A Textbook Process Finally Observed in Nature

on
By the end of this piece, you will understand what the Arsia Mons Elongated Cloud is, why it resisted explanation for several years after its first observation, and what the newly published model reveals about a physical process that has never been directly confirmed on any planet's atmosphere. The cloud that broke the model Every morning during Martian spring and summer, a thin, extremely long cloud forms near the Arsia Mons volcano and gradually elongates until it stretches to more than 1,000 miles before dissipating. The cycle then repeats for several months. The formation was first recorded by a probe in 2018, and from that point onward it became a persistent puzzle for atmospheric scientists working with Mars data. The cloud's geometry, its timing, and its seasonal recurrence all pointed to a real and reproducible physical process, yet no existing model could reproduce it. For a technical audience familiar with simulation work, the situation is analogous to a product...