Local LLM Setup with Docker Containers
This guide describes my setup for running a local LLM using Docker containers. The model and the frontend UI run in separate containers, connected via a Docker network.
Stack Overview: Final Architecture (Docker Version)
┌──────────────────────────────┐
│ Browser │
└──────────────┬───────────────┘
│ http://localhost:3000
┌──────────────▼───────────────┐
│ Open WebUI (Docker container)│
│ - Chat UI │
│ - Connects to API backend │
└──────────────┬───────────────┘
│ http://<server-container-name>:8080/v1
┌──────────────▼───────────────┐
│ llama.cpp server (Docker) │
│ - Loads GGUF model │
│ - Inference engine │
│ - Token streaming │
└──────────────┬───────────────┘
│ Vulkan backend
┌──────────────▼───────────────┐
│ GPU (Strix Halo RDNA 3.5) │
│ via Vulkan compute │
└──────────────┬───────────────┘
│
Unified System RAM
Install Docker
sudo apt install docker.ioRun Docker without sudo:
https://devopslog.pl/2026/05/06/using-docker-without-sudo/
(A reboot is required after configuration.)
Model Storag & Recommended Engine
Best place for the latest .guff models.
Hugging Face .gguf Models:
https://huggingface.co/unsloth
Recommended Engine for Strix Halo Chips (as of now):llama.cpp with Vulkan backend:
https://github.com/ggml-org/llama.cpp/blob/master/docs/docker.md
Create Docker Network
The server and Open WebUI containers will communicate via this llm-net network.
Check if it already exists:
docker network lsIf not, create it:
docker network create llm-netRun the Server Container
Example command to run the server container:
docker run -d \
--name gemma-4-E4B-it-UD-Q8_K_XL \
--network llm-net \
--device /dev/dri \
-p 8080:8080 \
-v $PWD/data/models:/models \
ghcr.io/ggml-org/llama.cpp:server-vulkan \
-m /models/gemma-4-E4B-it-UD-Q8_K_XL.gguf \
--host 0.0.0.0 \
--port 8080 \
-ngl 999API Endpoint URL:http://gemma-4-E4B-it-UD-Q8_K_XL:8080/v1
Command Explanation:
# Run in detached mode (`-d` starts the container in the background and returns the container ID)
docker run -d \
# Assign a container name (using the model name is recommended for clarity)
--name gemma-4-E4B-it-UD-Q8_K_XL \
# Connect to the `llm-net` Docker network (created earlier)
--network llm-net \
# Expose GPU devices (`llama.cpp` will automatically select the appropriate device)
--device /dev/dri \
# Map host port to container port (`-p` publishes a container port to the host)
-p 8080:8080 \
# Mount a host directory into the container (`-v` binds a local path to a container path)
-v $PWD/data/models:/models \
# Specify the Docker image
ghcr.io/ggml-org/llama.cpp:server-vulkan \
# Specify the path to the model file inside the container
-m /models/gemma-4-E4B-it-UD-Q8_K_XL.gguf \
# Listen on all network interfaces (allows access within the Docker network)
--host 0.0.0.0 \
# Listen for connections on port 8080
--port 8080 \
# Offload as many model layers to the GPU as possible (999 effectively means all layers)
-ngl 999
Run the Frontend UI Container
To avoid connection issues, start the server container first and wait for it to reach a healthy status before running the frontend (Open WebUI) container. This prevents the UI from failing to detect the model.
Run the Open WebUI container:
docker run -d \
--name open-webui \
--network llm-net \
-p 3000:8080 \
-e OPENAI_API_BASE_URL=http://gemma-4-E4B-it-UD-Q8_K_XL:8080/v1 \
-e OPENAI_API_KEY=without \
-v open-webui:/app/backend/data \
ghcr.io/open-webui/open-webui:main
Go to:
http://127.0.0.1:3000/
Command Explanation:
# Run in detached mode (`-d` starts the container in the background and returns the container ID)
docker run -d \
# Assign a container name
--name open-webui \
# Connect to the `llm-net` Docker network (enables direct communication & DNS resolution between the UI and the LLM server)
--network llm-net \
# Map host port to container port (`-p` publishes a container port to the host; access the UI at http://localhost:3000)
-p 3000:8080 \
# Set environment variable to point Open WebUI to the LLM server's OpenAI-compatible API endpoint
-e OPENAI_API_BASE_URL=http://gemma-4-E4B-it-UD-Q8_K_XL:8080/v1 \
# Set an API key placeholder (Open WebUI requires this field but ignores it for local/self-hosted backends)
-e OPENAI_API_KEY=without \
# Mount a named Docker volume to persist UI settings, chat history, and user data across container restarts
-v open-webui:/app/backend/data \
# Specify the Docker image (official Open WebUI repository, latest `main` branch)
ghcr.io/open-webui/open-webui:main
Troubleshooting
If the GPU remains busy after the model responds, it may be generating “Follow-Up Auto-Generation” suggestions (hints for the next prompt/question). You can disable this feature in Open WebUI:
User Icon > Settings > Interface > Follow-Up Auto-Generation (toggle it off)
Server with MTP Enhancement
This example shows an alternative server configuration. It is recommended to stop the first server container before running this one.
Qwen3.6 35B A3B MTP
docker run -d \
--name Qwen3.6-35B-A3B-MTP-UD-Q8_K_XL \
--network llm-net \
--device /dev/dri \
-p 8080:8080 \
-v $PWD/data/models:/models \
ghcr.io/ggml-org/llama.cpp:server-vulkan \
-m /models/Qwen3.6-35B-A3B-MTP-UD-Q8_K_XL.gguf \
--host 0.0.0.0 \
--port 8080 \
-ngl 999 \
-c 16384 \
-np 1 \
-cram 0 \
--spec-type draft-mtp \
--spec-draft-n-max 2
Add the API Endpoint to Open WebUI Connection Settings:http://Qwen3.6-35B-A3B-MTP-UD-Q8_K_XL:8080/v1
Useful Commands
To check the current tokens per second (t/s) and view other model statistics:
docker logs -f <model-container-name>Done! What next?
This setup currently provides a convenient and efficient local LLM environment. The next step is to implement RAG with web search capabilities.
01.06.2026 update
To run multiple models in one session, you have to publish each model on a different port.
e.g.
– gemma
-p 8081:8080 \
– Qwen
-p 8082:8080 \
