hidden
My setup of running local LLM by using containers, where I’m running model and frontend UI in separate containers connected by docker network.
stack overview:
Final architecture (Docker version)
┌──────────────────────────────┐
│ Browser │
└──────────────┬───────────────┘
│ http://localhost:3000
┌──────────────▼───────────────┐
│ Open WebUI (Docker container)│
│ - chat UI │
│ - connects to API backend │
└──────────────┬───────────────┘
│ http://llama:8080/v1
┌──────────────▼───────────────┐
│ llama.cpp server (Docker) │
│ - loads GGUF model │
│ - inference engine │
│ - token streaming │
└──────────────┬───────────────┘
│ Vulkan backend
┌──────────────▼───────────────┐
│ GPU (Strix Halo RDNA 3.5) │
│ via Vulkan compute │
└──────────────┬───────────────┘
│
Unified system RAM
install docker:
sudo apt install docker.io
Do docker without sudo:
https://devopslog.pl/2026/05/06/using-docker-without-sudo/
reboot required
Best place for .guff models to convinient run inside container by llama.cpp
HuggingFace .guff models
https://huggingface.co/unsloth
Best engine for strix halo chip. (to this date 🙂
llama.cpp engines:
https://github.com/ggml-org/llama.cpp/blob/master/docs/docker.md
Create docker network
By this llm-net network the server and open-webUI containers will be communicate.
check if is created:
docker network ls
if not, create:
docker network create llm-net
RUN SERVER
example of run server container:
docker run -d \
--name gemma-4-E4B-it-UD-Q8_K_XL \
--network llm-net \
--device /dev/dri \
-p 8081:8080 \
-v $PWD/data/models:/models \
ghcr.io/ggml-org/llama.cpp:server-vulkan \
-m /models/gemma-4-E4B-it-UD-Q8_K_XL.gguf \
--host 0.0.0.0 \
--port 8080 \
-ngl 999
API endpoint URL
http://gemma-4-E4B-it-UD-Q8_K_XL:8080/v1
Command explain:
# run docker (`-d` starts the container in the background and immediately returns the container ID: e.g f54b8049a5e42ca...)
docker run -d \
# specify the name of container (best approach the same as model)
--name gemma-4-E4B-it-UD-Q8_K_XL \
# assign container into docker llm-net network (created earlier)
--network llm-net \
# show path for GPU devices (llama.cpp choose device itself)
--device /dev/dri \
# HOST_PORT:CONTAINER_PORT (`-p` publish a port from the container to the host)
-p 8080:8080 \
# HOST_PATH:CONTAINER_PATH (`-v` mount a directory from your host into the container.)
-v $PWD/data/models:/models \
# Docker Image
ghcr.io/ggml-org/llama.cpp:server-vulkan \
# `-m` specify path to model INSIDE CONTAINER
-m /models/gemma-4-E4B-it-UD-Q8_K_XL.gguf \
# Listen on all network interfaces. (Locally inside network)
--host 0.0.0.0 \
# Listen for connections on port 8080. (on localhost 127.0.0.1)
--port 8080 \
# Try to put as many model layers on the GPU as possible (practically: all of them).
-ngl 999
RUN FRONTEND UI
To do this without issues, run or start server container and wait to (healthly) status, then next run frontend (open-webUI) container. This approach avoid problems with undetected models in open-webUI.
run open-webUI container:
docker run -d \
--name open-webui \
--network llm-net \
-p 3000:8080 \
-e OPENAI_API_BASE_URL=http://gemma-4-E4B-it-UD-Q8_K_XL:8080/v1 \
-e OPENAI_API_KEY=without \
-v open-webui:/app/backend/data \
ghcr.io/open-webui/open-webui:main
command explain:
# run docker (`-d` starts the container in the background and immediately returns the container ID)
docker run -d \
# specify the name of the container
--name open-webui \
# assign container into docker llm-net network (enables direct communication & DNS resolution between this UI and the LLM server)
--network llm-net \
# HOST_PORT:CONTAINER_PORT (`-p` publish a port from the container to the host; access the UI at http://localhost:3000)
-p 3000:8080 \
# set environment variable to point Open WebUI to the LLM server's OpenAI-compatible API endpoint
-e OPENAI_API_BASE_URL=http://gemma-4-E4B-it-UD-Q8_K_XL:8080/v1 \
# set environment variable for API key (Open WebUI requires this field but ignores it for local/self-hosted backends; "without" acts as a placeholder)
-e OPENAI_API_KEY=without \
# HOST_VOLUME:CONTAINER_PATH (`-v` mount a named Docker volume to persist UI settings, chat history, and user data across container restarts)
-v open-webui:/app/backend/data \
# Docker Image (official Open WebUI repository, latest `main` branch)
ghcr.io/open-webui/open-webui:main
TROUBLESHOOTING
If after model respond, the GPU is still busy, then probably the model is generating Follow-Up Auto-Generation (hint for next prompt/question).
You can disable it in open-web UI:
User icon > Settings > Interface > Follow-Up Auto-Generation (disable it)
SERVER with MTP enchance
QWEN3.6 35B A3B MTP
docker run -d \
--name Qwen3.6-35B-A3B-MTP-UD-Q8_K_XL \
--network llm-net \
--device /dev/dri \
-p 8082:8080 \
-v $PWD/data/models:/models \
ghcr.io/ggml-org/llama.cpp:server-vulkan \
-m /models/Qwen3.6-35B-A3B-MTP-UD-Q8_K_XL.gguf \
--host 0.0.0.0 \
--port 8080 \
-ngl 999 \
-c 131072 \
-np 1 \
-cram 0 \
--spec-type draft-mtp \
--spec-draft-n-max 2
ADD API ENDPOINTS INTO OPEN-WEBUI CONNECTION SETTINGS
http://Qwen3.6-35B-A3B-MTP-UD-Q8_K_XL:8080/v1
QWEN3.6 27B MTP Q6
docker run -d \
--name Qwen3.6-27B-MTP-Q6_K \
--network llm-net \
--device /dev/dri \
-p 8083:8080 \
-v $PWD/data/models:/models \
ghcr.io/ggml-org/llama.cpp:server-vulkan \
-m /models/Qwen3.6-27B-MTP-Q6_K.gguf \
--host 0.0.0.0 \
--port 8080 \
-ngl 999 \
-c 131072 \
-np 1 \
-cram 0 \
--spec-type draft-mtp \
--spec-draft-n-max 2
ADD API ENDPOINTS INTO OPEN-WEBUI CONNECTION SETTINGS
http://Qwen3.6-27B-MTP-Q6_K:8080/v1
Qwen3.6-35B-A3B-Uncensored-Q6
docker run -d \
--name Qwen3.6-35B-A3B-Uncensored-Q6 \
--network llm-net \
--device /dev/dri \
-p 8084:8080 \
-v $PWD/data/models:/models \
ghcr.io/ggml-org/llama.cpp:server-vulkan \
-m /models/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-Q8_K_P.gguf \
--host 0.0.0.0 \
--port 8080 \
-ngl 999 \
-c 131072 \
-np 1 \
-cram 0
ADD API ENDPOINTS INTO OPEN-WEBUI CONNECTION SETTINGS
http://Qwen3.6-35B-A3B-Uncensored-Q6:8080/v1
USEFUL COMMANDS
checking current t/s and other useful info about model:
docker logs -f <model-container-name>
TROUBLESHOOTING!
REMEMBER TO RUN CONTAINERS (especially models) IN HOME DIR:
home/noise/
Leave a Reply