DevOpsLog.pl

The Art of Technical Troubleshooting

Author: Konrad

  • Posted on

Local AI like ChatGPT via Docker Model Runner

The most powerful and convenient way to deploy AI with Docker is Docker Model Runner (DMR). High-level architecture and major components. Docker Model Runner – is a runtime process working outside the containers that provides models direct access to host hardware. OpenAI-compatible endpoints – the same API endpoints as the official OpenAI API uses. Provides
Read More
  • Posted on

Docker Model Runner: context size

To change the default context size of a model, which is typically 4096 tokens, use the following command: Example usage: NOTE! If you run a model after increasing the context size, RAM usage can immediately spike to the maximum amount reserved for that context size, even if the conversation only contains a small number of
Read More
  • Posted on

using Docker without sudo

Have you seen this error: Run Docker commands without sudo 1. Add the docker group if it doesn’t already exist 2. Add the connected user $USER to the docker group Optionally change the username to match your preferred user. IMPORTANT: Log out and log back in so that your group membership is re-evaluated. 3. Restart
Read More
  • Posted on

Docker and AI in Containers using Ollama

Real Use Case Using Docker as an environment and Ollama as a containerized application with a specific model, e.g. Phi-3. What is Ollama A tool that lets you run LLMs locally. It is often based on the llama.cpp engine.It handles: It runs as a server on port 11434. Ollama inside Docker Docker creates a container
Read More
  • Posted on

Docker Compose – Running a Multi-Container App

In short, Compose is a tool (now shipped by default with Docker) that allows you to easily define and run multi-container applications using a YAML file. The structure of the YAML file is defined by Compose. The Compose file should be called compose.yaml or compose.yml. Its structure includes the following top-level keys: Digging deeper:The core
Read More
  • Posted on

Docker Self Healing – Restart Policies

Restart policies are rules for restarting containers. There are four:●​ no (default)●​ on-failure●​ always●​ unless-stopped In the table: Y = Restart, N = No Restart How to use it To apply these policies, use --restart <policy> when running a container: or when update:
  • Posted on

Docker – external tools (debugging)

A typical Docker image doesn’t have built-in tools that you can find in a standard Linux system. The best way to use tools like vim, ping, etc. is to use Docker Debug. Docker Debug is a useful tool for working with slim images and containers. It gives you shell access even when the container or
Read More

PROJECTS