All work

Infrastructure

Retiring Portainer for Komodo

Replaced a container dashboard with a GitOps control plane in one day, so every stack in the fleet became recreatable from a git repository instead of from memory.

Stacks, one control plane
26
Servers managed
5
Discovery to cutover
1 day

Problem

My homelab ran around two dozen Docker compose projects across four hosts, and the only source of truth for any of them was a docker-compose.yml sitting in whatever directory it happened to be created in, on the host itself. Secrets were inline. Most images floated on latest. If a host died, the recovery plan was my memory.

Portainer was already running, and it was not the answer. Portainer shows you what is running. It has no concept of what should be running, so when the display is the truth, configuration drift is undetectable by construction. A control plane that reads the hosts can only tell you what happened. One that reads a git repository can tell you what was supposed to happen, and that is the version worth having at two in the morning.

FIG 01 · CONTROL PLANE
Komodo control plane dashboard showing five healthy servers, the stack fleet broken down by running, stopped and unhealthy state, and three scheduled procedures all reporting OK

The goal was never a nicer dashboard. It was making the fleet recreatable from zero.

Approach

I swept all four hosts first and changed nothing, inventorying every stack, every bind mount and every inline secret, because you cannot migrate what you have not counted. Then the design, which is four coupled decisions.

Every stack moves into one private git repository as host/stack/docker-compose.yml, with images pinned to explicit versions and digests instead of floating tags. Komodo runs as the control plane, Core on one host and a lightweight agent on every other, and stacks deploy from the repository via Komodo’s own clone rather than by editing files on a box. Secrets are stripped out of the repo entirely and replaced with variable references, with the real values held in my self-hosted password manager and injected per stack, so no secret value ever lands in git. Finally, every stack gets a redacted mirror in my knowledge base with a recreate runbook, which is what lets the AI system I run reason about the fleet without touching a host.

FIG 02 · ONE SOURCE OF TRUTH
Komodo stacks list showing every stack sharing an identical git repository in the Source column, distributed across four hosts, with live running and stopped states

That identical Source column on every row is the whole point of the project. The fleet has a desired state now, not just a current one, and the handful of exceptions are declared rather than accidental. The repository path is blurred because it is private.

FIG 03 · FLEET
Komodo servers list showing five servers, four on the local network plus one cloud VPS, all running the same agent version and all reporting OK
FIG 04 · DEPLOY FROM GIT
A single Komodo stack detail page showing it deploys from a git repository, with its real deploy history from creation onward listed alongside, and a red remote-error badge on the repository row

The fleet later grew to include a cloud VPS, which joined over an encrypted overlay rather than an exposed management port. Each stack keeps its real deploy history, and because every lifecycle action is available over an API, the whole fleet became queryable, which is what made automated fleet health reporting possible afterwards.

That red badge on the repository row is not a staging accident. It is my own fleet, right now, in exactly the failure mode this architecture has, and it is worth leaving in rather than reshooting around. What it means is at the end of this page.

Data never moved. That was the migration law that made a single-day cutover reasonable. Relative bind mounts were rewritten to the original absolute host paths, so every stack’s data directory was untouched and rolling back any stack stayed a one-line docker compose up in its old directory. I never had to trust the migration, which is exactly why I was willing to do the whole fleet in one pass. Nineteen stacks moved that day, each verified after a restart, and then the Portainer instance and both of its agents were removed.

FIG 05 · UNATTENDED OPERATIONS
Komodo update log showing scheduled procedures for server key rotation, a global image update sweep and a control plane database backup, all reporting SUCCESS across consecutive days

Result

Twenty-six stacks across five servers now run from one control plane, and any of them can be rebuilt on a bare host from the repository, the password manager and the runbook. Image updates arrive as reviewable pull requests instead of drifting silently. Key rotation, an image update sweep and a control plane database backup run unattended every day.

The failure mode is worth naming, because it is the one this architecture actually has, and because the screenshot above is it. A reconciler that cannot reach its repository keeps every container running exactly as before, so container-level alerting stays green while the desired-state loop is dead. Every stack in that fleet has been running perfectly for weeks while quietly no longer being reconciled against the repository, and nothing that was watching containers had any reason to say so. Monitoring the outcome is not the same as monitoring the mechanism, and the mechanism needs a check of its own. That lesson cost me more than the migration did, and it is the one I would want to be judged on.