Quiet VPS Watchdog

Status: Retired (August 2026) · Source available on request

The server this watchdog guarded was shut down on 31 August 2026 as a deliberate downsizing: the site moved to Cloudflare Pages, ShiftScan to Render, the agent gateway to a workstation. Managed platforms bring their own health checks, so the watchdog’s job disappeared with the host. The page is kept as a record of the design, which would apply unchanged to the next server.

Problem

A personal server that ran n8n, an AI agent gateway, a static site, a feed reader and a handful of Coolify-managed containers needed watching. The usual failure mode of agent-driven monitoring is not missing incidents — it is noise: an LLM asked “is everything fine?” every few minutes will eventually say something, and an alert that repeats every cycle stops being read.

The goal was the opposite: a watchdog that produces nothing when the server is healthy, exactly one message when a problem appears, silence while that same problem persists, and exactly one message when it clears.

What it does

A single dependency-free Python script ran every five minutes from the agent’s scheduler. It checked:

• Disk and inode usage on the root and Docker filesystems, against warning and critical thresholds
• Memory pressure, measured from MemAvailable rather than free memory
• Required systemd units (system and user scope) and any unit that has entered the failed state
• Expected Docker containers by name pattern — present, running, and not unhealthy or stuck in health: starting — plus any other container that is dead or restarting
• HTTP readiness of the public and Tailscale-internal endpoints, with per-target allowed status codes
• TLS certificate expiry for every public hostname
• Age of the newest n8n database backup

Every finding becomes an Issue(severity, key, detail). The list is sorted, serialised and hashed; that hash is the run’s signature.

The quiet part

The script’s own output is the whole notification policy:

issues and signature changed   → print the issue list      (one alert)
issues and signature unchanged → print nothing             (no repeat)
no issues, previous run had    → print one recovery line   (one all-clear)
no issues, previous run clean  → print nothing             (silence)

The scheduler’s instruction to the agent was a single sentence: deliver stdout verbatim; empty stdout means healthy and must remain silent. The agent never decided whether something was wrong — it was a delivery channel for a deterministic result.

State is one small JSON file (signature, had_issues, issue_count, checked_at) written atomically through a temp file and os.replace, so a crash mid-write cannot corrupt the memory of the previous run. A --mode report flag prints a full human-readable readiness summary on demand, for when someone actually wants to see the numbers.

Design decisions

• Zero LLM calls in the check path. Every judgment is a threshold, a status string or a certificate date. The cost of a healthy cycle is a few subprocess calls, and the result is the same every time for the same inputs.
• Deduplicate on content, not on time. A new disk warning next to an existing container failure changes the signature, so it is reported; the same two problems ten minutes later do not.
• No autonomous remediation. The watchdog restarts nothing and touches no service. A container that is down stays down until a person decides what to do — the operator, not the script, owns the blast radius.
• Fail loudly on its own errors. If the config is missing a required section or a check raises unexpectedly, the script exits non-zero with the error on stderr instead of reporting “healthy” by omission.
• Config is data. Targets, thresholds, container patterns and backup globs live in a JSON file next to the script; adding a service is a config change, not a code change.

Part of a wider self-hosted agent setup

The watchdog was one piece of running an AI agent (Hermes) against real infrastructure with narrow permissions:

• Read-only n8n access for the agent. The agent reached the self-hosted n8n instance (2.36.7) through a local stdio MCP bridge — a third-party project, hermes-n8n-mcp, not my code. The bridge exposes eleven tools; the agent’s configuration allowed eight, all read-only (health, list_workflows, get_workflow, find_workflows, list_executions, get_execution, recent_failures, export_workflow). The mutation tools — activate_workflow, deactivate_workflow — and raw container logs were excluded at the client. The n8n API key lived in a separate environment file, never in the agent’s configuration.
• No public ports for the bridge. Stdio only; n8n was reached on the loopback interface.

The same principle ran through both: the agent got a deterministic, bounded view of production, and anything that changed production stayed with a human.

Retrospective: what carried forward

Retiring a system is a result too. What this one proved, and what moved into later work:

• Silence is a feature you design, not a lucky outcome. The signature-hash dedupe — report on change, not on time — is the single idea most worth keeping, and it reappears in how my later automations decide when to notify a human.
• Keep the LLM out of the judgment path. The watchdog’s agent delivered results; it never interpreted them. The same boundary — deterministic checks decide, the model drafts or transports — became the core contract of the AI Support Operations Copilot.
• Fail loudly on your own errors. A monitor that reports “healthy” when its config is broken is worse than no monitor. Exiting non-zero instead of defaulting to green is now a standing rule in everything I build.
• Narrow, read-only agent access works in practice. Eight read-only tools out of eleven, mutations excluded at the client, keys outside the agent’s configuration — running an agent against real infrastructure taught me more about permission design than any amount of theory.
• Know when the job is gone. Managed platforms brought their own health checks, so the watchdog was retired rather than kept alive for sentiment. Deleting a working system when its reason disappears is part of operating honestly.

Known limitations

• Single host. The script has no notion of a fleet; it was written for one VPS and reads local /proc, systemctl and docker.
• Thresholds are static. There is no anomaly detection, trend, or rate-of-change alerting.
• The alert channel is whatever the agent’s scheduler delivers to; the script itself has no notifier.
• Source is private because the configuration names internal hosts and containers. A sanitised copy can be shared on request.
• Retired with the host. Nothing here runs today; the managed platforms that replaced the server are monitored by their providers.