Prometheus Alertmanager alert to host investigation
An Alertmanager alert brings signed evidence from inside the host in 10 seconds, and Claude traces a disguised process with no shell access.
The scenario
Two identical web servers, web-1 and web-2, and a database host labelled as holding card data ran in AWS. We planted a harmless stand-in for a compromise on web-2: a small script in a hidden directory, named to look like a time-sync helper, kept running by a systemd service and an obfuscated cron entry. Every few seconds it tried to reach an address in a documentation-only range that does not answer.
The alert
Prometheus fired FailingOutboundConnections for web-2, and Alertmanager routed it to the Secronyx alert bridge through a standard webhook receiver whose key can post alerts and nothing else.
At a glance
Prometheus fires FailingOutboundConnections for web-2
Alertmanager routes the alert to the Secronyx alert bridge
The bridge runs a host-compromise playbook of 17 read-only checks
web-2 signs every result within ten seconds; they are saved as an evidence case
Claude follows up with Prometheus and Secronyx, comparing web-2 with web-1
Claude reports a likely compromise and changes nothing
What Secronyx gathered, and how it was restricted
What it gathered from the host
- Running processes: the process, the user it runs as, its script's path, size and hash, and whether a package owns it
- Systemd units and cron entries that keep a process alive
- The host's own connection table, showing the address being called
- SELinux status and the attempts it logged
- The same checks on the twin host, for comparison
How it was restricted
- Read-only checks: no check returns file contents, and none changes the host
- A label policy decides what each caller may see; the card-data host is reserved for the security team
- Before a host runs anything, it checks a single-use pass signed by the commander, naming the caller and the rule that allowed the call
- Each host writes every call to its own tamper-evident audit log
- Claude had two tools only, Prometheus's MCP server and the commander: no shell and no file access
What the AI concluded
Claude's verdict was likely compromise on web-2. It named the process and its script, both ways it was kept alive with the cron command decoded, the address it kept calling, and that none of this exists on web-1. It could not see how the script got onto web-2 or read its contents, and it proposed isolating and rebuilding web-2 while changing nothing.
The proof
Figures from the recorded run, as reported in the walkthrough.
10 s
from the alert reaching the bridge to signed host evidence
40 calls
made by Claude: 32 to Secronyx and 8 to Prometheus
36 of 36
signed findings verified offline; changing one digit makes verification fail
0 calls
reached the card-data host

See every step
The walkthrough has the setup, the full story with screenshots, the evidence and how to run it yourself.
Product names belong to their owners and are used only to say what each demo connects to. Secronyx is not affiliated with, sponsored or endorsed by them.