Pro plan from £5 per endpoint per month · see pricing
Secronyx
PrometheusAlertmanagerClaude

Prometheus Alertmanager alert to host investigation

An Alertmanager alert brings signed evidence from inside the host in 10 seconds, and Claude traces a disguised process with no shell access.

The scenario

Two identical web servers, web-1 and web-2, and a database host labelled as holding card data ran in AWS. We planted a harmless stand-in for a compromise on web-2: a small script in a hidden directory, named to look like a time-sync helper, kept running by a systemd service and an obfuscated cron entry. Every few seconds it tried to reach an address in a documentation-only range that does not answer.

The alert

Prometheus fired FailingOutboundConnections for web-2, and Alertmanager routed it to the Secronyx alert bridge through a standard webhook receiver whose key can post alerts and nothing else.

At a glance

  1. Prometheus fires FailingOutboundConnections for web-2

  2. Alertmanager routes the alert to the Secronyx alert bridge

  3. The bridge runs a host-compromise playbook of 17 read-only checks

  4. web-2 signs every result within ten seconds; they are saved as an evidence case

  5. Claude follows up with Prometheus and Secronyx, comparing web-2 with web-1

  6. Claude reports a likely compromise and changes nothing

What Secronyx gathered, and how it was restricted

What it gathered from the host

  • Running processes: the process, the user it runs as, its script's path, size and hash, and whether a package owns it
  • Systemd units and cron entries that keep a process alive
  • The host's own connection table, showing the address being called
  • SELinux status and the attempts it logged
  • The same checks on the twin host, for comparison

How it was restricted

  • Read-only checks: no check returns file contents, and none changes the host
  • A label policy decides what each caller may see; the card-data host is reserved for the security team
  • Before a host runs anything, it checks a single-use pass signed by the commander, naming the caller and the rule that allowed the call
  • Each host writes every call to its own tamper-evident audit log
  • Claude had two tools only, Prometheus's MCP server and the commander: no shell and no file access

What the AI concluded

Claude's verdict was likely compromise on web-2. It named the process and its script, both ways it was kept alive with the cron command decoded, the address it kept calling, and that none of this exists on web-1. It could not see how the script got onto web-2 or read its contents, and it proposed isolating and rebuilding web-2 while changing nothing.

The proof

Figures from the recorded run, as reported in the walkthrough.

10 s

from the alert reaching the bridge to signed host evidence

40 calls

made by Claude: 32 to Secronyx and 8 to Prometheus

36 of 36

signed findings verified offline; changing one digit makes verification fail

0 calls

reached the card-data host

Terminal output of the Secronyx offline evidence check for web-2: VERIFIED with 36 findings, then FAILED after one digit in a stored result is changed
The alert bridge's evidence case verifies offline. Change one digit in one stored result and it fails.

See every step

The walkthrough has the setup, the full story with screenshots, the evidence and how to run it yourself.

Product names belong to their owners and are used only to say what each demo connects to. Secronyx is not affiliated with, sponsored or endorsed by them.