Turn a Prometheus alert into a host-level investigation, without giving AI shell access

Prometheus saw web-2's outbound connections going unanswered, but not why; through Secronyx, Claude found a disguised process, the systemd unit and hidden cron job that kept it alive, and the address it was calling, with no shell access and no changes to the host.

Request accessSee the investigation

This is a recorded run in AWS, not a mock-up. Prometheus fired an alert, Alertmanager sent it to Secronyx, and Claude, Anthropic's AI assistant, picked up from there. We recorded it on 12 November 2026 in AWS's London region. All times are UTC.

The problem

• Metrics show symptoms, not causes. Prometheus can see that a host's connection attempts go unanswered. It can't see which process makes them, how it got there or what keeps it running.
• Giving an AI a shell is too risky. A shell can change anything and read anything. You don't hand that to an AI on a production server.
• There's no trail of what the AI looked at. When an AI reaches a verdict, you need to know what it saw, on which host, and who allowed it.

The investigation: a quiet beacon on web-2

We planted a harmless stand-in for a compromise on web-2, one of two identical web servers. It is a small script in a hidden directory, named to look like a time-sync helper. A systemd service and an obfuscated cron entry keep it running, and every few seconds it tries to reach an address that never answers (a documentation-only range, so nothing on the internet receives it). It changes nothing and is reachable from nowhere.

In the screenshots, teal boxes mark Secronyx and what it found. Tool names are blanked.

At 10:36:15 Prometheus fired FailingOutboundConnections for web-2, and Alertmanager routed it to the Secronyx alert bridge.

The metric behind the alert. It shows that something is retrying, not what.

At 10:36:20 the alert reached the Secronyx alert bridge on the fleet proxy. The bridge matched it to the host-compromise playbook and ran its 17 read-only checks on web-2. By 10:36:30, ten seconds later, every result was signed by web-2 and saved as an evidence case.

At 10:37:06 we started Claude with two tools only: Prometheus's MCP server for the metrics and the Secronyx fleet proxy for the hosts. No shell and no file access. Nobody steered it after the first prompt. It read the bridge's results, went further where the evidence pointed, compared web-2 with its twin web-1, and wrote its report at 10:39:17. It made 40 calls: 32 to Secronyx and 8 to Prometheus.

Its verdict was likely compromise on web-2, and it named:

• the process, running as root, its script's path, size and hash, and the fact that no package owns it;
• both ways it was kept alive, the systemd unit and the cron entry, with the cron command decoded;
• the address it kept calling, from the host's own connection table, and that SELinux logged the attempts but was not enforcing;
• that none of this exists on web-1.

Claude's report, unedited apart from the blanked tool names.

It was also clear about its limits. It could not see how the script got onto web-2. We planted it with AWS Systems Manager, which Claude listed as a place for a person to look. It could not read the script's contents, because Secronyx doesn't return file contents. It proposed isolating and rebuilding web-2 and changed nothing, because Secronyx can't change a host.

What Secronyx proves

What the AI actually saw

Before a host runs anything, it checks a single-use pass signed by the fleet proxy, naming the caller and the rule that allowed the call. The host then writes the call to its own tamper-evident audit log. web-2's log shows the bridge's 17 checks, then Claude's 9 follow-ups, each under its own rule.

web-2's audit log. Each call names who it was for (the alert bridge or the AI investigator), the run and the rule that allowed it.

What it wasn't allowed to see

pci-db-1, a database host labelled as holding card data, is reserved for the security team. Prometheus still monitors it, so Claude could see its metrics. But on the fleet it is outside Claude's access, and Claude reported it as unchecked rather than trying to reach it. pci-db-1's own audit log has no calls from this investigation, because nothing reached it.

Evidence you can check later

The bridge's evidence case can be checked offline by anyone with secronyx evidence verify. It checks the proxy's signature, the fingerprint (hash) of every result and web-2's signature on its own results. All 36 findings verified. Then we changed one stored result, the beacon's address, by a single digit, and verification failed, naming the result.

Green: the case verifies. Amber: change one digit in one stored result and it fails.

Why Secronyx alongside Alertmanager

Alertmanager routes the alert and Claude does the reasoning. Secronyx provides the controlled, read-only access to host-level evidence, with independently verifiable records of every investigation.

• Evidence within seconds of the alert. The bridge runs a fixed playbook as soon as the alert arrives, before anyone opens a laptop, so the evidence reflects the moment it fired.
• No model inside the product. The bridge is deterministic. Any AI that speaks MCP can pick up from its results, and so can a person.
• One webhook receiver. Alertmanager needs no new plugins: a standard webhook_configs entry with its own key, which can post alerts and nothing else.
• A label policy decides what each caller may see, the bridge and the AI alike, and restricted hosts stay invisible.
• Every check is in the answering host's own audit log, and the results are signed so they can be checked long after the incident.

Why not just run a script from the alert?

Alertmanager can call any webhook, so a team could point it at a script that logs in to the host and runs commands. That works, but it moves the risk rather than removing it.

How it fits together

The bridge and Claude talk only to the Secronyx fleet proxy, which passes read-only checks to the hosts each of them may see. The hosts connect out to the proxy, so none of them opens a listening port.

How it's set up

1. Install Secronyx on each host. Here that's three EC2 instances: web-1 and web-2, the web tier, and pci-db-1, a database labelled as holding card data. Prometheus scrapes all three with node_exporter.

2. Run the Secronyx fleet proxy with the alert bridge. One config file maps each alert name to a playbook and the alert's instance label to the host.

3. Write the label policy. The bridge and the AI may see hosts labelled env=demo. Hosts labelled data=pci are reserved for people in the secops group.

4. Add one receiver to Alertmanager:

   receivers:
     - name: secronyx
       webhook_configs:
         - url: https://fleet-proxy.example.com/integrations/alertmanager/alerts
           send_resolved: true
           http_config:
             authorization: {type: ApiKey, credentials_file: /etc/alertmanager/secronyx-key}

5. Connect your AI. Claude Code connected to the fleet proxy as an MCP server with a key in the ai-investigators group, alongside Prometheus's MCP server.

Try it on your fleet

Secronyx is in beta. Tell us about your fleet and we'll help you connect it to Alertmanager.

Request accessTalk to us

Read more: fleet deployment · the Grafana demo · the AWS DevOps Agent demo

Prometheus is a registered trademark of The Linux Foundation. Claude is a trademark of Anthropic. AWS is a trademark of Amazon.com, Inc. or its affiliates. Other names are trademarks of their owners. Secronyx is not affiliated with, sponsored or endorsed by The Linux Foundation, the Prometheus project or Amazon.