Pro plan from £5 per endpoint per month · see pricing
Secronyx
ZabbixClaude

Zabbix problem to host investigation

A Zabbix problem on a crash-looping service brings signed host evidence in under a second, and Claude finds the config change behind it.

The scenario

Two web servers, web-1 and web-2, ran a small service, orders-api, with systemd set to restart it whenever it stops, and Zabbix tracked the restart count. We broke web-2 the way a bad deploy would: one line in the service's env file moved it onto a port another program already held, so it could not start.

The alert

A Zabbix trigger raised a High problem, orders-api is restarting repeatedly, on web-2. A standard Zabbix webhook media type, with its key kept in a secret macro, sent the problem to the Secronyx alert bridge.

At a glance

  1. Zabbix raises a High problem: orders-api is restarting repeatedly on web-2

  2. A Zabbix webhook sends the problem to the Secronyx alert bridge

  3. The bridge runs a service-failure-triage playbook of 13 read-only checks

  4. web-2 signs every result about half a second later

  5. Claude follows up through Secronyx, comparing web-2 with web-1

  6. Claude reports a misconfiguration and leaves the fix to a person

What Secronyx gathered, and how it was restricted

What it gathered from the host

  • Recent configuration file changes, with the time each changed
  • Service failures and the error the service logged on each start
  • Listening ports on the host
  • Logins, deploys and package changes around the time of the problem
  • The same checks on the twin host, for comparison

How it was restricted

  • Read-only checks: no check changes the host, so the fix was left for a person to make
  • Reading the env file or the process's environment needs a scope this policy does not give the AI, so Claude could not read the new port number
  • A label policy decides what each caller may see; the card-data host is reserved for the security team
  • Each host checks a single-use pass signed by the commander and writes every call to its own tamper-evident audit log
  • Claude had one tool, the commander: no shell, no file access and no Zabbix access beyond the data in its first prompt

What the AI concluded

Claude's verdict was misconfiguration. It named the changed file and the time, the healthy listener stopping 17 milliseconds later, the address-in-use error on every start since, and that web-1 still runs the service on its usual port. Its fix was to restore the port setting to match web-1 and confirm the restart counter stops, and it asked a person to confirm the change was sanctioned.

The proof

Figures from the recorded run, as reported in the walkthrough.

< 1 s

from the Zabbix notification to signed host evidence

26 calls

made by Claude, all to Secronyx

28 of 28

signed findings verified offline; changing one letter makes verification fail

17 ms

between the config change and the healthy listener stopping, as Claude reported

Terminal output of the Secronyx offline evidence check for web-2: VERIFIED with 28 findings, then FAILED after one letter in a stored result is changed
The alert bridge's evidence case verifies offline. Change one letter in one stored result and it fails.

See every step

The walkthrough has the setup, the full story with screenshots, the evidence and how to run it yourself.

Product names belong to their owners and are used only to say what each demo connects to. Secronyx is not affiliated with, sponsored or endorsed by them.