100% free and open source · every tool, every host, no licence key · read the licence
Secronyx
Get started

Tutorials

Tutorial: sweep a fleet from one agent

Issue scoped API keys, call tools/call over HTTPS against many hosts from one place, and aggregate the answers into operator-readable evidence, with a working bash sweep and a Python aggregator that handle scope denials, rate limits and result caps.

One host at a time is fine for an incident. A fleet question — "which of these 400 hosts is short of disk", "which are missing the patch", "which have a listener on 6379" — needs a sweep. Secronyx has no central component: a sweep is one caller fanning out over many independent read-only HTTP servers, each authenticating and auditing for itself. This walkthrough builds that sweep, from key issuance to an aggregated table, with scripts you can run.

Fleet deployment covers the topology and rollout. This page is the operator's side: making the calls and turning the answers into evidence.

Step 1: put an HTTP listener on each host

Each host runs its own server. The minimum that is safe off-loopback is TLS plus an authenticator; the server refuses to start on a non-loopback address without both, unless you pass --allow-unauthenticated (or set SECRONYX_ALLOW_UNAUTHENTICATED=1), which is for development only.

secronyx --transport http \
  --listen 0.0.0.0:8443 \
  --server-url https://app-07.example.com:8443 \
  --tls-cert /etc/secronyx/cert.pem \
  --tls-key  /etc/secronyx/key.pem \
  --api-keys-file /etc/secronyx/keys.json \
  --scopes core,triage,hooks,logs,resources \
  --audit-output /var/log/secronyx/audit.jsonl

--scopes decides what exists on that host at all: tools outside the listed scopes are never registered, so they cannot be called even by a key that claims them. Sensitive-scope tools need --enable-sensitive on top of that and stay off here. Rate limiting is on by default.

Step 2: issue a scoped, expiring, address-pinned key

Do not reuse one key everywhere. Issue one per caller, scoped to what the sweep actually reads:

secronyx apikey create \
  --file /etc/secronyx/keys.json \
  --name fleet-sweep \
  --scopes core,triage,hooks,logs \
  --expires 30d \
  --cidr 10.4.12.0/24
Created API key 3fa9c1e2 for "fleet-sweep" (scopes: core,triage,hooks,logs, expires 2026-10-17T10:14:02Z, from 10.4.12.0/24)
This key is shown once and cannot be recovered:
msk_3fa9c1e2_Yk9sQ2h2Vm5nNkJ4UnRLd3pFcDFhTHM4dWRJNGpYcWM

The format is msk_<id>_<secret>: a fixed msk prefix, an eight-character lowercase-hex public id, and a 32-byte secret in base64url. The store keeps only sha256:<hex> of the secret, so the printed line above is the only copy. --scopes * is refused outright — the CLI will not issue a wildcard key.

--cidr pins the key to source addresses. A key presented from outside them fails with API key 3fa9c1e2 is not permitted from this address, and if the server sits behind a proxy it only believes X-Forwarded-For when started with --trust-proxy-headers. --expires accepts Go durations plus a d suffix (90d, 12h, 30m).

Audit and rotate with the other two actions:

secronyx apikey list --file /etc/secronyx/keys.json
secronyx apikey revoke --file /etc/secronyx/keys.json --id 3fa9c1e2

list prints ID, NAME, STATUS (active, revoked or expired), CREATED, EXPIRES and SCOPES, and never prints a secret. Tutorial: per-operator API keys goes deeper; Authentication covers mutual TLS and SSH signing if a shared secret is not acceptable in your environment.

Step 3: make one call by hand before you script anything

The HTTP transport is a single JSON-RPC endpoint: POST /. Each POST is independent — the server does not require an initialize handshake before tools/call, which is what makes a stateless sweep possible.

curl -sS https://app-07.example.com:8443/ \
  -H 'Content-Type: application/json' \
  -H "X-API-Key: $MCP_KEY" \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/call",
       "params":{"name":"get_disk_info","arguments":{}}}' | jq .

The key may go in X-API-Key or as Authorization: ApiKey msk_...; the server checks X-API-Key first. A successful reply wraps the collector's JSON in MCP content:

{
  "jsonrpc": "2.0",
  "id": 1,
  "result": {
    "content": [
      { "type": "text",
        "text": "{\n  \"partitions\": [\n    {\n      \"device\": \"/dev/nvme0n1p2\",\n      \"mountpoint\": \"/\",\n      \"fstype\": \"ext4\",\n      \"total\": 214748364800,\n      \"used\": 203158092185,\n      \"free\": 11590272615,\n      \"used_percent\": 94.6\n    }\n  ],\n  \"timestamp\": \"2026-09-17T10:21:44Z\"\n}"
      }
    ]
  }
}

The payload is a JSON string inside content[0].text, not a nested object. Every aggregator you write has to parse twice. Two related checks before you trust a result:

  • result.isError is true when the tool ran and failed. The text then reads Error: <message>, or, past the 4 MiB --max-result-bytes cap, Error: result too large: N bytes exceeds the M byte cap; narrow the query (limit, lines, filters). There is no JSON-RPC error member in either case, so a script that only checks for error will treat a failure as data.
  • A JSON-RPC error member means the request never reached a collector. Code -32003 with message Forbidden and data tool "X" requires scope "Y" is a scope denial; -32602 with Tool not found means the tool is not registered on that host, usually because of its --scopes; -32601 is Method not found.

Confirm reachability separately. GET /health is unauthenticated and returns {"status":"ok"} — deliberately nothing else, since the auth method and transport are reconnaissance. GET /metrics serves Prometheus and does require authentication.

Step 4: the sweep script

Keep an inventory file of one host:port per line, then fan out with bounded concurrency. This script reads a tool name and arguments, writes one JSON Lines record per host, and never lets a single slow or unreachable host stall the run.

#!/usr/bin/env bash
# sweep.sh — call one tool across a fleet, emit JSON Lines to stdout.
# usage: MCP_KEY=msk_... ./sweep.sh hosts.txt get_disk_info '{}' [parallelism]
set -euo pipefail

HOSTS_FILE=${1:?inventory file required}
TOOL=${2:?tool name required}
ARGS=${3:-'{}'}
PARALLEL=${4:-16}
: "${MCP_KEY:?export MCP_KEY with the msk_ key}"

sweep_one() {
  local host="$1" tool="$2" args="$3"
  local body resp http rpc_err is_err payload
  body=$(printf '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"%s","arguments":%s}}' \
         "$tool" "$args")

  # --max-time bounds the whole call; --retry covers 429/503 with Retry-After.
  resp=$(curl -sS --max-time 25 --retry 3 --retry-delay 2 --retry-all-errors \
              -w '\n%{http_code}' \
              -H 'Content-Type: application/json' \
              -H "X-API-Key: ${MCP_KEY}" \
              -d "$body" \
              "https://${host}/" 2>/dev/null) || {
    jq -nc --arg host "$host" --arg tool "$tool" \
       '{host:$host,tool:$tool,ok:false,reason:"unreachable"}'
    return
  }

  http=$(tail -n1 <<<"$resp")
  resp=$(sed '$d' <<<"$resp")

  if [[ "$http" != "200" ]]; then
    jq -nc --arg host "$host" --arg tool "$tool" --arg code "$http" \
           --arg detail "$(jq -r '.error_description // .error // empty' <<<"$resp" 2>/dev/null)" \
       '{host:$host,tool:$tool,ok:false,reason:("http_"+$code),detail:$detail}'
    return
  fi

  rpc_err=$(jq -r '.error.data // .error.message // empty' <<<"$resp")
  if [[ -n "$rpc_err" ]]; then
    jq -nc --arg host "$host" --arg tool "$tool" --arg detail "$rpc_err" \
       '{host:$host,tool:$tool,ok:false,reason:"rpc_error",detail:$detail}'
    return
  fi

  is_err=$(jq -r '.result.isError // false' <<<"$resp")
  payload=$(jq -r '.result.content[0].text // empty' <<<"$resp")
  if [[ "$is_err" == "true" ]]; then
    jq -nc --arg host "$host" --arg tool "$tool" --arg detail "$payload" \
       '{host:$host,tool:$tool,ok:false,reason:"tool_error",detail:$detail}'
    return
  fi

  # content[0].text is a JSON *string*; --argjson re-parses it.
  jq -nc --arg host "$host" --arg tool "$tool" --argjson data "$payload" \
     '{host:$host,tool:$tool,ok:true,data:$data}'
}
export -f sweep_one
export MCP_KEY

grep -vE '^\s*(#|$)' "$HOSTS_FILE" \
  | xargs -P "$PARALLEL" -I{} bash -c 'sweep_one "$@"' _ {} "$TOOL" "$ARGS"
export MCP_KEY='msk_3fa9c1e2_Yk9sQ2h2Vm5nNkJ4UnRLd3pFcDFhTHM4dWRJNGpYcWM'
./sweep.sh hosts.txt get_disk_info '{}' 24 > disk.jsonl

Keep PARALLEL well under the number of hosts, not the number of requests one host can take: each server's default limits are 20 requests per second with a burst of 40 and 32 concurrent requests, and you are sending one request to each of many hosts. Where the sweep does bunch up on one host — a retry loop, a shared VIP — you will see 429 with a Retry-After header, or 503 server busy with Retry-After: 1.

Get the key wrong and something worse happens: the brute-force lockout counts 401 responses and, after 10 within the window, locks that client address out for 15 minutes. Test one host by hand before you fan out to four hundred.

Step 5: aggregate into evidence

Raw JSON Lines is not an answer. This Python script takes the sweep output and produces the operator-facing table, keeping the failures visible rather than dropping them — a sweep that silently skips unreachable hosts is worse than no sweep.

#!/usr/bin/env python3
"""aggregate.py — turn sweep.sh JSON Lines into an operator table.

    ./sweep.sh hosts.txt get_disk_info '{}' | ./aggregate.py disk
"""
import json
import sys
from collections import Counter


def load(stream):
    ok, failed = [], []
    for line in stream:
        line = line.strip()
        if not line:
            continue
        rec = json.loads(line)
        (ok if rec.get("ok") else failed).append(rec)
    return ok, failed


def report_disk(ok, threshold=85.0):
    """get_disk_info -> types.DiskInfo: partitions[] with used_percent."""
    rows = []
    for rec in ok:
        for part in rec["data"].get("partitions", []):
            if part.get("used_percent", 0.0) >= threshold:
                rows.append((
                    rec["host"],
                    part["mountpoint"],
                    part.get("device", ""),
                    part.get("fstype", ""),
                    part["used_percent"],
                    part.get("free", 0) / 2**30,
                ))
    rows.sort(key=lambda r: r[4], reverse=True)
    print(f"{'HOST':<28} {'MOUNT':<18} {'FSTYPE':<8} {'USED%':>7} {'FREE GiB':>9}")
    for host, mount, _dev, fstype, pct, free_gib in rows:
        print(f"{host:<28} {mount:<18} {fstype:<8} {pct:>7.1f} {free_gib:>9.1f}")
    return rows


def report_ports(ok, watch=(6379, 11211, 9200, 27017)):
    """get_listening_ports -> types.ListeningPortsResult: ports[]."""
    rows = []
    for rec in ok:
        for p in rec["data"].get("ports", []):
            if p["port"] in watch and not p["address"].startswith("127."):
                rows.append((rec["host"], p["protocol"], p["address"],
                             p["port"], p.get("process_name", "?"), p.get("pid", 0)))
    rows.sort()
    print(f"{'HOST':<28} {'PROTO':<6} {'ADDRESS':<18} {'PORT':>6} {'PROCESS':<16} {'PID':>7}")
    for host, proto, addr, port, proc, pid in rows:
        print(f"{host:<28} {proto:<6} {addr:<18} {port:>6} {proc:<16} {pid:>7}")
    return rows


REPORTS = {"disk": report_disk, "ports": report_ports}

if __name__ == "__main__":
    kind = sys.argv[1] if len(sys.argv) > 1 else "disk"
    ok, failed = load(sys.stdin)

    hits = REPORTS[kind](ok)

    print(f"\n{len(ok)} hosts answered, {len(failed)} did not, "
          f"{len(hits)} findings.", file=sys.stderr)
    if failed:
        print("\nNot answered:", file=sys.stderr)
        for reason, count in Counter(r["reason"] for r in failed).most_common():
            print(f"  {reason:<14} {count}", file=sys.stderr)
        for rec in failed[:10]:
            detail = (rec.get("detail") or "")[:100]
            print(f"  {rec['host']:<28} {rec['reason']:<14} {detail}", file=sys.stderr)
    # Non-zero exit when any host could not be reached, so CI notices.
    sys.exit(1 if failed else 0)
./sweep.sh hosts.txt get_disk_info '{}' | ./aggregate.py disk
HOST                         MOUNT              FSTYPE     USED%  FREE GiB
app-07.example.com:8443      /                  ext4        94.6      10.8
app-19.example.com:8443      /var/lib/docker    xfs         91.2      31.4
db-02.example.com:8443       /var/lib/pgsql     xfs         88.0      96.7

3 hosts answered, 1 did not, 3 findings.

Not answered:
  rpc_error      1
  app-31.example.com:8443      rpc_error      tool "get_disk_info" requires scope "core"

That last line is the point of keeping failures. app-31 is not healthy-by-omission; it answered with a scope denial, which means the tool is registered there but the identity it authenticated was not granted core — the key stores are per host, so one stale copy of keys.json is enough to cause it. Until someone fixes that host you have no disk reading for it at all. (Had the agent been started without core in its --scopes, the tool would not be registered and the reply would be Tool not found instead.)

The same shape works for any tool. get_listening_ports (scope hooks) for exposure sweeps, as in report_ports above; get_os_info (triage) for a build inventory; get_failed_units (triage) for service health; get_uptime (core) to find hosts that rebooted; get_journal_logs (logs) for unit logs; get_recently_installed_software (triage) or get_windows_hotfixes (software) for patch state. A tool answers only when its scope is both registered on the host and carried by the key, so get_windows_hotfixes needs software added to the host's --scopes and to the key — otherwise you get Tool not found from the host or Forbidden from the scope check. Arguments go in the third positional argument as JSON:

./sweep.sh hosts.txt get_journal_logs '{"lines":100,"unit":"nginx.service","priority":3}' > nginx.jsonl

Note priority is numeric here: the handler reads it as a number and ignores a name like err, and only values 1-7 reach journalctl --priority.

Step 6: keep the sweep cheap

A fleet sweep is a lot of collector runs. Three rules keep it from becoming the incident:

Ask for the narrowest tool. get_disk_info returns partitions; generate_system_report (scope report) returns everything and takes up to timeout_seconds (default 30) per host. Use the summary tools for summaries and the specific tools for specifics.

Bound every result. Tools that take lines or limit should get one. Past --max-result-bytes (4 MiB) you get a result with isError true and an explicit message telling you to narrow the query — which the sweep script above records as tool_error rather than crashing.

Never sweep the slow tools. On macOS hosts, get_recent_kernel_events and get_recent_critical_events are documented as taking a minute or more because they shell out to log show; get_recent_reboots, get_recent_service_failures, get_recent_resource_incidents, get_recent_config_changes and get_service_log_view carry the same warning in milder form. Across a fleet that is an hour of wall-clock time. Query those per host, after the sweep has narrowed the candidates.

Step 7: collect the other side of the record

Your sweep produced findings. Each host produced its own, independent, tamper-evident record of what you asked it. Every tools/call wrote a line to that host's audit log with timestamp, seq, event_id, action, resource (the tool name), identity (the key's name, so fleet-sweep), client_ip, params, result, duration_ns, prev_hash and hash. action is the literal string tools/call, and empty fields are omitted, so a call with no arguments carries no params and the very first record of a chain has no prev_hash (a restart resumes the chain from the last line already on disk).

while read -r host; do
  ssh "${host%%:*}" \
    "sudo secronyx --audit-verify --audit-output /var/log/secronyx/audit.jsonl" \
    && echo "$host chain ok" || echo "$host CHAIN FAILED"
done < hosts.txt

Authentication is audited too, successes and failures alike, so a sweep run with a stale key leaves a visible trail of denied results on every host it touched. That is the intended behaviour: the evidence that someone swept the fleet lives on the fleet, not in the sweeper's terminal. Audit logging and Tutorial: verify the audit chain cover the chain format and verification.

Where to go next

Built 2026-09-19. Source: levantar-ai/secronyx. Found a mistake? Tell us.