Fleet deployment

A single Secronyx process is simple. A thousand of them is an identity, inventory and evidence problem. This page is about the parts that only appear at scale: how credentials are issued and revoked per host, what the rate limiter does to a fan-out sweep, how much of the tool catalogue each class of host should expose, and how the audit trail gets off the machines. Everything here uses the flags and file formats in the repository; nothing is aspirational.

Quick start: a fleet proxy with dial-in hosts

The fastest way to query a whole estate from one place is the fleet proxy. One host runs secronyx --fleet-proxy and becomes the single authenticated endpoint; every other host runs secronyx --fleet-connect and dials out to it, so no host needs an inbound port. Callers authenticate once, at the proxy, and it fans each read-only call out and aggregates the replies — up to 5,000 hosts. The four steps below get a fleet live; the sections after this are the deeper reference. A print version is available: Fleet Installation Guide (PDF).

1 · Install & verify the binary (every host)

Put a verified binary and a licence on every host, including the proxy. Downloads are open; verify the checksum, signature and provenance before you run. See Installation for per-OS detail.

curl -fLO https://secronyx.com/downloads/latest/secronyx-linux-amd64
curl -fLO https://secronyx.com/downloads/latest/SHA256SUMS
sha256sum -c SHA256SUMS --ignore-missing
cosign verify-blob --bundle secronyx-linux-amd64.cosign.bundle secronyx-linux-amd64
sudo install -m 0755 secronyx-linux-amd64 /usr/local/bin/secronyx
sudo mkdir -p /etc/secronyx && sudo cp acme.license /etc/secronyx/license
secronyx --version

The release build is licence-enforced: a host with no valid licence refuses to start, so place the licence first, or pass --license-file <path> explicitly.

2 · Run the fleet server (the proxy)

On the proxy host, mint a caller key and one dial-in agent key per host, declare the inventory, then start the server. Callers connect on :8443; members dial in on :9443.

# caller key — needs the fleet scope plus the tool scopes they may use
secronyx apikey create --file callers.json --name soc-analyst --scopes fleet,core,triage,logs,security
# one dial-in agent key per host — name it after the host
secronyx apikey create --file agent-keys.json --name web-1   # prints msk_… for that host

A fleet-inventory.json lists the hosts. A host with no url is expected to dial in, and listing it lets you declare its tags up front.

{
  "version": 1,
  "defaults": { "tags": { "env": "prod" } },
  "hosts": [
    { "name": "web-1", "os": "linux",   "tags": { "role": "web" } },
    { "name": "dc-1",  "os": "windows", "tags": { "role": "dc" } }
  ]
}

secronyx --fleet-proxy --fleet-inventory /etc/secronyx/fleet-inventory.json \
  --transport http --listen 0.0.0.0:8443 --server-url https://fleet.example.com:8443 \
  --tls-cert /etc/secronyx/tls.crt --tls-key /etc/secronyx/tls.key \
  --api-keys-file /etc/secronyx/callers.json \
  --fleet-agent-listen 0.0.0.0:9443 \
  --fleet-agent-tls-cert /etc/secronyx/tls.crt --fleet-agent-tls-key /etc/secronyx/tls.key \
  --fleet-agent-api-keys-file /etc/secronyx/agent-keys.json \
  --audit-output /var/log/secronyx/fleet-proxy-audit.jsonl --enable-sensitive

Prefer certificates to keys? Swap --api-keys-file for --tls-client-ca + --mtls-identity-file for callers, and --fleet-agent-client-ca + --fleet-agent-identities for members.

3 · Enrol each host as a dial-in member

On each host, drop in its agent key (printed in step 2) and its licence, then start the member. It opens one outbound websocket to the proxy and serves its tools over it — no inbound port, nothing to firewall. A member started with --fleet-connect is fleet-only: it opens no listener, and setting --transport alongside it is refused. secronyx --query … still works locally for troubleshooting.

printf '%s' "msk_…" | sudo tee /etc/secronyx/fleet-agent-key >/dev/null

secronyx --fleet-connect wss://fleet.example.com:9443/fleet/connect \
  --fleet-name web-1 --fleet-tags env=prod,os=linux,role=web \
  --fleet-key-file /etc/secronyx/fleet-agent-key \
  --license-file /etc/secronyx/license \
  --audit-output /var/log/secronyx/audit.jsonl

Run it as a service with systemd, a Windows scheduled task, or a macOS LaunchDaemon — see Windows service and Running as a service.

4 · Connect a client & verify

Point an MCP client at the proxy and ask one question of the whole fleet. Tool arguments are JSON; omit targets to hit every host, or filter by os, tags or names. (get_os_info is in the triage scope, which the caller key above includes.)

{ "mcpServers": { "fleet": {
    "type": "http",
    "url": "https://fleet.example.com:8443/",
    "headers": { "X-API-Key": "msk_your_caller_key" } } } }

fleet_hosts   { }                                  # who dialled in, with tags + reachability
fleet_run   { "tool": "get_os_info" }              # run one tool on every host (omit targets = all)
fleet_run   { "tool": "get_os_info", "targets": { "os": ["windows"] } }
fleet_ask   { "prompt": "which hosts have Defender disabled?" }

Caller scopes carry through every forwarded call, each host records the call under the proxy's identity with the caller and run id, and the proxy logs each fan-out as fleet/fanout. For topology trade-offs, per-host credentials, scope budgets and audit collection, read on.

Choose a topology first

There are three, and the choice is mostly about firewalls. The fleet proxy above is the quickest path; the two base topologies are:

Inbound. Each host runs --transport http with TLS, and one orchestrator calls them. You control the credential, the scope grant and the audit sink on each host, and the whole thing works with no external service. It requires an inbound path to every host — a listening port, a firewall rule, and a reachable address. Best for data centres, managed VM estates and clusters.

Outbound. Each host runs --mode hybrid and connects out to the Secronyx service; work arrives over that connection. Nothing listens, no inbound rule is needed, and hosts behind NAT, on laptops or in customer networks are reachable. In exchange, host identity and authorisation move to the service. See Hybrid and SaaS mode.

Proxy (fan-out). A middle path you own end to end: one host runs --fleet-proxy as the single authenticated endpoint, and the rest run --fleet-connect to dial in to it. Callers authenticate once, no host needs an inbound port, and one call fans out to the whole estate. See the quick start above.

Most real fleets end up mixed: inbound in the data centre, outbound for everything at the edge. The tool catalogue and the redaction and audit behaviour are identical either way.

Per-host credentials

Never share one bearer token across a fleet. A static token grants Scopes: ["*"] — the wildcard — to whoever holds it, and rotating it means touching every host at once. Use a credential type that carries its own identity and scope grant.

API keys

The key store is a JSON file of hashed records; the secret is never stored. A key looks like msk_<id>_<secret>, where the id is an 8-character hex handle used for O(1) lookup and the secret is 32 bytes from crypto/rand, held as a plain SHA-256 digest.

secronyx apikey create \
  --file /etc/secronyx/keys.json \
  --name "ops-orchestrator" \
  --scopes core,logs,triage,state \
  --expires 90d \
  --cidr 10.20.0.0/24

secronyx apikey list   --file /etc/secronyx/keys.json
secronyx apikey revoke --file /etc/secronyx/keys.json --id 3fa9c1e2

The key is printed once at creation. Each record carries id, name, hash, scopes, created_at, optional expires_at, disabled and optional allowed_cidrs; --expires accepts Go durations plus a d suffix for days. The server re-reads the store when its modification time changes (checked at most every two seconds), so apikey revoke takes effect fleet-wide as soon as your configuration tool has written the file — no restart. Clients present the key as X-API-Key: msk_... or Authorization: ApiKey msk_....

--cidr is what makes a stolen key much less useful: the CIDR check runs against the client IP taken from the request context, which is RemoteAddr unless --trust-proxy-headers is set. Get that setting wrong behind a proxy and you have pinned every key to the proxy's address.

Mutual TLS

Where you already have an internal CA or a SPIFFE issuer, mutual TLS avoids distributing a secret at all. --tls-client-ca makes the listener demand a certificate; --mtls-identity-file maps certificates to names and scopes:

{
  "version": 1,
  "identities": [
    {
      "match": { "spiffe_id": "spiffe://corp/ns/ai-ops/sa/orchestrator" },
      "name": "orchestrator",
      "scopes": ["core", "logs", "triage"]
    },
    {
      "match": { "spki_sha256": "9f86d0818884c7d6..." },
      "name": "incident-bot",
      "scopes": ["core", "state"],
      "disabled": false
    }
  ]
}

A match can be spki_sha256, spiffe_id, dns, cn or ou. A verified certificate with no entry gets --mtls-default-scopes, which is empty — that is, denied — unless you set it; --mtls-require-mapping denies it regardless. --mtls-trust-domain restricts acceptable SPIFFE trust domains, and --mtls-crl points at a PEM or DER revocation list checked against every client certificate. Both files are loaded at startup, so a broken one fails the process rather than the first request. The identity file is re-read when its modification time changes, checked at most every two seconds; the CRL is not, so replacing it needs a restart.

SSH-signed requests

If your fleet already trusts an SSH CA, you can reuse it without issuing anything new. The server takes --ssh-authorized-keys (and --ssh-ca-keys for certificate-based signing); each line accepts the options scopes=, expires=, from= and, on CA lines, principals=. An unknown option is an error, so a typo cannot silently widen access.

scopes="core,logs",from="10.20.0.0/24",expires="2026-12-31" ssh-ed25519 AAAAC3Nz... ops@example.com

Clients sign with secronyx ssh-sign --url <url> (--key <path> | --agent [--fingerprint <fp>]) [--cert <path>] [--body <file>|-] [--curl], which produces an Authorization: SSH-Sig keyid="…", ts="…", nonce="…", sig="…" header. See Tutorial: SSH key request signing.

Per-host scope budgets

The scope policy is applied at registration, ahead of the per-caller check that runs on every call: a scope that is not registered has no tools in tools/list and cannot be called by anyone, whatever their grant. That makes --scopes the cheapest and strongest fleet control you have, and it should vary by host class:

# Tier-1 application servers: triage only
--scopes core,logs,triage,state

# Database hosts: add storage and filesystem visibility
--scopes core,logs,triage,state,storage,hooks

# Windows IIS estate
--scopes core,logs,triage,state,windows

# Build agents: software inventory for SBOM work
--scopes core,software,state

SECRONYX_SCOPES carries the same value through the environment, which is usually easier to template. sensitive needs --enable-sensitive as well as being listed, and should be absent from a fleet default; when a specific investigation needs it, enable it on the one host, for the duration, and let the audit log record the calls. See Scopes and authorization.

Sweeping a fleet

The rate limiter is per server and per client address, so a sweep is bounded by both. Each server allows 20 requests per second sustained with a burst of 40, caps in-flight requests at 32 across all clients, and locks out an address for 15 minutes after 10 authentication failures inside a 15-minute window. Over-rate requests get 429 with Retry-After; over-concurrency gets 503 server busy with Retry-After: 1. One orchestrator calling a thousand hosts therefore has plenty of headroom per host — but a retry loop that reuses a bad credential will lock itself out of a host quickly, so make credential failures fatal rather than retried.

An inventory-driven sweep needs nothing more than the CMDB, curl, jq and xargs:

#!/usr/bin/env bash
# sweep.sh - call one tool on every host in an inventory CSV: host,port,keyfile
set -euo pipefail

TOOL="${1:-get_incident_triage_snapshot}"
OUT="${2:-./sweep-$(date +%Y%m%dT%H%M%S)}"
mkdir -p "$OUT"

call_one() {
  IFS=, read -r host port keyref <<<"$1"
  local key; key="$(cat "$keyref")"
  local body; body=$(printf '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"%s"}}' "$TOOL")

  # /health is unauthenticated: skip hosts that are simply down.
  if ! curl -fsS --max-time 5 "https://${host}:${port}/health" >/dev/null; then
    printf '%s\tunreachable\n' "$host" >>"$OUT/errors.tsv"; return
  fi

  curl -fsS --max-time 30 -X POST "https://${host}:${port}/" \
       -H "X-API-Key: ${key}" -H 'Content-Type: application/json' \
       -d "$body" \
    | jq -c --arg host "$host" '{host:$host, result:.}' >>"$OUT/results.jsonl" \
    || printf '%s\tcall-failed\n' "$host" >>"$OUT/errors.tsv"
}
export -f call_one; export TOOL OUT

tail -n +2 inventory.csv | xargs -P 32 -I{} bash -c 'call_one "$@"' _ {}

jq -s 'length as $n | {hosts:$n}' "$OUT/results.jsonl"

-P 32 is a sensible starting point: the limit that bites first is your orchestrator's sockets and the CMDB, not the servers. If you push concurrency high, spread it across hosts rather than hammering one — the per-server MaxConcurrent of 32 is shared by every caller.

Triage tools are the right thing to sweep with. get_incident_triage_snapshot and get_security_posture_snapshot combine many collectors into one result, so one call per host beats twenty. A worked example is in Tutorial: sweep a fleet from one agent.

Issuing credentials from the CMDB

Key issuance is a loop over the same inventory. Run it on each host through your configuration tool so the secret never travels:

# Run on each host, via Ansible/Salt/DSC; emits the one-time key to stdout.
secronyx apikey create \
  --file /etc/secronyx/keys.json \
  --name "orchestrator@$(hostname -f)" \
  --scopes "${MCP_SCOPES:-core,logs,triage,state}" \
  --expires 90d \
  --cidr "${ORCHESTRATOR_CIDR}"

- name: Install and configure Secronyx
  hosts: linux_fleet
  become: true
  tasks:
    - name: Install the binary
      ansible.builtin.copy:
        src: secronyx
        dest: /usr/local/bin/secronyx
        mode: "0755"

    - name: Install the unit
      ansible.builtin.copy:
        dest: /etc/systemd/system/secronyx.service
        mode: "0644"
        content: |
          [Unit]
          Description=Secronyx Server
          After=network.target

          [Service]
          Type=simple
          User=secronyx
          Restart=always
          ExecStart=/usr/local/bin/secronyx \
            --transport http \
            --listen 0.0.0.0:8443 \
            --tls-cert /etc/secronyx/tls/cert.pem \
            --tls-key /etc/secronyx/tls/key.pem \
            --api-keys-file /etc/secronyx/keys.json \
            --scopes {{ mcp_scopes | default('core,logs,triage,state') }} \
            --audit-output /var/log/secronyx/audit.jsonl \
            --trust-proxy-headers=false

          [Install]
          WantedBy=multi-user.target

Run the binary directly from the unit rather than through secronyx service install. The bundled service wrapper runs secronyx service run, which understands only the hybrid-mode options and answers --transport http with "HTTP transport mode not yet implemented"; it is for the outbound topology, not this one.

On Windows the enterprise MSI takes an API_KEY property — the SaaS registration key, not a key-store entry — and installs the agent in outbound hybrid mode. It is deployable through win_package, PowerShell DSC or a GPO software installation policy; see Windows service.

Rotation is the same loop with a new --name and a later --expires, followed by apikey revoke on the old id once the orchestrator has switched. Because the store is reloaded on change, neither step restarts anything.

Collecting the audit trail

Each host writes JSON Lines to --audit-output (default /var/log/secronyx/audit.jsonl), rotating at --audit-max-file-size (100 MiB) and keeping --audit-max-files (10). Records are hash-chained, so tampering with a line breaks verification of everything after it.

At fleet scale, do both of these:

• Ship the lines. Point your existing agent at the file, or set --audit-output /dev/stdout under a supervisor that captures stdout (this is what the container and Kubernetes deployments do). Centralised copies give you cross-host correlation.
• Verify locally, on the host. The chain can only be checked where the file is:

secronyx --audit-verify --audit-output /var/log/secronyx/audit.jsonl
# Audit verification OK: 128431 events verified

A non-zero exit is a finding. Run it on a schedule and alert on failure; a shipped copy that disagrees with a local verification is exactly the signal you want. --audit-sync-write fsyncs every record for hosts where losing the last few seconds is unacceptable, at a throughput cost. The record format and the verification procedure are in Audit logging and Tutorial: verify the audit chain.

Fleet checklist

• One credential per host, or per host class, with an expiry and a CIDR pin.
• --scopes set per host class; sensitive off everywhere by default.
• TLS on every listener, or loopback plus a proxy; never --allow-unauthenticated in production.
• --trust-proxy-headers set to match reality, because audit IPs, API key CIDRs and SSH from= all depend on it.
• Rate limiting left on; treat 429/503 as backpressure and credential failures as fatal.
• Audit shipped centrally and verified locally, on a schedule.
• Revocation tested: revoke a key, confirm the next call fails without a restart.