Fleet deployment

A single Secronyx process is simple. A thousand of them is an identity, inventory and evidence problem. This page is about the parts that only appear at scale: how credentials are issued and revoked per host, what the rate limiter does to a fan-out sweep, how much of the tool catalogue each class of host should expose, and how the audit trail gets off the machines. Everything here uses the flags and file formats in the repository; nothing is aspirational.

Choose a topology first

There are two, and the choice is about firewalls rather than preference.

Inbound. Each host runs --transport http with TLS, and one orchestrator calls them. You control the credential, the scope grant and the audit sink on each host, and the whole thing works with no external service. It requires an inbound path to every host — a listening port, a firewall rule, and a reachable address. Best for data centres, managed VM estates and clusters.

Outbound. Each host runs --mode hybrid and connects out to the Secronyx service; work arrives over that connection. Nothing listens, no inbound rule is needed, and hosts behind NAT, on laptops or in customer networks are reachable. In exchange, host identity and authorisation move to the service. See Hybrid and SaaS mode.

Most real fleets end up mixed: inbound in the data centre, outbound for everything at the edge. The tool catalogue and the redaction and audit behaviour are identical either way.

Per-host credentials

Never share one bearer token across a fleet. A static token grants Scopes: ["*"] — the wildcard — to whoever holds it, and rotating it means touching every host at once. Use a credential type that carries its own identity and scope grant.

API keys

The key store is a JSON file of hashed records; the secret is never stored. A key looks like msk_<id>_<secret>, where the id is an 8-character hex handle used for O(1) lookup and the secret is 32 bytes from crypto/rand, held as a plain SHA-256 digest.

secronyx apikey create \
  --file /etc/secronyx/keys.json \
  --name "ops-orchestrator" \
  --scopes core,logs,triage,state \
  --expires 90d \
  --cidr 10.20.0.0/24

secronyx apikey list   --file /etc/secronyx/keys.json
secronyx apikey revoke --file /etc/secronyx/keys.json --id 3fa9c1e2

The key is printed once at creation. Each record carries id, name, hash, scopes, created_at, optional expires_at, disabled and optional allowed_cidrs; --expires accepts Go durations plus a d suffix for days. The server re-reads the store when its modification time changes (checked at most every two seconds), so apikey revoke takes effect fleet-wide as soon as your configuration tool has written the file — no restart. Clients present the key as X-API-Key: msk_... or Authorization: ApiKey msk_....

--cidr is what makes a stolen key much less useful: the CIDR check runs against the client IP taken from the request context, which is RemoteAddr unless --trust-proxy-headers is set. Get that setting wrong behind a proxy and you have pinned every key to the proxy's address.

Mutual TLS

Where you already have an internal CA or a SPIFFE issuer, mutual TLS avoids distributing a secret at all. --tls-client-ca makes the listener demand a certificate; --mtls-identity-file maps certificates to names and scopes:

{
  "version": 1,
  "identities": [
    {
      "match": { "spiffe_id": "spiffe://corp/ns/ai-ops/sa/orchestrator" },
      "name": "orchestrator",
      "scopes": ["core", "logs", "triage"]
    },
    {
      "match": { "spki_sha256": "9f86d0818884c7d6..." },
      "name": "incident-bot",
      "scopes": ["core", "state"],
      "disabled": false
    }
  ]
}

A match can be spki_sha256, spiffe_id, dns, cn or ou. A verified certificate with no entry gets --mtls-default-scopes, which is empty — that is, denied — unless you set it; --mtls-require-mapping denies it regardless. --mtls-trust-domain restricts acceptable SPIFFE trust domains, and --mtls-crl points at a PEM or DER revocation list checked against every client certificate. Both files are loaded at startup, so a broken one fails the process rather than the first request. The identity file is re-read when its modification time changes, checked at most every two seconds; the CRL is not, so replacing it needs a restart.

SSH-signed requests

If your fleet already trusts an SSH CA, you can reuse it without issuing anything new. The server takes --ssh-authorized-keys (and --ssh-ca-keys for certificate-based signing); each line accepts the options scopes=, expires=, from= and, on CA lines, principals=. An unknown option is an error, so a typo cannot silently widen access.

scopes="core,logs",from="10.20.0.0/24",expires="2026-12-31" ssh-ed25519 AAAAC3Nz... ops@example.com

Clients sign with secronyx ssh-sign --url <url> (--key <path> | --agent [--fingerprint <fp>]) [--cert <path>] [--body <file>|-] [--curl], which produces an Authorization: SSH-Sig keyid="…", ts="…", nonce="…", sig="…" header. See Tutorial: SSH key request signing.

Per-host scope budgets

The scope policy is applied at registration, ahead of the per-caller check that runs on every call: a scope that is not registered has no tools in tools/list and cannot be called by anyone, whatever their grant. That makes --scopes the cheapest and strongest fleet control you have, and it should vary by host class:

# Tier-1 application servers: triage only
--scopes core,logs,triage,state

# Database hosts: add storage and filesystem visibility
--scopes core,logs,triage,state,storage,hooks

# Windows IIS estate
--scopes core,logs,triage,state,windows

# Build agents: software inventory for SBOM work
--scopes core,software,state

SECRONYX_SCOPES carries the same value through the environment, which is usually easier to template. sensitive needs --enable-sensitive as well as being listed, and should be absent from a fleet default; when a specific investigation needs it, enable it on the one host, for the duration, and let the audit log record the calls. See Scopes and authorization.

Sweeping a fleet

The rate limiter is per server and per client address, so a sweep is bounded by both. Each server allows 20 requests per second sustained with a burst of 40, caps in-flight requests at 32 across all clients, and locks out an address for 15 minutes after 10 authentication failures inside a 15-minute window. Over-rate requests get 429 with Retry-After; over-concurrency gets 503 server busy with Retry-After: 1. One orchestrator calling a thousand hosts therefore has plenty of headroom per host — but a retry loop that reuses a bad credential will lock itself out of a host quickly, so make credential failures fatal rather than retried.

An inventory-driven sweep needs nothing more than the CMDB, curl, jq and xargs:

#!/usr/bin/env bash
# sweep.sh - call one tool on every host in an inventory CSV: host,port,keyfile
set -euo pipefail

TOOL="${1:-get_incident_triage_snapshot}"
OUT="${2:-./sweep-$(date +%Y%m%dT%H%M%S)}"
mkdir -p "$OUT"

call_one() {
  IFS=, read -r host port keyref <<<"$1"
  local key; key="$(cat "$keyref")"
  local body; body=$(printf '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"%s"}}' "$TOOL")

  # /health is unauthenticated: skip hosts that are simply down.
  if ! curl -fsS --max-time 5 "https://${host}:${port}/health" >/dev/null; then
    printf '%s\tunreachable\n' "$host" >>"$OUT/errors.tsv"; return
  fi

  curl -fsS --max-time 30 -X POST "https://${host}:${port}/" \
       -H "X-API-Key: ${key}" -H 'Content-Type: application/json' \
       -d "$body" \
    | jq -c --arg host "$host" '{host:$host, result:.}' >>"$OUT/results.jsonl" \
    || printf '%s\tcall-failed\n' "$host" >>"$OUT/errors.tsv"
}
export -f call_one; export TOOL OUT

tail -n +2 inventory.csv | xargs -P 32 -I{} bash -c 'call_one "$@"' _ {}

jq -s 'length as $n | {hosts:$n}' "$OUT/results.jsonl"

-P 32 is a sensible starting point: the limit that bites first is your orchestrator's sockets and the CMDB, not the servers. If you push concurrency high, spread it across hosts rather than hammering one — the per-server MaxConcurrent of 32 is shared by every caller.

Triage tools are the right thing to sweep with. get_incident_triage_snapshot and get_security_posture_snapshot combine many collectors into one result, so one call per host beats twenty. A worked example is in Tutorial: sweep a fleet from one agent.

Issuing credentials from the CMDB

Key issuance is a loop over the same inventory. Run it on each host through your configuration tool so the secret never travels:

# Run on each host, via Ansible/Salt/DSC; emits the one-time key to stdout.
secronyx apikey create \
  --file /etc/secronyx/keys.json \
  --name "orchestrator@$(hostname -f)" \
  --scopes "${MCP_SCOPES:-core,logs,triage,state}" \
  --expires 90d \
  --cidr "${ORCHESTRATOR_CIDR}"

- name: Install and configure Secronyx
  hosts: linux_fleet
  become: true
  tasks:
    - name: Install the binary
      ansible.builtin.copy:
        src: secronyx
        dest: /usr/local/bin/secronyx
        mode: "0755"

    - name: Install the unit
      ansible.builtin.copy:
        dest: /etc/systemd/system/secronyx.service
        mode: "0644"
        content: |
          [Unit]
          Description=Secronyx Server
          After=network.target

          [Service]
          Type=simple
          User=secronyx
          Restart=always
          ExecStart=/usr/local/bin/secronyx \
            --transport http \
            --listen 0.0.0.0:8443 \
            --tls-cert /etc/secronyx/tls/cert.pem \
            --tls-key /etc/secronyx/tls/key.pem \
            --api-keys-file /etc/secronyx/keys.json \
            --scopes {{ mcp_scopes | default('core,logs,triage,state') }} \
            --audit-output /var/log/secronyx/audit.jsonl \
            --trust-proxy-headers=false

          [Install]
          WantedBy=multi-user.target

Run the binary directly from the unit rather than through secronyx service install. The bundled service wrapper runs secronyx service run, which understands only the hybrid-mode options and answers --transport http with "HTTP transport mode not yet implemented"; it is for the outbound topology, not this one.

On Windows the enterprise MSI takes an API_KEY property — the SaaS registration key, not a key-store entry — and installs the agent in outbound hybrid mode. It is deployable through win_package, PowerShell DSC or a GPO software installation policy; see Windows service.

Rotation is the same loop with a new --name and a later --expires, followed by apikey revoke on the old id once the orchestrator has switched. Because the store is reloaded on change, neither step restarts anything.

Collecting the audit trail

Each host writes JSON Lines to --audit-output (default /var/log/secronyx/audit.jsonl), rotating at --audit-max-file-size (100 MiB) and keeping --audit-max-files (10). Records are hash-chained, so tampering with a line breaks verification of everything after it.

At fleet scale, do both of these:

• Ship the lines. Point your existing agent at the file, or set --audit-output /dev/stdout under a supervisor that captures stdout (this is what the container and Kubernetes deployments do). Centralised copies give you cross-host correlation.
• Verify locally, on the host. The chain can only be checked where the file is:

secronyx --audit-verify --audit-output /var/log/secronyx/audit.jsonl
# Audit verification OK: 128431 events verified

A non-zero exit is a finding. Run it on a schedule and alert on failure; a shipped copy that disagrees with a local verification is exactly the signal you want. --audit-sync-write fsyncs every record for hosts where losing the last few seconds is unacceptable, at a throughput cost. The record format and the verification procedure are in Audit logging and Tutorial: verify the audit chain.

Fleet checklist

• One credential per host, or per host class, with an expiry and a CIDR pin.
• --scopes set per host class; sensitive off everywhere by default.
• TLS on every listener, or loopback plus a proxy; never --allow-unauthenticated in production.
• --trust-proxy-headers set to match reality, because audit IPs, API key CIDRs and SSH from= all depend on it.
• Rate limiting left on; treat 429/503 as backpressure and credential failures as fatal.
• Audit shipped centrally and verified locally, on a schedule.
• Revocation tested: revoke a key, confirm the next call fails without a restart.