Tutorial: verify the audit chain

Secronyx writes every tool call and every authentication decision to an append-only JSON Lines file, and links the records with a SHA-256 hash chain so that an edit or a deletion can be detected after the fact. This walkthrough shows the record format, the built-in check, what each kind of tampering looks like, and how to verify the chain with a script that does not trust the binary that wrote it. Every command below was run against the current build and the outputs are what it printed. The reference page is Audit logging; the compliance framing is on Compliance mapping.

One disambiguation first. This page is about the server's own audit log. The get_audit_trail tool is a different thing: it reads the operating system's security audit events (auditd, the Windows Security log) and is part of the compliance tool group on the Tool reference.

1. Produce a log to verify

Auditing is on by default and writes to /var/log/secronyx/audit.jsonl. For a lab, point it somewhere writable and turn off buffering so every event hits the disk before the next command runs.

mkdir -p ~/audit-lab && cd ~/audit-lab

secronyx --transport http --listen 127.0.0.1:8093 \
  --audit-output ./audit.jsonl --audit-buffer-size 0 --audit-sync-write

2026/09/17 16:34:38 Audit logging enabled: provider=default output=./audit.jsonl
2026/09/17 16:34:38 Scope policy: 9 tools not registered (sensitive=false, scopes="")
2026/09/17 16:34:38 MCP HTTP Server starting on 127.0.0.1:8093
2026/09/17 16:34:38   Server URL: http://127.0.0.1:8093
2026/09/17 16:34:38   Auth:       none

Those lines go to stderr through Go's standard logger, which is where the date and time prefix comes from. The --audit-verify output in section 3 is printed without a prefix.

Make some traffic, including one call that fails:

for t in get_uptime get_cpu_info get_memory_info; do
  curl -s -X POST http://127.0.0.1:8093/ -H 'Content-Type: application/json' \
    -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"tools/call\",\"params\":{\"name\":\"$t\",\"arguments\":{}}}" > /dev/null
done

curl -s -X POST http://127.0.0.1:8093/ -H 'Content-Type: application/json' \
  -d '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"get_processes","arguments":{"limit":5,"sort_by":"cpu"}}}' > /dev/null

curl -s -X POST http://127.0.0.1:8093/ -H 'Content-Type: application/json' \
  -d '{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"get_nonexistent","arguments":{}}}' > /dev/null

jq -c '{seq,action,resource,result,prev:(.prev_hash//""|.[0:8]),h:(.hash[0:8])}' audit.jsonl

{"seq":1,"action":"tools/call","resource":"get_uptime","result":"success","prev":"","h":"b3c4d57a"}
{"seq":2,"action":"tools/call","resource":"get_cpu_info","result":"success","prev":"b3c4d57a","h":"f09f7f6a"}
{"seq":3,"action":"tools/call","resource":"get_memory_info","result":"success","prev":"f09f7f6a","h":"16724b9f"}
{"seq":4,"action":"tools/call","resource":"get_processes","result":"success","prev":"16724b9f","h":"b099e138"}
{"seq":5,"action":"tools/call","resource":"get_nonexistent","result":"error","prev":"b099e138","h":"22e4c789"}

Each record's prev_hash is the previous record's hash. The first record has no prev_hash at all, because the field is omitted when empty.

2. The record format

One JSON object per line, written with O_APPEND to a file created with mode 0640 in a directory created with mode 0750. The fields, in the order they are always serialised:

{
  "timestamp": "2026-09-17T15:34:41.085982496Z",
  "seq": 4,
  "event_id": "2733e1de-bb15-447b-8c31-95a20164a227",
  "action": "tools/call",
  "resource": "get_processes",
  "client_ip": "127.0.0.1",
  "params": {
    "limit": 5,
    "sort_by": "cpu"
  },
  "result": "success",
  "duration_ns": 80027126,
  "prev_hash": "16724b9fd02f43e964aba373d149ed1675ea0130c1e20600b54b16ba35bdcd17",
  "hash": "b099e138d47dc1e710c6824e57520ba752c8e1ea344c4faa8744f6e4b89a9287"
}

Omitted fields are absent rather than null, so a record carries only what applies. params is stored as it arrived; output redaction protects tool results, not the arguments, so do not design tooling that passes secrets as tool arguments.

Two shapes are worth recognising. An error:

{"timestamp":"2026-09-17T15:34:41.09648572Z","seq":5,"event_id":"f127733c-af55-4ba5-9428-628c8b75b271","action":"tools/call","resource":"get_nonexistent","client_ip":"127.0.0.1","result":"error","error":"tool not found","prev_hash":"b099e138...","hash":"22e4c789..."}

And an authentication success, whose metadata records how the caller proved who they were:

{"seq":1,"action":"auth/token_validation","identity":"carol","client_ip":"127.0.0.1","result":"success","metadata":{"client_id":"ssh:SHA256:DYnL7mPN9YrDFHjTivuXNYz9tCFVBX+2r5mnoVxr5l0","method":"ssh-signature","scopes":["core"]}}

A scope refusal appears as "result":"denied" with "error":"scope denied" on the tools/call event; tools/list is not itself audited.

3. Run the built-in check

--audit-verify walks the file, recomputes each hash, checks each link and exits non-zero on the first problem.

secronyx --audit-verify --audit-output ./audit.jsonl

Audit verification OK: 5 events verified

echo $?
# 0

It is a standalone mode: the server does not start, nothing listens, and the command is safe to run from cron or a monitoring check. Two behaviours to know before you wire it into alerting.

It opens the file for append, because it configures the ordinary file provider first. A file the checking user cannot write is not an error you will enjoy debugging:

2026/09/17 16:34:52 WARNING: audit provider "default" could not open "./ro.jsonl" (failed to open audit file: open ./ro.jsonl: permission denied); audit events are being written to stderr instead
Audit verification FAILED: stderr audit output cannot be verified; configure a file output

And a path that does not exist is created empty and reported as clean:

secronyx --audit-verify --audit-output ./ghost.jsonl
# Audit verification OK: 0 events verified
ls -l ghost.jsonl
# -rw-r----- 1 ops ops 0 Sep 17 16:34 ghost.jsonl

So a monitoring check must assert on the event count, not only on the exit status. A typo in the path otherwise reports success forever.

4. What tampering looks like

Work on copies. Change one field in the middle of the file:

cp audit.jsonl t1.jsonl
sed -i 's/get_memory_info/get_env_vars/' t1.jsonl
secronyx --audit-verify --audit-output ./t1.jsonl; echo "exit=$?"

Audit verification FAILED: hash mismatch at event 3 (seq=3)
exit=1

Delete an event entirely:

cp audit.jsonl t2.jsonl
sed -i '3d' t2.jsonl
secronyx --audit-verify --audit-output ./t2.jsonl; echo "exit=$?"

Audit verification FAILED: hash chain broken at event 3 (seq=4)
exit=1

Note the two different messages. hash mismatch means a record's own contents no longer hash to its stored hash. hash chain broken means a record's prev_hash does not match the hash of the record before it — the signature of a removed or reordered event. The reported seq tells you which event, and the surviving gap in seq tells you how many are missing.

Now the case the chain does not catch. Remove the last line:

cp audit.jsonl t3.jsonl
sed -i '5d' t3.jsonl
secronyx --audit-verify --audit-output ./t3.jsonl; echo "exit=$?"

Audit verification OK: 4 events verified
exit=0

A truncation at the tail leaves a perfectly valid chain. Nothing inside a single file can prove how long it should have been, which is why section 6 is about anchoring the head hash somewhere the server cannot reach.

A line that is not JSON at all fails earlier, in the provider: the hash chain is initialised from the last line of the file, so a corrupt tail stops the provider from opening the file and the fallback refuses to verify.

cp audit.jsonl t4.jsonl; printf 'not json\n' >> t4.jsonl
secronyx --audit-verify --audit-output ./t4.jsonl

2026/09/17 16:35:02 WARNING: audit provider "default" could not open "./t4.jsonl" (failed to parse last event for hash chain: invalid character 'o' in literal null (expecting 'u')); audit events are being written to stderr instead
Audit verification FAILED: stderr audit output cannot be verified; configure a file output

5. Verify it independently

The check above is run by the same binary that wrote the file. For evidence, verify with something else.

The hash input is recoverable textually. The stored hash is SHA-256 over the event serialised with that one field removed, and hash is always the last member of the object, so the bytes that were hashed are the line with its trailing ,"hash":"..." removed and the closing brace put back. No JSON re-serialisation, and therefore no dependence on how another language orders keys or formats numbers:

sed -n 4p audit.jsonl | python3 -c '
import sys, hashlib, re
line = sys.stdin.readline().rstrip("\n")
m = re.search(r",\"hash\":\"([0-9a-f]{64})\"\}$", line)
print("stored  ", m.group(1))
print("computed", hashlib.sha256((line[:m.start()] + "}").encode()).hexdigest())
'

stored   b099e138d47dc1e710c6824e57520ba752c8e1ea344c4faa8744f6e4b89a9287
computed b099e138d47dc1e710c6824e57520ba752c8e1ea344c4faa8744f6e4b89a9287

A complete verifier is about thirty lines. It also reads rotated .gz segments, and checks that seq never goes backwards — something the built-in check does not do:

#!/usr/bin/env python3
"""Independently verify an secronyx audit hash chain. Pass files oldest first."""
import gzip, hashlib, json, re, sys

TAIL = re.compile(r',"hash":"([0-9a-f]{64})"\}$')


def lines(path):
    opener = gzip.open if path.endswith(".gz") else open
    with opener(path, "rt", encoding="utf-8") as fh:
        for raw in fh:
            raw = raw.rstrip("\n")
            if raw:
                yield raw


def main(paths):
    prev_hash, prev_seq, count = "", 0, 0
    for path in paths:
        for raw in lines(path):
            count += 1
            m = TAIL.search(raw)
            if not m:
                sys.exit(f"{path}: event {count}: no trailing hash field")
            event = json.loads(raw)
            computed = hashlib.sha256(raw[: m.start()].encode() + b"}").hexdigest()
            if computed != m.group(1):
                sys.exit(f"{path}: hash mismatch at event {count} (seq={event['seq']})")
            if event.get("prev_hash", "") != prev_hash:
                sys.exit(f"{path}: hash chain broken at event {count} (seq={event['seq']})")
            if event["seq"] <= prev_seq:
                sys.exit(f"{path}: sequence went backwards at event {count} (seq={event['seq']})")
            prev_hash, prev_seq = m.group(1), event["seq"]
    print(f"OK: {count} events verified, last seq={prev_seq}")


if __name__ == "__main__":
    if len(sys.argv) < 2:
        sys.exit("usage: verify-audit.py <audit.jsonl[.gz]> [more files, oldest first]")
    main(sys.argv[1:])

python3 verify-audit.py audit.jsonl
# OK: 5 events verified, last seq=5
python3 verify-audit.py t1.jsonl
# t1.jsonl: hash mismatch at event 3 (seq=3)
python3 verify-audit.py t2.jsonl
# t2.jsonl: hash chain broken at event 3 (seq=4)

Because it takes several files, it also solves the rotation problem.

6. Rotation, restarts and the limits of the chain

The chain does not restart when the file does. At --audit-max-file-size bytes the current file is renamed to audit.jsonl.<YYYYMMDD-HHMMSS>, gzipped in the background, and a new file is opened; --audit-max-files rotated segments are kept. The first event in the new file carries the last event's hash from the old one, so the built-in check — which only ever reads one file — fails immediately:

ls -l rot/
# -rw-r----- 1 ops ops 1101 Sep 17 16:35 audit.jsonl
# -rw-rw-r-- 1 ops ops  605 Sep 17 16:35 audit.jsonl.20260917-153508.gz

secronyx --audit-verify --audit-output ./rot/audit.jsonl
# Audit verification FAILED: hash chain broken at event 1 (seq=6)

That is not corruption; it is a chain whose head is in the previous segment. Verify the segments together, oldest first:

python3 verify-audit.py rot/audit.jsonl.20260917-153508.gz rot/audit.jsonl
# OK: 8 events verified, last seq=8

Restarts behave the same way. On startup the provider reads the last line of the existing file, adopts its hash as the chain head and its seq as the counter, so a restarted server appends seq=6 with prev_hash equal to the hash of seq=5 and the file verifies as one continuous chain. The corollary is that two servers must never write to the same file: each would hold its own idea of the head and the chain would fork.

Now the honest limit. The chain is tamper-evident, not tamper-proof. Anyone who can write the file can rewrite every record and reseal the whole chain, and both verifiers will then say it is fine:

python3 verify-audit.py resealed.jsonl
OK: 6 events verified, last seq=6

secronyx --audit-verify --audit-output ./resealed.jsonl
Audit verification OK: 6 events verified

The chain's real value is that it makes surgical tampering impossible: you cannot change one line, and you cannot remove one event, without rewriting everything after it. To close the gap, remove the attacker's ability to rewrite history undetectably, by any of:

• Anchor the head. Record jq -r '.hash' audit.jsonl | tail -1 together with the event count, on a schedule, somewhere the audited host cannot write — a monitoring system, a ticket, a signed commit. A resealed chain will not reproduce a previously published head hash.
• Ship every line off the host as it is written, to a SIEM or log collector. The copy is written before an attacker gets root; the chain then lets you prove the copy and the original agree.
• Write to append-only storage. chattr +a on Linux, a WORM bucket, or a remote syslog sink.
• Alert on the gaps. Missing seq values, a seq that goes backwards, and a period with no events at all are all findings. The built-in check tests none of them directly — it only recomputes hashes and links — though a deleted event does surface as a broken chain.

A monitoring check that covers the realistic failure modes looks like this — note the count assertion, without which a mistyped path passes forever, and the 2>&1, because the failure message is printed on stderr:

#!/bin/sh
out=$(secronyx --audit-verify --audit-output /var/log/secronyx/audit.jsonl 2>&1) \
  || { echo "$out" >&2; exit 2; }
echo "$out"
case "$out" in
  "Audit verification OK: 0 events verified") echo "no events: wrong path or auditing off" >&2; exit 2 ;;
esac
exit 0

7. When there is no file to verify

Two configurations produce no verifiable log, and both announce themselves.

Auditing can be turned off outright, which logs a warning on every start:

2026/09/17 16:36:10 WARNING: audit logging is DISABLED

And if the configured output cannot be opened — an unwritable directory in a container is the usual cause — the provider falls back to JSON lines on stderr rather than going dark:

2026/09/17 16:36:24 WARNING: audit provider "default" could not open "/var/log/secronyx/audit.jsonl" (failed to create audit directory: mkdir /var/log/secronyx: permission denied); audit events are being written to stderr instead
2026/09/17 16:36:24 Audit logging enabled: provider=stderr output=/var/log/secronyx/audit.jsonl

Those stderr lines carry the same fields and the same hash chain, so if your platform captures stderr you can still verify them with the script in section 5 once the collector's own prefixes are stripped. But --audit-verify will not do it for you, and the chain head is lost on every restart. Treat the fallback as an alert, not a deployment model: mount a writable volume, or set --audit-output to a path the service user owns. Docker and Compose and Kubernetes and Helm cover the volume in each case.

Related pages

Audit logging documents every flag and the provider interface. Security model explains where the audit trail sits among the other controls, and Compliance mapping maps it to the control families that ask for one. The authentication tutorials — per-operator API keys, SSH key request signing and mutual TLS end to end — each end at the trail this page verifies. Back to the documentation index.