Tutorial: diagnose a slow Linux host

A host is slow. You have a read-only diagnostics plane in front of it and a page saying "app-07 latency up". This walkthrough takes that host from "something is wrong" to a written finding using Secronyx tools only. Every command below is real, every tool name is registered in internal/mcp/tools*.go, and every field named is a field of the collector struct the tool serialises.

Two ways of calling the same collectors appear throughout:

• secronyx --query <name> --json, which bypasses the MCP protocol entirely and runs the collector in-process. Useful when you are on the host, and useful for seeing a result shape before you wire the tool into an agent.
• A JSON-RPC tools/call over stdio or HTTP, which is what an AI client actually sends. The envelope is described in the JSON-RPC API; the transport in Remote access over HTTP.

Watch for one trap that catches people early. --query runs a fixed form of most queries. For get_processes it is always GetTopProcesses(10, "cpu"); there is no CLI flag that changes the sort or the count for that query. When you need arguments, use tools/call.

Step 1: establish what the agent will answer

Before reading any numbers, find out which tools this host's agent exposes. Scopes are decided at registration time by --scopes and --enable-sensitive, so two hosts in the same fleet can offer different inventories. Ask the server rather than assuming:

printf '%s\n' \
  '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"triage","version":"1"}}}' \
  '{"jsonrpc":"2.0","id":2,"method":"tools/list"}' \
  | secronyx \
  | jq -r 'select(.id==2) | .result.tools[].name' | sort

initialize answers with "protocolVersion":"2024-11-05" plus the server name and version. tools/list returns every tool the caller may invoke; over an authenticated HTTP transport that list is filtered to the caller's granted scopes, so what you see is exactly what you can call.

The tools this page uses, and the scope each is registered under:

If a tool you need is missing, the server was started without that scope, so the tool was never registered — calling it anyway returns a JSON-RPC error with code -32602 and message Tool not found. A tool that is registered but falls outside the scopes your identity was granted fails differently: code -32003, message Forbidden, and data of the form tool "get_cgroups" requires scope "resources", with an audit record whose result is denied. Either way that is a configuration answer, not a diagnosis; see Scopes and authorization.

Step 2: take the baseline

Start with the cheap, whole-host picture, so you know whether you are looking at a busy host or a broken one.

secronyx --query get_os_info --json
secronyx --query get_uptime --json
secronyx --query get_cpu_info --json
secronyx --query get_memory_info --json

get_cpu_info returns types.CPUInfo:

{
  "percent": 0,
  "count": 8,
  "physical_count": 4,
  "frequency": { "current": 2100, "min": 800, "max": 3600 },
  "load_average": { "load1": 14.82, "load5": 11.03, "load15": 6.47 },
  "timestamp": "2026-09-17T09:41:02.118Z"
}

Read that percent carefully, because it is the trap in this result. It is a delta between two readings of /proc/stat held on the collector, and both the CLI path and the get_cpu_info handler build a fresh collector per call — there is no previous reading, so a one-shot call returns exactly 0. Do not read it as an idle CPU; read it as no measurement. The utilisation signal in this result is load_average, and the saturation signal is the PSI figures in step 3.

count is logical cores and physical_count is physical ones, which is what turns a load average into a ratio: load1 of 14.82 against 8 logical cores is a run queue roughly 1.85x oversubscribed. frequency is in MHz and is read from cpu0's cpufreq entries alone, with current falling back to /proc/cpuinfo; where the kernel exposes no cpufreq limits, min and max come back as 0, and get_cpu_frequency_scaling in step 6 is the tool that reports per-policy figures.

per_cpu is absent because the CLI path calls Collect(false). Passing the argument, which means tools/call, adds the key but not the data:

{"jsonrpc":"2.0","id":3,"method":"tools/call",
 "params":{"name":"get_cpu_info","arguments":{"per_cpu":true}}}

On Linux that returns a per_cpu array of one zero per logical core: the collector counts the per-core entries in /proc/stat but does not yet track previous per-core times, so the percentages are placeholders. For the one-core-pinned case, use get_interrupts_summary (step 6) and the per-process sampling in step 4 instead.

get_uptime returns boot_time, uptime (a Go duration, so nanoseconds) and uptime_str. A short uptime under a long-lived service reframes the whole investigation, so check it before anything else.

get_memory_info returns total, available, used and free in bytes alongside used_percent, with Linux-only buffers, shared, slab, sreclaimable, sunreclaimable, page_tables and swap_cached, a cached figure on Linux and Windows, plus a nested swap object carrying total, used, free, used_percent, sin and sout. available is the number that matters; free on Linux routinely looks alarming and means nothing.

Step 3: separate utilisation from saturation

A fully busy CPU is not necessarily a problem. Work waiting to run is. On Linux 4.20 and later, pressure stall information distinguishes the two directly:

secronyx --query get_pressure_stall_info --json

{
  "supported": true,
  "resources": [
    { "resource": "cpu",
      "some": { "avg10": 61.22, "avg60": 48.90, "avg300": 22.14, "total": 918273645 },
      "full": { "avg10": 0.00, "avg60": 0.00, "avg300": 0.00, "total": 0 } },
    { "resource": "memory",
      "some": { "avg10": 0.11, "avg60": 0.08, "avg300": 0.04, "total": 44120 },
      "full": { "avg10": 0.00, "avg60": 0.00, "avg300": 0.00, "total": 0 } },
    { "resource": "io",
      "some": { "avg10": 3.41, "avg60": 2.98, "avg300": 3.10, "total": 51224899 } }
  ],
  "timestamp": "2026-09-17T09:41:05.402Z"
}

some is the share of wall-clock time in which at least one task was stalled on that resource; full is the share in which every runnable task was stalled. The collector fills in whichever of some and full the kernel's pressure file reports and omits the other, so a resource with no full object simply had no full line to parse. Kernels that do emit a full line under cpu report it as zero, as above; either way some is the number to read for CPU. Here cpu.some.avg10 at 61% against avg300 at 22% says the stall is recent and growing, while io.some is flat and memory is negligible. That is a CPU problem, and it started in the last few minutes.

If supported is false, there was no /proc/pressure directory to read: the kernel is older than 4.20, or PSI is disabled. Fall back to the load-to-core ratio from step 2 plus get_vmstat_summary:

secronyx --query get_vmstat_summary --json

Its fields are context_switches, forks, page_faults, major_page_faults, swap_ins, swap_outs, pages_in and pages_out. These are cumulative since boot, so take two samples a known interval apart and subtract. Rising major_page_faults alongside swap_ins moves the investigation to memory even when the CPU looks busy.

Step 4: find the process, accurately

The obvious call is the wrong one here:

secronyx --query get_processes --json

That returns a bare JSON array of types.ProcessInfo — the handler calls GetTopProcesses, which returns a slice, not an object — and on Linux its cpu_percent is hard-coded to 0: the collector takes a single /proc snapshot and never computes a delta. Because the cpu sort then compares a column of zeroes, the "top 10 by CPU" it hands back is not ranked by CPU at all. For a CPU investigation you want the sampled variant, which takes two CPU-time measurements separated by a delay and computes the delta:

{"jsonrpc":"2.0","id":4,"method":"tools/call",
 "params":{"name":"get_processes_sampled","arguments":{"sample_duration_ms":2000}}}

The result is types.ProcessList. Two entries of a much longer array:

{
  "processes": [
    { "pid": 20714, "name": "ruby", "username": "app",
      "cpu_percent": 392.7, "mem_percent": 4.1, "mem_rss": 1378238464,
      "status": "running",
      "cmdline": "puma: cluster worker 3: 20470 [api]" },
    { "pid": 1189, "name": "postgres", "username": "postgres",
      "cpu_percent": 71.4, "mem_percent": 11.6, "mem_rss": 3902341120,
      "status": "running",
      "cmdline": "postgres: app api 10.0.4.19(51244) SELECT" }
  ],
  "total": 412,
  "timestamp": "2026-09-17T09:41:11.907Z"
}

Three things about that shape. The sampled collector returns one entry per process it managed to sample, in no particular order and with no top-N cut, and total is simply the length of processes — so you sort and truncate on your side, and on a host with hundreds of processes this is the call most likely to meet the result cap described at the end of this page. create_time is absent because the sampled path never populates it — and do not reach for get_processes to fill the gap either: on Linux it sets create_time from /proc/[pid]/stat's start time, which the kernel counts in ticks since boot, then renders that as a Unix epoch, so PID 1 comes back as 1970-01-01T00:00:00Z. Add the offset to boot_time from get_uptime yourself if you need a wall-clock start. And a process that exits during the sampling window is dropped from the result rather than reported with a partial figure.

cpu_percent is per-process and can exceed 100 on a multi-core host: 392.7 is roughly four cores' worth of work, computed as the utime+stime delta over the sample window. mem_rss is bytes. cmdline has already been through redaction, so a process started with --password hunter2 shows --password [REDACTED]; see Redaction. sample_duration_ms defaults to 1000 and the handler only accepts a positive number, so keep it between one and a few seconds — longer is more accurate and blocks the call for that long.

Two follow-ups sharpen the picture:

secronyx --query get_process_tree --json
secronyx --query get_thread_summary --json

get_process_tree returns entries of pid, ppid, name and depth with total and truncated; the collector caps the tree at 500 entries, so on a busy host expect "truncated": true and read it as a shape, not a census. It answers "is this one worker or a fork storm under a supervisor". get_thread_summary returns total_threads, process_count and top_processes (each pid, name, threads); a process with thousands of threads and high CPU is usually a thread-pool misconfiguration rather than real work.

Step 5: rule out I/O and memory before you blame the CPU

High CPU that is really I/O wait is easy to misdiagnose. Check the storage path explicitly:

secronyx --query get_disk_io --json
secronyx --query get_io_latency --json
secronyx --query get_blocked_processes --json

get_io_latency returns per-device read_latency_ms, write_latency_ms, read_iops, write_iops, read_throughput_bytes, write_throughput_bytes, queue_depth and utilization_percent. A device near 100% utilisation with a deep queue and double-digit latency is the bottleneck regardless of what the CPU graph says.

get_blocked_processes lists processes in uninterruptible sleep (D state) under blocked, with count, and gives each one pid, ppid, name and — when /proc/[pid]/wchan is readable by the account the agent runs as — wchan. A non-empty list with a wchan in a filesystem or block-layer function is a storage stall, and the CPU number you were chasing is an artefact of it.

For memory, the decisive evidence is the OOM killer:

secronyx --query get_oom_events --json

{
  "events": [
    { "timestamp": "Sep 17 09:37:02",
      "killed_process": "ruby",
      "pid": 20486,
      "memory_info": "total-vm:6291456kB, anon-rss:5242880kB, file-rss:0kB",
      "raw_message": "Out of memory: Killed process 20486 (ruby) total-vm:6291456kB..." }
  ],
  "count": 1,
  "source": "journalctl",
  "timestamp": "2026-09-17T09:41:19.220Z"
}

source tells you whether the events came from journalctl or dmesg, which matters when you are judging how far back the evidence reaches — and so do the two limits behind it: the collector reads the last 500 kernel log lines (journalctl -k --no-pager -n 500, falling back to dmesg) and keeps at most the 50 most recent OOM lines out of them. An event's timestamp is a best-effort string lifted verbatim from the log line, so it is syslog short format here, an ISO prefix under some configurations, and seconds-since-boot in brackets under dmesg — not the RFC 3339 stamp the result's own top-level timestamp uses. A kill at 09:37 followed by a supervisor restart storm explains a CPU spike at 09:38 far better than any process listing does.

Step 6: check for containment, and for the hardware under-delivering

If the workload runs under systemd, Docker or Kubernetes, the limits it is held to matter as much as the usage:

secronyx --query get_cgroups --json

get_cgroups returns version (1 or 2) and groups, each with name, path, a controller on v1, and limits and usage as string maps of the cgroup's own control-file names and values. CPU throttling inside a cgroup presents as a slow application on a host that looks half idle, so compare the quota in limits against the usage counters before concluding the host is short of capacity.

Then check that the CPUs are actually running at speed:

secronyx --query get_cpu_frequency_scaling --json
secronyx --query get_temperature --json

get_cpu_frequency_scaling returns policies — each with policy, governor, available_governors, driver and min_freq_khz, max_freq_khz, cur_freq_khz — plus boost_supported and boost_enabled. A host pinned to powersave near its minimum frequency does a fraction of the work its core count implies. Where the kernel exposes no cpufreq sysfs at all, which is common on virtual machines, policies comes back as an empty array rather than an error. get_temperature returns sensors with name, temperature in degrees Celsius (millidegrees from hwmon or the thermal zones, divided by 1000) and, where the platform exposes them, high and critical; readings at or above critical mean thermal throttling, and the frequency figures will corroborate it. On a virtual machine an empty sensors array is normal — unsupported collectors return empty results rather than errors.

get_interrupts_summary is worth a look when one core is pinned and the rest idle. It returns total_interrupts and top_sources (each irq, count, device); a single IRQ dominating on a NIC or storage controller points at the driver, not the application.

Step 7: corroborate with the record

Numbers tell you the state now. Logs tell you when it changed.

{"jsonrpc":"2.0","id":5,"method":"tools/call",
 "params":{"name":"get_journal_logs","arguments":{"lines":200,"unit":"api.service","priority":3}}}

get_journal_logs accepts lines (default 50), unit and priority. Note that the handler reads priority as a number — it takes args["priority"].(float64) into types.LogQuery.Priority, the syslog numeric priority 0-7 — even though the schema describes the names. Send 3 for err, not "err". The result is types.JournalLogResult: a LogResult carrying entries (each timestamp, source, level, message, pid, unit, fields), a top-level source of journald, count and truncated. The type also declares a boots field, but the journal collector never fills it in, so do not write a client that waits for it; when a filter matches nothing, entries comes back as null rather than an empty array.

Two triage tools summarise the same material without you writing filters:

secronyx --query get_failed_units --json
secronyx --query get_recent_resource_incidents --limit 20 --json

get_failed_units returns units with name, load_state, active_state, sub_state, description, failed_at and result, which passes through whatever systemd's own Result property says — exit-code, signal, timeout and unit-start-limit-hit all appear in practice. get_recent_resource_incidents returns incidents with time, type (oom, cpu_throttle, io_throttle, memory_pressure), process, pid and details, plus roll-ups oom_kills and throttles. That pair of counters is the fastest way to confirm a throttling story across the whole boot.

Unlike most queries, get_recent_resource_incidents does honour --limit from the CLI, as do the other get_recent_* triage queries and get_deployment_events.

Step 8: write the finding

You now have an evidence chain rather than an impression. A finding for this host reads something like:

Between 09:37 and 09:41 UTC, cpu.some.avg10 rose from 22% to 61% while io.some stayed flat at ~3% and memory pressure stayed at zero, so the stall is CPU, not storage. get_processes_sampled over 2s attributes 392.7% CPU to PID 20714 (puma: cluster worker 3) against 8 logical cores. get_oom_events shows one kill at 09:37:02 of PID 20486, a sibling worker; the supervisor's restart is the trigger. get_cgroups shows the pool's CPU quota unchanged, so this is real work, not throttling. Failed units: none.

Every clause is a field from a named tool, which is the point: a reader can re-run the same calls and get the same shape of answer.

Step 9: the investigation audits itself

Audit logging is on by default and writes JSON Lines to /var/log/secronyx/audit.jsonl, falling back to stderr when that path cannot be opened. Each tools/call above produced a record carrying timestamp, seq, event_id, action (here the literal tools/call), resource (the tool name), result (success, error or denied), duration_ns and hash. Four more fields are written only when there is something to write: params when the call had arguments, identity and client_ip when the HTTP transport authenticated the caller, and prev_hash on every event after the first. The --query runs in this tutorial produce no such record — only the server's tool calls are audited.

sudo tail -n 5 /var/log/secronyx/audit.jsonl | jq -c '{seq,resource,result,duration_ns}'
secronyx --audit-verify --audit-output /var/log/secronyx/audit.jsonl

--audit-verify walks the hash chain and exits; it is how you prove afterwards that the record of the investigation was not edited. Audit logging covers the chain, rotation (--audit-max-file-size, default 104857600 bytes; --audit-max-files, default 10) and durability (--audit-sync-write); Tutorial: verify the audit chain walks verification end to end.

Two failure modes you will hit

The result is too large. Tool results are capped by --max-result-bytes, default 4 MiB (DefaultMaxResultBytes = 4 << 20). Exceeding it does not produce a JSON-RPC error; it produces a normal result with "isError": true whose text reads Error: result too large: N bytes exceeds the M byte cap; narrow the query (limit, lines, filters). Do what it says: lower lines, add unit, or take a limit.

A collector fails rather than returning nothing. Platform-unsupported collectors return empty results by design, so an empty sensors array on a VM is normal. A genuine failure comes back as a result with "isError": true and text Error: <message>, not as a protocol error — so a client that only checks for a JSON-RPC error member will silently treat a failure as data. Check isError.

Where to go next

• Tool reference for every tool, its scope and its arguments.
• Command line for the full flag set and which queries read which flags.
• Tutorial: sweep a fleet from one agent to run this sequence across many hosts at once.
• Troubleshooting when a tool is missing, denied or empty.
• Documentation home.