Diagnostics¶
Diagnostics are small point-in-time tables of what is going on on the host right now: the processes using the most CPU, the ones using the most memory, the services that are not running as they should, the last errors written to the system logs, the size of the places that usually fill a disk, the last failed checks. Modules keep them up to date and the core collects them on an interval, the way it collects metrics. The latest diagnostics snapshot is kept with every failed check and travels with it to an enrolled fleet server, so whoever reads a failure sees what was running when it happened. These routes serve the same snapshot live.
- List the sections —
GET /api/v2/diagnostics - Read one section —
GET /api/v2/diagnostics/{section}
Facts say what the host is, metrics say how much, as numbers over time, and diagnostics say what: rows and columns that name things - processes, failed checks, services, log entries - which a metric key cannot carry without turning every process name into a series of its own.
Turning it on¶
Nothing is collected until two things are switched on: the core’s capture, and each table in the module that produces it.
[/settings/core/failed checks]
; Collect diagnostics every interval, keep them with each failed check and
; serve them on /api/v2/diagnostics.
capture diagnostics = true
; The most rows kept per table (1-50).
diagnostics rows = 10
[/settings/core]
; How often to collect them. Empty (the default) collects them at the end of
; every metrics round, on the same thread.
diagnostics interval =
; Windows: the process tables and the services in trouble (CheckSystem)...
[/settings/system/windows/diagnostics]
processes = true
services = true
; ...the service crash events and the event log errors (CheckEventLog).
[/settings/eventlog/diagnostics]
services = true
errors = true
; Linux: all of them from CheckSystemUnix - services covers both service
; tables, errors the journal's errors.
[/settings/system/unix/diagnostics]
processes = true
services = true
errors = true
; Both: the temp and log directories, page and swap files and (Windows)
; shadow-copy storage, measured every growth spots interval (CheckDisk).
[/settings/disk/diagnostics]
growth spots = true
growth spots interval = 10m
With capture diagnostics on and no table switched on in any module, the
snapshot holds only the core’s own failures.recent. Every key is re-read on a
settings reload except diagnostics interval, which is read when the service
starts. A table switched off is absent from the next snapshot.
Enabling a table enables what it needs. processes = true turns on the
per-process sampling the tables are read off: on Windows the collector’s
process cpu sample (one extra process-table query a second), on Linux a
read of /proc/<pid>/stat and statm for every process once a second. With
the table off none of it runs. The service and log tables start a reader of
their own:
| Key | Starts |
|---|---|
services (CheckSystem) |
A refresh every 30 seconds on the collector’s auxiliary thread: one walk of the service control manager, opening only the services whose status already looks wrong. |
services, errors (CheckEventLog) |
One thread with its own event log subscriptions (System for the service events; Application and System for the errors). They are separate from the real-time filters, so they work whether or not real-time is enabled. |
services, errors (CheckSystemUnix) |
One thread that runs systemctl every 30 seconds - reading the services that look troubled each time, and every running service (for its restart count) once in five minutes, so a restart of a service that is running again can take that long to appear in services.problems - and reads the journal with journalctl every 10 seconds, from where the last read stopped. Each run is bounded to 10 seconds. The build does not link libsystemd. |
growth spots (CheckDisk) |
One thread that measures the growth spots 15 seconds after it starts and then every growth spots interval (default 10m, at least 10s; a settings reload keeps the last round and its schedule rather than measuring again): one directory at a time with a half-second pause between them, at background priority (thread background mode on Windows; the idle I/O class and nice 19 on Linux), at most 30 seconds per directory and 2 minutes of walking per round. |
An event table starts from one backward read of the last ten minutes of its logs, so a check that fails soon after the agent started - or after a settings reload, which restarts the readers - still has the events that led up to it. After that each event is rendered once, as it is written, and the last 50 are kept.
Nothing is collected because a check failed or a request arrived. A round asks each module for what it already holds - the process table its collector sampled a moment ago - and never queries, walks or waits. A failure attaches the snapshot the last round left, at most one interval old, and these routes serve it. A host that is failing checks is often bogged down; collecting on failure would add load at the worst moment.
The tables¶
| Section | Produced by | Enabled by | Columns |
|---|---|---|---|
failures.recent |
the core | capture diagnostics |
time, command, status, subject, message |
processes.top_cpu |
CheckSystem (Windows), CheckSystemUnix (Linux) | processes |
name, pid, cpu (percent), working_set (bytes), private_bytes (bytes) |
processes.top_memory |
CheckSystem (Windows), CheckSystemUnix (Linux) | processes |
the same, sorted by working set |
services.problems |
CheckSystem (Windows), CheckSystemUnix (Linux) | services |
name, state, exit_code, binary_path, account, started (timestamp), restart_count |
services.events |
CheckEventLog (Windows), CheckSystemUnix (Linux) | services |
time, service, id, message |
eventlog.errors |
CheckEventLog (Windows), CheckSystemUnix (Linux) | errors |
time, log, source, id, level, message |
disk.growth_spots |
CheckDisk | growth spots |
path, kind, size (bytes), note |
failures.recentis the last failed checks from the failed-check log, newest first: when, which command, its status, who asked for it and the first line of its message. In a failure’s own context it holds the failures before that snapshot was taken - the run-up to it.processes.top_cpuranks the processes by their CPU share over the last second, as a share of all cores together: 100 is every core busy, so a process pegging one core of four shows as 25.processes.top_memoryranks them by working set (resident memory).private_bytesis memory the process alone owns: private bytes on Windows, resident pages not shared with another process on Linux. A process is its name and pid: never its command line, which carries credentials far more often than check arguments do. On macOS the tables report that they are not supported yet.-
services.problemslists the services that are not where they should be. Modules are not told which service a failed check was about, so the table is every service in trouble, the ones restarted most often first, then those that exited with an error:- Windows: what
check_serviceflags by default - an automatic service that is not running, unless it is delayed, trigger-started, still starting or stopped cleanly (exit code 0) - plus any service that stopped with an error other than “never started”. An automatic service that was stopped by hand and exited with 0 is therefore not listed.exit_codeis the Win32 exit code, or the service’s own code when it reported one;restart_countis alwaysnull, because the service control manager keeps no such count. - Linux: the systemd units that failed, are waiting to be restarted,
were restarted automatically (
NRestarts, which systemd only resets when the unit is started or stopped by hand) or whose last run did not succeed.stateisActiveState (SubState),exit_codethe main process’s exit status,accountthe unit’sUser.
binary_pathis the executable alone: the image path’s orExecStart’s arguments are cut off, because they can carry credentials.startedis when the service’s current process started,nullwhen it has none. *services.eventsholds the service manager’s own account of services that crashed: on Windows the System log’s Service Control Manager events 7031 and 7034 (terminated unexpectedly) and 7024 (terminated with an error), withservicetranslated from the display name in the event to the service’s name; on Linux the systemd manager’s journal messages for a unit whose start job failed, that failed, whose main process exited with an error, or whose restart was scheduled (idisnull: a journal entry has none). *eventlog.errorsholds the last errors written to the system logs: on Windows every critical or error event in the Application and System logs; on Linux every journal entry at priorityerror worse, withlogthe unit that wrote it (kernelfor a kernel message,journalfor an entry that belongs to no unit),sourceits syslog identifier andlevelone ofemerg,alert,critorerr. The agent must be able to read the system journal: root, or a member ofadmorsystemd-journal. A message is event text, which applications fill with whatever they like; on Windows it is rendered in the host’s language, and an event whose source has no message for its id shows the event’s own strings. - Windows: what
-
disk.growth_spotsis the size of the places that usually fill a disk, for whoever reads a failedcheck_drivesize- thedisk.free.<drive>.used_change_1handused_change_24hmetrics say which volume grew, this table where to look.kindis one of:kindWindows Linux tempthe agent’s TEMPand the WindowsTempdirectoryTMPDIR(when set),/tmp,/var/tmplogsthe agent’s log directory ( ${log-path}), the WindowsLogsdirectory and the event logs (System32\winevt\Logs)the agent’s log directory, /var/logpage_fileeach page file the memory manager has open - swap_fileswapfile.syson the system driveeach swap file in /proc/swaps(a swap partition takes no filesystem space)shadow_storagethe shadow-copy storage in use, one row per volume ( Win32_ShadowStorage)- Every path comes from the system (environment, registry, WMI), never a hard-coded drive letter, and a spot that does not exist has no row. A directory’s
sizeis the total of the files beneath it; links, junctions and mount points are not followed and the walk stays on the directory’s filesystem. A directory that hits its time limit reports the size counted so far, says so innoteand marks the tabletruncated; entries that cannot be read are skipped and counted innote, so an unelevated agent still reports what it can see. A row with nosizesays why innote: the folder itself could not be read, or - for the shadow storage, which needs an elevated agent - the query failed.captured_atis when the last round finished, so this table can be up to an interval older than the rest of the snapshot; a settings reload keeps it until the next round.
On Linux the three service and log tables need systemd: a host without it
(/run/systemd/system) produces none of them, macOS included.
Caps¶
| Cap | Default |
|---|---|
| Rows per table | diagnostics rows, 10 (at most 50) |
| Cell length | 1024 bytes |
| Tables per snapshot | 16 |
| One snapshot | max context size, shared with metrics snapshots |
A table cut by a cap is marked truncated, so a reader can tell “10 rows” from
“at least 10”. A top list is always truncated when there were more processes
than rows. Every text cell passes through the same redaction as the
failed-check log: a password= or token= pair loses its value. prune
pattern under [/settings/core/failed checks] applies to table names too:
prune pattern = processes.top_memory leaves that table out. The metrics
pruning rules (zeros, empty, duplicates) do not apply to tables.
List the sections¶
GET /api/v2/diagnostics
Privilege: diagnostics.list
The tables in the latest snapshot, without their rows:
$ curl -k -s -u admin https://localhost:8443/api/v2/diagnostics | python -m json.tool
{
"enabled": true,
"snapshot_id": 812,
"captured_at": "2026-11-15T10:42:07Z",
"max_rows": 10,
"dropped": 0,
"excluded": [],
"count": 3,
"sections": [
{
"source": "core",
"description": "The last failed checks, newest first",
"captured_at": "2026-11-15T10:42:07Z",
"truncated": false,
"name": "failures.recent",
"rows": 2
},
{
"source": "CheckSystem",
"description": "The processes using the most CPU over the last second",
"captured_at": "2026-11-15T10:42:06Z",
"truncated": true,
"name": "processes.top_cpu",
"rows": 10
},
{
"source": "CheckSystem",
"description": "The processes with the largest working set",
"captured_at": "2026-11-15T10:42:06Z",
"truncated": true,
"name": "processes.top_memory",
"rows": 10
}
]
}
| Field | Description |
|---|---|
enabled |
Whether diagnostics are collected (capture diagnostics). false with no sections means switched off, not idle. |
snapshot_id |
Increasing for the life of the process; 0 before the first round. |
captured_at |
When the core finished the round. Each section has its own captured_at, which can be older: a table refreshed on a slow cadence is handed over as it was last refreshed. |
max_rows |
diagnostics rows, as applied. |
dropped |
Tables left out because the snapshot already held 16, or one of the same name, or because it did not fit max context size even with every row given up (the agent then also logs a warning). |
excluded |
Tables prune pattern left out. |
sections |
Each table’s name, source (the module that produced it, or core), description, captured_at, truncated, rows (the row count) and, when it could not be collected, error. |
Read one section¶
GET /api/v2/diagnostics/{section}
Privilege: diagnostics.list
One table, with its columns and its rows. Each row is an object keyed by column
name, so a client never counts positions. Timestamps are ISO 8601 UTC, and a
missing value is null.
$ curl -k -s -u admin https://localhost:8443/api/v2/diagnostics/processes.top_cpu | python -m json.tool
{
"source": "CheckSystem",
"description": "The processes using the most CPU over the last second",
"captured_at": "2026-11-15T10:42:06Z",
"truncated": true,
"columns": [
{ "name": "name", "type": "string" },
{ "name": "pid", "type": "int" },
{ "name": "cpu", "type": "float", "unit": "percent", "description": "Share of all cores over the last second" },
{ "name": "working_set", "type": "int", "unit": "bytes", "description": "Resident memory" },
{ "name": "private_bytes", "type": "int", "unit": "bytes", "description": "Memory not shared with other processes" }
],
"rows": [
{ "name": "acme-indexer.exe", "pid": 4242, "cpu": 87.5, "working_set": 412000256, "private_bytes": 398458880 },
{ "name": "svchost.exe", "pid": 1288, "cpu": 2.1, "working_set": 85983232, "private_bytes": 48349184 }
],
"name": "processes.top_cpu",
"snapshot_id": 812
}
The growth spots of an unelevated agent: the folder and the WMI class it may
not read say so in note rather than going missing.
$ curl -k -s -u admin https://localhost:8443/api/v2/diagnostics/disk.growth_spots | python -m json.tool
{
"source": "CheckDisk",
"description": "Size of the places that usually fill a disk: temp and log directories, page and swap files, shadow-copy storage",
"captured_at": "2026-11-15T09:33:05Z",
"truncated": false,
"columns": [
{ "name": "path", "type": "string" },
{ "name": "kind", "type": "string", "description": "temp, logs, page_file, swap_file or shadow_storage" },
{ "name": "size", "type": "int", "unit": "bytes", "description": "The directory tree's files, the file, or the shadow storage in use" },
{ "name": "note", "type": "string", "description": "What limited the size (a time limit, unreadable entries), or why there is none; empty when complete" }
],
"rows": [
{ "path": "C:\\Windows\\Temp", "kind": "temp", "size": null, "note": "Unable to read folder C:\\Windows\\Temp: 5: Access is denied." },
{ "path": "C:\\Windows\\Logs", "kind": "logs", "size": 600678673, "note": "1 folder or entry could not be read" },
{ "path": "C:\\Windows\\system32\\winevt\\Logs", "kind": "logs", "size": 435499008, "note": null },
{ "path": "C:\\pagefile.sys", "kind": "page_file", "size": 36507222016, "note": null },
{ "path": "C:\\swapfile.sys", "kind": "swap_file", "size": 16777216, "note": null },
{ "path": "all volumes", "kind": "shadow_storage", "size": null, "note": "could not be read (needs an elevated agent): ..." }
],
"name": "disk.growth_spots",
"snapshot_id": 9
}
A column’s type is string, int, float, timestamp or bool; unit
and description are only there when set. A section that could not be
collected carries an error: a failed table and an absent one mean different
things, the first that something is wrong, the second that it is switched
off. An error does not rule out rows - eventlog.errors reads two logs, and
when one of them cannot be read the section names it in error and still
carries the rows from the other.
A section the snapshot does not hold is a 404, as is any section while
capture diagnostics is off - the message says which.
Privileges¶
| Privilege | Grants |
|---|---|
diagnostics.list |
GET /api/v2/diagnostics, GET /api/v2/diagnostics/{section} |
No built-in role holds diagnostics.list other than full, which holds
everything: the tables name what runs on the host, which is more than a
monitoring or metrics scraper needs. Give it to a role of your own:
[/settings/WEB/server/roles]
diagnostics = diagnostics.list,login.get
A caller with failures.list reads the diagnostics a failure was recorded
with through /api/v2/failures/{index}/context,
as it already reads the metrics snapshot: grant failures.list with that in
mind once capture diagnostics is on.
Other ways in¶
- The web UI’s Diagnostics page shows every table of the latest snapshot,
failures.recentfirst, with the refresh selector the Metrics page has. It reads these routes, so it needsdiagnostics.list: a role without it is told so on the page. Above the tables, a Diagnostics tables card holds the switches:capture diagnosticsand every table a loaded module can produce, each written to the settings of the module that owns it (so it needs the settings grants, and a change takes effect when the configuration is saved and reloaded). With capture off the page says how to turn it on. The Failed checks page shows the tables a failure was recorded with underfailures.list. - The
nscp testprompt hasdiagnosticsto list the tables anddiagnostics <section>to print one. - Modules read the snapshot through the
NSAPIDiagnosticsQuerycore API (core_wrapper::get_diagnostics()), which exchanges thePB.Diagnosticsmessages indiagnostics.proto. A module produces tables by declaring"diagnostics": "produce"in itsmodule.jsonand implementingfetchDiagnostics()with the builder innscapi/nscapi_diagnostics_helper.hpp. - An enrolled agent sends the diagnostics a failure was recorded with to a fleet server that asks for failure context.