Skip to content

Add ECC support for Nvidia professional GPUs - #486

Merged
Syllo merged 6 commits into
Syllo:masterfrom
LH-and-FPGA:master
Sep 16, 2026
Merged

Syllo merged 6 commits into
Syllo:masterfrom
LH-and-FPGA:master

Conversation

@LH-and-FPGA

Copy link
Copy Markdown
Contributor

Summary

Adds an ECC error counter to the device header, shown right after the power field. NVIDIA professional and datacenter GPUs (e.g. RTX PRO, Quadro, Tesla, A100/H100) expose volatile ECC error counts through NVML; this surfaces them in nvtop.

Display format is ECC <corrected>/<uncorrected>:

  • corrected — single-bit errors automatically fixed by ECC (normal)
  • uncorrected — double-bit errors; a non-zero value is highlighted in red since it indicates a hardware fault

The field follows the existing "valid bit" pattern: on GPUs that do not support ECC (consumer cards, older drivers), NVML return NVML_ERROR_NOT_SUPPORTED, the valid bit stays unset, and the field is simply left blank — no special-casing required.

Changes

  • extract_gpuinfo_common.h: add ecc_corrected / ecc_uncorrected dynamic fields and their valid bits
  • extract_gpuinfo_nvidia.c: load nvmlDeviceGetTotalEccErrors (optional symbol, so older NVML libs still work) and query volatile ECC counts
  • interface.c / interface_internal_common.h: add the ecc_info window and draw it after the power field

Notes

  • Volatile counters are used (errors since last driver reload/reboot).
  • Tested on a machine with an RTX PRO 4000 Blackwell (ECC enabled, shows ECC 0/0) and an RTX 5090 (no ECC, field correctly hidden).

LH-and-FPGA and others added 6 commits June 26, 2026 12:56
# Conflicts:
#	src/interface.c
Only reserve panel width and allocate the ECC window when a monitored GPU
actually exposes volatile ECC counters, mirroring the NVLink probe pattern.
Consumer GPUs no longer get a widened device header.
The NVLink window was deliberately excluded from device_length(), so it
extended past the panel edge. When several devices share a row this
overlapped the next device header and was painted over. Factor the line-2
NVLink geometry into shared helpers and include it in the panel width so
the allocated window and reserved width cannot drift apart.
Corrected (SBE) errors are routine and can accumulate into the hundreds,
while uncorrected (DBE) errors should be zero and signal a fault. Cap
corrected at three digits and uncorrected at two, drop the two leading
spaces, and size device_ecc for "ECC 999/99" (10 columns).
@Syllo
Syllo merged commit 4769d53 into Syllo:master Sep 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants