Stale iSCSI node record blocks all Longhorn engine startups on a node (prod-worker-2)

🤖 Generated by the Agentic Engineer

Root cause

A stale iSCSI node record for a deleted Longhorn volume poisons the whole node-level iSCSI database, so every Longhorn engine startup on that node fails and no volume can attach there. Observed on prod-worker-2.

The engine's frontend startup runs iscsiadm -m node -T <target> -o show, which parses every record under /var/lib/iscsi/nodes/. One unparseable record makes the whole call exit 7:

Failed to startup frontend … iscsiadm -m node -T iqn.2019-10.io.longhorn:pvc-e1a09374… -o show
iSCSI ERROR: Unknown parameter name node.session.conn_reopen_log_freq
iSCSI ERROR: config file /var/lib/iscsi/nodes/iqn.2019-10.io.longhorn:pvc-7733aefd-fa85-4a67-b99b-f75fd6ef1a09/10.244.22.17,3260,1/default invalid
exit status 7

pvc-7733aefd-fa85-4a67-b99b-f75fd6ef1a09 no longer exists as a Longhorn volume — its node record outlived it, and carries a parameter the current iscsiadm rejects.

Why it surfaced now, and why it looked node-specific

/var/lib/iscsi survives reboots. prod-worker-2 rebooted at 2026-08-16T20:13:03Z (fresh bootID, 50 of 60 pods restarted in that window), which made iscsid re-read the on-disk database and start failing on the invalid record.

The signature is distinctive and worth recognising directly:

Replicas keep working; only engines fail. Replica instances never call iscsiadm; the engine frontend does. So the node looks healthy — instance-manager Running, Ready=True, Schedulable=True, disks fine — while attaching anything to it is impossible.
Every Longhorn Ready/RequiredPackages/KernelModulesLoaded condition reports True, so no Longhorn-level health check catches this.
The visible symptom is ControllerPublishVolume timing out with DeadlineExceeded, which reads like a slow attach rather than a node fault.

Live impact: 11 volumes attached cluster-wide, zero on prod-worker-2, and two CNPG instances (coroot-db-1, wedding-db-11) stuck in a pod-recreate/attach-timeout loop for 25+ minutes. Both clusters stayed served at ready=2 of 3, so this degraded redundancy rather than causing an outage.

Not a fleet-wide upgrade issue

Ruled out by measurement: every node runs iscsi-tools=v0.2.0 and Talos v1.13.8, and prod-worker-1/prod-worker-3 rebooted on 2026-08-12 and attach volumes normally (4 and 5 respectively). The fault is a node-local corrupt record, not a version skew.

Fix

Remove the stale record from the affected node, then let the engines start.

Prevention (the actual ask here)

Nothing removes an iSCSI node record when its Longhorn volume is deleted, and nothing detects an unparseable one. Two candidates, in the spirit of the existing longhorn-stale-node-cleanup CronJob:

A health check that runs iscsiadm -m node -o show per node and reports non-zero exit — this is the gap that let a storage node sit silently unable to attach.
A cleanup that prunes /var/lib/iscsi/nodes/ records whose IQN has no corresponding Longhorn volume.

Acceptance criteria

iscsiadm -m node -o show exits 0 on every storage node
All Longhorn volumes reach attached; coroot-db and wedding-db return to ready=3
A stale or invalid iSCSI record is detected automatically rather than by a stuck database

Size: the immediate repair is one directory removal per affected node; the detection/cleanup work is a small CronJob.

⚠️ Possible Duplicate Issue(s)

Stale iSCSI node record on prod-worker-3 breaks all Longhorn volume attachment, stalling wedding-db and coroot-db at 2/3 #3093

🔗 Related PRs

#3109 - fix(deploy): avoid redundant GHCR node reboots [merged]

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

Create Plan

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

docs(autoscaling): correct the affinity explanation and add a pre-dra…

🤖 Generated by the Agentic Engineer

This issue is no longer linked for auto-closure by PR #3181, and remains open with its ask
untouched.

#3181 carried Fixes #3180 while turning on orphanResourceAutoDeletion: replica-data. That setting
reclaims orphaned Longhorn replica data; it neither inspects iscsiadm -m node -o show output nor
removes a stale record under /var/lib/iscsi/nodes/. So it cannot meet any of this issue's acceptance
criteria, and merging it would have closed this issue while the failure mode described here — a node
silently unable to attach any volume, with every Longhorn health condition reporting True — stayed
live and undetected.

The link is corrected: #3181 now points at #3201, the separately-evidenced orphaned-replica-data
incident it genuinely fixes (14 orphaned directories, 24% free, DiskPressure, two databases unable to
rebuild a third replica). The two are different failure modes that happen to share a component.

Still outstanding here, unchanged from the Prevention section above:

a health check running iscsiadm -m node -o show per node and reporting non-zero exit
a cleanup pruning /var/lib/iscsi/nodes/ records whose IQN has no corresponding Longhorn volume

One sequencing note for whoever picks this up: a CronJob with a ServiceAccount lands in this
repository's rendered authorization surface, so it will move the aggregate fingerprint and needs
the re-approval procedure. Do not land it concurrently with another fingerprint-touching PR — the two
are mutually exclusive until one merges.

docs(longhorn): point the incident reference at the issue this change…

🤖 Generated by the Agentic Engineer

Root cause found upstream: an open-iscsi 2.1.12 regression, fixed in 2.1.13

The Unknown parameter name node.session.conn_reopen_log_freq failure is not unique to this cluster. It is longhorn/longhorn#13733, traced to open-iscsi/open-iscsi#540. Longhorn QA reproduced it by upgrading open-iscsi from ≤2.1.11 to 2.1.12 while volumes were attached, and a reporter confirmed that open-iscsi 2.1.13 no longer fails. Longhorn now has a knowledge-base update for it (longhorn/website#1446).

The platform picked up 2.1.12 through the Talos iscsi-tools extension. I read storage/iscsi-tools/vars.yaml at each siderolabs/extensions tag:

Talos extensions	open-iscsi
v1.13.0 – v1.13.7	2.1.11
v1.13.8 – v1.13.10 (prod runs the 1.13 line)	2.1.12 (the regression)
v1.14.0	2.1.12
v1.14.1 (released 2026-09-15)	2.1.13 (the fix)

The release-1.13 branch is still on 2.1.12, so no 1.13 patch release carries the fix. That matches the timing here: both incidents (#3093 on 2026-08-11 and prod-worker-2 on 2026-08-16) came after nodes rebooted into the new extension.

What this changes

The root-cause fix is Talos ≥ v1.14.1, not v1.14.0. The Talos 1.14 path (Migrate the Talos machine config onto the v1.14 configuration documents #3850, and the Renovate PR chore(deps): update dependency siderolabs/talos to v1.14.4 #4044, which still proposes v1.14.0) needs to land on v1.14.1 or later. v1.14.0 would keep the regression.
The prevention asks here are still worth doing, but as a safety net rather than the fix. Once nodes run 2.1.13, new records should parse. Records written under 2.1.12 may still be on disk, though, and the whitelist reaper from Stale iSCSI node record on prod-worker-3 breaks all Longhorn volume attachment, stalling wedding-db and coroot-db at 2/3 #3093 is what clears them. Detection is still the only thing that would catch a storage node that cannot attach while no pod is waiting on it.

Suggested order: (1) make sure the Talos 1.14 migration targets ≥ v1.14.1; (2) ship the detection check; (3) run the whitelist reap once after the upgrade, and keep it as a CronJob only if records keep turning up. Per the note above, (2) and (3) touch the authorization fingerprint, so neither should land at the same time as another change to it.

Re-verified 2026-10-07 (interactive Claude Code session): not delivered, nothing has changed.

Every production node (7 of 7) still runs Talos v1.13.10, the line that ships open-iscsi 2.1.12. The fixed open-iscsi arrives with Talos v1.14.1 or later, and the Talos 1.14 pull request (chore(deps): update dependency siderolabs/talos to v1.14.4 #4044) is still a draft that proposes v1.14.0.
main carries no iSCSI record check and no record cleanup, so the third acceptance criterion (automatic detection) is unmet.

Still missing, in order: Talos 1.14.1 or later on the nodes (through #3850), then the detection check. The detection check does not wait on the upgrade and can be built now.

Navigation Menu

Repository navigation

Stale iSCSI node record blocks all Longhorn engine startups on a node (prod-worker-2) #3180

Root cause

Why it surfaced now, and why it looked node-specific

Not a fleet-wide upgrade issue

Fix

Prevention (the actual ask here)

Acceptance criteria

Root cause found upstream: an open-iscsi 2.1.12 regression, fixed in 2.1.13

What this changes

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Stale iSCSI node record blocks all Longhorn engine startups on a node (prod-worker-2) #3180

Description

Root cause

Why it surfaced now, and why it looked node-specific

Not a fleet-wide upgrade issue

Fix

Prevention (the actual ask here)

Acceptance criteria

Activity

coderabbitai commented on Aug 16, 2026

devantler commented on Aug 17, 2026

devantler commented on Sep 27, 2026

Root cause found upstream: an open-iscsi 2.1.12 regression, fixed in 2.1.13

What this changes

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions