[sankalpsthakur]

Proposed changes

The c8y mapper requests a software list every time it receives the retained
te/<entity>/cmd/software_update capability message, minting a new
software_list command with a fresh c8y-mapper-<timestamp> id.
A mapper in a crash-restart loop therefore mints one command per restart and
never clears any of them (#4255).

This change makes the mapper keep track of the software_list commands it has
issued, from the moment their retained messages are received back from the
broker until they are cleared, and skip the request when one is still pending
for the target entity, whatever its status (non-terminal, or terminal but not
yet cleared). Only commands carrying the mapper's own id prefix are considered,
and the check is per target entity.

Implementation notes:

• CumulocityConverter gets a pending_software_list_requests: HashMap<CmdId, EntityTopicId>
  maintained next to active_commands: populated in the arm handling the
  mapper's own command instances (retained or echoed), emptied by the clearing
  message. register_software_update_operation consults it before calling
  request_software_list and logs the pending command id when it skips.
• The refresh issued after a software_update operation
  (operations/handlers/software_update.rs) is intentionally left untouched:
  a refresh after a change of the installed software is wanted.
• The second expectation of the issue (unchanged capability metadata should not
  re-trigger a request at all) is not implemented here: the system test
  mapper-publishing-agent-supported-ops.robot::Agent gets the software list request once it comes up
  expects a new request after a restart of the agent, i.e. on unchanged
  metadata, and software.robot::Software list should be populated during startup
  relies on the refresh after a mapper restart. Happy to follow up if the
  expected behaviour should change.
• Caveat worth a maintainer's eye: whether the retained command instances are
  received before the retained capability message depends on the order of the
  topic filters in the SUBSCRIBE packet. The mapper lists AnyCommand before
  AnyCommandMetadata, but tedge_mqtt_ext::MqttActorBuilder::build_actor
  builds the initial filter list from a HashSet, so the order is not
  guaranteed. When the capability message is delivered first the behaviour is
  unchanged (one request is made); when the command instances are delivered
  first, no duplicate is minted.

Types of changes

• Bugfix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Improvement (general improvements like code refactoring that doesn't explicitly fix a bug or add any new functionality)
• Documentation Update (if none of the other choices apply)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)

Paste Link to the issue

Refs #4255: #4255

Checklist

• I have read the CONTRIBUTING doc
• I have signed the CLA (in all commits with git commit -s. You can activate automatic signing by running just prepare-dev once)
• I ran just format as mentioned in CODING_GUIDELINES (cargo fmt -p c8y_mapper_ext -- --check is clean)
• I used just check as mentioned in CODING_GUIDELINES (only cargo clippy -p c8y_mapper_ext --tests was run locally, see below)
• I have added tests that prove my fix is effective or that my feature works
• I have added necessary documentation (if appropriate)

Further comments

Seven unit tests added in converter.rs: the existing behaviour (request on capability), no request while a previous one is pending (init, executing, successful but uncleared), request again once cleared, and two guards (a request not issued by the mapper, or pending for another device, does not block a new one).

Run locally (crate scope): cargo fmt --check, cargo check --tests and cargo clippy --tests are clean; cargo test -p c8y_mapper_ext software_list_ gives 7 passed. With the new check disabled, the 3 "not requested while pending" cases fail with a second software_list/c8y-mapper-<ts> request. The full just check and the Robot system tests were not run locally.

AI/LLM disclosure

• AI coding tools (Claude Code) were used to write this change, the tests and this description.
• The change was reviewed and the tests above were run locally before submitting.

[Bravo555]

When the capability message is delivered first the behaviour is unchanged (one request is made); when the command instances are delivered first, no duplicate is minted.

This is the crux. If the capability message is delivered before one of the pending software_list commands, the problem still happens.

Even though the order of topic filters can be used with mosquitto to influence the order of delivery of retained messages (didn't know that, but I was able to verify it with claude, very cool), it's not something the MQTT spec requires, but mosquitto implementation detail, which could change or not apply to other brokers. Even though we use only mosquitto, I'm not sure if depending on this is the best approach.

Also, even without this change, after the mapper is able to start without fail, it clears the all stale software_list commands. So basically, the fix only applies as long as the mapper stays unhealthy (after restoring health it cleans the stale topics) and the issue could happen again after a mosquitto update if they changed the processing logic.

But, even though it could happen, the fix makes it not happen for now and is Not That Big and the code is cheap with AI after all. So despite above, I can approve it. But since @jarhodes314, who made the issue, is back on monday, I guess we can wait a bit with merging for a final vibe check from him.

EDIT: Nevermind, after retrying a few more times, approximately once per 3 runs I get the capability message first which results in the new command being published:

2026-10-09T16:47:20.231615108Z DEBUG MQTT recv: [te/device/main///cmd/software_list qos=1 retained] {"types":["apt","flow"]}
...
2026-10-09T16:47:20.232883092Z DEBUG MQTT recv: [te/device/main///cmd/software_list/c8y-mapper-2026-10-09T16-35-24.956587901Z qos=1 retained] {"@version": ...
2026-10-09T16:47:20.233506684Z DEBUG MQTT pub: [te/device/main///cmd/software_list/c8y-mapper-2026-10-09T16-47-20.233008450Z qos=1 retained] {"status":"init"}

This is too flaky to be considered a complete fix, either other approaches have to be considered, or we could reconsider if avoiding duplicate commands in a scenario where the mapper is already degraded is necessary as originally proposed.

---

Failing test is unrelated, seems to be flaky, I'll run tests again and investigate the failing one to potentially fix.

[sankalpsthakur]

thanks for running it repeatedly. that's the gap the caveat in the description pointed at, and i agree the fix shouldn't depend on subscribe or delivery order.

an order-independent version i'd propose: when the capability message arrives, don't mint right away. set a short timeout instead (the availability actor in this crate already uses tedge_timer_ext), and when it fires, mint only if no c8y-mapper-* software_list command for that target was received in the meantime. the retained instances arrive right after subscribing, so a short window should be enough, and a mapper in a crash loop that exits inside the window mints nothing. i'd cover both delivery orders with a test in shuffled_tests.rs.

if you'd rather drop it, since a healthy mapper already clears the stale commands, i'm fine closing it. happy to wait for jarhodes314 on monday.

[jarhodes314]

I have now pushed a revised fix to this PR that waits for the retained messages to arrive before sending the software list, along with a robot test to test for the bug in practice.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[sankalpsthakur]

merged main for the nfpm rpm fix, ptal

[github-actions]

Robot Results

✅ Passed	❌ Failed	⏭️ Skipped	Total	Pass %	⏱️ Duration
1040	2	8	1042	99.81	3h25m39.119397s

Failed Tests

Name	Message	⏱️ Duration	Suite
Update tedge version from base to current using Cumulocity	Software not installed. errors=[('tedge', 'VERSION_MISMATCH'), ('tedge-mapper', 'VERSION_MISMATCH'), ('tedge-agent', 'VERSION_MISMATCH'), ('tedge-watchdog', 'VERSION_MISMATCH'), ('tedge-apt-plugin', 'VERSION_MISMATCH')], wanted=(Software(name='tedge', version='2\.0\.2\~425\+gb997aec', url='', action='install', softwareType='apt'), Software(name='tedge-mapper', version='2\.0\.2\~425\+gb997aec', url='', action='install', softwareType='apt'), Software(name='tedge-agent', version='2\.0\.2\~425\+gb997aec', url='', action='install', softwareType='apt'), Software(name='tedge-watchdog', version='2\.0\.2\~425\+gb997aec', url='', action='install', softwareType='apt'), Software(name='tedge-apt-plugin', version='2\.0\.2\425\+gb997aec', url='', action='install', softwareType='apt')), got=[{'softwareType': 'apt', 'name': 'adduser', 'version': '3.134', 'url': ''}, {'softwareType': 'apt', 'name': 'apt', 'version': '2.6.1', 'url': ''}, {'softwareType': 'apt', 'name': 'apt-transport-https', 'version': '2.6.1', 'url': ''}, {'softwareType': 'apt', 'name': 'base-files', 'version': '12.4+deb12u15', 'url': ''}, {'softwareType': 'apt', 'name': 'base-passwd', 'version': '3.6.1', 'url': ''}, {'softwareType': 'apt', 'name': 'bash', 'version': '5.2.15-2+b13', 'url': ''}, {'softwareType': 'apt', 'name': 'bash-completion', 'version': '1:2.11-6', 'url': ''}, {'softwareType': 'apt', 'name': 'bsdextrautils', 'version': '2.38.1-5+deb12u3', 'url': ''}, {'softwareType': 'apt', 'name': 'bsdutils', 'version': '1:2.38.1-5+deb12u3', 'url': ''}, {'sof... [ Message content over the limit has been removed. ] ...name': 'tedge-full', 'version': '2.0.1', 'url': ''}, {'softwareType': 'apt', 'name': 'tedge-mapper', 'version': '2.0.1', 'url': ''}, {'softwareType': 'apt', 'name': 'tedge-watchdog', 'version': '2.0.1', 'url': ''}, {'softwareType': 'apt', 'name': 'tzdata', 'version': '2026c-0+deb12u1', 'url': ''}, {'softwareType': 'apt', 'name': 'ucf', 'version': '3.0043+nmu1+deb12u1', 'url': ''}, {'softwareType': 'apt', 'name': 'usr-is-merged', 'version': '37deb12u1', 'url': ''}, {'softwareType': 'apt', 'name': 'util-linux', 'version': '2.38.1-5+deb12u3', 'url': ''}, {'softwareType': 'apt', 'name': 'util-linux-extra', 'version': '2.38.1-5+deb12u3', 'url': ''}, {'softwareType': 'apt', 'name': 'vim-common', 'version': '2:9.0.1378-2+deb12u2', 'url': ''}, {'softwareType': 'apt', 'name': 'vim-tiny', 'version': '2:9.0.1378-2+deb12u2', 'url': ''}, {'softwareType': 'apt', 'name': 'wget', 'version': '1.21.3-1+deb12u1', 'url': ''}, {'softwareType': 'apt', 'name': 'zlib1g', 'version': '1:1.2.13.dfsg-1', 'url': ''}, {'softwareType': 'apt', 'name': 'zsh', 'version': '5.9-4+b15', 'url': ''}, {'softwareType': 'apt', 'name': 'zsh-common', 'version': '5.9-4', 'url': ''}, {'softwareType': 'flow', 'name': 'c8y/alarms', 'version': '2.0.1', 'url': ''}, {'softwareType': 'flow', 'name': 'c8y/events', 'version': '2.0.1', 'url': ''}, {'softwareType': 'flow', 'name': 'c8y/health', 'version': '2.0.1', 'url': ''}, {'softwareType': 'flow', 'name': 'c8y/measurements', 'version': '2.0.1', 'url': ''}, {'softwareType': 'flow', 'name': 'c8y/units', 'version': '2.0.1', 'url': ''}] [RETRY] FAIL on 3. retry.	121.429 s	Tedge Self Update
Update tedge whilst using common community plugins	Software not installed. errors=[('tedge', 'VERSION_MISMATCH'), ('tedge-mapper', 'VERSION_MISMATCH'), ('tedge-agent', 'VERSION_MISMATCH'), ('tedge-apt-plugin', 'VERSION_MISMATCH'), ('tedge-watchdog', 'VERSION_MISMATCH')], wanted=(Software(name='tedge', version='2\.0\.2\~425\+gb997aec', url='', action='install', softwareType='apt'), Software(name='tedge-mapper', version='2\.0\.2\~425\+gb997aec', url='', action='install', softwareType='apt'), Software(name='tedge-agent', version='2\.0\.2\~425\+gb997aec', url='', action='install', softwareType='apt'), Software(name='tedge-apt-plugin', version='2\.0\.2\~425\+gb997aec', url='', action='install', softwareType='apt'), Software(name='tedge-watchdog', version='2\.0\.2\425\+gb997aec', url='', action='install', softwareType='apt')), got=[{'softwareType': 'apt', 'name': 'adduser', 'version': '3.134', 'url': ''}, {'softwareType': 'apt', 'name': 'apt', 'version': '2.6.1', 'url': ''}, {'softwareType': 'apt', 'name': 'apt-transport-https', 'version': '2.6.1', 'url': ''}, {'softwareType': 'apt', 'name': 'base-files', 'version': '12.4+deb12u15', 'url': ''}, {'softwareType': 'apt', 'name': 'base-passwd', 'version': '3.6.1', 'url': ''}, {'softwareType': 'apt', 'name': 'bash', 'version': '5.2.15-2+b13', 'url': ''}, {'softwareType': 'apt', 'name': 'bash-completion', 'version': '1:2.11-6', 'url': ''}, {'softwareType': 'apt', 'name': 'bsdextrautils', 'version': '2.38.1-5+deb12u3', 'url': ''}, {'softwareType': 'apt', 'name': 'bsdutils', 'version': '1:2.38.1-5+deb12u3', 'url': ''}, {'sof... [ Message content over the limit has been removed. ] ...name': 'tedge-full', 'version': '2.0.1', 'url': ''}, {'softwareType': 'apt', 'name': 'tedge-mapper', 'version': '2.0.1', 'url': ''}, {'softwareType': 'apt', 'name': 'tedge-watchdog', 'version': '2.0.1', 'url': ''}, {'softwareType': 'apt', 'name': 'tzdata', 'version': '2026c-0+deb12u1', 'url': ''}, {'softwareType': 'apt', 'name': 'ucf', 'version': '3.0043+nmu1+deb12u1', 'url': ''}, {'softwareType': 'apt', 'name': 'usr-is-merged', 'version': '37deb12u1', 'url': ''}, {'softwareType': 'apt', 'name': 'util-linux', 'version': '2.38.1-5+deb12u3', 'url': ''}, {'softwareType': 'apt', 'name': 'util-linux-extra', 'version': '2.38.1-5+deb12u3', 'url': ''}, {'softwareType': 'apt', 'name': 'vim-common', 'version': '2:9.0.1378-2+deb12u2', 'url': ''}, {'softwareType': 'apt', 'name': 'vim-tiny', 'version': '2:9.0.1378-2+deb12u2', 'url': ''}, {'softwareType': 'apt', 'name': 'wget', 'version': '1.21.3-1+deb12u1', 'url': ''}, {'softwareType': 'apt', 'name': 'zlib1g', 'version': '1:1.2.13.dfsg-1', 'url': ''}, {'softwareType': 'apt', 'name': 'zsh', 'version': '5.9-4+b15', 'url': ''}, {'softwareType': 'apt', 'name': 'zsh-common', 'version': '5.9-4', 'url': ''}, {'softwareType': 'flow', 'name': 'c8y/alarms', 'version': '2.0.1', 'url': ''}, {'softwareType': 'flow', 'name': 'c8y/events', 'version': '2.0.1', 'url': ''}, {'softwareType': 'flow', 'name': 'c8y/health', 'version': '2.0.1', 'url': ''}, {'softwareType': 'flow', 'name': 'c8y/measurements', 'version': '2.0.1', 'url': ''}, {'softwareType': 'flow', 'name': 'c8y/units', 'version': '2.0.1', 'url': ''}] [RETRY] FAIL on 3. retry.	80.429 s	Tedge Self Update