Repository navigation
Synchronizing a client's clock #7
Description
I have an ESP32 microcontroller connected as a client to a router on a Linux host over a serial connection.
Micro-ROS has a rmw_uros_sync_session call that allows getting the time from the agent.
Is there a similar functionality with Pico-ROS? When connected over a network, I can synchronize my microcontroller using NTP, but what should I do when connected over a serial link? (rmw_uros_sync_session works in any link).
Activity
xopxe commentedon Sep 1, 2025
Just a couple of comments.
Micro-ROS' rmw_uros_sync_session doesn't actually sync the clock; it just updates an offset variable you can use (typically through a rmw_uros_epoch_millis() call).
I need an up-to-date time to stamp ROS topics.
g4sp3r commentedon Sep 1, 2025
Hey @xopxe, thanks for raising the issue and providing useful info.
Unfortunately nothing like this currently exists.
We rely on zenoh-pico for networking and system utilities and rmw_zenoh to get connection to ROS. Maybe I missed it but I haven't seen any network time sync features in zenoh.
I read that zenohd fixes wrong timestamps but probably you need the correct timestamp inside ROS Header.
What you suggested could probably be implemented in "application" layer for now. Publish the current time on a topic from ROS host, the ESP32 can subscribe and calculate the offset for current clock. Use this offset when sending other messages.
It would be even "nicer" to make a service call for it but only service server is currently supported for Pico-ROS.
xopxe commentedon Sep 2, 2025
That's unfortunate, @g4sp3r. In your opinion, such functionality should reside where? I guess this would be useful for all microcontroller-based Zenoh-Pico applications that do not connect via an IP network.
Yes, I thought about implementing this on the application side. My idea is to replicate the NTP mechanism: send an "echo request" and then take half the RTT as the latency. Add this latency to the timestamp on the response, and that's your time. NTP does this from both ends to account for asymmetric RTTs, but I don't think this would be important here.
g4sp3r commentedon Sep 2, 2025
@xopxe I agree, the feature would be useful in a lot of situations.
My understanding is that time sync protocol belongs to Application layer of OSI, so it makes sense that zenoh doesn't support any feature like that.
Disregarding the layers for a moment, there is possibility that router timestamp is sent in keep-alive messages. This is just my wild speculation, but then it could make sense to support it in zenoh-pico for "free" since messages are already sent. Zenoh-pico also has a system.c wrapper with access to platform clock functions so it seems like a natural place to add it.
On the other hand, following the model, you could argue that support should come from PicoROS, but then it's not totally clear who will provide the clock sync services. I am not aware of any standard ROS time sync service we could depend on in comparison with micoROS that has it's agent and can control both sides.
One option would be to vendor the time sync ROS node running on the host in Pico-ROS repo. Add support for service call in
picoros.c. Add a function likepicoros_time_sync(char* topic name, (sync_cb*)(z_clock_t now), (sync_failed_cb*)(void)). Save the current time when function is called and make a service call. If we get reply from above node your proposed RTT/2 is added to the time inside message and callback is called. The callback is needed since zenoh-pico system wrapper has no functions to set the clock, so the actual calling of current platform clock setting function would be delegated to application.Or alternatively provide
managing the internal offset variable like you outlined above.
My sense is to first investigate the zenoh-pico option since it could be more elegant.
Also in parallel support service call in PicoROS as first step since it will be needed for other cases.
What are your thoughts?
xopxe commentedon Sep 3, 2025
I agree with your analysis, @g4sp3r
Having this service integrated inside zenoh-pico would be the most convenient option, but its implementation is not trivial. In the generic case, it will need to work on all links that Zenoh supports, including those with high latency and/or jitter. NTP, which solves this problem over IP networks, has a lot of code handling outliers, drifting, and so on. This complexity makes it more reasonable to place it in the application layer, as you said (NTP is considered an application layer service).
So, implementing it as a standard application first makes sense. It would be easier to tailor to different use cases and experiment a bit. I started prototyping such a version based on notifications, which mimics NTP message exchange. I saw that you have a PR for service clients. I'll try it as soon as possible and also use it to test the simpler "RTT/2" method, which could be good enough on some links.
g4sp3r commentedon Sep 3, 2025
@xopxe, I think your plan to implement it as standard application at first is good.
For start we could link the time-sync example application in the readme, or add it as part of examples in this repo.
If we can find a good solution I am also very happy to integrate it in Pico-ROS.
When you come to testing the service client PR, please let me know if there are some things with the api you think could be improved, or any other ideas you might come up with.
xopxe commentedon Sep 5, 2025
Hi @g4sp3r
I did a proof of concept for time synchronization.
The C++ code is Arduino through platformio, and is copied straight from my codebase so it refers to a bunch of stuff that from my environment, but shows the basic idea:
On an ESP32 over a TCP on a 115200 serial link I get about 7-8ms in RTT, and consecutive synchronization sessions get under 1ms difference, sometimes spiking to under 10ms differences. I think averaging several synchronizations will be good enough.
Implementing this using services would be cleaner. As it is now for real use you would need to consider responses coming from other instances of the service. As soon as you consider the service request working, I'll port this.
g4sp3r commentedon Sep 5, 2025
Hey @xopxe, that's great! Thanks for sharing ❤️
I think the new service call api shouldn't change anymore. You could try porting and give some comments in the PR if you hit any difficulties. I wanted to do more testing with different embedded devices before merging and your set-up on esp32 with serial would definitely be another useful data point.
saikishor commentedon Sep 6, 2025
@xopxe can you share a working example of your setup?
With @g4sp3r we tried to setup the esp32s3 with a PicoROS joystick example over serial, we could establish the connection to the router but when we do
ros2 topic list. We don't see any topics at all and neither do the nodes.@g4sp3r has stm32 working over serial with topics exposed. It would be really helpful for me to know how did you solve the issue or if you share the codebase, I can debug it myself for the roscon2025 workshop
Thank you
g4sp3r commentedon Sep 7, 2025
Looks like we were were missing
"tcp/[::]:7447"in ourZENOH_CONFIG_OVERRIDEwhen testing with @saikishor . UsingDEFAULT_RMW_ZENOH_ROUTER_CONFIG.json5with added serial port works.🤦
xopxe commentedon Sep 7, 2025
Just not to disappoint, I realized in the timing tests I did in my proof of concept I was actually connected through WiFi and not the serial.
Meanwhile, I realized that the topic-based time synchronization runs on standard messages from
std_msgsandsensor_msgs, but the service-based version would depend on a custom type, which is annoying and probably overengineering without a proper design.g4sp3r commentedon Sep 8, 2025
@xopxe you are right, that is annoying.
I will still merge #8 and propose we leave this issue open for further discussions.
xopxe commentedon Sep 15, 2025
I have a notification-based synchronization service working. The server is available here, while the client code can be found here. The client code is integrated into a bigger system, but can be easily extracted into a self-contained example.