Clock skew issues megathread

[craigloewen-msft]

Megathread

Current status: waiting on backport for kernel patch to mitigate issue.

We're creating this megathread to track the clock skew issues in WSL in one place, and will keep this parent comment current with any updates.

Background

Sometimes the WSL clock can become skewed after resume from sleep (specifically S0). See some example related issues for more info:
#8318
#8204
#7255

Potential work arounds

Use systemd to force clock sync

See this comment: #8204 (comment)

Set the hardware clock via a command

Run sudo hwclock -s. More info here.

Run ntpdate on distro start up

Edit /etc/wsl.conf to have this content:

[boot]
command="ntpdate ntp.ubuntu.com"

This will force a clock reset on start up of the distro.

Build a private kernel with this patch

[marceloid]

This workaround using systemd was the best and only solution in my case: #8204 (comment)

[duaneking]

The number of open issues that relate to this temporal distortion worry me, because it tells me there isn't a critical security focus on time by the teams involved in all these regressions and supposed fixes; so who is running the show for temporal security at MS? Anybody? Who is the Time Czar in the Security Org?

Or perhaps a better question to ask: Why aren't these consistent temporal anomalies being considered more of a security issue?

Everybody in security knows that time is a critical security component; so if time is not correct on the host system, then it is simply out of security compliance by default, right?

[haroldiedema]

Here's a simple fix that invokes hwclock -s everytime your machine wakes up from sleep. Note: This assumes you configured your WSL environment in such a way that it should always be running (for daemons/webservers/etc.).

Assuming you've changed the default user to something other than root, you'll first need to allow passwordless sudo when invoking sudo hwclock -s by updating the sudoers file:

$ sudo visudo

Add the following line:

%sudo   ALL=(ALL) NOPASSWD: /usr/sbin/hwclock

Next, create a batch file somewhere on your machine, e.g.: C:\sync-clock.bat with the following contents:

@echo off
ubuntu.exe run "sudo hwclock -s"
exit

(Change "ubuntu.exe" if you need to)

Lastly, create a Task in the Task Scheduler that runs every time your computer wakes up from sleep.

1. Open the Task Scheduler and create a new "Basic Task".
2. Set the trigger to "When a specific event is logged" and click "Next".
3. Under "Log", select System.
4. Under "Source", select Kernel-Power.
5. Under "Event ID", type "507".
6. Under "action", select "Start a program" and click "next"
7. Specify the batch file we just created: C:\sync-clock.bat.

If the task is not executed on your machine, it may be because your version of Windows emits a different Event ID. Open up the Event Viewer and check under System for any log entries that have the source "Kernel-Power" that match a timestamp when your machine has woken up from sleep mode. Verify the correct event by reading its description. It should state something along the lines of "The system exited sleep mode". The correct "Event ID" should be listed in the same window.

Hope this helps.

[ManuInNZ]

I suppose one could argue that it should rather be a wsl Linux kernel level task to have a hardware clock synch at power up/restore? in the proposed temp solution, I have used in the past `wsl -l -v` to list running distro and then exec into them, `wsl -d <distro> hwclock -s`. Might require the video step though, I had issues with some distros. cheers manu

…

On Thu, 27 Apr 2023, 00:08 Harold Iedema, ***@***.***> wrote: Here's a simple fix that invokes hwclock -s everytime your machine wakes up from sleep. Note: This assumes you configured your WSL environment in such a way that it should always be running (for daemons/webservers/etc.). Assuming you've changed the *default user* to something other than root, you'll first need to allow passwordless sudo when invoking sudo hwclock -s by updating the *sudoers* file: $ sudo visudo Add the following line: %sudo ALL=(ALL) NOPASSWD: /usr/sbin/hwclock Next, create a batch file somewhere on your machine, e.g.: C:\sync-clock.bat with the following contents: @echo off ubuntu.exe run "sudo hwclock -s"exit (Change "ubuntu.exe" if you need to) Lastly, create a Task in the *Task Scheduler* that runs every time your computer wakes up from sleep. 1. Open the Task Scheduler and create a new "Basic Task". This time, we want to start a task on a specific event. 2. Set the trigger to "When a specific event is logged" and click "Next". 3. Under "Log", select System. 4. Under "Source", select Kernel-Power. 5. Under "Event ID", type "507". 6. Under "action", select "Start a program" and click "next" 7. Specify the batch file we just created: C:\sync-clock.bat. If the task is not executed on your machine, it may be because your version of Windows emits a different Event ID. Open up the Event Viewer and check under System for any log entries that have the source "Kernel-Power" that match a timestamp when your machine has woken up from sleep mode. Verify the correct event by reading its description. It should state something along the lines of "The system exited sleep mode". The correct "Event ID" should be listed in the same window. Hope this helps. — Reply to this email directly, view it on GitHub <#10006 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB4R3ABWJD77SHKIDZBFM4TXDEF3LANCNFSM6AAAAAAXG6OMXU> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[esumii]

1. Open the Task Scheduler and create a new "Basic Task".
2. Set the trigger to "When a specific event is logged" and click "Next".
3. Under "Log", select System.
4. Under "Source", select Kernel-Power.
5. Under "Event ID", type "507".

I'd add 107 as well.

6. Under "action", select "Start a program" and click "next"
7. Specify the batch file we just created: C:\sync-clock.bat.

If the task is not executed on your machine,

Another reason can be (lack of) power:

#5324 (comment)

to run without power, "Start the task only if the computer is on AC power" must be unchecked on the Task Scheduler of Windows)

Also see the above link to do without sudo.

[gaia]

Lastly, create a Task in the Task Scheduler that runs every time your computer wakes up from sleep.

Task Scheduler doesn't work to run tasks in WSL2, see #9231

[dboreham]

I don't think we need any more workarounds do we? Someone needs to go into the hypervisor code and fix the bug, no? Hypervisor's job is to present correct hardware clock functionality to its host kernels, presumably. Or have we moved on from that being the case?

[duaneking]

I don't think we need any more workarounds do we?

No. An actual fix would be best for security and compliance as systems having the wrong time can create GDPR violations in the worst case, and MSFT is a globally GDPR compliant company, right? So the team has a clear mandate as part of being One Microsoft that they need to fix this, right?

Someone needs to go into the hypervisor code and fix the bug, no?

Yes, if that is where it is.

Hypervisor's job is to present correct hardware clock functionality to its host kernels, presumably. Or have we moved on from that being the case?

I don't believe anybody's done that kind of due diligence. The correct people at that level don't seem to be aware of this issue, or I suspect it would have been resolved if they truly understood how big of an issue this was, so the fact nobody in technical leadership has freaked out and mandated a fix asap tells me that has not happened yet.

[Clockwork-Muse]

if they truly understood how big of an issue this was, so the fact nobody in technical leadership has freaked out and mandated a fix asap tells me that has not happened yet.

I don't think this is as large of an issue as you're trying to make it out to be.

Keep in mind that WSL is primarily intended to be a developer tool, and not something you'd run a production-level server on (also - if your server is allowed to sleep you probably have much larger problems). It would be simpler, easier, and cheaper to just run whatever distro "natively" (either in a dedicated hypervisor like Hyper-V, or on bare metal).

Yes, it's annoying. Yes, there are security issues (although exploiting them still requires stealing a private key, which shouldn't be trivial). If it's the end of the world for you, though, you likely have larger problems.

[duaneking]

I don't think this is as large of an issue as you're trying to make it out to be.

Then respectfully, you do not understand the issue or how this impacts the world at scale.

Keep in mind that WSL is primarily intended to be a developer tool, and not something you'd run a production-level server on

... and that's exactly why this is such a big issue. if you're making the bad assumption that because this is a developer's system that it's not going to be attacked, then sadly I have some bad news for you. We developers get attacked everyday, and even now people are trying to figure out ways to get on our machines. You ever hear of supply chain attacks? Developers are the supply chain.

(also - if your server is allowed to sleep you probably have much larger problems).

I agree but this is workstations, and that's even more heavily audited in some environments. If companies invested half as much in their production system security as they put into their corporate security, a lot of breaches would never happen.

It would be simpler, easier, and cheaper to just run whatever distro "natively" (either in a dedicated hypervisor like Hyper-V, or on bare metal).

Not for everybody. It would not be easier for me. I asked for many GB of ram for a reason: Docker/K8s.

Yes, it's annoying. Yes, there are security issues (although exploiting them still requires stealing a private key, which shouldn't be trivial). If it's the end of the world for you, though, you likely have larger problems.

Because looking in "%SystemDrive%\Documents and Settings\All Users\Application Data\Microsoft\Crypto\RSA\MachineKeys is hard, right? ;) I get what you're saying, but it's also clear to me that you don't have the same goals I do.

[gaia]

Even apt install/update fails when the clock is off. You can't develop if you can't install/update the tools you need.

This is an important issue.
This should be basic to fix.
The systemd workaround is enough for me.

[esumii]

Task Scheduler doesn't work to run tasks in WSL2, see #9231

When the user is not logged on to the desktop, I suppose?

[asampal]

I think another point to note is that WSL2 didn't always have this problem. So if it could do the right thing after waking up at one point, how come it's such a problem fixing the issue now?

[dboreham]

I don't think this is as large of an issue as you're trying to make it out to be.

Respectfully disagree. It's a very serious hypervisor bug. Presumably needs to be filed against the right product (not WSL, most likely) to get on the radar of someone who can fix it.