[🐛 Bug]: significant performance and memory usage - regressions when switching from Selenium 3 to 4

[christophs78]

What happened?

PrimeFaces comes with a suite of integrationstest: https://github.com/primefaces/primefaces/tree/master/primefaces-integration-tests

We currently use Selenium 3.141.59. When i run them on my dev-machine (8C/16T, 32 GB RAM) it looks like this:

With primefaces/primefaces#8145 we try to upgrade to Selenium 4.1.0. There it looks like this:

Additional my notebook get´s unresponsive.

We also run this integrationtests on Github actions. They take about twice the time with Selenium 4.1.0. (And we run into some follow-up problems probably because of worse performance due to resource starvation.)

There already has been a (to some degree) similar issue: #9536

How can we reproduce the issue?

See PrimeFaces-Github-links above.

Relevant log output

Selenium 4.1.0:
https://github.com/primefaces/primefaces/actions/runs/1539682360

Compare this with a run on 3.141.59:
https://github.com/primefaces/primefaces/runs/4413396941?check_suite_focus=true

Operating System

Windows 10, Ubuntu 20.04.3

Selenium version

Java 4.1.0

What are the browser(s) and version(s) where you see this issue?

Chrome 96.0.4664.45, Firefox 94.0.2

What are the browser driver(s) and version(s) where you see this issue?

ChromeDriver 96.0.4664.45, GeckoDriver 0.30.0

Are you using Selenium Grid?

No

[github-actions]

@christophs78, thank you for creating this issue. We will troubleshoot it as soon as we can.

---

Info for maintainers

Triage this issue by using labels.

If information is missing, add a helpful comment and then I-issue-template label.

If the issue is a question, add the I-question label.

If the issue is valid but there is no time to troubleshoot it, consider adding the help wanted label.

If the issue requires changes or fixes from an external project (e.g., ChromeDriver, GeckoDriver, W3C), add the applicable G-* label, and it will provide the correct link and auto-close the issue.

After troubleshooting the issue, please add the R-awaiting answer label.

Thank you!

[christophs78]

Maybe this issue is somehow related to parallelism / concurrency.
We use junit.jupiter.execution.parallel.enabled=true to create one thread / browser instance per hardware-thread. This worked very well with Selenium 3.x. (--> on my dev-machine 16 threads)
Is it possible Selenium 4.1.0 has some issues in this area?

[cocorossello]

I think this is related to #9359

A workaround is also provided in the issue

[christophs78]

I think this is related to #9359

A workaround is also provided in the issue

Played around with this for one and a half hour. Yes it´s the same issue, but the workaround does not work.
I also checked out https://github.com/titusfortner/bug9359 and tried it on my machine. I reduced his settings from 12 to 2. It does not matter which values are returned by our JUnit5-CustomStrategy-impl.

Ah. It depends on the Java-version. The work-around does not work for Java 8, but it works for Java 11.
Update: It´s also (at least partial) broken for Java 17. (reproduceable with https://github.com/titusfortner/bug9359)

IMO #9359 needs to be reopened.

[titusfortner]

Well, looks like everyone now has to use Java 11.

I'm dismayed to be able to duplicate your findings.

I'll close this one and re-open that one. :-/

[pujagani]

@christophs78 Thank you for your findings and effort in solving this! I am sorry the earlier fix was not the perfect fix for this problem.
junit-team/junit-framework#1858 (comment) Mentions that the potential problem lies due to

selenium/java/src/org/openqa/selenium/remote/service/DriverService.java

Line 67 in 28070d2

private final ExecutorService executorService = Executors.newFixedThreadPool(2, r -> {

That change was made to support Junit 5, which now based on your findings works only for Java 11 unfortunately. Apologies that it does not work for all versions.
It was made since

selenium/java/src/org/openqa/selenium/remote/service/DriverService.java

Line 204 in 28070d2

CompletableFuture<StartOrDie> serverStarted = CompletableFuture.supplyAsync(() -> {

by default uses a ForkJoinPool, and that seemed to be the problem so we switched it to use a fixed thread pool.
Based on your findings, what do you think is primarily causing the problem? Any suggestions on how we can tackle it?

Thank you!

[christophs78]

@pujagani Thank´s for your reply. To be honest i currently have no good idea how to fix this within Selenium.
My personal feeling (being a senior java-dev, but not having extensive experience with async on java) is nested thread-pools (ForkJoinPool used buy JUnit 5 and ThreadPoolExecutor used by Selenium 4) is not the best idea. Maybe even OpenJDK is not perfect in this constellation.
I´d liked to play around a bit with Selenium - source and trying ideas, but IntelliJ-Bazel-plugin does not support IntelliJ 2021.3 yet. Maybe i can come back on this in a few weeks.
So I´ll try to do some sandbox-test´s with a sandbox-project where i try to reproduce the ThreadPoolExecutor nested within a ForkJoinPool and come back afterwards.
(Maybe you already saw junit-team/junit-framework#2805 as an idea how to tackle this issue from the other side. I´ll also look what the JUnit - team thinks about this.)

[titusfortner]

@christophs78 thanks again for your help with tracking this down.

Fwiw, JetBrains Toolbox makes it really simple to downgrade a minor version to something that supports Bazel. Also, if you can join us on the Selenium chat (IRC or Slack)— https://www.selenium.dev/support/#ChatRoom
Devs hang out in #selenium-tlc chat room, and we'd love to collaborate with you as best we can.