Skip to content

Updates for St2 3.10#6367

Open
nzlosh wants to merge 77 commits into
StackStorm:masterfrom
nzlosh:updates_3.10
Open

Updates for St2 3.10#6367
nzlosh wants to merge 77 commits into
StackStorm:masterfrom
nzlosh:updates_3.10

Conversation

@nzlosh

@nzlosh nzlosh commented Feb 2, 2026

Copy link
Copy Markdown
Contributor
  • Add support for Rocky10 and Ubuntu24.04 (Noble)
  • Drop support for Rocky8 and Ubuntu20.04 (Focal)

In the local and remote runner the unit file test were moved leaving
a directory without tests.  pytest raises error code 5 in this case
which causes the unit-tests target to fail.
@nzlosh nzlosh added this to the 3.10.0 milestone Feb 2, 2026
@nzlosh nzlosh self-assigned this Feb 2, 2026
@pull-request-size pull-request-size Bot added the size/L PR that changes 100-499 lines. Requires some effort to review. label Feb 2, 2026
@pull-request-size pull-request-size Bot added size/XXL PR that changes 1000+ lines. You should absolutely split your PR into several. and removed size/L PR that changes 100-499 lines. Requires some effort to review. labels Mar 17, 2026
@nzlosh

nzlosh commented Mar 17, 2026

Copy link
Copy Markdown
Contributor Author

Regenerated lockfile result

Lockfile diff: lockfiles/flake8.lock [flake8]
==                    Upgraded dependencies                     ==
  setuptools                     75.3.2       -->   75.3.4

Lockfile diff: lockfiles/black.lock [black]
==                    Upgraded dependencies                     ==
  tomli                          2.2.1        -->   2.4.0

Lockfile diff: lockfiles/pylint.lock [pylint]
==                    Upgraded dependencies                     ==
  setuptools                     75.3.2       -->   75.3.4
  tomli                          2.2.1        -->   2.4.0

Lockfile diff: lockfiles/st2.lock [st2]
==                    Upgraded dependencies                     ==
  apscheduler                    3.11.1       -->   3.11.2
  certifi                        2025.11.12   -->   2026.2.25
  charset-normalizer             3.4.4        -->   3.4.6
  gitpython                      3.1.45       -->   3.1.46
  httplib2                       0.31.0       -->   0.31.2
  icdiff                         2.0.7        -->   2.0.10
  mongoengine                    0.29.1       -->   0.29.3
  oslo-config                    9.6.0        -->   9.6.1
  packaging                      25.0         -->   26.0
  psutil                         7.1.3        -->   7.2.2
  pyasn1                         0.6.1        -->   0.6.3
  pytz                           2025.2       -->   2026.1.post1
  setuptools                     75.3.2       -->   75.3.4
  smmap                          5.0.2        -->   5.0.3
  sseclient-py                   1.8.0        -->   1.9.0
  st2-auth-ldap                  3.9.dev0     -->   3.10.dev0
  st2-rbac-backend               3.9.dev0     -->   3.10.dev0
  tomli                          2.3.0        -->   2.4.0
  tzdata                         2025.2       -->   2025.3
  wcwidth                        0.2.14       -->   0.6.0

Lockfile diff: lockfiles/bandit.lock [bandit]
==                    Upgraded dependencies                     ==
  gitpython                      3.1.45       -->   3.1.46
  pbr                            7.0.1        -->   7.0.3
  rich                           14.1.0       -->   14.3.3
  setuptools                     75.3.2       -->   75.3.4
  smmap                          5.0.2        -->   5.0.3

Lockfile diff: lockfiles/twine.lock [twine]
==                    Upgraded dependencies                     ==
  certifi                        2025.8.3     -->   2026.2.25
  charset-normalizer             3.4.3        -->   3.4.6
  idna                           3.10         -->   3.11
  nh3                            0.3.0        -->   0.3.3
  rich                           14.1.0       -->   14.3.3

Lockfile diff: lockfiles/pants-plugins.lock [pants-plugins]
==                    Upgraded dependencies                     ==
  attrs                          25.3.0       -->   25.4.0
  certifi                        2025.8.3     -->   2026.2.25
  charset-normalizer             3.4.3        -->   3.4.6
  idna                           3.10         -->   3.11
  iniconfig                      2.1.0        -->   2.3.0
  pyparsing                      3.2.5        -->   3.3.2
  tomli                          2.2.1        -->   2.4.0
  ujson                          5.11.0       -->   5.12.0
  urllib3                        2.5.0        -->   2.6.3

nzlosh added 8 commits March 17, 2026 22:10
Lockfile diff: lockfiles/twine.lock [twine]
==                    Upgraded dependencies                     ==
  cffi                           1.17.1       -->   2.0.0
  cryptography                   43.0.3       -->   46.0.5
  docutils                       0.20.1       -->   0.22.4
  jaraco-context                 6.0.1        -->   6.1.1
  jaraco-functools               4.1.0        -->   4.4.0
  keyring                        25.5.0       -->   25.7.0
  markdown-it-py                 3.0.0        -->   4.0.0
  more-itertools                 10.5.0       -->   10.8.0
  pycparser                      2.23         -->   3.0
  readme-renderer                43.0         -->   44.0
  requests                       2.32.4       -->   2.32.5
  secretstorage                  3.3.3        -->   3.5.0
  urllib3                        2.2.3        -->   2.6.3
  zipp                           3.20.2       -->   3.23.0
==                      Added dependencies                      ==
  typing-extensions              4.15.0
==                     Removed dependencies                     ==
  importlib-resources            6.4.5

Lockfile diff: lockfiles/bandit.lock [bandit]
==                    Upgraded dependencies                     ==
  markdown-it-py                 3.0.0        -->   4.0.0
  setuptools                     75.3.4       -->   82.0.1
  stevedore                      5.3.0        -->   5.7.0
==                     Removed dependencies                     ==
  pbr                            7.0.3
  typing-extensions              4.13.2

Lockfile diff: lockfiles/flake8.lock [flake8]
==                    Upgraded dependencies                     ==
  setuptools                     75.3.4       -->   82.0.1

Lockfile diff: lockfiles/pylint.lock [pylint]
==                    Upgraded dependencies                     ==

  dill                           0.4.0        -->   0.4.1
  platformdirs                   4.3.6        -->   4.9.4
  setuptools                     75.3.4       -->   82.0.1
  tomlkit                        0.13.3       -->   0.14.0
  typing-extensions              4.13.2       -->   4.15.0

Lockfile diff: lockfiles/black.lock [black]
==                    Upgraded dependencies                     ==
  click                          8.1.8        -->   8.3.1
  pathspec                       0.12.1       -->   1.0.4
  platformdirs                   4.3.6        -->   4.9.4
==                     Removed dependencies                     ==
  typing-extensions              4.13.2
Lockfile diff: lockfiles/st2.lock [st2]
==                    Upgraded dependencies                     ==
  attrs                          25.4.0       -->   26.1.0
  chardet                        7.2.0        -->   7.3.0
  debtcollector                  3.0.0        -->   3.1.0
  futurist                       3.2.1        -->   3.3.0
  gunicorn                       25.1.0       -->   25.2.0
  importlib-metadata             8.7.1        -->   9.0.0
  jsonpointer                    3.0.0        -->   3.1.1
  kazoo                          2.10.0       -->   2.11.0
  python-discovery               1.1.3        -->   1.2.0
  redis                          7.3.0        -->   7.4.0
  werkzeug                       3.1.6        -->   3.1.7
==                !! Downgraded dependencies !!                 ==
  jsonschema                     4.26.0       -->   4.17.3
==                      Added dependencies                      ==
  pyrsistent                     0.20.0
==                     Removed dependencies                     ==
  jsonschema-specifications      2025.9.1
  referencing                    0.37.0
  rpds-py                        0.30.0
…ents

Lockfile diff: lockfiles/st2.lock [st2]
==                    Upgraded dependencies                     ==
  cryptography                   46.0.5       -->   46.0.6
  requests                       2.32.5       -->   2.33.0
  tomli                          2.4.0        -->   2.4.1
==                      Added dependencies                      ==
  importlib-resources            6.5.2
nzlosh added 2 commits March 26, 2026 08:36
…le/requirements

Lockfile diff: lockfiles/st2.lock [st2]
==                      Added dependencies                      ==
  jsonschema-spec                0.1.6
  lazy-object-proxy              1.12.0
  openapi-schema-validator       0.4.4
  openapi-spec-validator         0.5.7
  pathable                       0.4.4
  rfc3339-validator              0.1.4
==                     Removed dependencies                     ==
  flex                           6.14.1
  jsonpointer                    3.1.1
  rfc3987                        1.3.8
  strict-rfc3339                 0.7
  validate-email                 1.3
@nzlosh nzlosh force-pushed the updates_3.10 branch 2 times, most recently from bab44e1 to 3d2ee49 Compare March 26, 2026 21:49
@guzzijones

Copy link
Copy Markdown
Contributor

Do we need to merge this command line due to the required tests that are no longer run? @nzlosh @skiedude

@guzzijones

Copy link
Copy Markdown
Contributor

we should force push this to master so we can fix st2-packaging. I am seeing ubuntu using an old version of pip==25.0.1

@guzzijones

Copy link
Copy Markdown
Contributor

I cannot locally merge and push to master. i do not have bypass authority. we are stuck in a situation where this repo needs st2-packages updated and st2-packages need this repo updated. Someone needs to merge this who has bypass authority please.

@nzlosh nzlosh force-pushed the updates_3.10 branch 2 times, most recently from 4d87d54 to c6b6649 Compare June 26, 2026 19:43
@guzzijones

Copy link
Copy Markdown
Contributor

it looks to be using an older pip to actually make the wheels. then the newer pip to install from the existing wheel folder.

nzlosh added 2 commits June 30, 2026 17:20
Regnerated fixed requirements and pants lockfile.

Lockfile diff: lockfiles/st2.lock [st2]

==                    Upgraded dependencies                     ==

  apscheduler                    3.11.2       -->   3.11.3
  argcomplete                    3.6.3        -->   3.7.0
  beautifulsoup4                 4.14.3       -->   4.15.0
  certifi                        2026.4.22    -->   2026.6.17
  cffi                           2.0.0        -->   2.1.0
  charset-normalizer             3.4.7        -->   3.4.8
  click                          8.3.3        -->   8.4.2
  cryptography                   47.0.0       -->   49.0.0
  decorator                      5.2.1        -->   5.3.1
  distlib                        0.4.0        -->   0.4.3
  editor                         1.7.0        -->   1.8.0
  filelock                       3.29.0       -->   3.29.6
  gitpython                      3.1.49       -->   3.1.50
  greenlet                       3.5.0        -->   3.5.3
  gunicorn                       25.3.0       -->   26.0.0
  httplib2                       0.31.2       -->   0.32.0
  idna                           3.13         -->   3.18
  jsonschema-path                0.4.6        -->   0.5.0
  msgpack                        1.1.2        -->   1.2.1
  openapi-schema-validator       0.8.1        -->   0.9.0
  openapi-spec-validator         0.8.5        -->   0.9.0
  orjson                         3.11.8       -->   3.11.9
  oslo-config                    10.3.0       -->   10.4.0
  oslo-i18n                      6.7.2        -->   6.8.0
  oslo-serialization             5.9.1        -->   5.10.0
  oslo-utils                     10.0.1       -->   10.1.1
  pathable                       0.5.0        -->   0.6.0
  pika                           1.3.2        -->   1.4.1
  pip                            26.1         -->   26.1.2
  platformdirs                   4.9.6        -->   4.10.0
  prettytable                    3.17.0       -->   3.18.0
  pydantic                       2.13.3       -->   2.13.4
  pydantic-core                  2.46.3       -->   2.46.4
  pydantic-settings              2.14.0       -->   2.14.2
  pygal                          3.1.0        -->   3.1.3
  python-discovery               1.2.2        -->   1.4.3
  python-ldap                    3.4.5        -->   3.4.7
  pytz                           2026.1.post1   -->   2026.2
  redis                          7.4.0        -->   8.0.1
  repoze-lru                     0.7          -->   0.8
  requests                       2.33.1       -->   2.34.2
  setuptools                     80.10.2      -->   82.0.1
  soupsieve                      2.8.3        -->   2.8.4
  stevedore                      5.7.0        -->   5.8.0
  typing-extensions              4.15.0       -->   4.16.0
  tzlocal                        5.3.1        -->   5.4.4
  ujson                          5.12.0       -->   5.13.0
  urllib3                        2.6.3        -->   2.7.0
  virtualenv                     21.3.0       -->   21.6.0
  wcwidth                        0.6.0        -->   0.8.2
  webob                          1.8.9        -->   1.8.10
  wrapt                          2.1.2        -->   2.2.2
  xmod                           1.9.0        -->   1.10.0
  zipp                           3.23.1       -->   4.1.0

f
@guzzijones

Copy link
Copy Markdown
Contributor

Why are we specifying the specific python binary now? Previously we had a working build we just needed to force push so we could work on packaging.

@guzzijones

Copy link
Copy Markdown
Contributor

fwiw it looks like st2api is not starting for some reason. thus the failure.

@guzzijones

guzzijones commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

it appears that gunicorn may have removed the eventlet entrypoint. you may need to pin gunicorn?
st2api.log artifact:


Error: class uri 'eventlet' invalid or not found: 

[Traceback (most recent call last):
  File "/home/runner/work/st2/st2/dist/export/python/virtualenvs/st2/3.11.11/lib/python3.11/site-packages/gunicorn/util.py", line 110, in load_class
    return load_entry_point(
           ^^^^^^^^^^^^^^^^^
  File "/home/runner/work/st2/st2/dist/export/python/virtualenvs/st2/3.11.11/lib/python3.11/site-packages/gunicorn/util.py", line 75, in load_entry_point
    raise ImportError("Entry point %r not found" % ((group, name),))
ImportError: Entry point ('gunicorn.workers', 'eventlet') not found

@nzlosh

nzlosh commented Jul 8, 2026

Copy link
Copy Markdown
Contributor Author

Good catch! Gunicorn v26.0.0 was released 5th May 2026 and dropped eventlet support https://gunicorn.org/news/#breaking-changes. I'll pin to 25.3.0.

We need to specify an exact version of Python for Rocky9 or the system packages fail to build because the default python3 interpreter is python 3.9.

Lockfile diff: lockfiles/st2.lock [st2]
==                    Upgraded dependencies                     ==
  charset-normalizer             3.4.8        -->   3.4.9
  filelock                       3.29.6       -->   3.29.7
==                !! Downgraded dependencies !!                 ==
  gunicorn                       26.0.0       -->   25.3.0
@guzzijones

Copy link
Copy Markdown
Contributor

Good catch! Gunicorn v26.0.0 was released 5th May 2026 and dropped eventlet support https://gunicorn.org/news/#breaking-changes. I'll pin to 25.3.0.

We need to specify an exact version of Python for Rocky9 or the system packages fail to build because the default python3 interpreter is python 3.9.

I commonly set PYTHON_VERSION=python3.11 and the make file just works. once the virtualenv is created python points to the correct version. here introduced an if statement, but can't we just set PYTHON_VERSION using the original code?

@guzzijones

Copy link
Copy Markdown
Contributor

The issues i am seeing in the circle ci stage are

  1. ubuntu build for python3.10 is failing and i am not sure why as the error says
[package: st2] [07:33:18]      The conflict is caused by:
[package: st2] [07:33:18]          jsonschema 4.26.0 depends on referencing>=0.28.4
[package: st2] [07:33:18]          jsonschema-path 0.3.4 depends on referencing<0.37.0
  1. as an aside the version of pip in the build stage is different than the wheelhouse stage.
    possibly there is no version of referencing for python3.10 that fits those restraints.

  2. el9 build is trying to use python3.9.

    1. this is set in st2-packaging make file not the make file in this repository.
    2. I have an MR to fix this as part of a larger fix.

I suggest we revert the changes here, force merge this. then work on st2-packaging.
Unfortunately, the make file logic is duplicated from this repo and st2-packaging

@guzzijones

Copy link
Copy Markdown
Contributor

ah actually since it is only looking in the wheelhouse the referencing version it needs does not exist.

@nzlosh

nzlosh commented Jul 8, 2026

Copy link
Copy Markdown
Contributor Author

I've got package builds for rocky9 working successfully locally, with the addition of the change in the makefile. I'm curretly debugging dh-virtualenv issues for jammy. Once I've resolved those issues, this branch will be ready for merge.

@guzzijones

Copy link
Copy Markdown
Contributor

I was able to get past the st2api service issues by pinning gunicorn < 0.25. gunicorn was requiring a higher version of eventlet 0.40.3. now i am seeing issues using virtualenv to install packs causing pip, setuptools, sha mismatches due to virtualenv bin embedding it's own wheels for those.

packging error

switch to python -m venv ?

…ker requires eventlet 0.40.3 or higher"

Lockfile diff: lockfiles/st2.lock [st2]
==                    Upgraded dependencies                     ==
  pyasn1                         0.6.3        -->   0.6.4
  python-discovery               1.4.3        -->   1.4.4
@guzzijones

Copy link
Copy Markdown
Contributor

@nzlosh you need to also pin referencing The wheel file that is downloaded is not compatible otherwise.

@guzzijones

Copy link
Copy Markdown
Contributor

The core issue i am seeing now is that virtualenv has added a hash check to make sure the embedded setuptools and pip are not changed. I am not sure how to go about debugging this one.

st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Running command "/opt/stackstorm/st2/bin/virtualenv -p /opt/stackstorm/st2/bin/python --always-copy --verbose --no-download /opt/stackstorm/virtualenvs/hubot" to create virtualenv.
Traceback (most recent call last):
  File "/opt/stackstorm/st2/lib/python3.10/site-packages/python_runner/python_action_wrapper.py", line 397, in <module>
    obj.run()
  File "/opt/stackstorm/st2/lib/python3.10/site-packages/python_runner/python_action_wrapper.py", line 216, in run
    output = action.run(**self._parameters)
  File "/opt/stackstorm/packs/packs/actions/pack_mgmt/setup_virtualenv.py", line 93, in run
    setup_pack_virtualenv(
  File "/opt/stackstorm/st2/lib/python3.10/site-packages/st2common/util/virtualenvs.py", line 115, in setup_pack_virtualenv
    create_virtualenv(
  File "/opt/stackstorm/st2/lib/python3.10/site-packages/st2common/util/virtualenvs.py", line 241, in create_virtualenv
    raise Exception(
Exception: Failed to create virtualenv in "/opt/stackstorm/virtualenvs/hubot":
 stdout=find interpreter for spec PythonSpec(path=/opt/stackstorm/st2/bin/python)
proposed PythonInfo(spec=CPython3.10.12.final.0-64-x86_64, system=/usr/bin/python3.10, exe=/opt/stackstorm/st2/bin/python, platform=linux, version='3.10.12 (main, Jun 22 2026, 18:55:27) [GCC 11.4.0]', encoding_fs_io=utf-8-utf-8)
create virtual environment via CPython3Posix(dest=/opt/stackstorm/virtualenvs/hubot, clear=False, no_vcs_ignore=False, global=False)
add seed packages via FromAppData(download=False, pip=bundle, setuptools=bundle, via=copy, app_data_dir=/root/.cache/virtualenv)
fail
Traceback (most recent call last):
  File "/opt/stackstorm/st2/lib/python3.10/site-packages/virtualenv/seed/embed/via_app_data/via_app_data.py", line 98, in _get
    result = get_wheel(
  File "/opt/stackstorm/st2/lib/python3.10/site-packages/virtualenv/seed/wheels/acquire.py", line 67, in get_wheel
    wheel = from_bundle(distribution, version, for_py_version, search_dirs, app_data, do_periodic_update, env)
  File "/opt/stackstorm/st2/lib/python3.10/site-packages/virtualenv/seed/wheels/bundle.py", line 27, in from_bundle
    wheel = load_embed_wheel(app_data, distribution, for_py_version, of_version)
  File "/opt/stackstorm/st2/lib/python3.10/site-packages/virtualenv/seed/wheels/bundle.py", line 43, in load_embed_wheel
    wheel = get_embed_wheel(distribution, for_py_version)
  File "/opt/stackstorm/st2/lib/python3.10/site-packages/virtualenv/seed/wheels/embed/__init__.py", line 92, in get_embed_wheel
    _verify_bundled_wheel(path)
  File "/opt/stackstorm/st2/lib/python3.10/site-packages/virtualenv/seed/wheels/embed/__init__.py", line 107, in _verify_bundled_wheel
    raise RuntimeError(msg)
RuntimeError: bundled wheel setuptools-83.0.0-py3-none-any.whl sha256 mismatch: expected 29b23c360f22f414dc7336bb39178cc7bcbf6021ed2733cde173f09dba19abb3, got e1a9ce5e48ed62145a529ad25d16a420f9306487b9b939d5f2ed27f453965f5c

@guzzijones

Copy link
Copy Markdown
Contributor

Something in the build process for ubuntu is changing the hashes for virtualenv seed pip and setuptools on each build.
I am not sure where it is looking for pip and setuptools specifically as i do not see the values of the variables in the stack trace.
run 1 expected

RuntimeError('bundled wheel pip-26.1.2-py3-none-any.whl sha256 mismatch: expected 382ff9f685ee3bc25864f820aa50505825f10f5458ffff07e30a6d96e5715cab, got 

run 1 result

66c3d61daaadec9ff1bbd36b0bb50473997241b886a51e52c9440cb1a224e649')

run 2 expected

RuntimeError: bundled wheel pip-26.1.2-py3-none-any.whl sha256 mismatch: expected 382ff9f685ee3bc25864f820aa50505825f10f5458ffff07e30a6d96e5715cab, got 

run 2 result

5f98ffef4002004b0fa1ea4287646a4174d4bf2380a48e7945ae8eae457e171b

run 3 expected

RuntimeError: bundled wheel pip-26.1.2-py3-none-any.whl sha256 mismatch: expected 382ff9f685ee3bc25864f820aa50505825f10f5458ffff07e30a6d96e5715cab, got 

run 3 result

6efcf36fd071aa902b3a6330ef3378b588b482754633a1c45b29ce2c8e989e75

@nzlosh

nzlosh commented Jul 10, 2026

Copy link
Copy Markdown
Contributor Author

I'm seeing the same issue building locally, I'll try debug today.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/XXL PR that changes 1000+ lines. You should absolutely split your PR into several.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants