[alnzng]

Linked issue: #xxx

Purpose of change

Setuptools v82.0.0 (released yesterday) removed pkg_resources as a top-level module. From the https://setuptools.pypa.io/en/stable/history.html:

pkg_resources has been removed from Setuptools. Most common uses of pkg_resources have been superseded by the [importlib.resources](https://docs.python.org/3/library/importlib.resources.html) and [importlib.metadata](https://docs.python.org/3/library/importlib.metadata.html) projects. Projects and environments relying on pkg_resources for namespace packages or other behavior should depend on older versions of setuptools. ([#3085](https://github.com/pypa/setuptools/issues/3085))

This change could cause problems because apache-beam < 2.66.0 imports pkg_resources at module load time, and current apache-flink still use apache-beam version < 2.61.0:

• v2.65.0 (still has pkg_resources): https://github.com/apache/beam/blob/v2.65.0/sdks/python/setup.py
• v2.66.0 (removed): https://github.com/apache/beam/blob/v2.66.0/sdks/python/setup.py

Example Errors:

Example error 1: https://github.com/apache/flink-agents/actions/runs/21844360007/job/63036590931

   Building apache-beam==2.61.0
  × Failed to build `apache-beam==2.61.0`
  ├─▶ The build backend returned an error
  ╰─▶ Call to `setuptools.build_meta.build_wheel` failed (exit status: 1)

      [stderr]
      Traceback (most recent call last):
        File "<string>", line 14, in <module>
        File
      "/Users/runner/work/_temp/setup-uv-cache/builds-v0/.tmpxdQVyc/lib/python3.11/site-packages/setuptools/build_meta.py",
      line 333, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=[])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File
      "/Users/runner/work/_temp/setup-uv-cache/builds-v0/.tmpxdQVyc/lib/python3.11/site-packages/setuptools/build_meta.py",
      line 301, in _get_build_requires
          self.run_setup()
        File
      "/Users/runner/work/_temp/setup-uv-cache/builds-v0/.tmpxdQVyc/lib/python3.11/site-packages/setuptools/build_meta.py",
      line 317, in run_setup
          exec(code, locals())
        File "<string>", line 34, in <module>
      ModuleNotFoundError: No module named 'pkg_resources'

      hint: This error likely indicates that `apache-beam@2.61.0` depends
      on `pkg_resources`, but doesn't declare it as a build dependency. If
      `apache-beam` is a first-party package, consider adding `pkg_resources`
      to its `build-system.requires`. Otherwise, either add it to your
      `pyproject.toml` under:

      [tool.uv.extra-build-dependencies]
      apache-beam = ["pkg_resources"]

      or `uv pip install pkg_resources` into the environment and re-run with
      `--no-build-isolation`.
  help: `apache-beam` (v2.61.0) was included because `apache-flink` (v2.2.0)
        depends on `apache-beam`

Example error 2 - this is from local testing

2026-02-09 14:26:47
java.lang.RuntimeException: Failed to create stage bundle factory! INFO:root:Initializing Python harness: /Users/alazhang/Workspaces/Python/linkedin_extend/flink-agent/.venv/lib/python3.11/site-packages/pyflink/fn_execution/beam/beam_boot.py --id=2-1 --provision_endpoint=localhost:62283
INFO:root:Starting up Python harness in a standalone process.
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/alazhang/Workspaces/Python/linkedin_extend/flink-agent/.venv/lib/python3.11/site-packages/pyflink/fn_execution/beam/beam_boot.py", line 121, in <module>
    from pyflink.fn_execution.beam import beam_sdk_worker_main
  File "/Users/alazhang/Workspaces/Python/linkedin_extend/flink-agent/.venv/lib/python3.11/site-packages/pyflink/fn_execution/beam/beam_sdk_worker_main.py", line 24, in <module>
    import pyflink.fn_execution.beam.beam_operations # noqa # pylint: disable=unused-import
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alazhang/Workspaces/Python/linkedin_extend/flink-agent/.venv/lib/python3.11/site-packages/pyflink/fn_execution/beam/beam_operations.py", line 21, in <module>
    from apache_beam.runners.worker import bundle_processor, operation_specs
  File "/Users/alazhang/Workspaces/Python/linkedin_extend/flink-agent/.venv/lib/python3.11/site-packages/apache_beam/runners/worker/bundle_processor.py", line 52, in <module>
    import apache_beam as beam
  File "/Users/alazhang/Workspaces/Python/linkedin_extend/flink-agent/.venv/lib/python3.11/site-packages/apache_beam/__init__.py", line 88, in <module>
    from apache_beam import io
  File "/Users/alazhang/Workspaces/Python/linkedin_extend/flink-agent/.venv/lib/python3.11/site-packages/apache_beam/io/__init__.py", line 21, in <module>
    from apache_beam.io.avroio import *
  File "/Users/alazhang/Workspaces/Python/linkedin_extend/flink-agent/.venv/lib/python3.11/site-packages/apache_beam/io/avroio.py", line 53, in <module>
    from apache_beam.io import filebasedsink
  File "/Users/alazhang/Workspaces/Python/linkedin_extend/flink-agent/.venv/lib/python3.11/site-packages/apache_beam/io/filebasedsink.py", line 29, in <module>
    from apache_beam.io import iobase
  File "/Users/alazhang/Workspaces/Python/linkedin_extend/flink-agent/.venv/lib/python3.11/site-packages/apache_beam/io/iobase.py", line 54, in <module>
    from apache_beam.transforms import Impulse
  File "/Users/alazhang/Workspaces/Python/linkedin_extend/flink-agent/.venv/lib/python3.11/site-packages/apache_beam/transforms/__init__.py", line 24, in <module>
    from apache_beam.transforms.external import *
  File "/Users/alazhang/Workspaces/Python/linkedin_extend/flink-agent/.venv/lib/python3.11/site-packages/apache_beam/transforms/external.py", line 41, in <module>
    from apache_beam.runners import pipeline_context
  File "/Users/alazhang/Workspaces/Python/linkedin_extend/flink-agent/.venv/lib/python3.11/site-packages/apache_beam/runners/pipeline_context.py", line 48, in <module>
    from apache_beam.transforms import environments
  File "/Users/alazhang/Workspaces/Python/linkedin_extend/flink-agent/.venv/lib/python3.11/site-packages/apache_beam/transforms/environments.py", line 53, in <module>
    from apache_beam.runners.portability import stager
  File "/Users/alazhang/Workspaces/Python/linkedin_extend/flink-agent/.venv/lib/python3.11/site-packages/apache_beam/runners/portability/stager.py", line 63, in <module>
    import pkg_resources
ModuleNotFoundError: No module named 'pkg_resources'

	at org.apache.flink.streaming.api.runners.python.beam.BeamPythonFunctionRunner.createStageBundleFactory(BeamPythonFunctionRunner.java:677)
	at org.apache.flink.streaming.api.runners.python.beam.BeamPythonFunctionRunner.open(BeamPythonFunctionRunner.java:290)
	at org.apache.flink.streaming.api.operators.python.process.AbstractExternalPythonFunctionOperator.open(AbstractExternalPythonFunctionOperator.java:57)
	at org.apache.flink.streaming.api.operators.python.process.AbstractExternalDataStreamPythonFunctionOperator.open(AbstractExternalDataStreamPythonFunctionOperator.java:85)
	at org.apache.flink.streaming.api.operators.python.process.AbstractExternalOneInputPythonFunctionOperator.open(AbstractExternalOneInputPythonFunctionOperator.java:117)
	at org.apache.flink.streaming.api.operators.python.process.ExternalPythonProcessOperator.open(ExternalPythonProcessOperator.java:64)
	at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.initializeStateAndOpenOperators(RegularOperatorChain.java:107)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreStateAndGates(StreamTask.java:858)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$restoreInternal$5(StreamTask.java:812)
	at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:812)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:771)
	at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:970)
	at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:939)
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:763)
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException: java.lang.IllegalStateException: Process died with exit code 0
	at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2050)
	at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache.get(LocalCache.java:3952)
	at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974)
	at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4958)
	at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4964)
	at org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory$SimpleStageBundleFactory.<init>(DefaultJobBundleFactory.java:498)
	at org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory$SimpleStageBundleFactory.<init>(DefaultJobBundleFactory.java:482)
	at org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory.forStage(DefaultJobBundleFactory.java:342)
	at org.apache.flink.streaming.api.runners.python.beam.BeamPythonFunctionRunner.createStageBundleFactory(BeamPythonFunctionRunner.java:675)
	... 16 more
Caused by: java.lang.IllegalStateException: Process died with exit code 0
	at org.apache.beam.runners.fnexecution.environment.ProcessManager$RunningProcess.isAliveOrThrow(ProcessManager.java:75)
	at org.apache.beam.runners.fnexecution.environment.ProcessEnvironmentFactory.createEnvironment(ProcessEnvironmentFactory.java:110)
	at org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory$1.load(DefaultJobBundleFactory.java:284)
	at org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory$1.load(DefaultJobBundleFactory.java:240)
	at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3528)
	at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2277)
	at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2154)
	at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2044)
	... 24 more

Tests

• Deployed a test Flink Agents app in local Flink cluster and verified its logs

API

N/A

Documentation

• doc-needed
• doc-not-needed
• doc-included

[alnzng]

@wenjin272 @xintongsong Any way to just rerun the failed CI tests? Above failed CI test failure seems to be not related to this change.

[xintongsong]

@alnzng, The code changes LGTM.

I think we might want to also create an issue ticket for this. Being limited to setuptools<82 is definitely not a long-term solution and we should push the beam community to resolve this issue, and then pyflink, which obviously takes time. An issue ticket would be a good place to gather all the historical context and progress, compared to a hotfix pr.

[xintongsong]

Ported to:

• release-0.1: 766f7c3
• release-0.2: c3935a6

[alnzng]

@xintongsong Good suggestion! Created a issue for tracking the long term fix: #529