Skip to content

test(php): cover OpenFeature in all weblog environments - #7393

Open
leoromanovsky wants to merge 10 commits into
mainfrom
leo.romanovsky/php-openfeature-all-environments
Open

test(php): cover OpenFeature in all weblog environments#7393
leoromanovsky wants to merge 10 commits into
mainfrom
leo.romanovsky/php-openfeature-all-environments

Conversation

@leoromanovsky

@leoromanovsky leoromanovsky commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Motivation

A customer found that an application error handler can convert a provider warning into an exception. OpenFeature then returns the code default with a GENERAL error.

The existing PHP FFE tests called the native Datadog client. They did not cross the OpenFeature provider and logger boundary where this failure occurred.

flowchart LR
    App["Application"] --> Client["OpenFeature client"]
    Client --> Provider["Datadog provider"]
    Provider --> Evaluator["Datadog evaluator"]
    Provider -. "warning" .-> Handler["Application error handler"]
    Handler --> Exception["Exception"]
    Exception --> Error["Code default plus GENERAL error"]

    Test["Previous FFE test"] --> Endpoint["POST /ffe"]
    Endpoint --> Native["Native Datadog client"]
    Native --> Evaluator
    Test -. "missed this boundary" .-> Provider
Loading

The OpenFeature PHP SDK requires PHP 8 or later. System tests still support PHP 7. PHP 7 must keep the /ffe contract when the SDK is unavailable.

Changes and Decisions

The /ffe endpoint now uses OpenFeature by default. Tests that need the native client can set evaluationApi to native.

The generic test omits evaluationApi. An available provider must return the configured value. An unavailable provider must return the supplied default with ERROR and PROVIDER_NOT_READY.

flowchart TD
    Test["Generic FFE test"] --> Endpoint["POST /ffe"]
    Endpoint --> Default["Default to OpenFeature"]
    Default --> Available{"Provider available?"}
    Available -->|"Yes"| Success["Configured value and no error"]
    Available -->|"No"| Fallback["Supplied default<br/>ERROR<br/>PROVIDER_NOT_READY"]
    Success --> Contract["Contract passes"]
    Fallback --> Contract
Loading

The shared PHP installer adds the OpenFeature SDK on PHP 8 and later. Laravel and Symfony use the shared /ffe handler. PHP 7 marks provider-dependent tests as unavailable but keeps the generic contract test enabled.

The PHP 8 success path depends on DataDog/dd-trace-php#4071.

Validation

These local runs used this PR at 0dad5d48bec63cffdd206b724013d26f497c0c8c. They used the CI-built ARM64 package from dd-trace-php bf360ef80839f483179e37215536d1713116167c.

TEST_LIBRARY=php WEBLOG_VARIANT=php-fpm-8.2 ./run.sh FEATURE_FLAGGING_AND_EXPERIMENTATION
37 passed, 9 skipped, 2735 deselected in 153.98s

TEST_LIBRARY=php WEBLOG_VARIANT=laravel11x ./run.sh FEATURE_FLAGGING_AND_EXPERIMENTATION tests/ffe/test_dynamic_evaluation.py::Test_FFE_OpenFeature_Evaluation
1 passed in 34.02s

TEST_LIBRARY=php WEBLOG_VARIANT=symfony7x ./run.sh FEATURE_FLAGGING_AND_EXPERIMENTATION tests/ffe/test_dynamic_evaluation.py::Test_FFE_OpenFeature_Evaluation
1 passed in 34.07s

TEST_LIBRARY=php WEBLOG_VARIANT=php-fpm-7.4 ./run.sh FEATURE_FLAGGING_AND_EXPERIMENTATION
3 passed, 9 skipped, 29 xfailed, 5 xpassed, 2735 deselected in 210.78s

The PHP-FPM 8.2 image used PHP 8.2.32. All four images reported tracer version 1.24.0+dev.bf360ef80839f483179e37215536d1713116167c.

@github-actions

github-actions Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

CODEOWNERS have been resolved as:

docs/understand/weblogs/end-to-end_weblog.md                            @DataDog/system-tests-core
manifests/php.yml                                                       @DataDog/apm-php @DataDog/asm-php
tests/ffe/test_dynamic_evaluation.py                                    @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
utils/build/docker/php/apache-mod/build.sh                              @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/common/ffe.php                                   @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/common/install_ddtrace.sh                        @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/weblogs/laravel11x/routes/web.php                @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/weblogs/symfony7x/src/Controller/AppController.php  @DataDog/apm-php @DataDog/system-tests-core

@datadog-prod-us1-3

datadog-prod-us1-3 Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Pipelines  Tests

Unblock PR with BitsAI

⚠️ Warnings

🚦 41 Pipeline jobs failed

Testing the test | System Tests (dotnet, dev) / End-to-end #2 / poc 2   View in Datadog   GitHub Actions

🔧 Fix in code. This looks caused by changes in this PR. 1 failed test. Assertion Error: Unavailable providers must return an error result: {'reason': 'DEFAULT', 'value': False, 'count': 1}

🧪 1 Test failed · 🎯 related to the job error above

🎯 tests.ffe.test_dynamic_evaluation.Test_FFE_OpenFeature_Unavailable.test_openfeature_unavailable[poc] from system_tests_suite   View in Datadog
AssertionError: Unavailable providers must return an error result: {&#39;reason&#39;: &#39;DEFAULT&#39;, &#39;value&#39;: False, &#39;count&#39;: 1}
assert &#39;DEFAULT&#39; == &#39;ERROR&#39;
  - ERROR
  &#43; DEFAULT

self = &lt;tests.ffe.test_dynamic_evaluation.Test_FFE_OpenFeature_Unavailable object at 0x7fb5b4665df0&gt;

    def test_openfeature_unavailable(self) -&gt; None:
        assert self.response.status_code == 200, f&#34;Flag evaluation failed: {self.response.text}&#34;
        result = json.loads(self.response.text)
...

Testing the test | System Tests (dotnet, dev) / End-to-end #2 / uds 2   View in Datadog   GitHub Actions

🔧 Fix in code. This looks caused by changes in this PR. 1 failed test. Assertion Error: Unavailable providers must return an error result: {'reason': 'DEFAULT', 'value': False, 'count': 1}. Expected 'ERROR', got 'DEFAULT' at tests/ffe/test_dynamic_evaluation.py:99.

🧪 1 Test failed · 🎯 related to the job error above

🎯 tests.ffe.test_dynamic_evaluation.Test_FFE_OpenFeature_Unavailable.test_openfeature_unavailable[uds] from system_tests_suite   View in Datadog
AssertionError: Unavailable providers must return an error result: {&#39;reason&#39;: &#39;DEFAULT&#39;, &#39;value&#39;: False, &#39;count&#39;: 1}
assert &#39;DEFAULT&#39; == &#39;ERROR&#39;
  - ERROR
  &#43; DEFAULT

self = &lt;tests.ffe.test_dynamic_evaluation.Test_FFE_OpenFeature_Unavailable object at 0x7fcc818ca8d0&gt;

    def test_openfeature_unavailable(self) -&gt; None:
        assert self.response.status_code == 200, f&#34;Flag evaluation failed: {self.response.text}&#34;
        result = json.loads(self.response.text)
...

Testing the test | System Tests (golang, dev) / End-to-end #2 / chi 2   View in Datadog   GitHub Actions

🔧 Fix in code. This looks caused by changes in this PR. 1 failed test. Assertion Error: Unavailable providers must return an error result: {'value': False, 'count': 1}.

🧪 1 Test failed · 🎯 related to the job error above

🎯 tests.ffe.test_dynamic_evaluation.Test_FFE_OpenFeature_Unavailable.test_openfeature_unavailable[chi] from system_tests_suite   View in Datadog
AssertionError: Unavailable providers must return an error result: {&#39;value&#39;: False, &#39;count&#39;: 1}
assert None == &#39;ERROR&#39;
 &#43;  where None = &lt;built-in method get of dict object at 0x7fd8bb8d7040&gt;(&#39;reason&#39;)
 &#43;    where &lt;built-in method get of dict object at 0x7fd8bb8d7040&gt; = {&#39;count&#39;: 1, &#39;value&#39;: False}.get

self = &lt;tests.ffe.test_dynamic_evaluation.Test_FFE_OpenFeature_Unavailable object at 0x7fd8e892c170&gt;

    def test_openfeature_unavailable(self) -&gt; None:
        assert self.response.status_code == 200, f&#34;Flag evaluation failed: {self.response.text}&#34;
        result = json.loads(self.response.text)
...

View all 41 failed jobs.

📋 Copy prompt for your agent
CI on my pull request is failing. Help me find and fix the root cause of each failing job below — they were flagged as caused by changes in this PR, so focus on the diff. For each job, explain the failure and propose a fix.

Branch: leo.romanovsky/php-openfeature-all-environments

Testing the test | System Tests (dotnet, dev) / End-to-end #2 / poc 2
Commit: fcba98ec77b58a72613037d412f4ab1e33136a44
Error (code / test):
1 failed test. Assertion Error: Unavailable providers must return an error result: {'reason': 'DEFAULT', 'value': False, 'count': 1}
CI job: https://github.com/DataDog/system-tests/actions/runs/31187127729/job/92919509488

Testing the test | System Tests (dotnet, dev) / End-to-end #2 / uds 2
Commit: fcba98ec77b58a72613037d412f4ab1e33136a44
Error (code / test):
1 failed test. Assertion Error: Unavailable providers must return an error result: {'reason': 'DEFAULT', 'value': False, 'count': 1}. Expected 'ERROR', got 'DEFAULT' at tests/ffe/test_dynamic_evaluation.py:99.
CI job: https://github.com/DataDog/system-tests/actions/runs/31187127729/job/92919509743

Testing the test | System Tests (golang, dev) / End-to-end #2 / chi 2
Commit: fcba98ec77b58a72613037d412f4ab1e33136a44
Error (code / test):
1 failed test. Assertion Error: Unavailable providers must return an error result: {'value': False, 'count': 1}.
CI job: https://github.com/DataDog/system-tests/actions/runs/31187127729/job/92919513672

Plus 38 more failing jobs not shown here.

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

🔄 Datadog auto-retried 1 job - 0 passed on retry View in Datadog

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: d3e449b | Docs | Datadog PR Page | Give us feedback!

@leoromanovsky

Copy link
Copy Markdown
Contributor Author

Validated against the current PHP fix build.

  • system-tests head: 0dad5d48bec63cffdd206b724013d26f497c0c8c
  • dd-trace-php fix head: bf360ef80839f483179e37215536d1713116167c
  • CI-built ARM64 PHP package: 1.24.0+dev.bf360ef80839f483179e37215536d1713116167c
  • PHP-FPM runtime: PHP 8.2.32

Local results:

php-fpm-8.2, complete FEATURE_FLAGGING_AND_EXPERIMENTATION scenario
37 passed, 9 skipped, 2735 deselected in 153.98s

laravel11x, Test_FFE_OpenFeature_Evaluation
1 passed in 34.02s

symfony7x, Test_FFE_OpenFeature_Evaluation
1 passed in 34.07s

php-fpm-7.4, complete FEATURE_FLAGGING_AND_EXPERIMENTATION scenario
3 passed, 9 skipped, 29 xfailed, 5 xpassed, 2735 deselected in 210.78s

The current GitHub Actions run does not use this PHP build:

  • All 12 production PHP jobs use php@1.23.1. Their only failed step is the FFE scenario. These jobs need a release that contains the PHP fix.
  • The 12 PHP 8 development jobs use php@1.24.0+dev.5bf45ce4ececff3bc5c046464636ce758c839a3c. That build predates the fix. Their FFE failures should stop after [java] Update mssql-jdbc version #4071 merges and the development package refreshes. They do not need a public release.
  • All 27 development PHP jobs also fail unrelated AppSec telemetry scenarios. The 15 PHP 7 development jobs do not fail the FFE step. Their red status comes only from those AppSec failures.

Therefore, the PHP fix will unblock the FFE failures. It will not make every PHP job green because the AppSec failures are separate.

@leoromanovsky
leoromanovsky marked this pull request as ready for review July 29, 2026 16:14
@leoromanovsky
leoromanovsky requested review from a team as code owners July 29, 2026 16:14
@leoromanovsky
leoromanovsky requested review from pavlokhrebto and typotter and removed request for a team July 29, 2026 16:15

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 54a205d2c7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread tests/ffe/test_dynamic_evaluation.py Outdated
Comment thread manifests/php.yml
Comment thread tests/ffe/test_dynamic_evaluation.py
Comment thread utils/build/docker/php/common/install_ddtrace.sh Outdated

@nccatoni nccatoni left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM (for @DataDog/system-tests-core) but you should get a review from someone more familiar with the php tracer

Split supported and unavailable-provider contracts, and avoid installing the OpenFeature SDK in parametric images.

Environment: Datadog workspace
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants