diff --git a/.github/workflows/split-publish.yml b/.github/workflows/split-publish.yml index d6bdea0d..ff65833f 100644 --- a/.github/workflows/split-publish.yml +++ b/.github/workflows/split-publish.yml @@ -29,6 +29,7 @@ on: # yamllint disable-line rule:truthy - 'convention-[0-9]*' - 'data-[0-9]*' - 'facade-[0-9]*' + - 'error-handler-[0-9]*' - 'filter-[0-9]*' - 'inline-[0-9]*' - 'lifecycle-[0-9]*' diff --git a/bridge/rector/FEATURE_PARITY.md b/bridge/rector/FEATURE_PARITY.md index ffbda363..629c0c7e 100644 --- a/bridge/rector/FEATURE_PARITY.md +++ b/bridge/rector/FEATURE_PARITY.md @@ -30,6 +30,7 @@ Conversion coverage across the three directions supported by `testo/bridge-recto | **ExpectNoAssertions** (`#[\Testo\Assert\ExpectNoAssertions]` ↔ `#[\PHPUnit\Framework\Attributes\DoesNotPerformAssertions]`) | ✅ *`ExpectNoAssertionsToPhpUnitRector` (attribute rename; both sides method/function-level only — no fan-out)* | ✅ *`DoesNotPerformAssertionsToTestoRector` (attribute rename)* | ➖ | | **Mocks** (`createMock`/`createStub` + `expects`/`method`/`will*`/`with`) | ➖ | 🟡 *`CreateMockToDoubleRector` converts onto the Double bridge (`testo/bridge-double`): `createMock`/`createStub` → `Double::for`, `create{Mock,Stub}ForIntersectionOfInterfaces([A, B])` → `Double::for(A, B)`, and the configuration chain onto `expects`/`allows`/`with`/`returns`/`throws`/`resolves` — the invocation matcher moves onto the verb (`once`→`times(1)`, `exactly`→`times`, `never`→`never`, `atLeastOnce`/`atLeast`/`atMost`→`times(minimum:/maximum:)`, `any`→`allows`), the method name off `->method()` onto `expects('m')`, `withAnyParameters()` drops away, the returns (incl. `willReturnArgument($n)`→`resolves(fn (...$a) => $a[$n])`, `willReturnSelf()`→`returns()`, and legacy `will(...)` incl. `onConsecutiveCalls`/`returnArgument`/`returnSelf`), the builder chain `getMockBuilder(X)->disableOriginalConstructor()->getMock()`→`Double::for(X)`, and `with()` constraints onto `Argument::*` (`anything`→`any`, `identicalTo`→`same`, `isInstanceOf`/`isType`→`type`, `callback`→`satisfies`, `contains`→`contains`, `matchesRegularExpression`→`matches`; `equalTo($x)`→bare `$x`, `isNull`/`isTrue`/`isFalse`→`null`/`true`/`false`; `greaterThan`/`lessThan`/`…OrEqual`, `isEmpty`, `stringContains`, `stringStartsWith`/`EndsWith`, `arrayHasKey`→`satisfies(fn ($value) => …)`; `logicalNot`→`Argument::not(...)`, `logicalOr`→`Argument::any(...)`). All-or-nothing per chain: `willReturnMap`, a variable matcher, a builder step beyond `disableOriginalConstructor` (or the bare constructor-calling `getMockBuilder(X)->getMock()`), `prophesize`, and `with()` constraints with no faithful form (`logicalAnd`, `equalToWithDelta`/`equalToCanonicalizing`, case-insensitive `stringContains`) leave the statement untouched — see `MockToTestoRector` (stub) and TODO.md* | ➖ | | **Memory-leak expectations** | ⛔ *no PHPUnit equivalent* | ➖ | ➖ | +| **Error-handler declarations** (`#[ExpectErrorHandlerChange]` ↔ `#[WithoutErrorHandler]`) | ⛔ *`ExpectErrorHandlerChangeRector` stub: PHPUnit always flags a test that leaves a handler on the stack as risky and has no attribute to waive that* | ⛔ *`WithoutErrorHandlerRector` stub: Testo core installs no error handler and the `testo/error-handler` plugin has no per-test opt-out; the two attributes mean different things* | ➖ | | **Retry / Repeat** (`#[Retry]`/`#[Repeat]`) | 🟡 *`RepeatRetryRector` converts `#[\Testo\Repeat]`/`#[\Testo\Retry]` → PHPUnit `#[Repeat]`/`#[Retry]` (PHPUnit 13.3+): `maxFailures`→`failureThreshold` (+1), Testo defaults made explicit. PHPUnit's are `TARGET_METHOD` only, so a class-level Testo attribute is fanned out onto each test method (a method's own attribute overrides it, not doubled); `markFlaky` is dropped (no PHPUnit equivalent)* | 🟡 *`RepeatRetryToTestoRector` converts `#[Repeat]`/`#[Retry]` → Testo's attributes: `failureThreshold`→`maxFailures` (−1; the default 1 folds to Testo's default 0 and is omitted)* | ➖ | | **Fiber** (`#[RunInFiber]`, `Coroutine::spawn/await/concurrently`) | ⛔ *no PHPUnit/Pest equivalent — neither has a fiber/coroutine test attribute or an in-test coroutine scope* | ➖ | ➖ | | **HTML report** (`HtmlPlugin`, `--log-html`) | ⛔ *not test code — a reporter configured in `testo.php` or by a flag, with nothing in a test file to convert* | ➖ | ➖ | @@ -102,7 +103,7 @@ attributes / body statements. It bails (leaves the statement untouched) on a non a `use (...)`-capturing closure, or any unrecognised modifier — see `src/PestToTesto/TODO.md`. The remaining ⛔ rows are intentionally out of scope: a missing target feature (`arch()`, -memory-leak, PHPUnit `assertThat` constraints), the substring-vs-regex +memory-leak, the error-handler declarations, PHPUnit `assertThat` constraints), the substring-vs-regex exception-message mismatch, or Pest `uses()` (a function has no base class / traits / `$this`). Mocks moved off this list: with the Double bridge there is now a target API, so `createMock`/ `createStub` (and intersection mocks), their `expects`/`method`/`will*` chains, and `with()` diff --git a/bridge/rector/src/PhpunitToTesto/TODO.md b/bridge/rector/src/PhpunitToTesto/TODO.md index cc208fcc..3ca3e292 100644 --- a/bridge/rector/src/PhpunitToTesto/TODO.md +++ b/bridge/rector/src/PhpunitToTesto/TODO.md @@ -20,6 +20,10 @@ exist for each so the intent and blockers are discoverable in code. constraint objects (and composites/callbacks) with no Testo equivalent. - **ExpectExceptionMessageMatchesRector** — regex message matching; Testo's `withMessage()` does literal matching, not PCRE, so conversion would change meaning. +- **WithoutErrorHandlerRector** — `#[WithoutErrorHandler]` opts one test out of PHPUnit's own + error handler; Testo core installs none and the `testo/error-handler` plugin has no per-test + opt-out. `#[\Testo\ErrorHandler\ExpectErrorHandlerChange]` is not a counterpart (it declares a + deliberate handler-stack change), so the attribute is left for manual handling. ## Implemented since the first cut diff --git a/bridge/rector/src/PhpunitToTesto/WithoutErrorHandlerRector.php b/bridge/rector/src/PhpunitToTesto/WithoutErrorHandlerRector.php new file mode 100644 index 00000000..a70f8a4c --- /dev/null +++ b/bridge/rector/src/PhpunitToTesto/WithoutErrorHandlerRector.php @@ -0,0 +1,44 @@ +=8.2", + "testo/testo": "0.10.34 - 1" + }, + "autoload": { + "psr-4": { + "Testo\\ErrorHandler\\": "src/" + } + }, + "minimum-stability": "dev", + "prefer-stable": true, + "extra": { + "branch-alias": { + "dev-1.x": "1.x-dev" + } + } +} diff --git a/plugin/error-handler/src/CapturedError.php b/plugin/error-handler/src/CapturedError.php new file mode 100644 index 00000000..bbdc2ee3 --- /dev/null +++ b/plugin/error-handler/src/CapturedError.php @@ -0,0 +1,47 @@ +severity), $this->message, $this->file, $this->line); + } + + private static function label(int $severity): string + { + return match ($severity) { + \E_WARNING, \E_USER_WARNING, \E_CORE_WARNING, \E_COMPILE_WARNING => 'Warning', + \E_NOTICE, \E_USER_NOTICE => 'Notice', + \E_DEPRECATED, \E_USER_DEPRECATED => 'Deprecated', + \E_RECOVERABLE_ERROR => 'Recoverable fatal error', + default => 'Error', + }; + } +} diff --git a/plugin/error-handler/src/CapturedErrors.php b/plugin/error-handler/src/CapturedErrors.php new file mode 100644 index 00000000..68a55a9a --- /dev/null +++ b/plugin/error-handler/src/CapturedErrors.php @@ -0,0 +1,28 @@ + $errors + */ + public function __construct( + public array $errors, + ) {} + + public function isEmpty(): bool + { + return $this->errors === []; + } +} diff --git a/plugin/error-handler/src/ErrorHandlerPlugin.php b/plugin/error-handler/src/ErrorHandlerPlugin.php new file mode 100644 index 00000000..eb9d6e1b --- /dev/null +++ b/plugin/error-handler/src/ErrorHandlerPlugin.php @@ -0,0 +1,37 @@ +get(InterceptorCollector::class) + ->addInterceptor(new ErrorHandlerInterceptor($this->failOnError)); + } +} diff --git a/plugin/error-handler/src/Exception/ErrorHandlerUnchanged.php b/plugin/error-handler/src/Exception/ErrorHandlerUnchanged.php new file mode 100644 index 00000000..da56f6ad --- /dev/null +++ b/plugin/error-handler/src/Exception/ErrorHandlerUnchanged.php @@ -0,0 +1,20 @@ +run($info, $next, $scope); + + if ($result->status === Status::Passed) { + $declared = self::declaresHandlerChange($info); + if ($declared && !$scope->changed()) { + $result = $result->with(status: Status::Failed)->withFailure(new ErrorHandlerUnchanged()); + } elseif (!$declared && $scope->changed()) { + $result = $result + ->with(status: Status::Risky) + ->withMessages(new MessageLog([ + ...$result->messages->all(), + new Message(\microtime(true), self::CHANNEL, Level::Warning, $scope->removed() + ? 'Test code or tested code removed error handlers other than its own.' + : 'Test code or tested code did not remove its own error handlers.'), + ])); + } + } + + if ($scope->errors === []) { + return $result; + } + + $result = $result->withAttribute(CapturedErrors::class, new CapturedErrors($scope->errors)); + $scope->stderr === [] or $result = $result->withMessages(new MessageLog([...$result->messages->all(), ...$scope->stderr])); + + if ($this->failOnError && $result->status === Status::Passed) { + $first = $scope->errors[0]; + $result = $result + ->with(status: Status::Failed) + ->withFailure(new \ErrorException($first->message, 0, $first->severity, $first->file, $first->line)); + } + + return $result; + } + + private static function declaresHandlerChange(TestInfo $info): bool + { + if (Reflection::fetchFunctionAttributes( + $info->testDefinition->reflection, + attributeClass: ExpectErrorHandlerChange::class, + ) !== []) { + return true; + } + + $class = $info->caseInfo->definition->reflection; + + return $class !== null && Reflection::fetchClassAttributes( + $class, + attributeClass: ExpectErrorHandlerChange::class, + ) !== []; + } + + /** + * Inside a fiber the scope leaves the stack on every suspension and comes back on resumption, + * so errors fired by an interleaved sibling test never land here. + * + * @param callable(TestInfo): TestResult $next + */ + private function run(TestInfo $info, callable $next, HandlerScope $scope): TestResult + { + $scope->install(); + try { + if (\Fiber::getCurrent() === null) { + return $next($info); + } + + $fiber = new \Fiber(static fn(): TestResult => $next($info)); + $value = $fiber->start(); + while (!$fiber->isTerminated()) { + $scope->suspend(); + try { + $resume = \Fiber::suspend($value); + } catch (\Throwable $e) { + $scope->resume(); + $value = $fiber->throw($e); + continue; + } + + $scope->resume(); + $value = $fiber->resume($resume); + } + + /** @var TestResult $result */ + $result = $fiber->getReturn(); + return $result; + } finally { + $scope->release(); + } + } +} diff --git a/plugin/error-handler/src/Internal/HandlerScope.php b/plugin/error-handler/src/Internal/HandlerScope.php new file mode 100644 index 00000000..b79261ce --- /dev/null +++ b/plugin/error-handler/src/Internal/HandlerScope.php @@ -0,0 +1,203 @@ + */ + public array $errors = []; + + /** @var list What PHP would have printed for the errors nobody handled. */ + public array $stderr = []; + + private readonly \Closure $handler; + + /** @var list Stack as it was before {@see install()}, bottom first. */ + private array $before = []; + + /** @var list Handlers the test installed above ours, bottom first. */ + private array $above = []; + + private ?\Closure $previous = null; + private bool $removed = false; + private bool $left = false; + + public function __construct() + { + $this->handler = function (int $severity, string $message, string $file, int $line): bool { + // Forwarded first: a previous handler that throws turns the error into control flow of + // the test, and then there is nothing to capture. + $handled = $this->previous !== null && (bool) ($this->previous)($severity, $message, $file, $line); + + // Left to PHP, E_USER_ERROR ends the script; swallowing it would run the test past a fatal. + $severity === \E_USER_ERROR and throw new \ErrorException($message, 0, $severity, $file, $line); + + if ((\error_reporting() & $severity) === 0) { + return true; + } + + $error = new CapturedError($severity, $message, $file, $line, $handled); + $this->errors[] = $error; + $handled or $this->stderr[] = new Message( + \microtime(true), + Messenger::CHANNEL_STDERR, + self::level($severity), + (string) $error, + ['severity' => $severity, 'file' => $file, 'line' => $line], + ); + + // Always taken: an unhandled error is reported through the stderr channel instead of PHP's own printing. + return true; + }; + } + + public function install(): void + { + $this->before = self::snapshot(); + $previous = \set_error_handler($this->handler); + $this->previous = $previous === null ? null : $previous(...); + } + + /** + * Takes our handler and everything above it off the stack for the time the test is suspended. + */ + public function suspend(): void + { + $stack = self::snapshot(); + $position = self::position($stack, $this->handler); + if ($position === null) { + $this->removed = true; + $this->above = []; + return; + } + + $this->above = \array_slice($stack, $position + 1); + self::pop(\count($stack) - $position); + } + + public function resume(): void + { + if ($this->removed) { + return; + } + + \set_error_handler($this->handler); + self::push($this->above); + } + + /** + * Removes the test's slice and puts the stack back as it was before {@see install()}. + */ + public function release(): void + { + $stack = self::snapshot(); + $position = self::position($stack, $this->handler); + + if ($position === null) { + $this->removed = true; + self::pop(\count($stack)); + self::push($this->before); + return; + } + + $this->left = \count($stack) - $position > 1; + self::pop(\count($stack) - $position); + } + + /** + * Whether the test left the stack different from how it found it: a handler of its own still + * installed, or ours gone. + */ + public function changed(): bool + { + return $this->removed || $this->left; + } + + public function removed(): bool + { + return $this->removed; + } + + /** + * The whole stack, bottom first, put back untouched. + * + * @return list + */ + private static function snapshot(): array + { + $stack = []; + while (true) { + $top = \set_error_handler(static fn(): bool => false); + \restore_error_handler(); + if ($top === null) { + break; + } + + $stack[] = $top; + \restore_error_handler(); + } + + $stack = \array_reverse($stack); + self::push($stack); + + return $stack; + } + + /** + * @param list $stack + */ + private static function position(array $stack, \Closure $handler): ?int + { + foreach ($stack as $i => $entry) { + if ($entry === $handler) { + return $i; + } + } + + return null; + } + + private static function level(int $severity): Level + { + return match ($severity) { + \E_NOTICE, \E_USER_NOTICE, \E_DEPRECATED, \E_USER_DEPRECATED => Level::Notice, + \E_RECOVERABLE_ERROR => Level::Error, + default => Level::Warning, + }; + } + + private static function pop(int $count): void + { + for ($i = 0; $i < $count; $i++) { + \restore_error_handler(); + } + } + + /** + * @param list $handlers Bottom first. + */ + private static function push(array $handlers): void + { + foreach ($handlers as $handler) { + \set_error_handler($handler); + } + } +} diff --git a/plugin/error-handler/tests/Stub/HandlerChange.php b/plugin/error-handler/tests/Stub/HandlerChange.php new file mode 100644 index 00000000..c960991a --- /dev/null +++ b/plugin/error-handler/tests/Stub/HandlerChange.php @@ -0,0 +1,18 @@ + new TestResult(info: $info, status: Status::Passed); + + $result = $interceptor->runTest($info, $next); + + Assert::same($result->status, Status::Passed); + Assert::null($result->getAttribute(CapturedErrors::class)); + } + + public function capturedErrorIsStoredAsAttribute(): void + { + $interceptor = new ErrorHandlerInterceptor(); + $info = self::createTestInfo(); + $next = static function (TestInfo $info): TestResult { + \trigger_error('test warning', \E_USER_WARNING); + return new TestResult(info: $info, status: Status::Passed); + }; + + $result = $interceptor->runTest($info, $next); + + Assert::same($result->status, Status::Passed); + $errors = $result->getAttribute(CapturedErrors::class); + Assert::instanceOf($errors, CapturedErrors::class); + Assert::false($errors->isEmpty()); + Assert::same(\count($errors->errors), 1); + Assert::same($errors->errors[0]->message, 'test warning'); + Assert::same($errors->errors[0]->severity, \E_USER_WARNING); + } + + public function multipleErrorsAreAllCaptured(): void + { + $interceptor = new ErrorHandlerInterceptor(); + $info = self::createTestInfo(); + $next = static function (TestInfo $info): TestResult { + \trigger_error('first', \E_USER_NOTICE); + \trigger_error('second', \E_USER_WARNING); + \trigger_error('third', \E_USER_DEPRECATED); + return new TestResult(info: $info, status: Status::Passed); + }; + + $result = $interceptor->runTest($info, $next); + + $errors = $result->getAttribute(CapturedErrors::class); + Assert::instanceOf($errors, CapturedErrors::class); + Assert::same(\count($errors->errors), 3); + Assert::same($errors->errors[0]->message, 'first'); + Assert::same($errors->errors[1]->message, 'second'); + Assert::same($errors->errors[2]->message, 'third'); + } + + public function collectModePreservesPassingStatus(): void + { + $interceptor = new ErrorHandlerInterceptor(failOnError: false); + $info = self::createTestInfo(); + $next = static function (TestInfo $info): TestResult { + \trigger_error('deprecated usage', \E_USER_DEPRECATED); + return new TestResult(info: $info, status: Status::Passed); + }; + + $result = $interceptor->runTest($info, $next); + + Assert::same($result->status, Status::Passed); + Assert::notNull($result->getAttribute(CapturedErrors::class)); + } + + public function failModeUpgradesPassingTestToFailed(): void + { + $interceptor = new ErrorHandlerInterceptor(failOnError: true); + $info = self::createTestInfo(); + $next = static function (TestInfo $info): TestResult { + \trigger_error('user warning', \E_USER_WARNING); + return new TestResult(info: $info, status: Status::Passed); + }; + + $result = $interceptor->runTest($info, $next); + + Assert::same($result->status, Status::Failed); + Assert::instanceOf($result->failure, \ErrorException::class); + Assert::same($result->failure->getMessage(), 'user warning'); + Assert::same($result->failure->getSeverity(), \E_USER_WARNING); + } + + public function failModeUsesFirstErrorAsFailure(): void + { + $interceptor = new ErrorHandlerInterceptor(failOnError: true); + $info = self::createTestInfo(); + $next = static function (TestInfo $info): TestResult { + \trigger_error('first error', \E_USER_WARNING); + \trigger_error('second error', \E_USER_NOTICE); + return new TestResult(info: $info, status: Status::Passed); + }; + + $result = $interceptor->runTest($info, $next); + + Assert::same($result->status, Status::Failed); + Assert::instanceOf($result->failure, \ErrorException::class); + Assert::same($result->failure->getMessage(), 'first error'); + } + + public function failModeDoesNotOverrideAlreadyFailedTest(): void + { + $interceptor = new ErrorHandlerInterceptor(failOnError: true); + $info = self::createTestInfo(); + $originalFailure = new \RuntimeException('assertion failure'); + $next = static function (TestInfo $info) use ($originalFailure): TestResult { + \trigger_error('also an error', \E_USER_WARNING); + return new TestResult(info: $info, status: Status::Failed, failure: $originalFailure); + }; + + $result = $interceptor->runTest($info, $next); + + Assert::same($result->status, Status::Failed); + Assert::same($result->failure, $originalFailure); + } + + public function failModeDoesNotOverrideErrorStatus(): void + { + $interceptor = new ErrorHandlerInterceptor(failOnError: true); + $info = self::createTestInfo(); + $originalFailure = new \RuntimeException('unexpected throw'); + $next = static function (TestInfo $info) use ($originalFailure): TestResult { + \trigger_error('also triggered', \E_USER_WARNING); + return new TestResult(info: $info, status: Status::Error, failure: $originalFailure); + }; + + $result = $interceptor->runTest($info, $next); + + Assert::same($result->status, Status::Error); + Assert::same($result->failure, $originalFailure); + } + + public function handlerIsRestoredAfterTestCompletes(): void + { + $interceptor = new ErrorHandlerInterceptor(); + $info = self::createTestInfo(); + $next = static fn(TestInfo $info): TestResult => new TestResult(info: $info, status: Status::Passed); + + $count = 0; + \set_error_handler(static function () use (&$count): bool { + $count++; + return true; + }); + + try { + $interceptor->runTest($info, $next); + \trigger_error('after test', \E_USER_NOTICE); + } finally { + \restore_error_handler(); + } + + Assert::same($count, 1); + } + + public function handlerIsRestoredEvenWhenTestThrows(): void + { + $interceptor = new ErrorHandlerInterceptor(); + $info = self::createTestInfo(); + $next = static fn(): TestResult => throw new \RuntimeException('unexpected throw'); + + $count = 0; + \set_error_handler(static function () use (&$count): bool { + $count++; + return true; + }); + + try { + try { + $interceptor->runTest($info, $next); + } catch (\RuntimeException) { + } + \trigger_error('after throw', \E_USER_NOTICE); + } finally { + \restore_error_handler(); + } + + Assert::same($count, 1); + } + + public function restoresTheOuterHandlerWhileSuspendedAndReinstallsItsOwnOnResume(): void + { + $interceptor = new ErrorHandlerInterceptor(); + $info = self::createTestInfo(); + + $outer = []; + \set_error_handler(static function (int $severity, string $message) use (&$outer): bool { + $outer[] = $message; + return true; + }); + + try { + $next = static function (TestInfo $info): TestResult { + \trigger_error('before suspend', \E_USER_NOTICE); + \Fiber::suspend(); + \trigger_error('after resume', \E_USER_NOTICE); + return new TestResult(info: $info, status: Status::Passed); + }; + + $fiber = new \Fiber(static fn(): TestResult => $interceptor->runTest($info, $next)); + $fiber->start(); + + // Fired while the test is suspended: reaches the outer handler directly, not via the test. + \trigger_error('fired while suspended', \E_USER_NOTICE); + Assert::same($outer, ['before suspend', 'fired while suspended']); + + $fiber->resume(); + Assert::true($fiber->isTerminated()); + + $result = $fiber->getReturn(); + $errors = $result->getAttribute(CapturedErrors::class); + Assert::instanceOf($errors, CapturedErrors::class); + Assert::same(\count($errors->errors), 2); + Assert::same($errors->errors[0]->message, 'before suspend'); + Assert::same($errors->errors[1]->message, 'after resume'); + Assert::same($outer, ['before suspend', 'fired while suspended', 'after resume']); + } finally { + \restore_error_handler(); + } + } + + public function silencedErrorIsNotCaptured(): void + { + $interceptor = new ErrorHandlerInterceptor(failOnError: true); + $info = self::createTestInfo(); + $next = static function (TestInfo $info): TestResult { + @\trigger_error('silenced', \E_USER_WARNING); + return new TestResult(info: $info, status: Status::Passed); + }; + + $result = $interceptor->runTest($info, $next); + + Assert::same($result->status, Status::Passed); + Assert::null($result->getAttribute(CapturedErrors::class)); + } + + public function errorOutsideErrorReportingIsNotCaptured(): void + { + $interceptor = new ErrorHandlerInterceptor(failOnError: true); + $info = self::createTestInfo(); + $next = static function (TestInfo $info): TestResult { + \trigger_error('deprecated', \E_USER_DEPRECATED); + return new TestResult(info: $info, status: Status::Passed); + }; + + $level = \error_reporting(\E_ALL & ~\E_USER_DEPRECATED); + try { + $result = $interceptor->runTest($info, $next); + } finally { + \error_reporting($level); + } + + Assert::same($result->status, Status::Passed); + Assert::null($result->getAttribute(CapturedErrors::class)); + } + + public function handlerLeftByTestDoesNotShadowOuterHandler(): void + { + $interceptor = new ErrorHandlerInterceptor(); + $info = self::createTestInfo(); + $next = static function (TestInfo $info): TestResult { + \set_error_handler(static fn(): bool => true); + return new TestResult(info: $info, status: Status::Passed); + }; + + $count = 0; + \set_error_handler(static function () use (&$count): bool { + $count++; + return true; + }); + + try { + $interceptor->runTest($info, $next); + \trigger_error('after test', \E_USER_NOTICE); + } finally { + \restore_error_handler(); + } + + Assert::same($count, 1); + } + + public function handlerLeftByTestDoesNotCaptureSiblingErrorsWhileSuspended(): void + { + $interceptor = new ErrorHandlerInterceptor(); + $info = self::createTestInfo(); + $next = static function (TestInfo $info): TestResult { + \set_error_handler(static fn(): bool => true); + \Fiber::suspend(); + return new TestResult(info: $info, status: Status::Passed); + }; + + $outerCount = 0; + \set_error_handler(static function () use (&$outerCount): bool { + $outerCount++; + return true; + }); + + try { + $fiber = new \Fiber(static fn(): TestResult => $interceptor->runTest($info, $next)); + $fiber->start(); + + \trigger_error('fired while suspended', \E_USER_NOTICE); + Assert::same($outerCount, 1); + + $fiber->resume(); + Assert::null($fiber->getReturn()->getAttribute(CapturedErrors::class)); + } finally { + \restore_error_handler(); + } + } + + public function previousHandlerStillReceivesCapturedErrors(): void + { + $interceptor = new ErrorHandlerInterceptor(); + $info = self::createTestInfo(); + $next = static function (TestInfo $info): TestResult { + \trigger_error('forwarded', \E_USER_WARNING); + return new TestResult(info: $info, status: Status::Passed); + }; + + $seen = []; + \set_error_handler(static function (int $severity, string $message) use (&$seen): bool { + $seen[] = $message; + return true; + }); + + try { + $result = $interceptor->runTest($info, $next); + } finally { + \restore_error_handler(); + } + + Assert::same($seen, ['forwarded']); + $errors = $result->getAttribute(CapturedErrors::class); + Assert::instanceOf($errors, CapturedErrors::class); + Assert::same($errors->errors[0]->message, 'forwarded'); + } + + public function previousHandlerTurningErrorIntoExceptionLeavesNothingCaptured(): void + { + $interceptor = new ErrorHandlerInterceptor(failOnError: true); + $info = self::createTestInfo(); + $next = static function (TestInfo $info): TestResult { + try { + \trigger_error('becomes exception', \E_USER_WARNING); + } catch (\ErrorException) { + return new TestResult(info: $info, status: Status::Passed); + } + + return new TestResult(info: $info, status: Status::Failed, failure: new \RuntimeException('not thrown')); + }; + + \set_error_handler(static fn(int $severity, string $message): bool => throw new \ErrorException($message, 0, $severity)); + + try { + $result = $interceptor->runTest($info, $next); + } finally { + \restore_error_handler(); + } + + Assert::same($result->status, Status::Passed); + Assert::null($result->getAttribute(CapturedErrors::class)); + } + + public function previousHandlerObservesTheRealErrorReportingLevel(): void + { + $interceptor = new ErrorHandlerInterceptor(); + $info = self::createTestInfo(); + $next = static function (TestInfo $info): TestResult { + \trigger_error('probe', \E_USER_WARNING); + return new TestResult(info: $info, status: Status::Passed); + }; + + $observed = null; + \set_error_handler(static function () use (&$observed): bool { + $observed = \error_reporting(); + return true; + }); + + $level = \error_reporting(\E_ALL & ~\E_NOTICE); + try { + $interceptor->runTest($info, $next); + } finally { + \error_reporting($level); + \restore_error_handler(); + } + + Assert::same($observed, \E_ALL & ~\E_NOTICE); + } + + public function unhandledErrorIsWrittenToStderrChannel(): void + { + $interceptor = new ErrorHandlerInterceptor(); + $info = self::createTestInfo(); + $next = static function (TestInfo $info): TestResult { + \trigger_error('to stderr', \E_USER_WARNING); + return new TestResult(info: $info, status: Status::Passed); + }; + + $result = $interceptor->runTest($info, $next); + + $errors = $result->getAttribute(CapturedErrors::class); + Assert::instanceOf($errors, CapturedErrors::class); + Assert::false($errors->errors[0]->handled); + + $stderr = $result->messages->channel(Messenger::CHANNEL_STDERR); + Assert::same(\count($stderr), 1); + Assert::same($stderr[0]->level, Level::Warning); + Assert::same($stderr[0]->content, \sprintf('Warning: to stderr in %s on line %d', __FILE__, $errors->errors[0]->line)); + Assert::same($stderr[0]->context['severity'], \E_USER_WARNING); + } + + public function errorDeclinedByPreviousHandlerIsWrittenToStderr(): void + { + $interceptor = new ErrorHandlerInterceptor(); + $info = self::createTestInfo(); + $next = static function (TestInfo $info): TestResult { + \trigger_error('declined', \E_USER_NOTICE); + return new TestResult(info: $info, status: Status::Passed); + }; + + \set_error_handler(static fn(): bool => false); + try { + $result = $interceptor->runTest($info, $next); + } finally { + \restore_error_handler(); + } + + Assert::false($result->getAttribute(CapturedErrors::class)->errors[0]->handled); + Assert::same(\count($result->messages->channel(Messenger::CHANNEL_STDERR)), 1); + } + + public function errorHandledByPreviousHandlerIsCapturedButKeptOutOfStderr(): void + { + $interceptor = new ErrorHandlerInterceptor(); + $info = self::createTestInfo(); + $next = static function (TestInfo $info): TestResult { + \trigger_error('handled upstream', \E_USER_WARNING); + return new TestResult(info: $info, status: Status::Passed); + }; + + \set_error_handler(static fn(): bool => true); + try { + $result = $interceptor->runTest($info, $next); + } finally { + \restore_error_handler(); + } + + $errors = $result->getAttribute(CapturedErrors::class); + Assert::instanceOf($errors, CapturedErrors::class); + Assert::true($errors->errors[0]->handled); + Assert::same($result->messages->channel(Messenger::CHANNEL_STDERR), []); + } + + public function failModeFailsOnErrorHandledUpstreamToo(): void + { + $interceptor = new ErrorHandlerInterceptor(failOnError: true); + $info = self::createTestInfo(); + $next = static function (TestInfo $info): TestResult { + \trigger_error('handled upstream', \E_USER_WARNING); + return new TestResult(info: $info, status: Status::Passed); + }; + + \set_error_handler(static fn(): bool => true); + try { + $result = $interceptor->runTest($info, $next); + } finally { + \restore_error_handler(); + } + + Assert::same($result->status, Status::Failed); + } + + public function unhandledErrorIsNotPrintedByPhp(): void + { + $interceptor = new ErrorHandlerInterceptor(); + $info = self::createTestInfo(); + $printed = null; + $next = static function (TestInfo $info) use (&$printed): TestResult { + \ob_start(); + \trigger_error('would be printed', \E_USER_NOTICE); + $printed = \ob_get_clean(); + return new TestResult(info: $info, status: Status::Passed); + }; + + $display = \ini_set('display_errors', '1'); + try { + $interceptor->runTest($info, $next); + } finally { + $display === false or \ini_set('display_errors', $display); + } + + Assert::same($printed, ''); + } + + public function userErrorBecomesAnException(): void + { + $interceptor = new ErrorHandlerInterceptor(); + $info = self::createTestInfo(); + $next = static function (TestInfo $info): TestResult { + \trigger_error('fatal by contract', \E_USER_ERROR); + return new TestResult(info: $info, status: Status::Passed); + }; + + try { + $interceptor->runTest($info, $next); + Assert::fail('E_USER_ERROR must not be swallowed'); + } catch (\ErrorException $e) { + Assert::same($e->getMessage(), 'fatal by contract'); + Assert::same($e->getSeverity(), \E_USER_ERROR); + } + } + + public function handlerLeftByTestMarksPassingTestRisky(): void + { + $interceptor = new ErrorHandlerInterceptor(); + $info = self::createTestInfo(); + $next = static function (TestInfo $info): TestResult { + \set_error_handler(static fn(): bool => true); + return new TestResult(info: $info, status: Status::Passed); + }; + + $result = $interceptor->runTest($info, $next); + + Assert::same($result->status, Status::Risky); + Assert::false($result->messages->isEmpty()); + } + + public function handlerLeftByTestDoesNotOverrideFailedStatus(): void + { + $interceptor = new ErrorHandlerInterceptor(); + $info = self::createTestInfo(); + $failure = new \RuntimeException('assertion failure'); + $next = static function (TestInfo $info) use ($failure): TestResult { + \set_error_handler(static fn(): bool => true); + return new TestResult(info: $info, status: Status::Failed, failure: $failure); + }; + + $result = $interceptor->runTest($info, $next); + + Assert::same($result->status, Status::Failed); + Assert::same($result->failure, $failure); + } + + public function handlerRemovedByTestMarksPassingTestRiskyAndKeepsOuterHandler(): void + { + $interceptor = new ErrorHandlerInterceptor(); + $info = self::createTestInfo(); + $next = static function (TestInfo $info): TestResult { + \restore_error_handler(); + return new TestResult(info: $info, status: Status::Passed); + }; + + $count = 0; + \set_error_handler(static function () use (&$count): bool { + $count++; + return true; + }); + + try { + $result = $interceptor->runTest($info, $next); + \trigger_error('after test', \E_USER_NOTICE); + } finally { + \restore_error_handler(); + } + + Assert::same($result->status, Status::Risky); + Assert::same($count, 1); + } + + public function declaredHandlerChangeKeepsPassedStatus(): void + { + $interceptor = new ErrorHandlerInterceptor(); + $info = self::createTestInfo(new \ReflectionMethod(HandlerChange::class, 'declared')); + $next = static function (TestInfo $info): TestResult { + \set_error_handler(static fn(): bool => true); + return new TestResult(info: $info, status: Status::Passed); + }; + + $count = 0; + \set_error_handler(static function () use (&$count): bool { + $count++; + return true; + }); + + try { + $result = $interceptor->runTest($info, $next); + \trigger_error('after test', \E_USER_NOTICE); + } finally { + \restore_error_handler(); + } + + Assert::same($result->status, Status::Passed); + Assert::same($count, 1); + } + + public function declaredHandlerChangeOnClassAppliesToItsTests(): void + { + $interceptor = new ErrorHandlerInterceptor(); + $info = self::createTestInfo(new \ReflectionMethod(HandlerChangeCase::class, 'inherited')); + $next = static function (TestInfo $info): TestResult { + \set_error_handler(static fn(): bool => true); + return new TestResult(info: $info, status: Status::Passed); + }; + + $result = $interceptor->runTest($info, $next); + + Assert::same($result->status, Status::Passed); + } + + public function declaredHandlerChangeThatDoesNotHappenFails(): void + { + $interceptor = new ErrorHandlerInterceptor(); + $info = self::createTestInfo(new \ReflectionMethod(HandlerChange::class, 'declared')); + $next = static fn(TestInfo $info): TestResult => new TestResult(info: $info, status: Status::Passed); + + $result = $interceptor->runTest($info, $next); + + Assert::same($result->status, Status::Failed); + Assert::instanceOf($result->failure, ErrorHandlerUnchanged::class); + } + + public function undeclaredStubMethodIsHeldToThePlainContract(): void + { + $interceptor = new ErrorHandlerInterceptor(); + $info = self::createTestInfo(new \ReflectionMethod(HandlerChange::class, 'undeclared')); + $next = static function (TestInfo $info): TestResult { + \set_error_handler(static fn(): bool => true); + return new TestResult(info: $info, status: Status::Passed); + }; + + $result = $interceptor->runTest($info, $next); + + Assert::same($result->status, Status::Risky); + } + + public function handlerInstalledByTestIsBackAfterResume(): void + { + $interceptor = new ErrorHandlerInterceptor(); + $info = self::createTestInfo(new \ReflectionMethod(HandlerChange::class, 'declared')); + + $ownCount = 0; + $next = static function (TestInfo $info) use (&$ownCount): TestResult { + \set_error_handler(static function () use (&$ownCount): bool { + $ownCount++; + return true; + }); + \Fiber::suspend(); + \trigger_error('after resume', \E_USER_NOTICE); + return new TestResult(info: $info, status: Status::Passed); + }; + + $outerCount = 0; + \set_error_handler(static function () use (&$outerCount): bool { + $outerCount++; + return true; + }); + + try { + $fiber = new \Fiber(static fn(): TestResult => $interceptor->runTest($info, $next)); + $fiber->start(); + + \trigger_error('fired while suspended', \E_USER_NOTICE); + Assert::same($outerCount, 1); + Assert::same($ownCount, 0); + + $fiber->resume(); + Assert::same($ownCount, 1); + Assert::same($fiber->getReturn()->status, Status::Passed); + Assert::null($fiber->getReturn()->getAttribute(CapturedErrors::class)); + + \trigger_error('after test', \E_USER_NOTICE); + Assert::same($outerCount, 2); + } finally { + \restore_error_handler(); + } + } + + private static function createTestInfo(?\ReflectionMethod $reflection = null): TestInfo + { + $reflection ??= new \ReflectionMethod(self::class, 'createTestInfo'); + $caseDefinition = new CaseDefinition( + name: 'TestCase', + type: 'test', + file: Path::create(__FILE__), + reflection: $reflection->getDeclaringClass(), + ); + $caseInfo = new CaseInfo(definition: $caseDefinition, suiteIdentity: new SuiteIdentity('ErrorHandler/Unit')); + $testDefinition = new TestDefinition(reflection: $reflection); + + return new TestInfo( + name: 'testMethod', + caseInfo: $caseInfo, + testDefinition: $testDefinition, + ); + } +} diff --git a/plugin/error-handler/tests/suites.php b/plugin/error-handler/tests/suites.php new file mode 100644 index 00000000..cf7146f9 --- /dev/null +++ b/plugin/error-handler/tests/suites.php @@ -0,0 +1,15 @@ +register(); + + Assert::true(ErrorBootstrap::isRegistered()); +} +``` + +Allowed on a method, a function, or a class (then it covers every test of the class). The +declaration is a two-way contract: a marked test that leaves the stack unchanged is +`Status::Failed` with `Testo\ErrorHandler\Exception\ErrorHandlerUnchanged`. + +A test that installs a handler and removes it again before returning needs no attribute. + +## Fibers + +The plugin's handler and anything the test installed above it leave the stack whenever the test +fiber suspends and come back when it resumes. A test under `#[RunInFiber]` or `#[RunInRevolt]` +sees its own handler after every `await`, and a sibling running in the gap never has it. + +## Pitfalls + +- Without the plugin a warning is not a failure and appears in no report. Add the plugin before + writing a test that expects a warning to matter. +- Turning a warning into an exception is the previous handler's job, not the plugin's. Check + `CapturedErrors` or use `failOnError`; do not `Expect::exception(\ErrorException::class)` unless a + bootstrap handler throws it. +- `trigger_error(..., E_USER_ERROR)` is deprecated since PHP 8.4 and always throws here; do not use + it to "fail from inside" a test, throw or `Assert::fail()` instead. +- `#[ExpectErrorHandlerChange]` is for tests *about* error handlers. A test that merely triggers + errors never needs it. diff --git a/skills/testo-write-tests/SKILL.md b/skills/testo-write-tests/SKILL.md index 51f8bc19..33876ab8 100644 --- a/skills/testo-write-tests/SKILL.md +++ b/skills/testo-write-tests/SKILL.md @@ -8,15 +8,6 @@ description: 'Write or modify tests in a project that uses the Testo PHP testing The attribute set, assertion facade, exception expectations, and lifecycle hooks are Testo's own. Write them the Testo way described below — don't transliterate idioms from other test frameworks. -## Before you write code - -Fetch the canonical API surface: - -- `https://php-testo.github.io/llms.txt` — concise index. Always start here. -- `https://php-testo.github.io/llms-full.txt` — escalate when `llms.txt` doesn't answer the question. - -If the project ships an `AGENTS.md`, honour it. - ## Canonical shape of a test class ```php diff --git a/testo.php b/testo.php index 859dc758..48c46828 100644 --- a/testo.php +++ b/testo.php @@ -26,6 +26,7 @@ 'plugin/codecov/tests', 'plugin/convention/tests', 'plugin/data/tests', + 'plugin/error-handler/tests', 'plugin/facade/tests', 'plugin/filter/tests', 'plugin/inline/tests', @@ -66,6 +67,7 @@ require 'plugin/codecov/tests/suites.php', require 'plugin/convention/tests/suites.php', require 'plugin/data/tests/suites.php', + require 'plugin/error-handler/tests/suites.php', require 'plugin/facade/tests/suites.php', require 'plugin/filter/tests/suites.php', require 'plugin/inline/tests/suites.php', diff --git a/tools/phpunit/phpunit.xml b/tools/phpunit/phpunit.xml index 4b56a7fd..904c0ef4 100644 --- a/tools/phpunit/phpunit.xml +++ b/tools/phpunit/phpunit.xml @@ -42,6 +42,11 @@ ../../tests/PhpUnit/*/Stub ../../tests/PhpUnit/*/*/Stub ../../tests/PhpUnit/*/*/*/Stub + + ../../tests/PhpUnit/ErrorHandler