EBS: check Twitch PubSub credentials at startup; app: warn when Twitch refuses them

extension/README.md

@@ -203,7 +203,7 @@ and is not repeated here.
 ### 0. Before touching the console
 
 - [ ] The EBS is deployed and reachable at `https://ednexus.signal-and-thread.com` over valid TLS
-      (`GET /healthz` returns `{"status":"ok"}`), and the same origin is what the desktop app's
+      (`GET /healthz` returns `status: "ok"`, plus `pubsub: "unknown"`, `"accepted"` or `"rejected"`: a `rejected` means Twitch is refusing the EBS's credentials and viewers get no live updates, see the EBS README's credential self-check), and the same origin is what the desktop app's
       `AppSettings.Twitch.EbsBaseUrl` defaults to.
 - [ ] The desktop app build that publishes a **heartbeat** has shipped (see
       [Publishing contract](#publishing-contract-with-the-desktop-app)). Without it, an idle docked

src/EDNexus.App/ViewModels/MainWindowViewModel.cs

@@ -265,6 +265,7 @@ private EngineHost BuildHost()
             {
                 if (result.IsSuccess) Trace.TraceInformation("Twitch: stream card published.");
                 else Trace.TraceWarning($"Twitch: stream card publish failed ({result.Status}) — {result.Error}");
+                UpdateTwitchWarning(result);
                 TwitchPublishCompleted?.Invoke(result);
             };
             twitchCard.ReauthRequired += () =>
@@ -374,6 +375,24 @@ private void UpdateJournalStatus()
     private void OnDiscordSettingsChanged(DiscordSettings settings) => _host.ApplyDiscordSettings(settings);
 
     [ObservableProperty] private string _journalStatus = "";
+
+    /// <summary>
+    /// A warning, shown on the main window, that the stream card is failing for a reason the commander
+    /// cannot fix (Twitch refusing the backend's credentials). Empty when there is nothing to say. The
+    /// Settings window's status line is only seen with Settings open; this is seen while playing.
+    /// </summary>
+    [ObservableProperty, NotifyPropertyChangedFor(nameof(HasTwitchWarning))] private string _twitchWarning = "";
+
+    public bool HasTwitchWarning => TwitchWarning.Length > 0;
+
+    /// <summary>Raises or clears <see cref="TwitchWarning"/> from a publish result; runs on the UI thread, since results arrive on the publisher's.</summary>
+    private void UpdateTwitchWarning(EDNexus.Core.Twitch.StreamStatePublishResult result)
+    {
+        if (EDNexus.Core.Twitch.StreamCardWarning.For(result) is { } warning)
+            Dispatcher.UIThread.Post(() => TwitchWarning = warning);
+        else if (EDNexus.Core.Twitch.StreamCardWarning.Clears(result))
+            Dispatcher.UIThread.Post(() => TwitchWarning = "");
+    }
     [ObservableProperty] private string _privacyStatus = "";
     [ObservableProperty] private string _commanderName = "—";
     [ObservableProperty] private string _balance = "0 cr";

src/EDNexus.App/Views/MainWindow.axaml

@@ -217,6 +217,14 @@
       </Grid>
     </Border>
 
+    <!-- Twitch warning bar: the stream card is failing for a reason the commander cannot fix (Twitch is
+         refusing the backend's credentials). Seen while playing, unlike the Settings status line. -->
+    <Border DockPanel.Dock="Bottom" Background="{StaticResource AmberChipBg}" BorderBrush="{StaticResource CardBorder}"
+            BorderThickness="0,0,0,1" Padding="20,8" IsVisible="{Binding HasTwitchWarning}">
+      <TextBlock Text="{Binding TwitchWarning, StringFormat='⚠ {0}'}" Foreground="{StaticResource AmberBrush}"
+                 TextWrapping="Wrap" />
+    </Border>
+
     <!-- Footer / status bar -->
     <Border DockPanel.Dock="Bottom" Background="{StaticResource Surface100}" BorderBrush="{StaticResource CardBorder}"
             BorderThickness="0,1,0,0" Padding="20,8">

src/EDNexus.App/Views/SettingsWindow.axaml.cs

@@ -623,6 +623,8 @@ private void OnTwitchPublishCompleted(StreamStatePublishResult result) =>
                 StreamStatePublishStatus.Unauthorized => "The backend rejected this machine's credential — sign in again.",
                 StreamStatePublishStatus.TooLarge => "The last card was too large for Twitch — try hiding a section.",
                 StreamStatePublishStatus.RateLimited => "Publishing is being rate-limited; updates are slowing down.",
+                StreamStatePublishStatus.BackendRejectedByTwitch =>
+                    "Twitch is rejecting the backend's credentials, so viewers are not getting live updates. It is a backend problem, not your sign-in.",
                 StreamStatePublishStatus.Cleared => "Card taken off the air.",
                 StreamStatePublishStatus.ClearedNotDelivered =>
                     "Card taken off the air; viewers already watching will be told on the next retry.",

src/EDNexus.Core/Twitch/StreamCardWarning.cs

@@ -0,0 +1,28 @@
+namespace EDNexus.Core.Twitch;
+
+/// <summary>
+/// The warning the app shows when publishing the stream card is failing for a reason the commander
+/// cannot fix from their side. Kept apart from the UI so the wording and the rule for when it appears
+/// and clears are tested.
+/// </summary>
+public static class StreamCardWarning
+{
+    /// <summary>
+    /// What to show after a publish, or <c>null</c> for nothing. Only a <see cref="StreamStatePublishStatus.BackendRejectedByTwitch"/>
+    /// raises one: a plain failure (the network dropped, a rate limit) already has its own status line
+    /// and clears itself on the next success, whereas a credential the backend operator must fix would
+    /// otherwise go unnoticed, since the card keeps looking fine on the commander's own machine.
+    /// </summary>
+    public static string? For(StreamStatePublishResult result) =>
+        result.Status == StreamStatePublishStatus.BackendRejectedByTwitch
+            ? "Twitch is rejecting the EDNexus backend's credentials, so viewers are not getting live updates. " +
+              "This is a problem with the backend's configuration, not with your sign-in; it keeps retrying and " +
+              "the card will start updating again as soon as it is fixed."
+            : null;
+
+    /// <summary>
+    /// True when this result shows the problem is over: the card reached viewers (or was taken down
+    /// cleanly). A failure of some other kind proves nothing either way, so it leaves a shown warning alone.
+    /// </summary>
+    public static bool Clears(StreamStatePublishResult result) => result.IsSuccess;
+}

src/EDNexus.Core/Twitch/StreamStateApiClient.cs

@@ -34,6 +34,14 @@ public enum StreamStatePublishStatus
     /// already watching. Worth retrying for their sake; nothing is left public to anyone new.
     /// </summary>
     ClearedNotDelivered,
+
+    /// <summary>
+    /// Twitch is refusing the <i>backend's</i> own PubSub credentials (<c>503</c> with the
+    /// <see cref="StreamStateApiClient.PubSubRejectedHeader"/> header). The card is stored, so new
+    /// viewers still get it, but nothing reaches viewers already watching. It is a server
+    /// misconfiguration: not the commander's sign-in (so no re-login) and not worth retrying hard.
+    /// </summary>
+    BackendRejectedByTwitch,
 }
 
 /// <param name="Status">Whether the snapshot reached viewers, and if not, why not.</param>
@@ -103,6 +111,12 @@ public StreamStateApiClient(HttpClient? http = null)
     /// </summary>
     public const string SnapshotRemovedHeader = "X-EDNexus-Snapshot-Removed";
 
+    /// <summary>
+    /// Header the EBS sets when Twitch is refusing the EBS's own PubSub credentials. Mirrors
+    /// <c>EDNexus.Ebs.Services.TwitchCredentialStatus.RejectedHeader</c>.
+    /// </summary>
+    public const string PubSubRejectedHeader = "X-EDNexus-Pubsub-Rejected";
+
     public Task<StreamStatePublishResult> PublishAsync(
         string updateStateEndpoint, string token, StreamCardSnapshot snapshot, CancellationToken ct = default)
     {
@@ -156,6 +170,7 @@ private async Task<StreamStatePublishResult> SendAsync(
             var status = response.StatusCode switch
             {
                 _ when response.Headers.Contains(SnapshotRemovedHeader) => StreamStatePublishStatus.ClearedNotDelivered,
+                _ when response.Headers.Contains(PubSubRejectedHeader) => StreamStatePublishStatus.BackendRejectedByTwitch,
                 HttpStatusCode.Unauthorized => StreamStatePublishStatus.Unauthorized,
                 HttpStatusCode.RequestEntityTooLarge => StreamStatePublishStatus.TooLarge,
                 HttpStatusCode.TooManyRequests => StreamStatePublishStatus.RateLimited,

src/EDNexus.Core/Twitch/TwitchStreamCardService.cs

@@ -193,6 +193,15 @@ internal TwitchStreamCardService(
         // the EBS's initial state. The host calls RequestPublish() once the replay is done.
     }
 
+    /// <summary>
+    /// How long to wait between retries while the backend's Twitch credentials are being refused:
+    /// 24 times the publish floor, capped at <see cref="MaxRetryAfter"/> (so two minutes at the default
+    /// five-second floor). Scaled from the floor rather than fixed so a test with a tiny floor does not
+    /// have to wait minutes.
+    /// </summary>
+    internal TimeSpan BackendFaultRetryInterval =>
+        TimeSpan.FromTicks(Math.Min(MaxRetryAfter.Ticks, _minInterval.Ticks * 24));
+
     /// <summary>The floor between two publishes this service was built with.</summary>
     public TimeSpan MinInterval => _minInterval;
 
@@ -290,7 +299,8 @@ private async Task PumpAsync(CancellationToken ct)
             try
             {
                 var published = await PublishIfChangedAsync(ct).ConfigureAwait(false);
-                if (published is { Status: StreamStatePublishStatus.RateLimited or StreamStatePublishStatus.Failed })
+                if (published is { Status: StreamStatePublishStatus.RateLimited or StreamStatePublishStatus.Failed
+                    or StreamStatePublishStatus.BackendRejectedByTwitch })
                 {
                     // Give a struggling EBS (or a tighter-than-expected rate limit) room to recover
                     // rather than retrying at the floor interval.
@@ -299,7 +309,21 @@ private async Task PumpAsync(CancellationToken ct)
                     // Capped, so a misconfigured proxy's header cannot silence the card for the session.
                     if (published.RetryAfter is { } wait && wait > backoff)
                         backoff = wait < MaxRetryAfter ? wait : MaxRetryAfter;
-                    retry = ++_consecutiveFailures <= MaxPublishRetries;
+                    if (published.Status == StreamStatePublishStatus.BackendRejectedByTwitch)
+                    {
+                        // A backend fault lasts as long as it takes an operator to fix a secret: minutes or
+                        // hours, not the seconds a bounded retry covers. The warning on screen promises it keeps
+                        // retrying, and with no earlier success even the heartbeat wake is skipped, so giving up
+                        // would leave an idle commander's card stale until something changed. So retry for as long
+                        // as it takes, slowly (one small request every BackendFaultRetryInterval), and outside
+                        // the bounded budget that stops a dead EBS from being polled forever.
+                        if (backoff < BackendFaultRetryInterval) backoff = BackendFaultRetryInterval;
+                        retry = true;
+                    }
+                    else
+                    {
+                        retry = ++_consecutiveFailures <= MaxPublishRetries;
+                    }
                 }
                 else if (published is not null)
                 {

src/EDNexus.Ebs/Program.cs

@@ -122,6 +122,14 @@
         // long a single misbehaving/slow call can hold resources.
         client.Timeout = TimeSpan.FromSeconds(10);
     });
+    // Whether Twitch accepts the credentials PubSub calls are signed with: written by every real
+    // broadcast and by the startup check, read by /healthz and by the update-state response.
+    builder.Services.AddSingleton<TwitchCredentialStatus>();
+    builder.Services.AddHttpClient<ITwitchCredentialProbe, TwitchCredentialProbe>(client =>
+    {
+        client.Timeout = TimeSpan.FromSeconds(10);
+    });
+    builder.Services.AddHostedService<TwitchCredentialCheckBackgroundService>();
     builder.Services.AddHttpClient<ITwitchOAuthClient, TwitchOAuthClient>(client =>
     {
         // Same reasoning as above, and a login/refresh waits on this: fail in seconds, not the 100s default.
@@ -306,7 +314,20 @@ await Results.Problem(
     // project site rather than a 404.
     app.MapGet("/", () => Results.Redirect("https://signal-thread.github.io/EDNexus/"));
 
-    app.MapGet("/healthz", () => Results.Ok(new { status = "ok" }));
+    // "ok" means the process is up and answering, which is all a container health check should
+    // gate on, so a credential problem is reported alongside it, never instead of it: a wrong secret
+    // should not get the container restarted in a loop. `pubsub` is "unknown", "accepted" or "rejected".
+    app.MapGet("/healthz", (TwitchCredentialStatus credentials) =>
+    {
+        var (state, httpStatus, at) = credentials.Current;
+        return Results.Ok(new
+        {
+            status = "ok",
+            pubsub = state.ToString().ToLowerInvariant(),
+            pubsubStatus = httpStatus,
+            pubsubCheckedAt = at,
+        });
+    });
 
     app.MapOAuthEndpoints();
 
@@ -349,7 +370,7 @@ await Results.Problem(
             // not the request's original whitespace, which the limit never saw.
             using var compactState = JsonDocument.Parse(pubSubRequest.Message);
 
-            bool published;
+            PubSubBroadcastResult broadcast;
             using (await channelLocks.AcquireAsync(channelId, cancellationToken).ConfigureAwait(false))
             {
                 // Authenticated again now that this channel is ours alone: a sign-out (POST /oauth/revoke) that
@@ -362,12 +383,27 @@ await Results.Problem(
                 }
 
                 stateStore.Set(channelId, compactState.RootElement);
-                published = await pubSubClient.BroadcastAsync(channelId, compactState.RootElement, cancellationToken).ConfigureAwait(false);
+                broadcast = await pubSubClient.BroadcastAsync(channelId, compactState.RootElement, cancellationToken).ConfigureAwait(false);
             }
 
-            if (!published)
+            if (broadcast != PubSubBroadcastResult.Published)
             {
                 logger.LogWarning("Failed to publish state update for channel {ChannelId} to Twitch PubSub.", channelId);
+
+                // Twitch refusing the EBS's own credentials is a server misconfiguration, not an outage,
+                // and no amount of retrying fixes it. Say so with a header (a proxy in front can replace
+                // a 502's body) and a 503 (which one does not rewrite), so the desktop app can tell the
+                // commander it is not their sign-in, instead of a bare "502". Decided from THIS call's
+                // outcome, not the shared status: that stays "rejected" after an earlier refusal, and a
+                // later 400 or Twitch outage must not be reported as a credentials problem.
+                if (broadcast == PubSubBroadcastResult.CredentialsRejected)
+                {
+                    httpRequest.HttpContext.Response.Headers[TwitchCredentialStatus.RejectedHeader] = "credentials";
+                    return Results.Problem(
+                        "Twitch is rejecting this service's PubSub credentials, so live updates cannot be delivered. The card is stored and new viewers still see it. This is a server configuration problem.",
+                        statusCode: StatusCodes.Status503ServiceUnavailable);
+                }
+
                 return Results.Problem("Failed to publish the update to Twitch PubSub.", statusCode: StatusCodes.Status502BadGateway);
             }
 

src/EDNexus.Ebs/README.md

@@ -111,6 +111,47 @@ end to end instead of finding out when a viewer reports a blank card:
 5. Switch the card off (`DELETE /api/update-state`, or the app's toggle) and confirm the card disappears
    and `initial-state` answers `404`.
 
+### Credential self-check
+
+A wrong `Twitch__ExtensionSecret` used to be silent: the service started, stored every card, answered
+`/healthz` with `ok`, and never delivered a live update. Viewers who load the channel still saw the
+card (it is served from the store), so nothing looked broken until someone noticed a card that never
+changed. Three things now make it loud:
+
+1. **A startup check.** About 10 seconds after start, then every 6 hours, the EBS sends the real
+   Helix PubSub endpoint a request signed exactly like a broadcast but aimed at a target Twitch cannot
+   deliver to, so **nothing is ever sent to a viewer**. Twitch refusing the signed token
+   (`401`/`403`) logs at **error**: `Twitch REJECTED the EBS's PubSub credentials (403: ...)`, naming
+   the three settings to check. A `400` (the credentials got through and only the bogus target was
+   refused) logs `Twitch did not refuse the EBS's PubSub credentials`. Anything else (a network error,
+   a 5xx, a 429) is "inconclusive" and changes nothing. It never throws or stops the service.
+2. **Real broadcasts keep the record.** Every actual PubSub call updates the same status: a refusal
+   sets it, a success clears it. This is the authority, so a stale startup verdict cannot outlive the
+   first real publish.
+3. **`/healthz` reports it, and publishes carry it.** `/healthz` still answers `status: "ok"` (a wrong
+   secret must not get the container restarted in a loop, which would not fix it) with
+   `pubsub: "unknown" | "accepted" | "rejected"`, plus `pubsubStatus` (the HTTP status behind a
+   rejection) and `pubsubCheckedAt`. A `POST /api/update-state` that fails *because* Twitch is
+   refusing these credentials returns **`503`** with the header `X-EDNexus-Pubsub-Rejected: credentials`
+   instead of a bare `502` (a proxy in front can replace a 502's body, so the header and the status
+   code carry the meaning). The desktop app turns that into a warning on its main window.
+
+```bash
+curl -s https://<your ebs>/healthz
+# {"status":"ok","pubsub":"rejected","pubsubStatus":403,"pubsubCheckedAt":"2026-10-18T17:50:43+00:00"}
+```
+
+What `rejected` means and how to fix it: `Twitch__ExtensionSecret` must be this extension's own base64
+secret from the Developer Console (not another extension's, a rotated one, or the application's Client
+Secret); `Twitch__ExtensionId` must be this extension's Client ID; and set `Twitch__OwnerUserId` to the
+numeric Twitch user id of the extension's owner. Restart the service and watch `/healthz` (it updates
+on the next real publish; the check also re-runs on its 6 hour cycle).
+
+> **What the check does not prove.** Twitch verifying the token before it looks at the request body is
+> reasoned from its documentation, not observed: the refusal side was seen in production (`403 JWT could
+> not be verified`), the acceptance side was not. So `accepted` from the startup check means "not
+> refused"; only a real broadcast proves the credentials work, and that is why it overrides the check.
+
 ## Behind a reverse proxy
 
 The container speaks plain HTTP and expects a TLS-terminating proxy (Caddy in the compose file) in
@@ -203,7 +244,10 @@ Called by the desktop client on behalf of the broadcaster.
 - **Responses**: `200 OK` on success, `401 Unauthorized` for a missing/invalid/revoked token,
   `400 Bad Request` for a missing/`null`/non-object `state`, `413 Payload Too Large` if the state
   or the request body exceeds its limit, `429 Too Many Requests` if
-  the per-channel rate limit is exceeded, `502 Bad Gateway` if Twitch PubSub rejects the message.
+  the per-channel rate limit is exceeded, `502 Bad Gateway` if Twitch PubSub fails the message, and
+  `503 Service Unavailable` with `X-EDNexus-Pubsub-Rejected: credentials` when the failure is Twitch
+  refusing **this service's own credentials** (see [Credential self-check](#credential-self-check)).
+  The state is stored in every one of these cases, so new viewers still get the card.
 
 ### `DELETE /api/update-state`