Skip to content

Reduce Astra input and preserve background cache keys - #903

Closed
thomasluizon wants to merge 3 commits into
mainfrom
feature/ticket-582-astra-model-calls
Closed

thomasluizon wants to merge 3 commits into
mainfrom
feature/ticket-582-astra-model-calls

Conversation

@thomasluizon

Copy link
Copy Markdown
Owner

Tracks thomasluizon/orbit-tickets#582.

Astra still declared all 69 tools, and background calls resolved a budget user without putting that identity on the OpenAI request. This finishes the remaining request changes while preserving D147's models, prices and daily ceilings.

Intended approach

  • AiCompletionClient.cs:156 and :296 fall back to AiCallContext for request routing and recorded usage. Explicit user IDs retain precedence. Tests cover stable/different users, HTTP/background identity, text/JSON, both tiers and legacy-model rollback.
  • ChatToolGroups.cs and ProcessUserChatCommand.Ai.cs:53 select cumulative English/Portuguese groups from the message plus history. The keywordless count is 16 including load_tools, or 17 with client-supported ask. get_profile stays core. Standalone thanks carries no tools. The registry still contains 69 chat operations; no business tool was deleted.
  • AiIntentService.cs:609 exposes missing server catalog names through a strict loader enum. Loading only enables declarations. It never executes operations. Two consecutive loading rounds are allowed; existing budgets, cancellation, authorization and approvals still govern execution. Loaded state survives continuations. Unsupported ask is excluded from both initial and recoverable declarations, with clarification in prose.
  • The two append-only properties in AiToolModels.cs carry internal orchestration state, not endpoint DTO changes. Card guidance stays available for tools loaded later. The JSON cancellation test repeats 200 independent calls while preserving its existing fake clock and network-like stream lifetime.

Remainder audit

Requirement Disposition and source
Models, low reasoning, compatible temperature and configuration rollback #801, #809; AiSettings.cs:7, AiCompletionClient.cs:81 and :150. Existing request tests remain green. No model, price or ceiling changed here.
Cache key on every chat/background request Chat and explicit JSON identity came from #801. This PR closes ambient background identity in AiCompletionClient.cs:156 and :296.
Strict schemas, enums and null/absent equivalence #816; StrictToolSchema.cs:23, AiIntentService.cs:437, StrictChatToolSchemaTests and Application optional-null tests. No existing parameter enum changed here; #816's Enum audit lists every prior conversion and allowed values. The loader adds an enum of fixed server tool names.
Description trimming #816 measured tool descriptions reduced 53.4% and schema descriptions 27.1%. Later load-bearing schedule/routine guidance is preserved.
Tool gating, count below 20, cumulative unlock and recovery Completed here. Tests cover English, accented Portuguese, turn-five history, exact count 16, byte-identical prefixes for equal active sets, loading beside a core call, invalid names and loading limits.
Deleted surfaces set_color_scheme is already absent, asserted by AgentCatalogServiceTests:121. No additional deletion was required or made; no memory tool changed.
Model history and cache-prefix layout #817; AppConstants.cs:58 caps model history at 20, AiIntentService.BuildHistoryTranscript keeps newest whole messages, and ProcessUserChatCommand.Ai.cs:44 separates static/dynamic context. API/validator history stays 40.
Overflow summary tier Obsolete per the work order's board correction: no overflow-summary call or history_summary path exists to reroute.
PostHog event, costs, traces, spans, errors and privacy #817; AiUsageRecorder.cs:22, AiIntentService.CompleteObservedRoundAsync, AiCompletionClient.ExecuteResponseAsync. Recorder tests cover daily-cost agreement, shared traces, unique spans, failed calls, no-op/throwing analytics and privacy. #817 records live HTTP 200 ingestion on 2026-10-20, UUID 100873a3-1abb-4761-9a05-42bcc5d7273c; unchanged fields retain that SDK evidence.
Quotas, transcription, endpoints, shipped DTOs and daily schema Unchanged. OpenAPI regeneration produced no tracked diff.
Live token reduction and final quality First-round provider usage shows 52.04% fewer Luna input tokens below. This is not production daily-row evidence or a completed conversation-quality gate.

External interface evidence

Verified 2026-10-24 UTC. OpenAI 2.14.0's installed manifest names commit 2e77b08828145f658ec04e49aec87abb1543c553. Read its options serializer: lines 132-135 map SafetyIdentifier to safety_identifier; lines 281-284 map PromptCacheKey to prompt_cache_key. #801/#816 establish the remaining Responses, stream, usage and strict-factory shapes. Transport tests inspect actual installed SDK serialization; responses come from its model writer.

Live GET /v1/models returned these complete matching entries. Re-derive through the same authenticated endpoint and select the exact IDs from data:

[
  {"id":"gpt-4.1-mini","object":"model","created":1744318173,"owned_by":"system","shutdown_date":"2027-01-14"},
  {"id":"gpt-6-luna","object":"model","created":1789406102,"owned_by":"system","shutdown_date":null}
]

The official pricing page confirms Luna's standard per-million prices: input 0.10 USD, cache read 0.01, 30-minute cache write 0.125, output 0.50. The Mini page confirms 0.40 input, 0.10 cached, 1.60 output. These match the unchanged pricing configuration.

A standalone probe exported declarations through production ConvertToSdkTool, submitted them to /v1/responses at Luna low and received HTTP 200. All 70 declarations from the broader schema audit were accepted as strict: 69 chat tools plus the MCP-only management tool. The returned function-call excerpt was:

{
  "id":"resp_0b5d2a5801e69d75016add00f08b2887d29fcc743d46678ff4",
  "status":"completed","model":"gpt-6-luna",
  "output":[{"id":"fc_0b5d2a5801e69d75016add00f35e6c87d2be453affc03418c4","type":"function_call","status":"completed","arguments":"{}","call_id":"call_gatx7S0htYOidni88wypQLn6","name":"get_profile"}],
  "usage":{"input_tokens":26879,"input_tokens_details":{"cache_write_tokens":26876,"cached_tokens":0},"output_tokens":14,"output_tokens_details":{"reasoning_tokens":0},"total_tokens":26893}
}

The repeated declarations and unrelated response fields are omitted. The submitted get_profile parameters were {"type":"object","properties":{},"required":[],"additionalProperties":false}, with strict:true. Gated probes also accepted the strict loader. No backend operation was executed and no app server started.

Live before and after

Requests came through the real handler, prompt builder and SDK, with synthetic Run/Read habits due on 2026-10-24, the same synthetic IDs and empty history. Full uses the pre-gating catalog at ab8547bf. Mini is gpt-4.1-mini with temperature 0.1; Luna is gpt-6-luna at low reasoning.

Prompts: 1 obrigado; 2 log my run today; 3 create a habit every wednesday and sunday; 4 log my run and show my progress; 5 show my profile and today's summary; 6 bulk log all habits today.

TTFT is the first text/function-argument delta; round time ends at the terminal response. Each row is one live model round, not an executed conversation. USD uses unchanged daily-recorder rates, including cache writes. Warmth varies, so these costs are not an invoice comparison.

Prompt Variant Input / cached / write / output tokens TTFT / round seconds USD First-round tools
1 Mini full 33603 / 0 / 0 / 24 3.400 / 3.523 0.013480 prose
1 Luna full 33608 / 33605 / 0 / 7 3.324 / 3.833 0.000340 prose
1 Luna gated 6864 / 0 / 6861 / 7 0.956 / 1.100 0.000861 prose
2 Mini full 33604 / 33536 / 0 / 47 2.508 / 2.631 0.003456 log_habit
2 Luna full 33609 / 33598 / 8 / 45 2.502 / 2.952 0.000360 log_habit
2 Luna gated 14761 / 0 / 14758 / 81 1.410 / 2.012 0.001886 query_habits
3 Mini full 33608 / 33536 / 0 / 669 3.062 / 6.913 0.004453 ask
3 Luna full 33613 / 33598 / 12 / 82 2.658 / 3.791 0.000379 ask
3 Luna gated 21946 / 0 / 21943 / 47 2.128 / 2.560 0.002767 prose
4 Mini full 33607 / 33536 / 0 / 53 2.530 / 2.645 0.003467 bulk_log_habits
4 Luna full 33612 / 33598 / 11 / 139 3.744 / 4.087 0.000407 query_habits, get_daily_summary
4 Luna gated 15517 / 0 / 15514 / 140 3.534 / 3.805 0.002010 query_habits, get_habit_metrics
5 Mini full 33606 / 33536 / 0 / 72 2.415 / 2.531 0.003497 get_profile, get_daily_summary
5 Luna full 33611 / 33598 / 10 / 75 3.650 / 4.344 0.000375 get_profile, get_daily_summary
5 Luna gated 15694 / 0 / 15691 / 73 1.907 / 2.052 0.001998 get_profile, get_daily_summary
6 Mini full 33605 / 33536 / 0 / 78 2.237 / 2.623 0.003506 bulk_log_habits
6 Luna full 33610 / 33598 / 9 / 81 2.868 / 3.510 0.000378 bulk_log_habits
6 Luna gated 21943 / 21931 / 9 / 76 2.524 / 3.204 0.000259 bulk_log_habits

Luna input falls from 201,663 to 96,725 tokens, a 52.04% reduction. Every prompt falls by at least 34.7%.

Prompt Correctness observed
1 All variants return Portuguese thanks without a call; gated declares no schemas.
2 Full resolves the correct Run ID; gated first queries Run on the correct date. Logging after lookup requires continuation.
3 No activity was named. Full calls unsupported ask; gated asks for the name in prose without inventing a placeholder. This is clarification, not a successful create.
4 Full Luna reads habits/summary; gated reads habits/metrics. A later logging call is not established by the first round.
5 Both Luna variants return profile and the requested day's summary.
6 Both select one bulk logging call with the all-habits selector.

Gated response IDs in prompt order: resp_08152180f3921d93016add03423ea087d28de5fa54220dd1dc, resp_07bfe78a75c20b34016add034241ac87d2805e4edf6375321b, resp_0c8095e6b259c1b4016add0343225487d296287044904f9254, resp_06909334ef50f80e016add0344017887d2832f2016d5ccd6a6, resp_0afcc6ca121e159f016add0345a7c487d28dc499049dc1aa48, resp_06ff5bcbaad83ede016add0347b9b887d2923b3192c25217e4.

Final answer correctness, #1568 account replay and deployed daily-row measurements remain acceptance work. #26 is still open in the UI repository, so its promptfoo gate was unavailable here. No writes or synthetic successful tool results were used to claim completion.

Test evidence

Before edits, the unchanged identity/background/fixed-turn tests passed four cases while the cache-key defect remained:

dotnet test tests/Orbit.Infrastructure.Tests --filter 'FullyQualifiedName~AiCompletionClientTests.CompleteJsonAsync_WithUserId_AttributesRecordedUsage|FullyQualifiedName~AiCompletionClientTests.BackgroundCalls_RecordPurposeLatencyAndOneTracePerRun|FullyQualifiedName~AiInputBudgetTests.FixedTurns' --logger 'console;verbosity=normal'

Strengthened tests, before the implementation fix:

dotnet test tests/Orbit.Infrastructure.Tests --no-restore --filter 'FullyQualifiedName~BudgetedBackgroundCalls_UseStableCacheKeyForResolvedUser|FullyQualifiedName~CompleteJsonAsync_ExplicitUserTakesPrecedenceOverAmbientUser' --logger 'console;verbosity=normal'

Exit 1: eight cases failed at the absent prompt_cache_key; explicit identity precedence passed. The identical command passes all nine after the fix.

The unchanged fixed-turn test passed with 69 tools. Before gating, dotnet test tests/Orbit.Infrastructure.Tests --no-restore --filter FullyQualifiedName~KeywordlessTurn_DeclaresFewerThanTwentyTools exited 1, observing 69 against the cap. It now exits 0 with exact count 16. The strengthened fixed-turn test separately enforces at least 30% serialized-request reduction against equal content with all registry schemas; this local metric is separate from live provider usage.

Focused runs passed 600 Application and 170 Infrastructure tests; the final strict/input/identity selection passed 41. The JSON deadline theory now repeats its independent call 200 times.

All required commands were judged by process exit code:

Command Result
dotnet build Orbit.slnx -t:Rebuild Exit 0, zero errors; 20 existing dependency/obsolete-API warnings
Focused Infrastructure selection (AiInputBudgetTests, BudgetedBackgroundCalls, StrictChatToolSchemaTests) Exit 0, 41 passed
dotnet test tests/Orbit.Domain.Tests Exit 0, 1,008 passed
dotnet test tests/Orbit.Application.Tests Exit 0, 7,456 passed
dotnet test tests/Orbit.Infrastructure.Tests Exit 0, 5,575 passed, 3 existing PostgreSQL-environment skips
dotnet test tests/Orbit.Analyzers.Tests --no-build --no-restore Exit 0, 40 passed
dotnet test tests/Orbit.Infrastructure.Tests --no-build --no-restore --filter FullyQualifiedName~CompletionDeadline_CancelsTheWholeCallIncludingRetryWaits --collect 'XPlat Code Coverage' (with a temporary results directory) Exit 0, four cases passed, including 200 independent JSON cancellations; Cobertura artifact produced

The final required-suite pass totals 14,039 passing tests plus the three environment skips. Both generated OpenAPI files remained unchanged. The final rebuild and focused/Domain/Application/Infrastructure sequence ran serially after coverage completed.

One Infrastructure attempt overlapped coverage instrumentation and was invalidated by BadImageFormatException; it is not counted as passing. Coverage and normal runs were then separated with a rebuild. Hooks passed formatting, dash, timeless-text, root and suppression checks. No new bare narration comments were added.

Assumptions

  • The latest remainder order requests the missing input cuts; restored cumulative groups and bounded recovery rather than preserving Reduce Astra input and record model analytics (#582) #817's final full-catalog reversal. Prefix equality applies to equal unlocked sets and client capability.
  • Preserved core habit/goal operations, profile, gamification and meta operations; gated tag management and rare reads/writes rather than preserving the older always-declared tag invariant, which conflicts with the requested count.
  • An unnamed-activity create requires clarification; rejected inventing a habit title to force the example into a create call.
  • Used provider first-round usage for worker measurement; rejected labeling it production daily-row evidence or a completed quality pass.
  • Preserved the absent overflow-summary path and subsequent load-bearing prompt changes rather than undoing later behavior fixes.

Manual steps

  1. After merge, deploy through GitHub Actions > Release API (release.yml), environment=staging, branch=main. Verify /health serves the merged commit. No new secret/environment value is required.
  2. In Orbit Staging > Hoje > Astra, replay these messages with continuations, the English/Portuguese weekday create with a named activity, and #1568's account scenarios. Retain final answers and opt-in eval traces. Verify both weekdays and every requested action; run feat: add calendar dismiss endpoint #26's gate when available.
  3. Read deployed AiUsageDaily chat rows for model, nonzero cost and input/cached-token reduction against baseline. In PostHog > AI Observability > Generations and Traces, verify one $ai_generation per call, shared chat traces, distinct spans and agreement with daily costs. This worker did not change analytics registration or production rows.

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes across all 10 files, including request identity, tool selection, loader continuations, and the existing execution and budget boundaries.

  • Background identity: Ambient AiCallContext identity supplies cache keys, safety identifiers, and usage attribution when no explicit user ID is provided.
  • Tool gating: Cumulative English/Portuguese keyword groups reduce initial declarations, standalone thanks omits tools, and unsupported ask is excluded from both initial and recoverable declarations.
  • Capability recovery: The strict load_tools catalog enables missing declarations without executing operations, preserves state across continuations, and keeps provider calls behind budget reservations and cancellation.
  • Verification: All 158 focused Infrastructure tests and 600 Application tests passed. Live end-to-end conversation replay and deployed usage measurements were not independently verified and remain the acceptance work identified in the PR description.

Pullfrog  | View workflow run | Using gpt-6.1-sol | 𝕏

@sonarqubecloud

Copy link
Copy Markdown

@thomasluizon

Copy link
Copy Markdown
Owner Author

Closing unmerged. Its background cache-key fix landed separately as #959; its keyword tool gating stays rejected under the approved thomasluizon/orbit-tickets#1740 plan.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant