Repository navigation
Description
Depends on #651 — the lexical/fuzzy port of
winui-searchintowinapp(full v1). This issue adds the semantic layer on top of that ported foundation.
Is your feature request related to a problem? Please describe.
When building a Windows UI, a developer (or agent) has to already know which control to use and where to find a working example. Discovery today means leaving the CLI and manually hunting through the WinUI Gallery, the Windows Community Toolkit gallery, and docs. For an agent this is especially costly — it burns turns guessing at control names and APIs instead of getting a grounded, working snippet.
winapp should be able to answer "what control does X, and show me a real example" from the command line, in natural language.
Current state
- Control/sample discovery lives in the separate
winui-searchtool (in the microsoft/win-dev-skills repo), which does fuzzy/lexical matching over a set of controls/gallery samples — you have to roughly know the control name; natural-language intent ("a list with swipeable rows") doesn't resolve well. - It covers controls + gallery samples today, but lexically and without docs — this issue makes it semantic and folds docs into the same corpus.
- Note: this is distinct from
winapp ui search, which searches a running app's UI tree (UI automation) — unrelated to control/sample discovery.
Describe the solution you'd like
Add a winapp find-ui "<query>" command: natural-language, semantic search across WinUI controls, samples, and docs that returns a working code example plus where it came from. This is WinUI-scoped (its corpus is WinUI + Windows Community Toolkit controls/samples/docs), so it's nested/scoped as a WinUI capability.
- Natural-language query — "a card with an image and title", "tabbed layout", not just exact control names.
- Semantic matching — ranks by intent; the existing fuzzy/lexical search stays as the fast exact-name path.
- Controls + samples + docs — searches the WinUI Gallery, Windows Community Toolkit gallery, and docs, with filter options to narrow (e.g.
--source toolkit). - Agent-friendly output —
--jsonwith a structured result (control, snippet, source, link) so an agent gets a grounded example in one shot. - Extensible — leave a seam for additional authoring styles later (e.g. Reactor controls).
Replaces the winui-search tool (from microsoft/win-dev-skills), upgrading it from fuzzy to semantic.
Proposed architecture
Semantic search over two kinds of corpus, unified in one shared vector space (shared with find-api) so a single query searches them together:
- Global corpus — controls, samples, and docs (same for every user): gathered and embedded centrally at release, delivered as a download-only, fetch-and-cached vector index. Refreshes by re-publishing server-side; the CLI pulls updates when online. No user data is ever uploaded.
- Query embedding — on-device with one small bundled model. The query is compared locally against the cached vectors; the existing fuzzy/lexical search stays as the fast exact-name path.
winapp find-ui "<query>"
├─ ensure global control/sample/doc index (bundled baseline + fetch/refresh when online)
├─ embed the query locally with the bundled model
├─ cosine match against the cached vectors (lexical/fuzzy = fast exact-name path)
└─ return ranked results (control, snippet, source, link) [+ --json]
- Payload minimized: the only heavy bundled artifact is the shared embedding model. The global index is fetched-and-cached, not shipped whole; only a small baseline ships for offline.
- Shared model / one vector space:
find-uiandfind-apiuse one embedding model, so UI + API + docs + sample hits can eventually merge into one ranked answer (fewer agent turns, fewer tokens). - Download-only cloud: the service serves public prebuilt vectors to the client — it never receives the query or user data; embedding + comparison happen locally.
- Feasibility: a local PoC with a 384-dim embedding model ranked the correct control from natural-language intent (e.g. "swipe rows to delete" → SwipeControl), with a sub-2 MB index and ~20 ms warm queries.
- The main thing to approve:
find-ui= bundled shared model + a fetch-and-cached global index over controls, samples, and docs.
Open questions
- Payload budget for the bundled model (~30–40 MB quantized)
- Global index delivery - how much ships "baseline" and what do we leave to fetch.
- How docs/samples are gathered and embedded centrally (sources, chunking, release pipeline).
- Command placement for a WinUI-scoped capability. Existing commands are either flat top-level (
init,run,sign) or grouped by capability noun (cert …,manifest …,ui …) — never by framework; framework differences are handled by flags (e.g.init --setup-sdks none). So: flatwinapp find-ui(WinUI-scoped by its corpus) vs. a grouping, given there's no framework-namespace precedent?
Additional context
Part of the single-winapp-CLI direction (new top-level command; WinUI-aware). Migrates the existing winui-search tool from microsoft/win-dev-skills into winapp, upgrading it from fuzzy to semantic. Sibling of winapp find-api (semantic API search) — the two share an embedding model/vector space toward a future unified semantic index across UI, docs, and APIs. Future function winapp ask <query> once each has been established?
Activity
Jaylyn-Barbee commentedon Sep 28, 2026
PoC results: semantic
find-ui, and one architecture correctionI built an end-to-end proof of concept for this, simulating what release-time baking would actually look like, and benchmarked it against the lexical search that landed in #651. Posting the results because they answer three of the four open questions above with measurements, and because one part of the proposed architecture does not survive contact with the data.
Everything below is measured on a 181-case labeled evaluation set (179 scored, 2 excluded as genuinely ambiguous) split into four segments that are scored separately: exact-name (65), intent (80), negative-scope (22 — out of find-ui's corpus but answerable from docs), and negative-hard (12 — out of scope for the whole product). Segments are scored separately on purpose, because an aggregate win reliably hides an exact-name regression.
Setup:
all-MiniLM-L6-v2, int8-quantized, 384-dim, mean pooling, ONNX Runtime.1. Semantic beats lexical on find-ui, without regressing exact names
The exact-name row is the important one. It is unchanged, which was the precondition for this being worth doing at all.
The winning configuration is not pure semantic ranking. It is semantic-gates-then-rank-fusion: the semantic score decides whether there is an answer at all, and reciprocal rank fusion decides the order within that gated set. Ranking with lexical first and re-ranking semantically was actively harmful (57.5% intent). Pure semantic ranking scored marginally higher on intent (76.2%) but collapsed abstention on out-of-scope queries to 50%, which is the failure mode this repo has already rejected three separate times.
Worth noting how the remaining intent misses fail: 17 of 20 are abstentions, not wrong answers. The system errs toward silence rather than confident nonsense.
2. The one thing above that the data refutes
The architecture section says the corpora are:
and that
The first half is true. The second half does not follow from it, and that turns out to matter a lot.
A shared model gives you a shared vector space. It does not give you comparable scores. Docs chunks are longer and more discursive than control records, so they sit at systematically higher cosine similarity to almost any query. A docs page titled "NavigationView" scored 0.765 against the query, while the actual NavigationView control record scored 0.704 — so the docs page wins, and the user asking for a control gets an article instead.
Merging both corpora into one ranked answer, measured:
Eleven exact-name queries were hijacked by docs pages this way. This is not a tuning problem — the two corpora need genuinely different thresholds (0.40 for controls, 0.60 for docs) on identical vectors from the same model, so there is no single cutoff that is correct for both.
What works instead is precedence. Ask each corpus independently, in a fixed order, each with its own calibrated threshold: find-ui answers first; docs are consulted only when find-ui abstains. Adding docs then cannot regress find-ui, because find-ui's decision is already made before docs are consulted.
This still delivers the outcome the issue is actually after — fewer agent turns, fewer tokens, docs reachable from the same query. It just gets there by routing rather than merging.
3. Token cost, measured rather than assumed
The issue argues a unified answer means "fewer agent turns, fewer tokens." I measured this, because showing more sources plausibly costs more tokens, not fewer. Counted with
o200k_baseagainst real capturedwinapp find-uistdout, with output rendering held identical across both arms so the only difference is what was retrieved:The saving does not come from semantic output being shorter — for exact-name queries the two are byte-identical (4,010 tokens each). It comes from abstaining correctly. Lexical burns tokens emitting near-miss controls for queries it cannot answer; semantic stays quiet and spends those tokens only where there is something to say.
One caveat on how to read this: today's shipped output averages ~2.6 control entries per query. Trimming that to 1 is a ~31% token reduction that has nothing to do with semantics and could be done tomorrow. The semantic change itself is the −1%-tokens-for-+35-answers trade.
This result is conditional on tight output. At three entries per section the same system costs +81% tokens for +5 answers. If this ships as two labeled sections, they need to be short ones.
4. Answers to the open questions
Payload budget for the bundled model (~30–40 MB quantized) — confirmed at 38.2 MB per architecture, but the split matters: the int8 model (21.96 MB) is architecture-neutral, and only
onnxruntime.dll(~16 MB) is architecture-specific. Two findings attached:onnxruntime.dllcannot be embedded in the single-file NativeAOT executable. It publishes as a loose native file alongside the exe. The CLI ships as one self-contained binary today, so this changes the distribution shape, not just the size. This is the part that most needs a decision.Global index delivery — how much ships "baseline" vs is fetched — the measurements invert the proposal above. The control index is 88.6 KB and the full docs index is 3.33 MB. The index is the small part; the model is the 22 MB part.
So rather than shipping a baseline index and fetching the rest, the better split is ship the index, fetch the model:
Microsoft.Windows.SDK.BuildToolsvia NuGet when a command needs it, which inherits the corporate-proxy story and gives a signed, hashed, pinned artifact. MiniLM is Apache-2.0, so redistribution is fine with attribution.How docs are gathered, chunked, and embedded centrally — prototyped against
MicrosoftDocs/windows-dev-docs(hub/apps/**, 963 files, 11.7 MB of Markdown, sourced from the Git repo under CC BY 4.0, never scraped). That produced 9,075 chunks → 3.33 MB index.One non-obvious constraint: chunks are capped at 170 words because MiniLM truncates at 256 tokens. Microsoft's own commonly recommended ~500-token chunk size would be silently truncated by this model — you would lose the tail of every long chunk with no error. Chunk size has to be chosen against the model's context window, not from a general guideline.
The fourth open question — command placement — this PoC does not address.
5. Two implementation landmines
Both of these produce silently degraded relevance with passing tests. I hit both; flagging them so the next person does not.
Batch padding corrupts embeddings. The int8 MiniLM ONNX graph does not fully honor
attention_mask— padded positions leak into the pooled vector, so a document's embedding depends on which other documents happened to share its batch. Measured cosine similarity of 0.994 between the same text embedded batched vs. alone. Nothing throws; the index is just quietly wrong. Baking unpadded at batch size 1 fixes it and was 33% faster here, though at scale the right fix is bucketing chunks by identical token length so batching can be restored without padding.An early test I wrote for this was vacuous — a single-item batch pads to its own length, so it cannot detect the bug. The test has to compare against a batch containing a longer item.
InvariantGlobalizationdestroys accented text. This CLI setsInvariantGlobalization=true.Microsoft.ML.Tokenizersstrips accents viastring.Normalize(FormD), which is a documented no-op under that setting. Clean A/B on the stringcafé résumé naïve:InvariantGlobalization=false→[101, 7668, 13746, 15743, 102](correct)InvariantGlobalization=true→[101, 100, 100, 100, 102]— every word becomes[UNK]That is total information loss, not subtle degradation (see
dotnet/machinelearning#7728). MiniLM's tokenizer config resolves to strip-accents, so this model definitively needs it. A ~20-line ASCII diacritic fold applied before tokenizing restores exact parity while keepingInvariantGlobalization=true.Because both failures are silent, I would land the Python↔C# parity check as a CI fixture, not run it once. Any model, tokenizer, or ONNX Runtime version bump can silently reintroduce either bug. After both fixes, parity is exact (cosine 1.000000, max elementwise difference 0.000000).
6. Runtime feasibility — confirmed
A NativeAOT spike mirroring this project's real publish settings (
PublishAot,SelfContained,TrimMode=full,InvariantGlobalization):7. Open risks
find-api(#626) is completely untested here. No index, no evaluation set. Given that controls and docs needed different thresholds despite sharing a model, I would expect find-api to need a third, and I would not assume its numbers until it has its own eval set. It needs that eval set before it gets an index, not after — otherwise there is no way to tell tuning from overfitting.Docs routing is reliable in bulk but inseparable at the margin. 17 of 22 in-scope docs queries score cleanly above 0.59. But the contested band overlaps: the highest-scoring hard negative ("deploy my app to azure app service", 0.566) sits 0.002 above a legitimate in-scope query ("tune garbage collection for throughput", 0.564). No global docs threshold both answers those and refuses Azure. Tuning cannot fix this — the separability ceiling across the whole set is 87.8% balanced accuracy.
The practical read is that this is an argument for showing docs as a clearly labeled, separate section and letting the reader judge, rather than silently deciding for them at a threshold that provably cannot be right in every case.
The evaluation set has a known bias risk. I originally authored the negative segments against the control corpus alone, before the docs corpus existed. Four cases labeled "out of scope for the whole product" turned out to have real answers in
hub/apps(GitHub Actions CI, Entity Framework, iOS/SwiftUI migration, GC tuning) and were relabeled. Two more were genuinely ambiguous and are now excluded from scoring rather than forced into a segment. Notably, "android recyclerview adapter" was kept as out-of-scope, because the migration docs cover WPF, UWP, iOS/SwiftUI, React Native, Electron, MAUI and Flutter — but not Android. That relabeling made the reported scope coverage worse (77.8% → 63.6%), which is the direction an honest correction should move.I am happy to land the eval set in the repo as a CI fixture if that is useful — it is the only thing standing between "semantic search" and "semantic search that we can prove did not regress exact-name lookups."
Jaylyn-Barbee commentedon Sep 28, 2026
Correction: the batching fix I described above is wrong
In the PoC write-up I said the batch-padding problem could be fixed by bucketing records by identical token length so batching could be restored without padding. That does not work, and padding was not the root cause. Correcting it here because the wrong version is an easy optimization for someone to reach for, and it fails silently.
What I actually measured
I built 128 strings that all tokenize to exactly 11 tokens, so no padding can occur at any batch size, and compared each batch size against embedding one at a time:
Vectors diverge at batch size 2, with no padding involved, and stay diverged. The same test on the fp32 model is exact (cosine 1.000000, max diff 0.000000) at every batch size.
So this is int8 quantization, not attention-mask handling: the quantized graph's numerics depend on batch shape. Padding made it worse and was what I originally caught, but removing padding does not remove the problem.
It changes retrieved results, not just digits. On a small probe, 1 of 5 queries returned a different top-1 hit depending only on the batch size the index was built with.
The corrected rule
A query is always embedded alone at runtime, so with the int8 model the index must be built one record at a time to live in the same vector space as the queries that search it. There is no batching workaround — bucketing, sorting, or padding tricks do not recover it.
The practical guidance in the original comment (bake unpadded at batch size 1) stands. The reason I gave for it, and the bucketing optimization I recommended for scale, do not.
This costs less than it sounds
Batching was not buying anything here anyway. Padded batching pads every record to the longest one in its batch, and real API/control records are short with a long tail (median 13 tokens, max 184), so batching mostly computes padding. Measured on 3,000 real API metadata records:
int8 at batch size 1 is both the only correct option and the fastest one. fp32 at batch 32 is ~8x slower than batch 1.
For scale: the full local API metadata cache on my machine is 141,822 records (21,851 types + 119,971 members), which is about 5.5 minutes to embed at that rate, or ~50 seconds for types only.
Why this needs a regression test rather than a note
Both of these — the original padding drift and this batch-shape drift — produce a silently wrong index. Nothing throws, no test goes red, and relevance just quietly degrades. A cosine of 0.992 looks fine in a log and still reorders results.
I originally suggested landing the Python/C# parity check as a CI fixture. This strengthens that: the fixture should assert that embedding a fixed set of records one at a time and in a batch produces identical vectors, and fail if they ever diverge. That single assertion catches both bugs, and would have caught my incorrect fix.
Jaylyn-Barbee commentedon Sep 28, 2026
Correction: I benchmarked
find-uiagainst the wrong baselineMy PoC write-up above justifies semantic
find-uion a +25pp intent gain over the lexical search in #651. That number is correct and I stand behind how it was measured. But lexical is the wrong thing to compare against, and once I measured against the right one, the case for this feature gets narrower and much more specific.Correcting it here because the original framing would justify building the wrong version of this.
The baseline I left out
find-uiis called overwhelmingly by a coding agent, not typed by a human. So the question that decides whether an index is worth its cost is not "does semantic beat BM25". It is "does this tell the agent anything it does not already know?"I never tested that. So I did: a model was given the same 80 intent queries with no tools, no index, no repo access, no lookups, required to name the control, and scored against the same eval set.
find-ui(#651, ships today)find-ui(what I recommended above)81.2% is the strict score. Crediting 8 individually enumerated cases where the model gave the real platform type and our corpus indexes a doc-page title instead —
MarkdownTextBlockvsmarkdown,CameraPreviewvscapture element / camera preview,ConnectedAnimationvsconnected animation— it is 91.2%. I did not creditTextBlockfortype ramp, which is a control-vs-guidance conflation rather than an alias.The caller already beats the index I recommended. On aggregate intent accuracy, this feature makes the agent worse.
Why this matters even though the +25pp is real
Both numbers are true at once. Semantic really does beat lexical by 25pp. And the agent really does beat semantic by ~9-19pp. The first comparison just doesn't establish that the feature is worth building, because it measures against a baseline nobody is actually using as their alternative. The agent's alternative to calling
find-uiis not "call lexicalfind-ui" — it is "answer from what I already know."Measured against that, aggregate intent accuracy is the wrong headline metric, and optimizing it optimizes the wrong thing.
What survives, and it is a sharper case than the one I made
Splitting the same 80 queries by what the model gets wrong:
All seven are long-tail Windows Community Toolkit or very new WinUI controls:
PagerControl,TabbedCommandBar,Docking,Card,MetadataControl,AnnotatedScrollBar,Typography.That is the actual product. Not "better control discovery" — the agent is already ahead of us there — but coverage of the specific long tail the agent does not have, which is exactly the Toolkit and newest-WinUI surface that this repo is in the best position to know about.
The concrete engineering consequence
The 0.40 abstain gate I recommended suppresses 2 of those 7.
So under the configuration I actually recommended, it is 5 of 7, not 7 of 7. I am reporting the gated number because that is the one that would ship.
The threshold was calibrated to maximize aggregate accuracy across all 80 queries. But the aggregate is 91% head, and the head is where the index has no value to add. Calibrating on the aggregate tunes the gate for the queries that do not matter, and it is currently silencing 29% of the queries that do. The gate should be calibrated against the tail.
That is a real, actionable change to the design, and I would not have found it without this baseline.
The other thing the index buys: refusing to invent
The model named two controls that do not exist in WinUI, UWP, or the Windows Community Toolkit —
PaginationandLabeledSlider.Both are plausible, and
Paginationis instructive about why: it exists as a third-party Telerik control, and the real Microsoft control isPagerControl.LabeledSliderdoes not exist anywhere; the control that actually does the job asked for isAnnotatedScrollBar. An agent that trusts either writes code against a type that was never there, and finds out at build time or later.A grounded index that can answer "no such control" prevents that, and this value is invisible to every accuracy metric in my original write-up. It is also a reason the abstain behavior is a feature rather than just a safety measure: silence is the correct answer to some queries, and it is one the agent cannot produce on its own.
Worth separating two things that look alike here: the model also named nine controls that are real but absent from our corpus —
WebView,ConnectedAnimation,ToastNotification,BadgeNotification,FileOpenPicker,AutomationProperties,MarkdownTextBlock,CameraPreview,ElementSoundPlayer. Some are legitimately out of scope (FileOpenPickeris an API, not a control) and several are the naming-alias cases above. "Absent from our corpus" is not the same claim as "does not exist", and only the first two were verified as genuinely nonexistent.Limitation that cuts against my own conclusion
I authored this eval set by hand, and I verified each expected control existed in the corpus before writing the case down. That process favors well-known controls, which inflates the model's cold score and shrinks the measured tail. The true tail is probably larger than 7 of 80.
That cuts in favor of the feature, not against it — but it means none of the numbers in my original write-up are measuring the right population. A head-dominated eval set mostly measures where an index cannot win.
What I would change about this issue
The rest of the original write-up stands: the exact-name non-regression, precedence routing over merged ranking, the two silent implementation landmines, the NativeAOT feasibility result, and the payload split.
One forward pointer: I ran this same baseline against #626 and the result there is much starker — the model scored 40/40 with zero hallucinations, leaving an empty tail. I will write that up on that issue rather than here.
find-ui: search control descriptions and add an API list to the sampl…
Jaylyn-Barbee commentedon Oct 13, 2026
Closing as not planned: the juice isn't worth the squeeze. In a test with real agents (157 tasks, 3 models), keyword search already led to an acceptable control about 97% of the time. Semantic search didn't improve that and didn't save tokens, but it would add a model or a hosted service to build and maintain. Docs support continues in #1070.