[Feature]: image 2 已经够厉害了，最需要的是如何把生成好的UI 变成组件化，再到原型的过程！

Problem / pain point

image 2 已经够厉害了，最需要的是如何把生成好的UI 拆解变成组件化，再到交互原型的过程！市面上都没有完整的方案，求求

Proposed solution

UI 拆解变成组件化，目前lovart 只能做到一点点。但是再到下一步他就完全不支持了

Alternatives considered

No response

Scope

UI

Hard-constraints check

I have read the hard constraints in CLAUDE.md and this proposal does not conflict with any of them.

感谢反馈，这个方向很重要。按当前公开仓库来看，我确认现在已经有的是“提示词/附件 -> HTML 或 JSX/React 原型”的生成与局部改写能力；但没有在公开代码里找到“把现有 UI 自动拆成可复用组件，再继续串成交互原型流程”的完整实现。

目前更接近的是：你可以附带文件/图片作为上下文，让模型生成或改写原型；生成后还会自动补一轮交互深化，例如 tab 切换、卡片展开、hover/press、空状态等。但这和“结构化拆组件 -> 组件树/状态机 -> 原型流”还是两件事。

这个项目现在仍是 pre-alpha，README 里的路线也显示很多导入/设计系统相关能力还在后续阶段，公开代码里我只看到更丰富的导入能力排在 v0.2、Figma 图层导出排在 1.0 之后，所以你提的这条链路目前更像是待规划能力，不是已经内置但没暴露的功能。

Relevant code:

README.zh-CN.md:224 — 当前公开描述是“提示词 → HTML 或 JSX/React 组件原型”
README.zh-CN.md:228 — 已有 Comment mode，支持只重写预览中的局部区域
apps/desktop/src/renderer/src/components/Sidebar.tsx:167 — 当前 UI 支持把文件作为上下文附加到会话里
apps/desktop/src/main/prompt-context.ts:148 — 图片附件会被作为 image input 传给模型
packages/core/src/index.ts:682 — 生成时会把附件转成 userImages 送入模型
packages/core/src/index.ts:736 — 局部改写时同样会把附件转成 userImages
apps/desktop/src/renderer/src/hooks/polishPrompt.ts:14 — 现有“二轮深化”主要是在已有页面上补交互深度，不是做组件拆解流水线
README.zh-CN.md:255 — roadmap 里 v0.2 提到的是更丰富的导入能力
README.zh-CN.md:267 — roadmap 里写了 Figma 图层导出在 1.0 之后

Need more info:

你更希望输入是什么：截图、生成后的 HTML/JSX、还是 Figma 设计稿？
你更希望输出是什么：React 组件树、设计系统 token、还是可点击的多页面原型？
你说的“到原型的过程”更偏向单页状态交互，还是页面之间的 flow（跳转、返回、分支）？

open-codesign Bot

这条思路我也很想看到落地，稍微构思一下，如果我们把gpt image 2 生成过程加到prompt -> composition 的pi-ai agent tool call kits, 那是否可以用生成出来的image当作context 来reference and enhance the final composition output of each ui component. We can also then show the gpt image 2 生成的原型图 next to or as a popover for the contexts referenced within the UI component display on a sidebar

这条思路我也很想看到落地，稍微构思一下，如果我们把gpt image 2 生成过程加到prompt -> composition 的pi-ai agent tool call kits, 那是否可以用生成出来的image当作context 来reference and enhance the final composition output of each ui component. We can also then show the gpt image 2 生成的原型图 next to or as a popover for the contexts referenced within the UI component display on a sidebar

Quick update on my last comment. After checking PR #193 ("Gpt image 2 support"), generate_image_asset is already a registered agent tool in packages/core/src/tools/generate-image-asset.ts with 7 purposes (hero, product, poster, background, illustration, logo, other), defaulted off in Settings. So that idea is more accurately framed as "build a new orchestration loop on top of the existing tool" rather than "add a new tool", smaller change, fully reuses PR #193's infrastructure. Saving that for Phase 2 as a separate Discussion once Phase 1 lands.

Answers to the bot's three open questions, focused on Phase 1:

Input: AI-generated UI mockup images (gpt image 2 / lovart / midjourney / plain screenshots). Plumbing already exists at prompt-context.ts:148 and index.ts:682 via userImages, what's missing is structured output.

Output: a ui_kits/<slug>/ folder shaped like this (mock data shown for clarity):

ui_kits/
└── saas-dashboard/
    ├── index.html          # full-page HTML, visually parity-matched to source mockup
    ├── components/
    │   ├── Sidebar.tsx     # <Sidebar nav={[...]} active="dashboard" />
    │   ├── MetricCard.tsx  # <MetricCard label="MRR" value="$12.4k" delta="+8.2%" />
    │   ├── ChartPanel.tsx  # <ChartPanel data={...} timeRange="30d" />
    │   └── ActivityTable.tsx
    ├── tokens.css          # :root { --color-brand: #c96442; --space-md: 16px; ... }
    ├── manifest.json       # { schemaVersion: 1, generator, components, tokens }
    └── README.md           # handoff notes for the downstream coding agent

So Claude Code / Cursor can pick the bundle up and implement it into real product code. This is the segment OP's title is calling out ("image 2 已经够厉害了, 最需要的是怎么变成组件再到原型").

Granularity: Phase 1 is single image → single ui_kit only. Multi-page flow / routing / state machines → Phase 3, separate issue.

Implementation plans to follow existing patterns strictly: decompose-to-ui-kit.ts mirrors done.ts / generate-image-asset.ts (factory + AgentTool + typebox params + adjacent .test.ts), trigger hook mirrors polishPrompt.ts but user-triggered instead of auto-fired, no new dependencies, decomposition is prompt-driven, output carries schemaVersion so downstream coding agents can evolve safely.

Next step: I'll prototype on my fork, render a before/after side-by-side video via Remotion, reply here with the MP4 before opening any PR.

One binary I'd like to align on first: should the emitted ui_kits/<slug>/ be (a) in-memory only, surfaced via the Files panel + existing ZIP export (downstream coding agent unzips the bundle, no SQLite schema change), or (b) persisted to disk in the design workspace (requires schemaVersion bump per CLAUDE.md)? My lean is (a), smaller blast radius, consistent with how polishPrompt.ts's second-pass only modifies the in-memory artifact.

中文版本 (Chinese version)

接着上一条更新一下, 查了 PR #193 ("Gpt image 2 support") 才发现 generate_image_asset 已经是 agent 的 tool 了 (packages/core/src/tools/generate-image-asset.ts), 7 个 purpose, Settings 里默认关。所以那条思路准确说是在已有 tool 之上做 orchestration loop, 不是加新工具, 改动更小, 完全复用 PR #193 的基础设施。这条放 Phase 2, 等 Phase 1 落了再单开 Discussion。

回 bot 那三个澄清, 聚焦 Phase 1:

输入: AI 生成的 UI mockup 图 (gpt image 2 / lovart / 截图都算)。地基已经在 prompt-context.ts:148 跟 index.ts:682 通过 userImages 通路上, 缺产出端的结构化。
输出: 一个 ui_kits/<slug>/ 目录, 形态如上 (mock 数据)。这样 Claude Code / Cursor 拿过去就能直接接到真实代码里。
粒度: Phase 1 只做单图 → 单 ui_kit。多页 flow / 路由 / 状态机 → Phase 3 单开 issue。

实现严格抄现有 pattern: decompose-to-ui-kit.ts 跟 done.ts / generate-image-asset.ts 一致 (factory + AgentTool + typebox + 邻近 .test.ts), trigger 抄 polishPrompt.ts 但用户显式触发不 auto-fire, 不引入新依赖, decomposition 走 prompt-driven, 输出带 schemaVersion 方便下游 coding agent。

下一步: 先在 fork 上跑 PoC, Remotion 渲个 before/after side-by-side 视频回贴这条 thread 再开 PR。

想先确认一个二选一: ui_kits/<slug>/ 是 (a) 内存里只在 Files panel + 走现有 ZIP export (编码 agent 解压就用, 不动 SQLite schema), 还是 (b) 持久化到磁盘 design workspace (走 schemaVersion bump 按 CLAUDE.md 要求)。我倾向 (a), blast radius 小, 跟 polishPrompt.ts 二轮深化只改 in-memory artifact 是一致的。

chore: add changeset for decompose-to-ui-kit feature

Status update on the Phase 1 PoC I committed to in the previous comment.

TL;DR: built it on a fork, validated cross-tier with real runs, opened PR #241. Self-answering my own open binary in option (a) (in-memory + Files panel + ZIP export, no SQLite schema bump). Surfaced 3 systemic dependencies during dogfood that I'm filing as separate Discussions, not bundling into this PR.

What landed

3 new agent tools in packages/core/src/tools/, all mirroring existing patterns (done.ts / generate-image-asset.ts):

decompose-to-ui-kit.ts — emits ui_kits/<slug>/{index.html, components/*.tsx, tokens.css, manifest.json, README.md} with schemaVersion: 1
verify-ui-kit-parity.ts — deterministic verifier (element count, visible-text coverage, token coverage) — no LLM, no floats
verify-ui-kit-visual-parity.ts — vision judge wrapped in 12-question boolean rubric across 5 dimensions; parityScore = passCount / totalChecks derived deterministically; bounded-enum status (verified | needs_review | needs_iteration | failed | unavailable)

Plus host-side BrowserWindow.capturePage() renderer + judge dispatcher in apps/desktop/src/main/, sidebar AddMenu trigger, locale-aware EN/ZH prompt walking the agent through decompose → verify → reconcile → iterate (max 2) → done. Per-tool-call cost surfaces inline as a toast.

PR #241 has the full breakdown.

Visual proof (real runs, not mocks)

Side-by-side hero — e2e-opus-final run, parityScore 0.90:

4-frame reconcile reel — e2e-nodebench-iter showing iter-0 (0.82) → iter-1 (0.78) honest score drift. Agent fixed some gaps but introduced new layout drift; the boolean rubric exposes the regression instead of hiding it. This is the intended behavior, not a bug:

BENCHMARKS.md has the full cross-tier methodology + per-run results (Opus / Pro+Pro+iterate / Kimi+Gemini3 / NodeBench iter), reproducibility instructions, and research citations (WebDevJudge, Prometheus-Vision, Trust-but-Verify ICCV 2025).

A live-recorded session demo (real Electron app, no slideshow stitching) is in progress and will be added to the PR description when ready.

Self-answer to my own open binary

in-memory + Files panel + ZIP export (option a) vs persisted-to-disk + schemaVersion bump (option b)

Going with (a). Rationale held up under implementation: smaller blast radius, consistent with polishPrompt.ts second-pass mutating only in-memory artifact, no SQLite migration, downstream coding agents (Claude Code / Cursor) unzip the bundle and treat it as filesystem input regardless of how Open CoDesign stored it internally. Easy to upgrade to (b) later if a multi-design "library of ui_kits" use case emerges.

Three systemic dependencies surfaced during dogfood (filing as Discussions)

End-to-end usage in a real session exposed three failure modes that this PR doesn't fix and shouldn't try to:

Spiral failures — agent kept retrying find-and-insert against fragile whitespace, broke the file structure, then tried to fix the breakage and broke it more. ~14min of wasted compute. No /rollback exists.
Rate-limit death — 4 consecutive 429 Provider returned error... is temporarily rate-limited upstream on deepseek-v4-flash and deepseek-v4-pro (free tier limits). Agent gave up. No auto-failover to next capable model.
Lesson amnesia — even after correction, the agent has no way to remember "last time find-and-insert against this file's whitespace failed, prefer read-then-replace" on the next turn.

Each deserves its own design conversation, not a bundled implementation. Filed 3 separate Discussions in Ideas:

These are explicitly not blocking for the Phase 1 decompose PR — that ships standalone. They're prereqs for the decompose flow to be robust under hands-off async use, which is a different bar.

What I'd value most from review

Whether the boolean rubric design (12 checks, 5 dimensions, threshold-derived status enum) is the shape you'd want, or if you'd push toward fewer/more checks
Whether the host-injection pattern for RenderUiKitFn + JudgeVisualParityFn matches your preference vs an alternative coupling
Whether option (a) (in-memory) for ui_kits/<slug>/ output is right, or you'd push for (b) (persisted) before merge
PR size feedback — happy to split into 3 separate PRs (per-tool) if the ~1500 LOC feels too big for one review

Branch is 11 commits behind upstream main (mostly chore/deps including the pi-agent-core 0.67.68 → 0.70.2 bump). I held off on rebasing to keep the as-built state visible for review; will rebase + adjust for the pi-ai 0.70.2 changes on request.

中文版本 (Chinese version)

接上一条 Phase 1 PoC 的状态更新。

TL;DR：fork 上跑通了，跨 tier 真数据验证完，开了 PR #241。自答上次留的二选一是 (a) 内存 + Files panel + ZIP export，不动 SQLite schema。Dogfood 过程中暴露 3 条系统级依赖，单独开 Discussion 不塞这个 PR。

已落地 — 3 个 agent tool，全部抄现有 pattern (done.ts / generate-image-asset.ts)：

decompose-to-ui-kit.ts — 输出 ui_kits/<slug>/{index.html, components/*.tsx, tokens.css, manifest.json, README.md}，带 schemaVersion: 1
verify-ui-kit-parity.ts — 确定性 verifier（元素数对齐、可见文本覆盖、token 覆盖），无 LLM 无浮点
verify-ui-kit-visual-parity.ts — vision judge 包一层 12 项 boolean rubric / 5 个维度；parityScore = passCount / totalChecks 确定性推导；状态用 bounded enum (verified | needs_review | needs_iteration | failed | unavailable)

加上 host 侧的 BrowserWindow.capturePage() renderer + judge dispatcher、sidebar AddMenu 入口、EN/ZH 双语 prompt 走 decompose → verify → reconcile → iterate (max 2) → done 流程。每次 tool call 的成本 toast 实时显示。

详细 breakdown 在 PR #241。

视觉证据（真实跑出来的，不是 mock）：

并排 hero — e2e-opus-final run，parityScore 0.90（见上面英文版图）
iter reel — e2e-nodebench-iter iter-0 (0.82) → iter-1 (0.78)，agent 修了一些 gap 但引入新的 layout drift，boolean rubric 把 regression 直接 surface 不藏，这是设计意图不是 bug

BENCHMARKS.md 有完整方法论 + 跨 tier 单跑结果 + 复现说明 + 研究引用。

实景录制（真 Electron app 不是切片拼接）正在录，好了贴到 PR 描述里。

Dogfood 暴露的 3 条系统级依赖（单独开 Discussion，不塞这个 PR）：

Spiral 失败 — agent 反复对脆弱空白做 find-and-insert，破坏文件结构后试图修复又破坏更多，浪费 ~14 分钟。没有 /rollback
Rate-limit 死锁 — deepseek-v4-flash 和 deepseek-v4-pro 连续 4 次 429（免费 tier 限制），agent 直接放弃。没有自动 failover
Lesson 失忆 — 即使被纠正，agent 没法记住「上次对这个文件做 find-and-insert 失败了，下次先 read 再 replace」

每条都值得单独讨论，不要打包实现。已开 3 条 Discussion 在 Ideas：

这些 不阻塞 Phase 1 decompose PR — 那个独立 ship。它们是 decompose 在 无人值守 async 场景下要 robust 的前置依赖，是另一个 bar。

最希望评审的方向：

boolean rubric 的形态（12 项 / 5 维 / 阈值推导 status）是不是你想要的，要不要增减
host 注入模式（RenderUiKitFn + JudgeVisualParityFn）跟你的偏好是否一致
ui_kits/<slug>/ 走 (a) 内存 vs (b) 持久化的判断 — merge 前需不需要切到 (b)
PR 大小（~1500 LOC）是不是过大 — 可以拆成 3 个 PR（每个 tool 一个）

分支落后 upstream main 11 commits（主要是 chore/deps 包括 pi-agent-core 0.67.68 → 0.70.2）。先没 rebase 是想保留 as-built 状态便于评审，需要的话立刻 rebase + 适配 pi-ai 0.70.2 的改动。

大佬，牛逼！我去试试

@HomenShum

感谢你的反馈！目前主干的公开代码中已有 generate_image_asset 工具（packages/core/src/tools/generate-image-asset.ts），支持基于提示生成图像（hero、产品图等），但尚未包含自动将截图/生成图像拆解为可复用组件并串联成交互原型的功能。

好消息是，贡献者 @HomenShum 已经提交了 PR #241，在其中实现了 3 个新工具：

decompose-to-ui-kit — 将 UI 截图分解为组件、token、workflow
verify-ui-kit-parity / verify-ui-kit-visual-parity — 验证与迭代

PR 目前处于 review 阶段，合并后将拥有你描述的“UI 图片 → 组件化 → 原型”核心能力。

如果你想现在就尝试，可以关注 PR #241 的进展，或直接在 fork 上体验。未来路线图还包括多页面流程、状态机等更高级的原型编排（Phase 3）。

再次感谢提出这个方向！

Open-CoDesign Bot

fix(decompose): address 4 Major review findings on PR OpenCoworkAI#241

补充一下 Open-CoDesign Bot 的回复 / Following up on Open-CoDesign Bot's reply:

1. PR #241 的 4 个 Major review findings 刚才已经修好 — 包括 (a) decompose 循环成功条件用错了 verifier 的状态词汇 (deterministic 只返回 ok|needs_iteration，不是 verified|needs_review)，(b) 视觉 verifier 默认从 source.png 读图，但运行时 agent FS 没有真把 attachments 持久化进去，导致 verify_ui_kit_visual_parity 在大部分跑里默默降级成 unavailable，(c) renderUiKit() 和 judgeVisualParity() 抛错没兜底 (text-only 模型、JSON 不规整都会直接炸而不是返回 unavailable)，(d) changeset 不应该写 Closes #225 — 这只是 Phase 1。Merge conflict 也解了，PR 状态从 CONFLICTING → MERGEABLE，现在等 CI + maintainer review。

The 4 Major findings flagged in the github-actions review are now fixed in commit e1afcdf — loop status vocabulary, source-image FS persistence, judge fallback, and changeset wording. Merge conflicts resolved; PR is now MERGEABLE.

2. 同时做了一个产品化的 web 版本 — parity-studio (Convex 后端 + React/Vite 前端，浏览器里直接用，不用装 desktop)。和 PR #241 是互补关系：

同样的 boolean-per-dimension 视觉 parity 判定 + verify-and-iterate 循环 + ui_kit/<slug>/ handoff bundle 这套核心原语
加了 Kilo Code 风格的 ✨ enhance prompt + 4 层 tier 路由 (frontier / balanced / free / small)，每一层每一阶段 (advise / execute / generate / decompose / iterate / judge / enhance) 单独配模型
free 层用 inclusionai/ling-2.6-1t:free，在我们的 5-query 工具调用 eval 上 15/15 通过，高于 paid claude-haiku-4-5 和 claude-sonnet-4-5（两个 paid 都是 14/15）。Eval 详情见仓库 docs/plans/2026-04-29-free-model-eval.md
MCP server parity-studio-mcp@0.1.0 已发，6 个 hosted-Convex 工具 (parity_chat_send、parity_chat_advise、parity_enhance_prompt、parity_run_listRecent、parity_chat_history、parity_export)，让 Claude Code / Cursor / Windsurf 直接通过 MCP 调用

Also built a productized web version — parity-studio — that ships the same boolean parity rubric + verify-and-iterate loop as a browser app (no desktop install). Adds a Kilo-style ✨ enhance + 4-tier auto router. The free tier (inclusionai/ling-2.6-1t:free) outscored paid claude-haiku-4-5 and claude-sonnet-4-5 on parity-studio's tool surface (15/15 vs 14/15). MCP server v0.1.0 ships 6 hosted-Convex tools so Claude Code / Cursor / Windsurf can drive the pipeline over MCP.

关系 / How they relate: open-codesign 还是 desktop-first，完整的"设计 ↔ 编辑器 ↔ 拆解"循环。parity-studio 是 web-only 的"上传图 → 拆 → 导出"切片，主要给"我已经有图了，只想要 ui_kit + Claude Code handoff"的场景用。两个共享同一套 parity primitive，PR #241 是这个 primitive 在 open-codesign 里的上游版本。

fix(decompose): address bot follow-up review (3 minor + 1 stale-diff …

feat(core): add decompose_to_ui_kit agent tool + sidebar trigger

chore: add changeset for decompose-to-ui-kit feature

fix(decompose): address 4 Major review findings on PR OpenCoworkAI#241

fix(decompose): address bot follow-up review (3 minor + 1 stale-diff …

feat(core): add decompose-to-ui-kit + boolean parity verifiers (Phase…

feat(runtime): switch preview screens inside one document (#458)

Navigation Menu

Repository navigation

[Feature]: image 2 已经够厉害了，最需要的是如何把生成好的UI 变成组件化，再到原型的过程！ #225

Problem / pain point

Proposed solution

Alternatives considered

Scope

Hard-constraints check

What landed

Visual proof (real runs, not mocks)

Self-answer to my own open binary

Three systemic dependencies surfaced during dogfood (filing as Discussions)

What I'd value most from review

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Feature]: image 2 已经够厉害了，最需要的是如何把生成好的UI 变成组件化，再到原型的过程！ #225

Description

Problem / pain point

Proposed solution

Alternatives considered

Scope

Hard-constraints check

Activity

github-actions commented on Apr 24, 2026

HomenShum commented on Apr 25, 2026

HomenShum commented on Apr 26, 2026

HomenShum commented on Apr 26, 2026

What landed

Visual proof (real runs, not mocks)

Self-answer to my own open binary

Three systemic dependencies surfaced during dogfood (filing as Discussions)

What I'd value most from review

chenjunyu-1990 commented on Apr 27, 2026

github-actions commented on May 1, 2026

HomenShum commented on May 1, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions