Repository navigation
Conversation
|
Tick the box to add this pull request to the merge queue (same as
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Design proposal (SIG: Data Lake Analytics) for StarRocks materialized views stored in an Iceberg REST catalog, per the Iceberg materialized-view spec (apache/iceberg#11041, impl apache/iceberg#9830).
Summary:
CREATE MATERIALIZED VIEW <irc_catalog>.<db>.<mv> AS <query>creates the MV in the REST catalog — an IcebergViewwhosecurrentVersion().storageTable()points at a<mv>__storagetable, with data materialized there and an IcebergRefreshStaterecorded in the storage snapshot summary. No native StarRocks MV is persisted; the catalog is the source of truth. StarRocks reuses its task scheduler (via a dedicated refresh processor) to drive refresh and keep run-history.Initial scope: full refresh, non-partitioned, all-Iceberg base tables; manual + scheduled refresh. Depends on the (unreleased) Iceberg MV spec — prototype to be ready when it lands.
Seeking SIG input, especially on the incremental-refresh direction (§6): extend the catalog
RefreshStatewith per-partition state (keeps the catalog authoritative) vs. keep a local MV object (reuses the native incremental pipeline). A prototype exists; happy to share.