Skip to content
This repository was archived by the owner on Nov 11, 2026. It is now read-only.

read the last recorded block per bridge from an index, not a full table scan - #540

Open
0xshubhs wants to merge 1 commit into
DefiLlama:masterfrom
0xshubhs:perf/last-recorded-blocks
Open

0xshubhs wants to merge 1 commit into
DefiLlama:masterfrom
0xshubhs:perf/last-recorded-blocks

Conversation

@0xshubhs

Copy link
Copy Markdown
Contributor

Fixes #539

One getLastRecordedBlocks(bridgeName?) in query.ts replaces the four copies of the full table aggregate. It reads the first and last tx_block per config row with two ORDER BY tx_block LIMIT 1 lookups instead of MIN/MAX over every row, and takes an optional bridge name so runAdapterByName stops loading every bridge's progress to run one.

The index it needs is declared in sql/data.sql:

CREATE INDEX IF NOT EXISTS transactions_bridge_id_tx_block ON bridges.transactions (bridge_id, tx_block)
    WHERE origin_chain IS NULL AND tx_block IS NOT NULL;

This has to exist before the code ships. Without it the lookups fall back to a bitmap heap scan per bridge, which is slower than what they replace. In production that means CREATE INDEX CONCURRENTLY first, then deploy.

Measured on postgres 16 with this schema, 1,607 config rows and 10m transactions:

plan buffers time
master Parallel Seq Scan, Partial HashAggregate 285,715 609 ms
this branch Index Only Scan, 1,607 loops, 0 heap fetches 12,858 23 ms

Driven through the function itself against that database rather than just as SQL: 28 ms against 502 ms, 1,607 keys both ways, 0 entries differing in either startBlock or endBlock.

Two behaviour changes, both deliberate:

  • getLastBlocks and runAdapterByName now apply WHERE origin_chain IS NULL like the other two call sites, so portal's solana slots stop being reported as an EVM chain's last recorded block.
  • the query moved from the write pool to querySql, which is where the rest of the reads live.

startBlock is kept in the result even though getBlocksForRunningAdapter ignores it, because GET /lastblocks puts it in the response and the index serves both ends anyway.

npm run test:server passes (170 tests) and tsc --noEmit reports the same 11 pre-existing errors as master, no more.

…le scan

four copies of the same query aggregate every row of bridges.transactions to
find where each adapter should resume. the cron job runs one of them every
cycle, and /lastblocks serves another. MIN(tx_block) is in all four and nothing
reads it: getBlocksForRunningAdapter only looks at endBlock.

on a 10m row copy of the schema the query is a parallel seq scan, 285,715
buffers and 609ms, and it grows with the table. the same values come back from
two ORDER BY ... LIMIT 1 lookups per config row against
(bridge_id, tx_block) WHERE origin_chain IS NULL AND tx_block IS NOT NULL,
as an index only scan with no heap fetches: 12,858 buffers and 23ms, 0 of 1607
bridge ids disagreeing.

two of the four copies were also missing the origin_chain filter, so they could
report portal's solana slots as an evm chain's last block. they now share one
query, which takes an optional bridge name so running a single adapter no longer
loads every bridge's progress.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

the last recorded block per bridge is found by aggregating the whole transactions table, in four places

1 participant