Skip to content

Add Vermont campaign finance support#138

Merged
shu1513 merged 5 commits into
mainfrom
codex/vermont-finance-phase1
Jun 28, 2026
Merged

Add Vermont campaign finance support#138
shu1513 merged 5 commits into
mainfrom
codex/vermont-finance-phase1

Conversation

@shu1513

@shu1513 shu1513 commented Jun 27, 2026

Copy link
Copy Markdown
Owner

Summary

  • Add Vermont campaign finance feature flags, scheduler commands, and sync/probe scripts
  • Wire Vermont finance summaries into ballot lookup for eligible offices
  • Add Vermont finance pipeline, writer, and aggregation support plus database tables
  • Expand tests for Vermont sync, scheduler, aggregators, and ballot lookup behavior

Testing

  • Added and updated unit tests for Vermont finance loading, syncing, scheduling, and ballot lookup integration
  • Not run (not requested)

Summary by CodeRabbit

  • New Features

    • Added Vermont candidate finance support, including scheduled syncing, manual triggers, and a live probe for checking data quality.
    • Candidate finance summaries now appear in ballot lookup results where Vermont finance is enabled.
    • Added richer finance breakdowns for direct contributions, outside spending, and outside-group support.
  • Bug Fixes

    • Improved matching and validation so candidate committee and office results are more reliable.
    • Added safer handling for missing or unsupported finance data, reducing failed lookups and syncs.

@coderabbitai

coderabbitai Bot commented Jun 27, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@shu1513, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 53 minutes and 35 seconds. Learn how PR review limits work.

Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file).

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits.

🚦 How do rate limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 8cec6959-e5a7-4880-9e98-8dc9b3aeccd2

📥 Commits

Reviewing files that changed from the base of the PR and between e4e4b31 and 7668bf1.

📒 Files selected for processing (5)
  • backend/package.json
  • backend/src/config/featureFlags.ts
  • backend/src/pipeline/address/ballotLookup.ts
  • backend/tests/pipeline/address/ballotLookup.test.ts
  • db/migrations/138_add_vermont_campaign_finance_tables.sql
📝 Walkthrough

Walkthrough

Adds a complete Vermont campaign finance integration: a Postgres migration for five new tables, a typed API client for the Vermont CFD API, aggregators for direct contributions and outside spending, a transactional finance snapshot writer, batch sync with BullMQ scheduling, CLI scripts for sync/trigger/probe, ballot lookup wiring that merges Vermont summaries, feature flags, and comprehensive Vitest suites.

Changes

Vermont Campaign Finance Pipeline

Layer / File(s) Summary
DB migration: Vermont finance tables
db/migrations/136_add_vermont_campaign_finance_tables.sql
Creates five Postgres tables (vt_candidate_finance_links, _summaries, _direct_breakdowns, _outside_groups, _outside_group_breakdowns) with CHECK constraints, FK cascades, compound uniqueness, indexes, and BEFORE UPDATE triggers.
Feature flags and npm scripts
backend/src/config/featureFlags.ts, backend/package.json
Adds isVermontCampaignFinanceEnabled and isVermontCampaignFinanceSyncEnabled(force?) flag helpers; adds six npm scripts for sync, dry-run, scheduler upsert/worker/trigger, and live probe.
Vermont Campaign Finance API client
backend/src/pipeline/vermontFinance/vermontCampaignFinanceClient.ts
Implements the full typed Vermont CFD REST client: constants, request/response contracts, payload builder, AbortController timeout fetch layer with error-code mapping, typed row parsers, and five exported async API functions.
Vermont eligible offices registry
backend/src/pipeline/vermontFinance/vermontFinanceEligibleOffices.ts
Defines the finite set of finance-eligible Vermont office keys with O(1) Set/Map lookups and exports toVermontFinanceOfficeKey, isVermontFinanceEligibleOffice, mapVermontOfficeSought, and toVermontOfficeSearchInput.
Vermont candidate committee resolver
backend/src/pipeline/vermontFinance/vermontCandidateCommitteeResolver.ts
Implements normalizeVermontCandidateNameKeys, buildVermontCandidateSearchPhrases, resolveVermontCandidateCommittee (matched/ambiguous/unmatched), row deduplication helpers, and async searchAndResolveVermontCandidateCommittee.
Direct contribution, outside spending, and outside group aggregators
backend/src/pipeline/vermontFinance/vermontDirectContributionAggregator.ts, vermontOutsideSpendingAggregator.ts, vermontOutsideGroupContributionAggregator.ts
Adds three pure aggregation modules: contributor-source/size breakdown accumulation, PAC outside-spending grouping by support/oppose, and outside-group contribution fetch-and-aggregate with donor/industry classification.
Vermont finance snapshot writer
backend/src/pipeline/vermontFinance/vermontFinanceWriter.ts
Implements upsertVermontFinanceLink and replaceVermontCandidateFinanceSnapshot with transactional upsert/stale-delete logic for all five Vermont finance table sections; validates inputs before any DB interaction.
Vermont candidate finance sync orchestration
backend/src/pipeline/vermontFinance/vermontCandidateFinanceSync.ts
Implements syncVermontCandidateFinance: validates inputs, resolves or trusts committee, paginates transactions, aggregates direct and outside data, classifies labels, and conditionally writes snapshot when not in dryRun.
Vermont candidate finance batch sync
backend/src/pipeline/vermontFinance/vermontCandidateFinanceBatchSync.ts
Adds listDueVermontCandidateFinanceSyncRows (parameterized CTE query with office/staleness filtering) and syncDueVermontCandidateFinance (serial per-row sync with error capture and structured summary return).
Ballot lookup Vermont finance integration
backend/src/pipeline/address/ballotLookup.ts, backend/src/pipeline/vermontFinance/vermontBallotLookupFinanceLoader.ts, backend/src/pipeline/vermontFinance/index.ts
Extends BallotLookupFinanceSummary.source with "VERMONT_CFD", adds shared missing-module detection, implements multi-query SQL loader returning a Map<string, BallotLookupFinanceSummary>, and merges Vermont summaries into the finance map. Also parameterizes buildOutsideIndustrySupportExplanation with supportAction.
BullMQ scheduler and worker
backend/src/scheduler/vermontCandidateFinanceSyncScheduler.ts
Defines job/queue constants and types, Redis URL parsing, recurring scheduler upsert/removal, manual job enqueueing with feature-flag gating, job execution with Postgres pool lifecycle, and BullMQ Worker factory with concurrency 1.
CLI scripts: sync, trigger, upsert, probe, worker
backend/src/scripts/syncDueVermontCandidateFinance.ts, triggerVermontCandidateFinanceSync.ts, upsertVermontCandidateFinanceSyncScheduler.ts, runVermontCandidateFinanceSyncSchedulerWorker.ts, probeVermontCandidateFinance.ts
Adds five CLI entrypoints with argument parsing, feature-flag gating, and structured JSON output: due-sync runner, BullMQ manual trigger, recurring scheduler upsert, BullMQ worker process with graceful shutdown, and live probe with optional CSV comparison.
Test suites
backend/tests/pipeline/vermontFinance/*, backend/tests/scheduler/vermontCandidateFinanceSyncScheduler.test.ts, backend/tests/scripts/*, backend/tests/pipeline/address/ballotLookup.test.ts
Adds Vitest suites for the API client, committee resolver, all three aggregators, finance writer, batch sync, candidate sync, scheduler, ballot lookup integration (positive and negative), probe, and CLI scripts.

Estimated code review effort

🎯 5 (Critical) | ⏱️ ~120 minutes

Possibly related PRs

  • shu1513/voteapp#126: Adds Alaska ballot-lookup finance integration with the same dynamic-import/missing-module pattern that this PR refactors into shared helpers and reuses for Vermont.
  • shu1513/voteapp#127: Adds New Jersey candidate finance ballot-lookup support using the same featureFlags.ts extension, BallotLookupFinanceSummary.source union expansion, and loadCandidateFinanceSummariesByCandidateElection merge pattern.
  • shu1513/voteapp#136: Adds Illinois finance ballot-lookup support with the same BallotLookupFinanceSummary.source extension and loadCandidateFinanceSummariesByCandidateElection merge wiring.

Poem

🐇 Hopping through Vermont's hills so green,
Finance data flows through tables unseen,
A scheduler ticks, the BullMQ hums,
Contributions tallied, outside money sums.
With feature flags and SQL neat and trim,
This rabbit syncs the CFD on a whim! 🌿

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: adding Vermont campaign finance support across backend, scheduler, scripts, and tests.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/vermont-finance-phase1

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@greptile-apps

greptile-apps Bot commented Jun 27, 2026

Copy link
Copy Markdown

Greptile Summary

This PR introduces Vermont campaign finance support, following the established pattern of other state finance integrations. It adds a full pipeline: API client, candidate committee resolver, direct/outside contribution aggregators, a finance writer, a ballot lookup loader, scheduled sync infrastructure, and DB tables for five new tables (vt_candidate_finance_links, vt_candidate_finance_summaries, vt_candidate_finance_direct_breakdowns, vt_candidate_finance_outside_groups, vt_candidate_finance_outside_group_breakdowns).

  • Vermont finance summaries are wired into the ballot lookup via a dynamically-imported optional loader, gated behind VERMONT_CAMPAIGN_FINANCE_ENABLED. The isMissingOptionalCampaignFinanceModule helper now correctly validates the missing module specifier from the error message, addressing the transitive-import-failure concern from the previous review round.
  • The sync pipeline fetches all statewide expenditure rows per-candidate without a filer filter, then performs in-memory candidate matching; this is intentional per the code comment and an acknowledged trade-off.
  • total_disbursements and cash_on_hand are intentionally left null in summaries (the Vermont public API doesn't expose reliable aggregates for these), with an inline comment explaining the gap.

Confidence Score: 4/5

The change introduces a large but self-contained new feature that is off-by-default behind a feature flag, with no destructive changes to existing tables or code paths.

The ballot lookup wiring is gated behind isVermontCampaignFinanceEnabled() and isolated via dynamic import, so no existing state is affected when the flag is off. The isMissingOptionalCampaignFinanceModule helper now correctly validates the missing module specifier from the error message, properly distinguishing loader-absent from loader-broken. The only new observation is that contributor_source_type direct breakdown data is written to the DB but never queried in the ballot lookup loader, leaving top_occupations and top_employers permanently empty.

backend/src/pipeline/vermontFinance/vermontBallotLookupFinanceLoader.ts — the direct breakdown query should either fetch contributor_source_type rows or document why they are intentionally excluded.

Important Files Changed

Filename Overview
backend/src/pipeline/vermontFinance/vermontCandidateFinanceSync.ts Core sync orchestrator: fetches contributions scoped to one filer and all statewide expenditures without a filer filter, aggregates, and writes via replaceVermontCandidateFinanceSnapshot. The statewide expenditure fetch is an acknowledged per-candidate inefficiency; the intentional omission of total_disbursements/cash_on_hand is now properly commented.
backend/src/pipeline/vermontFinance/vermontBallotLookupFinanceLoader.ts Ballot lookup loader running 5 sequential DB queries to build BallotLookupFinanceSummary. Uses SUM for direct totals and MAX for outside totals (correct deduplication across multiple links). top_occupations, top_employers, and top_industries for direct_campaign are always [], consistent with Vermont API limitations.
backend/src/pipeline/address/ballotLookup.ts Vermont finance loader wired in via isMissingOptionalCampaignFinanceModule, which now validates the missing module specifier in the error message — properly distinguishing loader absent from loader present but broken.
db/migrations/138_add_vermont_campaign_finance_tables.sql Well-structured migration with five tables, FK chains, CHECK constraints, unique indexes, and set_updated_at triggers.
backend/src/pipeline/vermontFinance/vermontCandidateFinanceBatchSync.ts Batch sync correctly loads only linked candidates, always passes trustedCommittee to bypass re-resolution, and handles errors per-candidate without aborting the batch.
backend/src/pipeline/vermontFinance/vermontDirectContributionAggregator.ts Aggregates direct contributions by source type and size bucket. totalReceipts and directContributionTotal are intentionally set to the same value since Vermont's transaction API doesn't expose a separate total-receipts endpoint.
backend/src/pipeline/vermontFinance/vermontOutsideSpendingAggregator.ts Correctly filters PAC expenditures by candidate name/entity ID from the full statewide set.
backend/src/pipeline/vermontFinance/vermontCampaignFinanceClient.ts Standard HTTP client wrapping the Vermont campaign finance API.
backend/src/pipeline/vermontFinance/vermontFinanceWriter.ts Atomically replaces all Vermont finance data for a candidate in a single DB transaction.
backend/src/pipeline/vermontFinance/vermontCandidateCommitteeResolver.ts Resolves candidate-to-committee by scanning contribution/expenditure rows with matched/unmatched/ambiguous status.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Scheduler / CLI Script] --> B[syncDueVermontCandidateFinance]
    B --> C{Has active link in DB?}
    C -->|No| D[Skip candidate]
    C -->|Yes| E[syncVermontCandidateFinance with trustedCommittee]
    E --> F[Fetch contributions by filerRegistrationGuid]
    E --> G[Fetch ALL statewide expenditures for electionYear]
    F --> H[aggregateVermontDirectContributions]
    G --> I[aggregateVermontOutsideSpending filter by candidate name/entityId]
    I --> J[fetchAndAggregateOutsideGroupContributions per matched PAC]
    H --> K[replaceVermontCandidateFinanceSnapshot]
    I --> K
    J --> K
    K --> L[(vt_candidate_finance_links vt_candidate_finance_summaries vt_candidate_finance_direct_breakdowns vt_candidate_finance_outside_groups vt_candidate_finance_outside_group_breakdowns)]
    M[Ballot Lookup Request] --> N{isVermontCampaignFinanceEnabled?}
    N -->|No| O[Return empty map]
    N -->|Yes| P[Dynamic import vermontBallotLookupFinanceLoader]
    P -->|Module missing| Q[console.warn + return empty]
    P -->|Loaded| R[loadVermontCandidateFinanceSummaries 5 sequential DB queries]
    R --> S[BallotLookupFinanceSummary source: VERMONT_CFD]
    L --> R
Loading
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A[Scheduler / CLI Script] --> B[syncDueVermontCandidateFinance]
    B --> C{Has active link in DB?}
    C -->|No| D[Skip candidate]
    C -->|Yes| E[syncVermontCandidateFinance with trustedCommittee]
    E --> F[Fetch contributions by filerRegistrationGuid]
    E --> G[Fetch ALL statewide expenditures for electionYear]
    F --> H[aggregateVermontDirectContributions]
    G --> I[aggregateVermontOutsideSpending filter by candidate name/entityId]
    I --> J[fetchAndAggregateOutsideGroupContributions per matched PAC]
    H --> K[replaceVermontCandidateFinanceSnapshot]
    I --> K
    J --> K
    K --> L[(vt_candidate_finance_links vt_candidate_finance_summaries vt_candidate_finance_direct_breakdowns vt_candidate_finance_outside_groups vt_candidate_finance_outside_group_breakdowns)]
    M[Ballot Lookup Request] --> N{isVermontCampaignFinanceEnabled?}
    N -->|No| O[Return empty map]
    N -->|Yes| P[Dynamic import vermontBallotLookupFinanceLoader]
    P -->|Module missing| Q[console.warn + return empty]
    P -->|Loaded| R[loadVermontCandidateFinanceSummaries 5 sequential DB queries]
    R --> S[BallotLookupFinanceSummary source: VERMONT_CFD]
    L --> R
Loading

Reviews (4): Last reviewed commit: "Merge remote-tracking branch 'origin/mai..." | Re-trigger Greptile

@shu1513

shu1513 commented Jun 27, 2026

Copy link
Copy Markdown
Owner Author

@coderabbitai full review

@coderabbitai

coderabbitai Bot commented Jun 27, 2026

Copy link
Copy Markdown
✅ Action performed

Full review finished.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 11

🧹 Nitpick comments (3)
backend/tests/pipeline/vermontFinance/vermontOutsideGroupContributionAggregator.test.ts (1)

142-203: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick win

Cover the multi-page fetch path.

fetchAndAggregateVermontOutsideGroupContributions() is supposed to walk paginated Vermont responses, but this test only proves the single-page case. An off-by-one or early-stop bug would still pass while undercounting donor and industry totals.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@backend/tests/pipeline/vermontFinance/vermontOutsideGroupContributionAggregator.test.ts`
around lines 142 - 203, The current test only exercises a single Vermont API
page, so it does not verify pagination handling in
fetchAndAggregateVermontOutsideGroupContributions. Expand this test to cover
multiple pages by having fetchImpl return page 1 and page 2 results for the same
filerRegistrationGuid, then assert the aggregated fetched/matched/included
counts and outsideGroupBreakdowns reflect both pages. Use
fetchAndAggregateVermontOutsideGroupContributions and fetchImpl to locate the
behavior, and make sure the request assertions confirm pageNumber advances
correctly until all pages are consumed.
backend/tests/pipeline/vermontFinance/vermontCampaignFinanceClient.test.ts (1)

384-405: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

Add a timeout/abort-path assertion.

This suite never exercises the client’s timeout mapping. A regression in the AbortError/deadline path would still pass here even though the scheduler and CLI workers depend on that branch to fail fast.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@backend/tests/pipeline/vermontFinance/vermontCampaignFinanceClient.test.ts`
around lines 384 - 405, The `getVermontContributionDetails` failure-mapping test
covers HTTP, API envelope, and malformed JSON cases but misses the timeout/abort
path. Add an assertion in this test suite that exercises the client’s timeout
handling by forcing the fetch path to abort or reject with an `AbortError`, and
verify the function maps it to the expected timeout-related error shape. Use the
existing `getVermontContributionDetails` helper and its `fetchImpl`/`timeoutMs`
options so the new case validates the same deadline logic used by the scheduler
and CLI workers.
backend/tests/pipeline/vermontFinance/vermontOutsideSpendingAggregator.test.ts (1)

45-151: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Add an explicit oppose-case.

This suite never exercises a row that should land in outsideOpposeTotal or emit supportOppose: "oppose". A regression in the oppose classification branch would still pass here while the ballot finance summary exposes both sides.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@backend/tests/pipeline/vermontFinance/vermontOutsideSpendingAggregator.test.ts`
around lines 45 - 151, The vermontOutsideSpendingAggregator test suite only
covers support paths, so add an explicit oppose-case to exercise the branch that
produces outsideOpposeTotal and supportOppose: "oppose". Update
vermontOutsideSpendingAggregator.test.ts by adding a scenario through
aggregateVermontOutsideSpending with a row that should classify as oppose, then
assert the summary totals and group fields reflect the oppose side. Use the
existing aggregateVermontOutsideSpending and summary.groups assertions to verify
the new classification.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@backend/src/config/featureFlags.ts`:
- Around line 230-233: The sync toggle in isVermontCampaignFinanceSyncEnabled is
still coupled to isVermontCampaignFinanceEnabled, so the sync-specific env flag
and force option cannot bypass the public rollout gate. Update the feature flag
logic in isVermontCampaignFinanceSyncEnabled to evaluate
VERMONT_CAMPAIGN_FINANCE_SYNC_ENABLED and force independently of
isVermontCampaignFinanceEnabled, using the existing helper names to keep the
sync path enabled for prewarming/backfills even when the public Vermont flag is
off.

In `@backend/src/pipeline/address/ballotLookup.ts`:
- Around line 6984-7007: The optional-module guard in
loadVermontCandidateFinanceSummariesByCandidateElection is too broad because
isOptionalModuleMissing currently treats any ERR_MODULE_NOT_FOUND from the
dynamic import path as optional. Narrow the check so only the exact top-level
import of VERMONT_BALLOT_LOOKUP_FINANCE_LOADER_PATH is swallowed, and let
transitive missing imports inside the loader rethrow; update the catch logic
around import(VERMONT_BALLOT_LOOKUP_FINANCE_LOADER_PATH) accordingly.

In `@backend/src/pipeline/vermontFinance/vermontCampaignFinanceClient.ts`:
- Around line 627-650: The endpoint-specific helpers in
getVermontContributionDetails and getVermontExpenditureDetails currently allow
mismatched transactionTypeCode values from input, which can send the wrong query
to the API. Update these helpers to either ignore any provided
transactionTypeCode and always use the matching hardcoded value for each
endpoint, or explicitly validate and throw when input.transactionTypeCode does
not match the helper’s expected type. Use the existing
getVermontContributionDetails, getVermontExpenditureDetails, and
buildVermontTransactionSearchPayload symbols to locate the change.

In `@backend/src/pipeline/vermontFinance/vermontCandidateCommitteeResolver.ts`:
- Around line 28-32: `officeName` is being used as both the lookup key and the
returned display label in `vermontCandidateCommitteeResolver`, which breaks
round-tripping. Update the resolver API so the input and output distinguish the
canonical office identifier from the mapped display name, and make
`mapVermontOfficeSought`/resolver entrypoints use the canonical value for
lookups while only presenting the display label in results. Ensure all resolver
paths that currently accept or return `officeName` are updated consistently so
reusing a prior result does not fall into `unsupported_office` for offices like
`Auditor of Accounts` and `State Representative`.
- Around line 358-384: The fallback candidate lookup in
vermontCandidateCommitteeResolver currently only fetches page 1 from
getVermontContributionDetails and getVermontExpenditureDetails, so valid matches
beyond the first 100 rows can be missed. Update the search loop around
candidateSearchPhrases to paginate through both result sets until no more rows
are returned, and keep appending via appendUniqueRows into rowsByGuid before
deciding no_candidate_committee_match. Use the existing
getVermontContributionDetails, getVermontExpenditureDetails, and
appendUniqueRows symbols to locate the logic.

In `@backend/src/pipeline/vermontFinance/vermontCandidateFinanceSync.ts`:
- Around line 185-198: The pagination helper fetchAllTransactionRows stops after
maxPages even when page.totalItems shows there are more rows, which can lead to
incomplete snapshot replacements. Update fetchAllTransactionRows and the
VermontCandidateFinanceSync call path so pagination-cap reached cases are
treated as a failure condition instead of returning partial rows; surface an
error before the snapshot write/replace logic runs, and ensure the caller that
aggregates finance totals does not persist undercounted data.

In `@backend/src/pipeline/vermontFinance/vermontOutsideSpendingAggregator.ts`:
- Around line 143-150: `toGroups()` is truncating the data too early, so the
summary totals for `outsideSupportTotal` and `outsideOpposeTotal` are being
computed from an already sliced list. Update the aggregation flow in
`vermontOutsideSpendingAggregator` so totals are calculated from the full
`groups` set before applying the `maxGroups` limit for display, and keep the
top-N slicing only for the returned group list. Use the `toGroups()` helper and
the `outsideGroups`/summary total computation path as the key places to adjust.

In `@backend/src/scheduler/vermontCandidateFinanceSyncScheduler.ts`:
- Around line 83-87: The Redis connection options in
vermontCandidateFinanceSyncScheduler are being shared between worker and
producer/admin paths, but maxRetriesPerRequest: null should only apply to the
Worker. Split the shared ConnectionOptions setup so the worker-specific override
stays on the worker connection, while the Queue.add() and upsertJobScheduler()
paths use a separate producer/admin connection without that retry setting.

In `@backend/src/scripts/probeVermontCandidateFinance.ts`:
- Around line 575-583: The committee resolution call in
probeVermontCandidateFinance currently drops the parsed district, which can lead
to incorrect matches for state_upper/state_lower candidates. Update the
resolveCandidateCommittee invocation to forward input.args.district alongside
candidateName, officeScope, officeName, and electionYear, using the existing
parseProbeVermontCandidateFinanceArgs and client.resolveCandidateCommittee flow.
Add or update a district-scoped case in
backend/tests/scripts/probeVermontCandidateFinance.test.ts to verify the
district is preserved through resolution.

In `@backend/tests/pipeline/vermontFinance/vermontCandidateFinanceSync.test.ts`:
- Around line 19-27: The transaction mock in createMockDb currently reuses the
same query spy for both db.query and client.query, which can hide
pool-versus-client misuse in the sync path. Split these into separate mocks,
keep connect() returning the transactional client, and update the
vermontCandidateFinanceSync tests to assert the write SQL is executed through
client.query rather than db.query.

In `@backend/tests/pipeline/vermontFinance/vermontFinanceWriter.test.ts`:
- Around line 19-27: Separate the pool and transactional client query spies in
createTransactionalMockDb so db.query and client.query are different mocks.
Update the test setup in vermontFinanceWriter.test.ts to verify transactional
behavior specifically through the client returned by connect(), ensuring
BEGIN/INSERT/COMMIT are asserted on client.query and not accidentally satisfied
by pool-level db.query. Use the createTransactionalMockDb helper and the
client/query symbols to keep the intent clear.

---

Nitpick comments:
In `@backend/tests/pipeline/vermontFinance/vermontCampaignFinanceClient.test.ts`:
- Around line 384-405: The `getVermontContributionDetails` failure-mapping test
covers HTTP, API envelope, and malformed JSON cases but misses the timeout/abort
path. Add an assertion in this test suite that exercises the client’s timeout
handling by forcing the fetch path to abort or reject with an `AbortError`, and
verify the function maps it to the expected timeout-related error shape. Use the
existing `getVermontContributionDetails` helper and its `fetchImpl`/`timeoutMs`
options so the new case validates the same deadline logic used by the scheduler
and CLI workers.

In
`@backend/tests/pipeline/vermontFinance/vermontOutsideGroupContributionAggregator.test.ts`:
- Around line 142-203: The current test only exercises a single Vermont API
page, so it does not verify pagination handling in
fetchAndAggregateVermontOutsideGroupContributions. Expand this test to cover
multiple pages by having fetchImpl return page 1 and page 2 results for the same
filerRegistrationGuid, then assert the aggregated fetched/matched/included
counts and outsideGroupBreakdowns reflect both pages. Use
fetchAndAggregateVermontOutsideGroupContributions and fetchImpl to locate the
behavior, and make sure the request assertions confirm pageNumber advances
correctly until all pages are consumed.

In
`@backend/tests/pipeline/vermontFinance/vermontOutsideSpendingAggregator.test.ts`:
- Around line 45-151: The vermontOutsideSpendingAggregator test suite only
covers support paths, so add an explicit oppose-case to exercise the branch that
produces outsideOpposeTotal and supportOppose: "oppose". Update
vermontOutsideSpendingAggregator.test.ts by adding a scenario through
aggregateVermontOutsideSpending with a row that should classify as oppose, then
assert the summary totals and group fields reflect the oppose side. Use the
existing aggregateVermontOutsideSpending and summary.groups assertions to verify
the new classification.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 75133335-fd36-4904-9d8d-7686dee14395

📥 Commits

Reviewing files that changed from the base of the PR and between e98ec97 and 70460d4.

📒 Files selected for processing (34)
  • backend/package.json
  • backend/src/config/featureFlags.ts
  • backend/src/pipeline/address/ballotLookup.ts
  • backend/src/pipeline/vermontFinance/index.ts
  • backend/src/pipeline/vermontFinance/vermontBallotLookupFinanceLoader.ts
  • backend/src/pipeline/vermontFinance/vermontCampaignFinanceClient.ts
  • backend/src/pipeline/vermontFinance/vermontCandidateCommitteeResolver.ts
  • backend/src/pipeline/vermontFinance/vermontCandidateFinanceBatchSync.ts
  • backend/src/pipeline/vermontFinance/vermontCandidateFinanceSync.ts
  • backend/src/pipeline/vermontFinance/vermontDirectContributionAggregator.ts
  • backend/src/pipeline/vermontFinance/vermontFinanceEligibleOffices.ts
  • backend/src/pipeline/vermontFinance/vermontFinanceWriter.ts
  • backend/src/pipeline/vermontFinance/vermontOutsideGroupContributionAggregator.ts
  • backend/src/pipeline/vermontFinance/vermontOutsideSpendingAggregator.ts
  • backend/src/scheduler/vermontCandidateFinanceSyncScheduler.ts
  • backend/src/scripts/probeVermontCandidateFinance.ts
  • backend/src/scripts/runVermontCandidateFinanceSyncSchedulerWorker.ts
  • backend/src/scripts/syncDueVermontCandidateFinance.ts
  • backend/src/scripts/triggerVermontCandidateFinanceSync.ts
  • backend/src/scripts/upsertVermontCandidateFinanceSyncScheduler.ts
  • backend/tests/pipeline/address/ballotLookup.test.ts
  • backend/tests/pipeline/vermontFinance/vermontCampaignFinanceClient.test.ts
  • backend/tests/pipeline/vermontFinance/vermontCandidateCommitteeResolver.test.ts
  • backend/tests/pipeline/vermontFinance/vermontCandidateFinanceBatchSync.test.ts
  • backend/tests/pipeline/vermontFinance/vermontCandidateFinanceSync.test.ts
  • backend/tests/pipeline/vermontFinance/vermontDirectContributionAggregator.test.ts
  • backend/tests/pipeline/vermontFinance/vermontFinanceWriter.test.ts
  • backend/tests/pipeline/vermontFinance/vermontOutsideGroupContributionAggregator.test.ts
  • backend/tests/pipeline/vermontFinance/vermontOutsideSpendingAggregator.test.ts
  • backend/tests/scheduler/vermontCandidateFinanceSyncScheduler.test.ts
  • backend/tests/scripts/probeVermontCandidateFinance.test.ts
  • backend/tests/scripts/syncDueVermontCandidateFinance.test.ts
  • backend/tests/scripts/triggerVermontCandidateFinanceSync.test.ts
  • db/migrations/127_add_vermont_campaign_finance_tables.sql

Comment on lines +230 to +233
export function isVermontCampaignFinanceSyncEnabled(force = false): boolean {
return (
isVermontCampaignFinanceEnabled() &&
(force || readBooleanEnv("VERMONT_CAMPAIGN_FINANCE_SYNC_ENABLED", false))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Decouple sync gating from the public rollout flag.

isVermontCampaignFinanceSyncEnabled() still requires isVermontCampaignFinanceEnabled(), so VERMONT_CAMPAIGN_FINANCE_SYNC_ENABLED=true or --force has no effect while the public Vermont flag is off. That makes the sync-specific flag ineffective and blocks prewarming/backfills before ballot lookup rollout.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@backend/src/config/featureFlags.ts` around lines 230 - 233, The sync toggle
in isVermontCampaignFinanceSyncEnabled is still coupled to
isVermontCampaignFinanceEnabled, so the sync-specific env flag and force option
cannot bypass the public rollout gate. Update the feature flag logic in
isVermontCampaignFinanceSyncEnabled to evaluate
VERMONT_CAMPAIGN_FINANCE_SYNC_ENABLED and force independently of
isVermontCampaignFinanceEnabled, using the existing helper names to keep the
sync path enabled for prewarming/backfills even when the public Vermont flag is
off.

Comment on lines +6984 to +7007
function isOptionalModuleMissing(error: unknown): boolean {
if (!(error instanceof Error)) {
return false;
}
const code = "code" in error ? String(error.code) : "";
return code === "ERR_MODULE_NOT_FOUND" || code === "MODULE_NOT_FOUND";
}

async function loadVermontCandidateFinanceSummariesByCandidateElection(
db: Queryable,
candidateRows: readonly CandidateRow[],
electionRows: readonly ElectionRow[]
): Promise<Map<string, BallotLookupFinanceSummary>> {
if (!isVermontCampaignFinanceEnabled()) {
return new Map();
}

try {
const loader = (await import(VERMONT_BALLOT_LOOKUP_FINANCE_LOADER_PATH)) as VermontBallotLookupFinanceLoaderModule;
return await loader.loadVermontCandidateFinanceSummariesByCandidateElection(db, candidateRows, electionRows);
} catch (error) {
if (isOptionalModuleMissing(error)) {
console.warn("Vermont campaign finance ballot lookup loader is unavailable; skipping Vermont finance summaries");
return new Map();

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

tmpdir="$(mktemp -d)"
cat > "$tmpdir/loader.mjs" <<'EOF'
import "./nested-missing.mjs";
export const ok = true;
EOF

node --input-type=module <<NODE
try {
  await import("file://$tmpdir/loader.mjs");
} catch (error) {
  console.log("code:", error.code);
  console.log("message:", error.message);
}
NODE

Repository: shu1513/voteapp

Length of output: 1982


🌐 Web query:

Node.js ESM import missing dependency error.message code ERR_MODULE_NOT_FOUND includes missing module specifier and parent path shape

💡 Result:

The Node.js ERR_MODULE_NOT_FOUND error is thrown by the ECMAScript module (ESM) loader when a requested module specifier cannot be resolved to an existing file or package [1][2]. The error message typically follows this format: Error [ERR_MODULE_NOT_FOUND]: Cannot find package '' imported from <parent_path> [3][4][5] Components of the error message: 1. Cannot find module/package '': This indicates the exact string used in the import statement that failed to resolve [3][5]. 2. imported from <parent_path>: This provides the absolute file system path of the module (the parent) that contained the failing import statement [3][4][5]. Key technical details: - ESM resolution: Unlike CommonJS, the ESM loader does not automatically append file extensions (like.js or.json) or look for index files in directories [6]. The specifier must resolve to a precise location [7][8]. - Cause: This error occurs when the specified path does not exist, the package is not defined in package.json exports/imports, or the module format is unsupported [1][8]. - Troubleshooting: Ensure the import specifier includes the correct file extension (e.g.,.js,.mjs), uses a relative path (starting with./ or../) if importing a local file, and that the file actually exists at the resolved location [9][8][6].

Citations:


Only exempt the missing top-level loader import.

ERR_MODULE_NOT_FOUND includes both the missing specifier and the importing file, so a substring check against VERMONT_BALLOT_LOOKUP_FINANCE_LOADER_PATH still swallows transitive missing imports inside that loader. Narrow this to the exact dynamic import target instead of any module-not-found on the loader's message.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@backend/src/pipeline/address/ballotLookup.ts` around lines 6984 - 7007, The
optional-module guard in loadVermontCandidateFinanceSummariesByCandidateElection
is too broad because isOptionalModuleMissing currently treats any
ERR_MODULE_NOT_FOUND from the dynamic import path as optional. Narrow the check
so only the exact top-level import of VERMONT_BALLOT_LOOKUP_FINANCE_LOADER_PATH
is swallowed, and let transitive missing imports inside the loader rethrow;
update the catch logic around import(VERMONT_BALLOT_LOOKUP_FINANCE_LOADER_PATH)
accordingly.

Comment on lines +627 to +650
export async function getVermontContributionDetails(
input: VermontTransactionSearchInput = {},
options?: VermontCampaignFinanceClientOptions
): Promise<VermontPagedResult<VermontContributionRow>> {
const payload = buildVermontTransactionSearchPayload({ ...input, transactionTypeCode: input.transactionTypeCode ?? "TCON" });
const response = await postVermontCampaignFinanceJson(
"PublicTransactionDetails/GetContributionsDetails",
payload,
options
);
return parsePagedData(response, "contribution details", parseContributionRow);
}

export async function getVermontExpenditureDetails(
input: VermontTransactionSearchInput = {},
options?: VermontCampaignFinanceClientOptions
): Promise<VermontPagedResult<VermontExpenditureRow>> {
const payload = buildVermontTransactionSearchPayload({ ...input, transactionTypeCode: input.transactionTypeCode ?? "TEXP" });
const response = await postVermontCampaignFinanceJson(
"PublicTransactionDetails/GetExpenditureDetails",
payload,
options
);
return parsePagedData(response, "expenditure details", parseExpenditureRow);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Reject mismatched transactionTypeCode values on endpoint-specific helpers.

These helpers accept input.transactionTypeCode, so callers can send "TEXP" to the contributions endpoint or "TCON" to the expenditure endpoint. That turns both public APIs into easy wrong-result footguns; hardcode the matching type or throw on mismatch.

Proposed fix
 export async function getVermontContributionDetails(
   input: VermontTransactionSearchInput = {},
   options?: VermontCampaignFinanceClientOptions
 ): Promise<VermontPagedResult<VermontContributionRow>> {
-  const payload = buildVermontTransactionSearchPayload({ ...input, transactionTypeCode: input.transactionTypeCode ?? "TCON" });
+  if (input.transactionTypeCode && input.transactionTypeCode !== "TCON") {
+    throw new VermontCampaignFinanceClientError(
+      "invalid_request",
+      "Contribution details only support transactionTypeCode=TCON"
+    );
+  }
+  const payload = buildVermontTransactionSearchPayload({ ...input, transactionTypeCode: "TCON" });
   const response = await postVermontCampaignFinanceJson(
     "PublicTransactionDetails/GetContributionsDetails",
     payload,
     options
   );
@@
 export async function getVermontExpenditureDetails(
   input: VermontTransactionSearchInput = {},
   options?: VermontCampaignFinanceClientOptions
 ): Promise<VermontPagedResult<VermontExpenditureRow>> {
-  const payload = buildVermontTransactionSearchPayload({ ...input, transactionTypeCode: input.transactionTypeCode ?? "TEXP" });
+  if (input.transactionTypeCode && input.transactionTypeCode !== "TEXP") {
+    throw new VermontCampaignFinanceClientError(
+      "invalid_request",
+      "Expenditure details only support transactionTypeCode=TEXP"
+    );
+  }
+  const payload = buildVermontTransactionSearchPayload({ ...input, transactionTypeCode: "TEXP" });
   const response = await postVermontCampaignFinanceJson(
     "PublicTransactionDetails/GetExpenditureDetails",
     payload,
     options
   );
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
export async function getVermontContributionDetails(
input: VermontTransactionSearchInput = {},
options?: VermontCampaignFinanceClientOptions
): Promise<VermontPagedResult<VermontContributionRow>> {
const payload = buildVermontTransactionSearchPayload({ ...input, transactionTypeCode: input.transactionTypeCode ?? "TCON" });
const response = await postVermontCampaignFinanceJson(
"PublicTransactionDetails/GetContributionsDetails",
payload,
options
);
return parsePagedData(response, "contribution details", parseContributionRow);
}
export async function getVermontExpenditureDetails(
input: VermontTransactionSearchInput = {},
options?: VermontCampaignFinanceClientOptions
): Promise<VermontPagedResult<VermontExpenditureRow>> {
const payload = buildVermontTransactionSearchPayload({ ...input, transactionTypeCode: input.transactionTypeCode ?? "TEXP" });
const response = await postVermontCampaignFinanceJson(
"PublicTransactionDetails/GetExpenditureDetails",
payload,
options
);
return parsePagedData(response, "expenditure details", parseExpenditureRow);
export async function getVermontContributionDetails(
input: VermontTransactionSearchInput = {},
options?: VermontCampaignFinanceClientOptions
): Promise<VermontPagedResult<VermontContributionRow>> {
if (input.transactionTypeCode && input.transactionTypeCode !== "TCON") {
throw new VermontCampaignFinanceClientError(
"invalid_request",
"Contribution details only support transactionTypeCode=TCON"
);
}
const payload = buildVermontTransactionSearchPayload({ ...input, transactionTypeCode: "TCON" });
const response = await postVermontCampaignFinanceJson(
"PublicTransactionDetails/GetContributionsDetails",
payload,
options
);
return parsePagedData(response, "contribution details", parseContributionRow);
}
export async function getVermontExpenditureDetails(
input: VermontTransactionSearchInput = {},
options?: VermontCampaignFinanceClientOptions
): Promise<VermontPagedResult<VermontExpenditureRow>> {
if (input.transactionTypeCode && input.transactionTypeCode !== "TEXP") {
throw new VermontCampaignFinanceClientError(
"invalid_request",
"Expenditure details only support transactionTypeCode=TEXP"
);
}
const payload = buildVermontTransactionSearchPayload({ ...input, transactionTypeCode: "TEXP" });
const response = await postVermontCampaignFinanceJson(
"PublicTransactionDetails/GetExpenditureDetails",
payload,
options
);
return parsePagedData(response, "expenditure details", parseExpenditureRow);
}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@backend/src/pipeline/vermontFinance/vermontCampaignFinanceClient.ts` around
lines 627 - 650, The endpoint-specific helpers in getVermontContributionDetails
and getVermontExpenditureDetails currently allow mismatched transactionTypeCode
values from input, which can send the wrong query to the API. Update these
helpers to either ignore any provided transactionTypeCode and always use the
matching hardcoded value for each endpoint, or explicitly validate and throw
when input.transactionTypeCode does not match the helper’s expected type. Use
the existing getVermontContributionDetails, getVermontExpenditureDetails, and
buildVermontTransactionSearchPayload symbols to locate the change.

Comment on lines +28 to +32
export type VermontCandidateCommitteeResolverInput = {
candidateName: string;
officeScope: string;
officeName: string;
electionYear: number;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

officeName is not round-trippable across this resolver API.

The input treats officeName as the canonical lookup key, but matched results return officeName as the display label from mapVermontOfficeSought. Reusing a returned officeName in either resolver entrypoint will therefore come back as unsupported_office for cases like Auditor of Accounts and State Representative.

Also applies to: 41-46, 239-242, 287-289, 345-348

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@backend/src/pipeline/vermontFinance/vermontCandidateCommitteeResolver.ts`
around lines 28 - 32, `officeName` is being used as both the lookup key and the
returned display label in `vermontCandidateCommitteeResolver`, which breaks
round-tripping. Update the resolver API so the input and output distinguish the
canonical office identifier from the mapped display name, and make
`mapVermontOfficeSought`/resolver entrypoints use the canonical value for
lookups while only presenting the display label in results. Ensure all resolver
paths that currently accept or return `officeName` are updated consistently so
reusing a prior result does not fall into `unsupported_office` for offices like
`Auditor of Accounts` and `State Representative`.

Comment on lines +358 to +384
const rowsByGuid = new Map<string, VermontCandidateCommitteeTransactionRow>();
for (const filerName of candidateSearchPhrases) {
const [contributions, expenditures] = await Promise.all([
getVermontContributionDetails(
{
pageNumber: 1,
pageSize: 100,
filerName,
electionYear: input.electionYear,
transactionTypeCode: "TCON",
},
options
),
getVermontExpenditureDetails(
{
pageNumber: 1,
pageSize: 100,
filerName,
electionYear: input.electionYear,
transactionTypeCode: "TEXP",
},
options
),
]);
appendUniqueRows(rowsByGuid, contributions.items);
appendUniqueRows(rowsByGuid, expenditures.items);
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Paginate candidate searches before resolving.

Each phrase only reads page 1 from contributions and expenditures. The fallback last-name search can easily exceed 100 rows, so a valid committee on page 2+ gets reported as no_candidate_committee_match.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@backend/src/pipeline/vermontFinance/vermontCandidateCommitteeResolver.ts`
around lines 358 - 384, The fallback candidate lookup in
vermontCandidateCommitteeResolver currently only fetches page 1 from
getVermontContributionDetails and getVermontExpenditureDetails, so valid matches
beyond the first 100 rows can be missed. Update the search loop around
candidateSearchPhrases to paginate through both result sets until no more rows
are returned, and keep appending via appendUniqueRows into rowsByGuid before
deciding no_candidate_committee_match. Use the existing
getVermontContributionDetails, getVermontExpenditureDetails, and
appendUniqueRows symbols to locate the logic.

Comment on lines +143 to +150
function toGroups(input: {
groups: Iterable<GroupAccumulator>;
sourceUrl: string | null;
maxGroups: number;
}): VermontOutsideSpendingGroup[] {
return [...input.groups]
.sort((left, right) => right.amountCents - left.amountCents || left.filerName.localeCompare(right.filerName))
.slice(0, input.maxGroups)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

maxGroups currently truncates the summary totals.

outsideSupportTotal and outsideOpposeTotal are computed from outsideGroups, but toGroups() has already sliced to the top maxGroups. Once there are more groups than the display limit, the summary under-reports actual outside spending.

Also applies to: 224-235

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@backend/src/pipeline/vermontFinance/vermontOutsideSpendingAggregator.ts`
around lines 143 - 150, `toGroups()` is truncating the data too early, so the
summary totals for `outsideSupportTotal` and `outsideOpposeTotal` are being
computed from an already sliced list. Update the aggregation flow in
`vermontOutsideSpendingAggregator` so totals are calculated from the full
`groups` set before applying the `maxGroups` limit for display, and keep the
top-N slicing only for the returned group list. Use the `toGroups()` helper and
the `outsideGroups`/summary total computation path as the key places to adjust.

Comment on lines +83 to +87
const opts: ConnectionOptions = {
host: parsed.hostname,
port: parsedPort,
db: parsedDb,
maxRetriesPerRequest: null,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Inspect the scheduler module and nearby call sites.
FILE="backend/src/scheduler/vermontCandidateFinanceSyncScheduler.ts"
wc -l "$FILE"
sed -n '1,220p' "$FILE"

echo
echo "=== package references ==="
rg -n "bullmq|ioredis|maxRetriesPerRequest|createVermontCandidateFinanceSyncScheduler" backend package.json . -g '!node_modules' -g '!dist' -g '!build' || true

echo
echo "=== relevant queue/worker constructors ==="
rg -n "new (Queue|Worker)|upsertJobScheduler|add\\(" backend/src -g '!node_modules' -g '!dist' -g '!build' || true

Repository: shu1513/voteapp

Length of output: 1952


🌐 Web query:

BullMQ ioredis maxRetriesPerRequest Queue Worker connection options official docs

💡 Result:

In BullMQ, the ioredis option maxRetriesPerRequest must be set to null when creating connections for Workers [1][2][3]. Why it is required for Workers Workers in BullMQ rely on blocking Redis commands to monitor the queue for new jobs [1][4]. If maxRetriesPerRequest is not set to null, these blocking commands may fail or behave unpredictably when a temporary disconnection occurs, potentially breaking the worker's ability to process jobs [4]. Setting this to null ensures the worker will keep attempting to execute commands indefinitely until the connection is restored [1]. How to configure it 1. When providing an existing ioredis instance: You must manually set maxRetriesPerRequest: null in the ioredis constructor [1][2][3]. const connection = new IORedis({ host: 'localhost', port: 6379, maxRetriesPerRequest: null, // Required for Workers }); const worker = new Worker('myQueue', processor, { connection }); 2. When providing connection options directly: If you pass a configuration object instead of an instance, BullMQ will automatically handle the configuration for you in most cases [2][3]. However, if you are passing an instance that is shared or pre-configured, you are responsible for ensuring this setting is correct [1][5]. Difference between Queue and Worker It is a common best practice to treat Queue and Worker connections differently [4]: - Queue instances (used to add jobs, pause queues, etc.) typically function better with a default or low maxRetriesPerRequest value, so that they fail quickly if Redis is unreachable [1][4]. - Worker instances require maxRetriesPerRequest: null to maintain their persistent, blocking connection to Redis [1][4]. Warning: If you provide an existing ioredis instance to a Worker with maxRetriesPerRequest set to anything other than null, BullMQ will emit a warning, as this configuration can lead to unintended side effects for worker functionality [1][4].

Citations:


Split the queue/admin Redis options from the worker connection. maxRetriesPerRequest: null is required for the Worker, but reusing it for Queue.add() / upsertJobScheduler() makes the trigger and scheduler-management paths slower to fail when Redis is unavailable. Keep the worker override separate from the producer/admin connection.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@backend/src/scheduler/vermontCandidateFinanceSyncScheduler.ts` around lines
83 - 87, The Redis connection options in vermontCandidateFinanceSyncScheduler
are being shared between worker and producer/admin paths, but
maxRetriesPerRequest: null should only apply to the Worker. Split the shared
ConnectionOptions setup so the worker-specific override stays on the worker
connection, while the Queue.add() and upsertJobScheduler() paths use a separate
producer/admin connection without that retry setting.

Comment on lines +575 to +583
const resolution = await client.resolveCandidateCommittee(
{
candidateName: input.args.candidateName,
officeScope: input.args.officeScope,
officeName: input.args.officeName,
electionYear: input.args.electionYear,
},
clientOptions
);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Forward --district into committee resolution.

parseProbeVermontCandidateFinanceArgs() captures district, but this resolver call drops it. For state_upper/state_lower probes, that can match the wrong committee when the same candidate name appears in multiple districts.

Suggested fix
   const resolution = await client.resolveCandidateCommittee(
     {
       candidateName: input.args.candidateName,
       officeScope: input.args.officeScope,
       officeName: input.args.officeName,
+      district: input.args.district,
       electionYear: input.args.electionYear,
     },
     clientOptions
   );

A district-scoped regression in backend/tests/scripts/probeVermontCandidateFinance.test.ts would help lock this in.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
const resolution = await client.resolveCandidateCommittee(
{
candidateName: input.args.candidateName,
officeScope: input.args.officeScope,
officeName: input.args.officeName,
electionYear: input.args.electionYear,
},
clientOptions
);
const resolution = await client.resolveCandidateCommittee(
{
candidateName: input.args.candidateName,
officeScope: input.args.officeScope,
officeName: input.args.officeName,
district: input.args.district,
electionYear: input.args.electionYear,
},
clientOptions
);
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@backend/src/scripts/probeVermontCandidateFinance.ts` around lines 575 - 583,
The committee resolution call in probeVermontCandidateFinance currently drops
the parsed district, which can lead to incorrect matches for
state_upper/state_lower candidates. Update the resolveCandidateCommittee
invocation to forward input.args.district alongside candidateName, officeScope,
officeName, and electionYear, using the existing
parseProbeVermontCandidateFinanceArgs and client.resolveCandidateCommittee flow.
Add or update a district-scoped case in
backend/tests/scripts/probeVermontCandidateFinance.test.ts to verify the
district is preserved through resolution.

Comment on lines +19 to +27
function createMockDb() {
const query = vi.fn().mockResolvedValue({ rows: [{ id: LINK_ID }], rowCount: 1 });
const client = { query, release: vi.fn() };
return {
query,
connect: vi.fn().mockResolvedValue(client),
client,
};
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

This mock also hides pool-vs-client transaction misuse.

db.query and client.query point to the same spy, so the sync tests still pass if the implementation accidentally mixes pool-level queries into the transactional write path. Split those mocks and assert the SQL runs through the connected client.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@backend/tests/pipeline/vermontFinance/vermontCandidateFinanceSync.test.ts`
around lines 19 - 27, The transaction mock in createMockDb currently reuses the
same query spy for both db.query and client.query, which can hide
pool-versus-client misuse in the sync path. Split these into separate mocks,
keep connect() returning the transactional client, and update the
vermontCandidateFinanceSync tests to assert the write SQL is executed through
client.query rather than db.query.

Comment on lines +19 to +27
function createTransactionalMockDb() {
const query = vi.fn().mockResolvedValue({ rows: [{ id: LINK_ID }], rowCount: 1 });
const client = { query, release: vi.fn() };
return {
query,
connect: vi.fn().mockResolvedValue(client),
client,
};
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

Separate the pool and client query spies.

createTransactionalMockDb() reuses one mock for both db.query and client.query, so these tests cannot catch an implementation that opens a client and then runs BEGIN/INSERT/COMMIT on the pool instead of the transaction-bound client. That would still pass here while breaking the atomicity this writer is supposed to guarantee.

Proposed fix
 function createTransactionalMockDb() {
-  const query = vi.fn().mockResolvedValue({ rows: [{ id: LINK_ID }], rowCount: 1 });
-  const client = { query, release: vi.fn() };
+  const poolQuery = vi.fn();
+  const clientQuery = vi.fn().mockResolvedValue({ rows: [{ id: LINK_ID }], rowCount: 1 });
+  const client = { query: clientQuery, release: vi.fn() };
   return {
-    query,
+    query: poolQuery,
     connect: vi.fn().mockResolvedValue(client),
     client,
   };
 }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
function createTransactionalMockDb() {
const query = vi.fn().mockResolvedValue({ rows: [{ id: LINK_ID }], rowCount: 1 });
const client = { query, release: vi.fn() };
return {
query,
connect: vi.fn().mockResolvedValue(client),
client,
};
}
function createTransactionalMockDb() {
const poolQuery = vi.fn();
const clientQuery = vi.fn().mockResolvedValue({ rows: [{ id: LINK_ID }], rowCount: 1 });
const client = { query: clientQuery, release: vi.fn() };
return {
query: poolQuery,
connect: vi.fn().mockResolvedValue(client),
client,
};
}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@backend/tests/pipeline/vermontFinance/vermontFinanceWriter.test.ts` around
lines 19 - 27, Separate the pool and transactional client query spies in
createTransactionalMockDb so db.query and client.query are different mocks.
Update the test setup in vermontFinanceWriter.test.ts to verify transactional
behavior specifically through the client returned by connect(), ensuring
BEGIN/INSERT/COMMIT are asserted on client.query and not accidentally satisfied
by pool-level db.query. Use the createTransactionalMockDb helper and the
client/query symbols to keep the intent clear.

@shu1513
shu1513 merged commit 603f4dd into main Jun 28, 2026
2 checks passed
@shu1513
shu1513 deleted the codex/vermont-finance-phase1 branch June 28, 2026 05:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant