Repository navigation
fix: drop hard CPU caps; equal PostgREST memory share on ≥2GB instances - #108
Conversation
…=2GB instances Palise Tools outage (feedback 2def74a5): postgrest on a t4g.medium was capped at 0.3 vCPU / 544MB while the host sat idle - 16.8k throttled cgroup periods and a heap high-water mark at the memory cap produced 10s proxy timeouts under modest concurrency. Container quotas, not instance class, were the ceiling. - Remove hard cpus: limits from all services. Single-tenant hosts: caps only prevented bursting into idle cores; under contention the kernel's equal cpu.weight default arbitrates. Memory caps stay (host OOM protection). - POSTGREST_BASE 50 -> 150 on instances >=1800MB so postgrest gets an equal third (a 4GB medium: 544MB -> 1268MB, GHC heap cap 1248M). Tiny instances keep the legacy split - the 1.0 scale-factor clamp leaves no slack there. - Keep legacy *_CPUS= keys in the .env cleanup so upgraded instances shed them. Verified per-tier via simulation (460MB-15.7GB): nano/micro allocations are byte-identical to current behavior; every tier stays within usable memory. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BpSh9Qb69rJYnKEdMDWPph
WalkthroughThe pull request changes resource scaling to memory-only behavior. The script selects PostgREST memory based on total RAM, corrects rounded allocations, and preserves memory settings in ChangesResource scaling
Estimated code review effort: 3 (Moderate) | ~15–30 minutes Possibly related PRs
Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
jwfing
left a comment
There was a problem hiding this comment.
Review: fix — drop hard CPU caps; equal PostgREST memory share on ≥2GB instances
Summary: A well-reasoned, tightly-scoped ops fix that removes container CPU caps on single-tenant hosts and rebalances the memory split so PostgREST gets an equal third on ≥1800MB instances — the allocator math and safety claims check out, no blocking issues.
Requirements context
No /docs/superpowers/ (or docs/specs/) directory exists in this repo — it's a minimal deployment repo (just auto-scale-memory.sh, docker-compose.yml, docker-init/, functions/), with no test harness or CI. No matching spec/plan found — assessed against the PR description, the linked incident evidence, and the surrounding code alone. I independently reproduced the verification table and confirmed the postgres-cap safety argument against docker-init/db/postgresql.conf.
Findings
Critical
(none)
Suggestion
-
Functionality — postgres/insforge memory caps shrink ~22% as a side-effect of raising
POSTGREST_BASE(auto-scale-memory.sh:32-45,75-77). RaisingTOTAL_BASE350→450 lowersSCALE_FACTOR, so on a medium the postgres and insforge caps drop 1631→1268MB, not just postgrest's rise. The postgres side is justified and verified:postgresql.confis static (shared_buffers=32MB,work_mem=2MB,maintenance_work_mem=16MB), so its RSS doesn't track the cgroup limit (matches the 437MB observed). One residual note: the composecommandruns postgres atmax_connections=${PG_MAX_CONNECTIONS:-100}= 150 on a medium (from #98); 150 backends × ~5–10MB aggregate RSS sits closer to the new 1268MB cap than it did to 1631MB. Empirically the margin is huge (437MB observed, pooled/idle-light connections), so this is not blocking — but a one-line note in the PR that the postgres cap reduction was weighed against the 150-connection ceiling (and that insforge's reduction is also fine) would close the reasoning gap the body leaves implicit. -
Software engineering — no committed simulation/assertion for the allocator (
auto-scale-memory.sh). The PR body describes simulating 460MB–15.7GB; that simulation isn't checked in. The repo has no test convention, so this is not a blocker, but a small checked-in shell harness asserting the tier table (nano184/61/184, medium1268/1268/1268, etc.) would guard against silent regressions the next time the split is tuned — this is the third consecutive change to this allocator (#98, GHC-heap cap, now CPU/rebalance).
Information
-
TOTAL_CPUS/nprocis now computed but only echoed (auto-scale-memory.sh:20,25). With CPU scaling gone,TOTAL_CPUSno longer feeds any allocation — it's retained only in the "Total system memory … CPUs: N" log line. Harmless; keep it for the log or drop it, author's call. -
Header-comment rationale for the 1800MB cutoff is slightly imprecise (
auto-scale-memory.sh:35-36). "Below ~2GB the scale factor clamps to 1.0 and there is no slack to redistribute" — at exactly 1800MB the factor is ~3.9, not clamped; the 1.0 clamp only bites far below (usable < base ≈ 450MB). The 1800 cutoff is really a deliberate policy choice for nano/micro tiers, which is fine — the comment just over-attributes it to the clamp. -
Security: No security-relevant changes. No new user input, no secrets newly logged or returned, no auth/authorization touched, no new dependencies. The
.envrewrite (backup + sed strip + append) is pre-existing behavior; the strip list now correctly sheds the legacy*_CPUS=/# CPU scaling factor:keys so upgraded instances don't carry stale caps. -
Performance: This is a performance fix and it's a net positive — removing hard CPU quotas lets containers burst into idle cores on single-tenant hosts (the 0.3-vCPU postgrest cap was the documented incident cause), and the larger postgrest heap (544→1268MB on medium, GHC
-Mcap 1248M) addresses the heap-at-cap timeouts. Memory caps are correctly retained for host OOM-killer protection.
Verdict
approved (informational — human approval is still a separate action). No Critical findings; the two Suggestions and the Information notes are non-blocking. The CPU-cap removal is safe for single-tenant hosts, the memory rebalance is arithmetically sound (reproduced the tier table byte-for-byte), and the reduced postgres cap is safe given the conf-driven postgresql.conf. Note this PR is currently marked draft.
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
auto-scale-memory.sh (1)
118-133: 🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick winHandle supported legacy
*_CPUSkeys explicitly. If older.envfiles can containDENO_CPUS,VECTOR_CPUS, orNODE_EXPORTER_CPUS, the current cleanup leaves them in place. Add only confirmed legacy keys.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@auto-scale-memory.sh` around lines 118 - 133, Update the cleanup command in the resource-settings flow to explicitly remove the confirmed legacy keys DENO_CPUS, VECTOR_CPUS, and NODE_EXPORTER_CPUS alongside the existing *_CPUS entries. Do not add any other unconfirmed legacy keys, and leave the generated settings block unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@auto-scale-memory.sh`:
- Around line 37-44: Update the allocation calculation in the auto-scaling flow
around TOTAL_BASE and TOTAL_ALLOCATED so independently rounded component
allocations cannot exceed USABLE_MEM; floor rounded values or reduce one
allocation to fit the remaining budget. Add an assertion that TOTAL_ALLOCATED is
less than or equal to USABLE_MEM before writing the .env file, preserving valid
allocations when the budget is sufficient.
- Around line 32-36: Update the comment near the PostgREST memory allocation
logic to accurately state that sub-2GB instances still scale SCALE_FACTOR above
1.0 while POSTGREST_BASE remains 50MB; remove the claim that the scale factor
clamps or that small instances retain a fixed legacy allocation.
---
Outside diff comments:
In `@auto-scale-memory.sh`:
- Around line 118-133: Update the cleanup command in the resource-settings flow
to explicitly remove the confirmed legacy keys DENO_CPUS, VECTOR_CPUS, and
NODE_EXPORTER_CPUS alongside the existing *_CPUS entries. Do not add any other
unconfirmed legacy keys, and leave the generated settings block unchanged.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 13661e82-54b3-4296-a2ae-a8884715d759
📒 Files selected for processing (2)
auto-scale-memory.shdocker-compose.yml
…strip more legacy CPU keys - Comment no longer claims the 1.0 scale-factor clamp drives the sub-2GB legacy base (factor is ~1.23 on a nano); the real reason is absolute slack. - Shave per-service round-up overshoot (up to 2MB near the 1800MB cutoff) off insforge so TOTAL_ALLOCATED never exceeds USABLE_MEM; verified at 1802MB -> 591/591/590 = 1772/1772. - .env cleanup also strips legacy DENO_CPUS/VECTOR_CPUS/NODE_EXPORTER_CPUS. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BpSh9Qb69rJYnKEdMDWPph
There was a problem hiding this comment.
2 issues found and verified against the latest diff
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="auto-scale-memory.sh">
<violation number="1" location="auto-scale-memory.sh:38">
P3: The new comment describing the POSTGREST_BASE threshold is misleading. It claims the scale factor "clamps to 1.0" below ~2GB and that small instances "keep the legacy base", but the actual clamp in this script only fires when usable memory drops below the base total (roughly <480MB), and the `>= 1800` check means the 1800–2000MB band (the whole "small" tier) does get the new equal-third base. Worth correcting the comment so an operator doesn't misread which instances keep the 50MB legacy base.</violation>
<violation number="2" location="auto-scale-memory.sh:38">
P2: With POSTGREST_BASE now equal to POSTGRES_BASE/INSFORGE_BASE (150MB) on instances >=1800MB, independently rounding each scaled allocation can push TOTAL_ALLOCATED slightly above USABLE_MEM near the threshold (e.g. TOTAL_MEM=1802MB yields TOTAL_ALLOCATED=1773MB against USABLE_MEM=1772MB). Consider flooring the allocations or adjusting one value after rounding, and assert TOTAL_ALLOCATED <= USABLE_MEM before writing .env.</violation>
</file>
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
| # cap after days of uptime. Below ~2GB there is little absolute slack — an | ||
| # equal third would shrink postgres/insforge caps on boxes where they are | ||
| # already tight — so tiny instances keep the legacy base. | ||
| if [ "$TOTAL_MEM" -ge 1800 ]; then |
There was a problem hiding this comment.
P2: With POSTGREST_BASE now equal to POSTGRES_BASE/INSFORGE_BASE (150MB) on instances >=1800MB, independently rounding each scaled allocation can push TOTAL_ALLOCATED slightly above USABLE_MEM near the threshold (e.g. TOTAL_MEM=1802MB yields TOTAL_ALLOCATED=1773MB against USABLE_MEM=1772MB). Consider flooring the allocations or adjusting one value after rounding, and assert TOTAL_ALLOCATED <= USABLE_MEM before writing .env.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At auto-scale-memory.sh, line 37:
<comment>With POSTGREST_BASE now equal to POSTGRES_BASE/INSFORGE_BASE (150MB) on instances >=1800MB, independently rounding each scaled allocation can push TOTAL_ALLOCATED slightly above USABLE_MEM near the threshold (e.g. TOTAL_MEM=1802MB yields TOTAL_ALLOCATED=1773MB against USABLE_MEM=1772MB). Consider flooring the allocations or adjusting one value after rounding, and assert TOTAL_ALLOCATED <= USABLE_MEM before writing .env.</comment>
<file context>
@@ -35,6 +29,21 @@ else
+# medium gave it 544MB against a 90-connection pool, with its heap grazing the
+# cap after days of uptime. Below ~2GB the scale factor clamps to 1.0 and
+# there is no slack to redistribute, so tiny instances keep the legacy base.
+if [ "$TOTAL_MEM" -ge 1800 ]; then
+ POSTGREST_BASE=150
+else
</file context>
| # cap after days of uptime. Below ~2GB there is little absolute slack — an | ||
| # equal third would shrink postgres/insforge caps on boxes where they are | ||
| # already tight — so tiny instances keep the legacy base. | ||
| if [ "$TOTAL_MEM" -ge 1800 ]; then |
There was a problem hiding this comment.
P3: The new comment describing the POSTGREST_BASE threshold is misleading. It claims the scale factor "clamps to 1.0" below ~2GB and that small instances "keep the legacy base", but the actual clamp in this script only fires when usable memory drops below the base total (roughly <480MB), and the >= 1800 check means the 1800–2000MB band (the whole "small" tier) does get the new equal-third base. Worth correcting the comment so an operator doesn't misread which instances keep the 50MB legacy base.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At auto-scale-memory.sh, line 37:
<comment>The new comment describing the POSTGREST_BASE threshold is misleading. It claims the scale factor "clamps to 1.0" below ~2GB and that small instances "keep the legacy base", but the actual clamp in this script only fires when usable memory drops below the base total (roughly <480MB), and the `>= 1800` check means the 1800–2000MB band (the whole "small" tier) does get the new equal-third base. Worth correcting the comment so an operator doesn't misread which instances keep the 50MB legacy base.</comment>
<file context>
@@ -35,6 +29,21 @@ else
+# medium gave it 544MB against a 90-connection pool, with its heap grazing the
+# cap after days of uptime. Below ~2GB the scale factor clamps to 1.0 and
+# there is no slack to redistribute, so tiny instances keep the legacy base.
+if [ "$TOTAL_MEM" -ge 1800 ]; then
+ POSTGREST_BASE=150
+else
</file context>
jwfing
left a comment
There was a problem hiding this comment.
Review: fix — drop hard CPU caps; equal PostgREST memory share on ≥2GB
Summary: A tight, well-justified resource-policy change that removes hard cpus: caps (single-tenant hosts) and gives PostgREST an equal memory third on ≥1800MB instances; the allocator logic is correct and I found no blocking issues.
Requirements context: No matching spec/plan found — insforge-standalone has no docs/superpowers/ (or any docs/ spec dir) and no test harness. Assessed against the PR description, the on-box incident evidence (Palise Tools throttling), and the surrounding script/compose code. Missing automated tests is not flagged, consistent with this repo having no test/CI harness.
I re-verified the head commit (e17cb0c8) — it adds the excess-shave block and extends the .env *_CPUS strip beyond the cubic-summarized commit.
Findings
Critical
(none)
- Functionality: I re-ran the allocator by hand against the PR's table and it matches.
POSTGREST_BASEis correctly set afterTOTAL_MEMis known (auto-scale-memory.sh:38-42),TOTAL_BASEis recomputed after that (:45), andUSABLE_MEMis defined (:53) before the new shave block references it (:104). No use-before-def from the reordering.- <1800MB tiers stay byte-identical to prior behavior (base
150:50:150); e.g. 460MB →184/61/184, pool 15. ✓ - ≥1800MB → equal thirds; 3835MB medium →
1268/1268/1268, total 3804 ≤ 3805 usable. ✓ - The excess-shave (
:100-107) is correctly bounded: unrounded sum equalsUSABLE_MEMexactly (TOTAL_BASE * USABLE/TOTAL_BASE), so rounding overshoot is ≤~1.5MB — well withinINSFORGE_MEM, never driving it negative.
- <1800MB tiers stay byte-identical to prior behavior (base
- Security: No security-relevant changes. No new user input, no SQL/shell-injection surface (the script reads
free/sysctland writes.env), no secrets logged, no auth/authz touched. Removing CPU caps relies on the kernel's default equalcpu.weightunder contention; memory caps are retained for OOM protection — a reasonable trade on single-tenant hosts. - Performance: This is a performance fix (removes cgroup throttling, ~2.3× PostgREST heap on a 4GB medium). No regressions introduced.
Suggestion
- Software engineering / maintainability (
auto-scale-memory.sh:38&:94): The1800threshold appears independently in thePOSTGREST_BASEselection and in the connection-tier ladder (-ge 1800small tier). They align intentionally today, but a shared constant would prevent future drift if one is ever retuned without the other. - Ops (
auto-scale-memory.sh:4-7, PR body): Uncapping CPU means burstablet4g/t3instances can now consume CPU credits without a per-container ceiling. The PR already calls out a CloudWatchCPUCreditBalancealarm as a separate lane — worth making sure that follow-up actually lands, since it's the main guardrail replacing the removed caps.
Information
auto-scale-memory.sh:20,25:TOTAL_CPUSis now vestigial — it no longer drives any allocation and survives only in the diagnostic echo. Harmless; fine to keep or drop.- Rollout (
:*): As the PR notes, only instances that re-runauto-scale-memory.sh+docker-compose uppick this up; the existing fleet needs the update pushed (tracked separately as the v2.2.9 rollout gap). Not a code issue. - Pre-existing (not this PR): The
SCALE_FACTOR < 1.0clamp (:67-71) is unreachable given the earlierUSABLE_MEM < TOTAL_BASEexit (:55-59) guarantees the factor ≥ 1. Predates this change; noting only for awareness.
Verdict
approved (informational — posted as a COMMENT; the GitHub green-check approval remains a separate human action). Zero Critical findings; the two Suggestions and the Information notes are non-blocking. Clean, focused, and the allocator math checks out.
There was a problem hiding this comment.
0 issues found across 1 file (changes from recent commits).
Requires human review: Auto-approval blocked by 2 unresolved issues from previous reviews.
Re-trigger cubic
Why
Palise Tools (b47a8ea8, ap-southeast, t4g.medium) hit a 45-min window of 10s proxy timeouts and socket resets (feedback
2def74a5). On-box evidence: the postgrest container was capped at 0.3 vCPU / 544MB while the 2-core host sat at load 0.01 — 16,847 throttled cgroup periods, heap high-water mark already at the memory cap, escalatingtimeout of 10000ms exceeded(0 → 6 → 56/day since Aug 3). Container quotas — not instance class — were the ceiling, so customer resizing cannot fix it.The legacy 150:50:150 memory split gives postgrest ~14% of RAM on every tier while it runs a RAM-tiered pool (90 connections on a medium), and CPU caps were hard limits that no service could burst past even on an idle host.
What
cpus:limits from all services. Single-tenant hosts: caps only prevented bursting into idle cores. Under contention the kernel's default equalcpu.weightarbitrates. (Ops follow-up, separate lane: CloudWatch alarm onCPUCreditBalancefor burstable instances.)POSTGREST_BASEbecomes 150 on instances ≥1800MB → equal thirds. A 4GB medium: postgrest 544MB → 1268MB (GHC heap cap 1248M). Below ~2GB the 1.0 scale-factor clamp leaves no slack, so nano/micro keep the legacy split..envcleanup still strips legacy*_CPUS=keys so upgraded instances shed them.Verification
Simulated the allocator across 460MB–15.7GB totals: nano/micro results byte-identical to current behavior; every tier stays within usable memory:
postgres's reduced cap on medium (1631→1268) is safe:
postgresql.confis conf-driven (shared_buffers=32MB,work_mem=2MB); observed RSS on the incident box is 437MB.Rollout note: this only affects instances that re-run
auto-scale-memory.sh+docker-compose up— existing fleet needs the update pushed (v2.2.9 rollout gap tracked separately).🤖 Generated with Claude Code
https://claude.ai/code/session_01BpSh9Qb69rJYnKEdMDWPph
Summary by cubic
Remove hard CPU caps and rebalance memory so PostgREST gets an equal share on instances with ≥2GB RAM. This prevents throttling on idle hosts and increases PostgREST headroom (4GB medium: 544MB → 1268MB) to avoid timeouts under load.
Bug Fixes
cpus:limits fromdocker-compose.yml; rely on kernel fair scheduling. Memory caps stay.auto-scale-memory.shto setPOSTGREST_BASE=150when total RAM ≥1800MB; tiny tiers keep the legacy 150:50:150 split..envcleanup drops legacy*_CPUSkeys so upgraded instances shed old caps.Migration
auto-scale-memory.shanddocker-compose upon each instance to apply.CPUCreditBalancefor burstable hosts.Written for commit adaee6a. Summary will update on new commits.
Summary by CodeRabbit
Note on the postgres cap reduction (medium: 1631→1268MB): weighed against the 150-connection ceiling — postgres memory is conf-driven (
shared_buffers=32MB,work_mem=2MBper sort/hash), so even 150 connections at several work_mem units each stay well under 1GB; observed RSS on the incident box is 437MB, leaving >2.5× headroom under the new cap.