Skip to content

Postgres connection leak: each (re)connect on a logged-out instance leaks an unclosed sqlstore pool (evogo_auth/evogo_users) #106

Open

Description

@gitlovable

Summary

Every (re)connect attempt on an instance whose WhatsApp session was logged out (pairing removed on the phone → whatsmeow LoggedOut) leaks a Postgres connection pool: a new whatsmeow client + sqlstore is created per attempt, the login fails with LoggedOut (expected), and the abandoned sql.DB pool is never Close()d. The idle connections stay parked in Postgres forever.

With any periodic reconnect source, this grows unbounded until the whole Postgres cluster hits max_connections and refuses every client. Aggravated by evolution-go connecting as the postgres superuser by default, which also exhausts superuser_reserved_connections — locking the DBA out during the incident.

We hit this twice in production in two days, through two different trigger paths:

Variant A — external POST /instance/connect (day 1)

Our supervisor retried /instance/connect every 2 min on a logged-out instance.

  • Postgres log: 742 × FATAL: sorry, too many clients already, origin user=postgres, db=evogo_users
  • Connection count of the evolution-go process (nsenter … ss -tn | grep :5432): 167 (healthy baseline for the same deployment: ~15–20)
  • Exact correlation: 88 authenticated POST /instance/connect (HTTP 200) since container start ≈ the accumulated leaked connections (+1 per attempt). 401 attempts (invalid token) do not leak — the leak is in the authenticated path that instantiates the client/sqlstore.
  • docker restart evolution-go instantly dropped the count 167 → 3 and the cluster recovered.

Variant B — internal whatsmeow auto-reconnect (day 2, no API calls at all)

After we stopped retrying via API, a logged-out instance kept cycling through whatsmeow's internal auto-reconnect:

  • 25 × LoggedOut connection events in 6h for one instance, only 2 authenticated /instance/connect in the same window (so not API-triggered)
  • 121 idle connections accumulated in db=evogo_auth (session store), ~7/hour, all state=idle, oldest from the moment the cycling started
  • Same fix: restart frees them instantly; the leak resumes while the logged-out instance keeps retrying.

Suspected root cause

On the connect path, a new whatsmeow client + sqlstore container (its own database/sql pool) is created per attempt. When login fails with LoggedOut, the client is discarded but the store/pool is not closed. An abandoned Go sql.DB keeps its idle connections open indefinitely.

Suggested fix

  1. Close the sqlstore/sql.DB on the connect error path (or reuse the instance's existing client/pool instead of creating a new one per attempt);
  2. Treat LoggedOut as terminal in the reconnect loop: stop auto-retrying a session that requires re-pairing (surface "re-pair required" instead);
  3. Independently: document/support running with a non-superuser DB role — with the default superuser connection, any leak escalates to a full-cluster outage including the reserved slots.

Workarounds we use meanwhile

  • Supervisor treats LoggedOut as terminal and stops calling /instance/connect;
  • docker restart evolution-go when the count grows;
  • Planning a dedicated DB role with CONNECTION LIMIT as a blast-radius guard.

Happy to provide more logs/details if useful. 🇧🇷 Podemos detalhar em PT-BR também, se preferirem.

Activity

  1. sendzai commented on Jul 14, 2026

    @sendzai

    Hey everyone,

    I ran into this exact same issue where connection pools were leaking during reconnect loops (especially with proxy rotations and logged-out instances) and eventually hitting the sorry, too many clients limit.

    Until the PR fix is fully merged/shipped, the best workaround I found is to configure PostgreSQL to clean up these abandoned connections automatically. Since the app loses reference to the old pools, they just sit in the database as idle.

    You can set idle_session_timeout to automatically close them after a few minutes.

    If you're using Docker Compose, just add command: postgres -c idle_session_timeout=300000 (this sets it to 5 minutes) to your postgres service:

    yaml
    postgres:
        image: postgres:15-alpine
        container_name: sendzai_db
        restart: always
        command: postgres -c idle_session_timeout=300000
        environment:
          POSTGRES_USER: admin
          POSTGRES_PASSWORD: your_password
          POSTGRES_DB: evolution
    
    If you are running Postgres natively, you can append this to your postgresql.conf and restart/reload:
    idle_session_timeout = 300000
    
    

    This completely stopped our DB from running out of connection slots and keeps everything running smoothly. Hopefully this helps anyone stuck on this right now!

  2. jonathanpog98 commented on Jul 14, 2026

    @jonathanpog98

    Hey everyone,

    I ran into this exact same issue where connection pools were leaking during reconnect loops (especially with proxy rotations and logged-out instances) and eventually hitting the sorry, too many clients limit.

    Until the PR fix is fully merged/shipped, the best workaround I found is to configure PostgreSQL to clean up these abandoned connections automatically. Since the app loses reference to the old pools, they just sit in the database as idle.

    You can set idle_session_timeout to automatically close them after a few minutes.

    If you're using Docker Compose, just add command: postgres -c idle_session_timeout=300000 (this sets it to 5 minutes) to your postgres service:

    yaml
    postgres:
        image: postgres:15-alpine
        container_name: sendzai_db
        restart: always
        command: postgres -c idle_session_timeout=300000
        environment:
          POSTGRES_USER: admin
          POSTGRES_PASSWORD: your_password
          POSTGRES_DB: evolution
    
    If you are running Postgres natively, you can append this to your postgresql.conf and restart/reload:
    idle_session_timeout = 300000
    

    This completely stopped our DB from running out of connection slots and keeps everything running smoothly. Hopefully this helps anyone stuck on this right now!

    Hello!

    Thanks for the help, @sendzai. To test this out, I ran the following on my database:
    ALTER DATABASE evogo_users SET idle_session_timeout = '5min'; ALTER DATABASE evogo_auth SET idle_session_timeout = '5min';

    It has worked so far, and I am currently testing it.
    I did this because I use the same container for other applications.

  3. guilhermeCassettari commented on Jul 14, 2026

    @guilhermeCassettari

    We're hitting the same leak through a third trigger path not yet documented here: the internal auto-reconnect on healthy instances — no external /instance/connect calls, no logged-out sessions involved.

    Whenever a connected instance drops (ordinary WhatsApp/network blip), evolution-go logs:

    [uuid] UserInfo cache cleared for token: ***
    [uuid] Disconnected detected, restarting instance
    [uuid] Starting reconnection process - simulating restart
    

    and each of these internal restarts orphans one sqlstore pool (~2 idle Postgres connections, Go database/sql default MaxIdleConns).

    Quantified over a 5h window on latest (2026-07), 7 instances, moderate traffic:

    Component Connections on the auth DB
    Baseline: 7 instances × 2 idle per pool 14
    9 internal restarts × 2 orphaned each +18
    Predicted / measured 32 / 31

    pg_stat_activity shows the orphaned connections being born in pairs exactly at the restart timestamps, and never closing (one sat idle for 3+ hours). On a day with unstable connectivity this exhausted the default max_connections=100 twice in 24h — and it feeds back on itself: pool saturation makes instances drop, which triggers more internal restarts.

    The diagnosis in #102 matches what we observe sqlstore.New() per (re)connect, container never closed) — that fix should cover this path too, as long as it also applies to the internal "simulating restart" flow, not only the external connect endpoints.

  4. guilhermeCassettari commented on Jul 15, 2026

    @guilhermeCassettari

    This should be fixed by #117 — it makes the whatsmeow auth-store use a single shared sqlstore.Container with a capped pool instead of opening a new unbounded one on every (re)connect. Reproduced the leak locally (15 reconnects: 1 → 16 connections, unbounded growth afterwards) and verified the fix keeps it at 1-2.

  5. ShowtimeProp commented on Aug 1, 2026

    @ShowtimeProp

    Confirming on 0.7.2 — the trigger is unpaired instances stuck in a QR loop

    We hit this on evoapicloud/evolution-go:0.7.2 and were able to pin down a
    specific trigger that doesn't seem to be documented yet: instances that have
    never been paired (no JID) get stuck in an endless QR cycle, and every cycle
    leaks a pool.

    Environment

    • Evolution GO 0.7.2, Docker Swarm, dedicated postgres:15-alpine
    • 12 instances total: 7 paired (with JID), 5 never paired (no JID)
    • CONNECT_ON_STARTUP=true

    Mechanism

    With CONNECT_ON_STARTUP=true, GO attempts to connect all instances at boot,
    including logged-out ones. The 5 unpaired instances then loop forever:

    [<id>] QR code generated #3 (max: 5)
    [<id>] QR code generated #4 (max: 5)
    [<id>] Maximum QR code count reached (5), forcing logout
    [<id>] No jid found. Creating new device
    [<id>] QR timeout — signaling kill channel
       ↳ and it starts over
    

    Each Creating new device builds a new client and a new sqlstore pool that is
    never closed. Connections then climb until:

    Failed to create container: failed to upgrade database:
    failed to check if version table is up to date: pq: sorry, too many clients already
    

    Once that happens, GO can no longer create clients at all: QR generation stops
    working and dropped instances can't reconnect.

    Numbers

    max_connections=300, few instances connected exhausted in ~7 h
    max_connections=500, 7 instances connected + 5 looping exhausted in ~90 min, repeatedly
    Restarts needed overnight (automated watchdog) 15 in 9 hours

    What did NOT help

    • Raising max_connections (300 → 500): only delays it.
    • idle_session_timeout=10min: verified active via SHOW idle_session_timeout,
      made no measurable difference. The leak is bursty — a runaway retry loop
      saturates faster than the timeout can reap.

    What DID fix it

    Setting CONNECT_ON_STARTUP=false, then reconnecting only the paired instances
    via POST /instance/reconnect:

    before:  ~500 connections, saturating every ~60-90 min
    after:   30 connections, stable, 0 QR codes generated
    

    Same 7 instances connected and working in both cases. The only difference is
    that the 5 unpaired instances are no longer being connected.

    Suggested angle

    Two things seem worth looking at independently:

    1. The sqlstore pool created per client isn't closed when a client is discarded
      (already covered by Postgres connection leak: each (re)connect on a logged-out instance leaks an unclosed sqlstore pool (evogo_auth/evogo_users) #106 / PostgreSQL connection leak: a new sqlstore pool is created on instance reconnect #118).
    2. The QR cycle for a logged-out instance never terminates — it forces a logout
      and immediately creates a new device, so it can loop indefinitely. Capping
      that (or requiring an explicit connect request) would remove the amplifier
      even before the pool leak is fixed.

    Happy to provide more logs if useful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions