Skip to content

0.7.2: fatal error: concurrent map writes in whatsmeowService.StartClient (kills the whole process) #203

Open

Description

@Rbertolli

Running evoapicloud/evolution-go:0.7.2 with ~12 instances, the process dies repeatedly with an unrecoverable Go runtime error. Not a recoverable panic — it takes down every instance in the deployment. We are at 168 restarts in 54 days.

Stack trace

fatal error: concurrent map writes

goroutine 18577 [running]:
internal/runtime/maps.fatal({0x195b5e4?, 0x0?})
	/usr/local/go/src/runtime/panic.go:1046 +0x18
internal/runtime/maps.(*Map).Delete(0xc000ffe9c0, 0x171eb60, 0xc003c3b228)
	/usr/local/go/src/internal/runtime/maps/map.go:652 +0x4c
github.com/evolution-foundation/evolution-go/pkg/whatsmeow/service.whatsmeowService.StartClient(...)
	/build/pkg/whatsmeow/service/whatsmeow.go:581 +0x28bc
github.com/evolution-foundation/evolution-go/pkg/whatsmeow/service.whatsmeowService.StartClient(...)
	/build/pkg/whatsmeow/service/whatsmeow.go:629 +0x3185
github.com/evolution-foundation/evolution-go/pkg/whatsmeow/service.whatsmeowService.StartClient(...)
	/build/pkg/whatsmeow/service/whatsmeow.go:629 +0x3185
github.com/evolution-foundation/evolution-go/pkg/whatsmeow/service.whatsmeowService.StartClient(...)
	/build/pkg/whatsmeow/service/whatsmeow.go:629 +0x3185
created by github.com/evolution-foundation/evolution-go/pkg/whatsmeow/service.whatsmeowService.StartInstance in goroutine 18558
	/build/pkg/whatsmeow/service/whatsmeow.go:2407 +0xa2d

The crashing write is a map delete at whatsmeow.go:581, reached from StartClient. Note StartClient appears recursively via whatsmeow.go:629 — it re-enters itself on retry, so a single instance can have several StartClient frames in flight while other instances are starting from StartInstance in their own goroutines. The map mutated at :581 looks like shared state across instances with no mutex.

When it happens

Every crash we have captured has a live handleQRCodes goroutine in the same dump:

goroutine 72775 [sleep]:
time.Sleep(0xdf8475800)
github.com/evolution-foundation/evolution-go/pkg/whatsmeow/service.(*MyClient).handleQRCodes.func1()
	/build/pkg/whatsmeow/service/whatsmeow.go:785 +0x85
created by github.com/evolution-foundation/evolution-go/pkg/whatsmeow/service.(*MyClient).handleQRCodes in goroutine 72773
	/build/pkg/whatsmeow/service/whatsmeow.go:704 +0x86

So the trigger for us is: pair an instance by QR while another instance is (re)connecting. The pairing never reaches the auth store — after the restart the instance has an empty JID and needs a new QR. From the outside it looks like "we scanned the QR and it dropped by itself".

Reproduction

  1. Have several instances registered, some offline.
  2. Call POST /instance/connect on more than one of them at roughly the same time (an external reconnect loop is enough).
  3. Pair one instance by QR while that is happening.

It is a race, so it does not reproduce every time — for us it lands about 3 times a day.

Environment

  • evoapicloud/evolution-go:0.7.2 (sha256:6fa601464bd76d3d19ece4c48d8bb5373d025d95d45b939ae4bf0c77b03f5aaa)
  • whatsmeow v0.0.0-20260630180629-b572e5bcb92b
  • Auth store in Postgres (POSTGRES_AUTH_DB set)
  • CONNECT_ON_STARTUP=false, ~12 instances

Impact

Because the failure is fatal error rather than a panic, no recover() helps and the process exits (code 2). With CONNECT_ON_STARTUP=false every instance is left disconnected after the restart, and any instance without a stored JID needs a human to scan a QR again. A mutex around the map at whatsmeow.go:581 would stop a single instance pairing from taking down the whole server.

Happy to provide the full goroutine dump if useful.

Activity

  1. Rbertolli commented on Sep 22, 2026

    @Rbertolli
    Author

    It happened again ~2 hours later, and this time it is not the QR pairing path — which I think widens the scope of the report.

    Same unsynchronized map, reached from the event handler instead:

    fatal error: concurrent map read and map write
    
    goroutine 388149 [running]:
    internal/runtime/maps.fatal({0x197342e?, 0xc007413b80?})
    	/usr/local/go/src/runtime/panic.go:1046 +0x18
    github.com/evolution-foundation/evolution-go/pkg/whatsmeow/service.whatsmeowService.ReconnectClient(...)
    	/build/pkg/whatsmeow/service/whatsmeow.go:188 +0x23c
    github.com/evolution-foundation/evolution-go/pkg/whatsmeow/service.(*MyClient).myEventHandler.func3({0xc002627020, 0x24})
    	/build/pkg/whatsmeow/service/whatsmeow.go:1987 +0xbb
    created by github.com/evolution-foundation/evolution-go/pkg/whatsmeow/service.(*MyClient).myEventHandler in goroutine 388135
    	/build/pkg/whatsmeow/service/whatsmeow.go:1985 +0x749f
    

    So the same map is touched from at least three places with no lock between them:

    • StartClient — whatsmeow.go:581 (map delete), the original report
    • ReconnectClient — whatsmeow.go:188 (read + write), this one
    • myEventHandler — spawns func3 per event at :1985, which calls ReconnectClient

    The practical consequence is worse than "pairing is fragile": myEventHandler spawns a goroutine per event, so any disconnect event can start a reconnect that races with whatever else is running. No QR involved, no operator action involved. In our deployment that is a few crashes a day — 169 container restarts in 54 days — and each one drops every instance on the server, because we run with CONNECT_ON_STARTUP=false.

    A mutex guarding that map would cover all three paths at once.

    Timeline of the two we captured:

    • 21:14 UTC-3 — concurrent map writes, StartClient, QR pairing in flight
    • 01:11 UTC-3 — concurrent map read and map write, ReconnectClient, no pairing, no human action

    Full goroutine dumps available on request for either.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions