redis-ha-4.27.0 - split brain #283

Activity

1. Pride1st1

   added

   bugSomething isn't workingSomething isn't working

   on Jun 26, 2024
2. Pride1st1

   assigned

   DandyDeveloper

   on Jun 26, 2024
3. mhkarimi1383 commented on Aug 18, 2024

   

   mhkarimi1383

   on Aug 18, 2024

   Contributor

   More actions

   +1
4. tschirmer commented on Oct 24, 2024

   

   tschirmer

   on Oct 24, 2024

   More actions

   I've had this too.

   I've found that when we've added a descheduler to the stack (https://github.com/kubernetes-sigs/descheduler) to balance nodes automatically, this kind of issue will disable the redis service frequently.

   Can the master allocation be done with kubernetes lease locks? https://kubernetes.io/docs/concepts/architecture/leases/
5. DandyDeveloper commented on Oct 24, 2024

   

   DandyDeveloper

   on Oct 24, 2024

   Owner

   More actions

   @tschirmer I'm trying to work out why this would happen unless the podManagementPolicy of the STS is set to Parallel?

   Is this happening in either of your cases? @tschirmer ??

   Because in theory, on first rollout, the first pod should start up and become master, way before -1/-2 start.
6. mhkarimi1383 commented on Oct 24, 2024

   

   mhkarimi1383

   on Oct 24, 2024

   Contributor

   More actions

   @DandyDeveloper
   Hi
   I'm having problem when my network becomes a bit unstable (for example pods are not able to each other for a sec.) and my redis pods can't see each other
7. tschirmer commented on Oct 30, 2024

   

   tschirmer

   on Oct 30, 2024

   More actions

   @tschirmer I'm trying to work out why this would happen unless the podManagementPolicy of the STS is set to Parallel?

   Is this happening in either of your cases? @tschirmer ??

   Because in theory, on first rollout, the first pod should start up and become master, way before -1/-2 start.

   Haven't set it to Parallel. I suspect it would be something like, pod when evicted isn't completing the trigger-failover-if-master.sh. We are running it with sentinel, which might add some complexity here. I haven't debugged it yet.

   So far we're getting a load of issues with the liveness probe not containing the SENTINELAUTH env from the secret, but it's clearly defined in the spec; and a restart of the pod works. It's happening very frequently though, so I'm wondering if there needs to be a grace period defined on startup and shutdown to prevent it both of these things from happening
8. mhkarimi1383 commented on Nov 3, 2024

   

   mhkarimi1383

   on Nov 3, 2024

   Contributor

   More actions

   I think being able to have separated Statefulsets for redises and sentinels will make this chart more stable and manageable,
   By creating two Statefulsets and giving sentinel monitor config to monitor an external host

   👍React with 👍1Reacted by Tomek Święcicki
9. tschirmer commented on Nov 7, 2024

   

   tschirmer

   on Nov 7, 2024

   More actions

   I like the idea of seperate stateful sets, I've been thinking of doing that and making a PR

   I suspect this is from preStop hooks not firing and completely successfully. trigger-failover-if-master.sh occasionally doesn't run as expected. When we had the descheduler running it was ~2min between turning on and off each pod, and found that every now and again, that would fail. The rate of failure is low, so it's unlikely occur unless you're hammering it (we haven't had an issue with the ah cluster once we turned off the descheduler.
10. mhkarimi1383 commented on Nov 8, 2024

    

    mhkarimi1383

    on Nov 8, 2024

    Contributor

    More actions

    I wanted to make a PR too. But there are a lot of configs that should propagate this change
11. tschirmer commented on Nov 22, 2024

    

    tschirmer

    on Nov 22, 2024

    More actions

    I found that there were a couple things wrong with my setup:

    • 1. because the permissions for the readonly config was set to 420, no preStop hooks were being triggered.
    • 2. we had a problem with our pvc csi driver that was attaching the same drive to any pvc with the same name in different namespaces.

    The permissions were the killer, because nothing was failing over on shutdown.

    I'm half way through writing a leader elector in golang for this based on k8s leases. Got it claiming the lease already. I'm not sure it's totally necessary after we've solved these other issues though.
12. tschirmer commented on Nov 22, 2024

    

    tschirmer

    on Nov 22, 2024

    Last edited by tschirmer

    More actions

    specifically. In the stateful set the volume definitions here:
    from:

          volumes:
            - configMap:
                defaultMode: 420  ####THIS ONE  ensured that the preStop Hooks didn't have the permissions to run. Changed it to 430
                name: redis-session-configmap
              name: config
            - hostPath:
                path: /sys
                type: ''
              name: host-sys
            - configMap:
                defaultMode: 493
                name: redis-session-health-configmap
              name: health
    

    to:

          volumes:
            - configMap:
                defaultMode: 430  ####THIS ONE  ensured that the preStop Hooks didn't have the permissions to run. Changed it to 430
                name: redis-session-configmap
              name: config
            - hostPath:
                path: /sys
                type: ''
              name: host-sys
            - configMap:
                defaultMode: 493
                name: redis-session-health-configmap
              name: health
    
13. tschirmer commented on Nov 22, 2024

    

    tschirmer

    on Nov 22, 2024

    More actions

    Also found that that preStopHook: /readonly-config/..data/trigger-failover-if-master.sh

    requires SENTINELAUTH, but it's not defined in the env for the redis container

    echo "[K8S PreStop Hook] Start Failover."
    get_redis_role() {
      is_master=$(
        redis-cli \
          -a "${AUTH}" --no-auth-warning \
          -h localhost \
          -p 6379 \
          info | grep -c 'role:master' || true
      )
    }
    get_redis_role
    
    echo "[K8S PreStop Hook] Got redis role."
    if [[ "$is_master" -eq 1 ]]; then
      echo "[K8S PreStop Hook] This node is currently master, we trigger a failover."
      response=$(
        redis-cli \
          -a "${SENTINELAUTH}" --no-auth-warning \
          -h 127.0.0.1 \
          -p 26379 \
          SENTINEL failover mymaster
      )
      if [[ "$response" != "OK" ]] ; then
        echo "[K8S PreStop Hook] Failover failed"
        echo "$response"
        exit 1
      fi
      timeout=30
      while [[ "$is_master" -eq 1 && $timeout -gt 0 ]]; do
        sleep 1
        get_redis_role
        timeout=$((timeout - 1))
      done
      echo "[K8S PreStop Hook] Failover successful"
    else
      echo "[K8S PreStop Hook] This node is currently replica, no failover needed."
    fi
    
14. tschirmer commented on Nov 22, 2024

    

    tschirmer

    on Nov 22, 2024

    Last edited by tschirmer

    More actions

    ^I'd modified the above so I could get some debug data. Along with this in the stateful set:

              lifecycle:
                preStop:
                  exec:
                    command:
                      - /bin/sh
                      - '-c'
                      - >-
                        echo "running preStop" >> /proc/1/fd/1 &&
                        /readonly-config/trigger-failover-if-master.sh | tee >>
                        /proc/1/fd/1 &&  echo "finished preStop" >> /proc/1/fd/1
    

    the >> /proc/1/fd/1 forces this output in the container log in k8s
15. tschirmer commented on Nov 22, 2024

    

    tschirmer

    on Nov 22, 2024

    More actions

    Found that running preStops would consistently fail.

    running preStop
    [K8S PreStop Hook] Start Failover.
    [K8S PreStop Hook] Got redis role.
    [K8S PreStop Hook] This node is currently master, we trigger a failover.
    [K8S PreStop Hook] Failover failed
    
    finished preStop
    

    Found that the Sentinel container had shut down before the command could be executed on the localhost., so it kept getting a failover failed. Changed the sentinel preStop to add in a 10 sec delay to keep it alive while this happened and it seems to work every time now.

              lifecycle:
                preStop:
                  exec:
                    command:
                      - /bin/sh
                      - '-c'
                      - >-
                        sleep 10
    
16. cm3brian commented on Nov 25, 2024

    

    cm3brian

    on Nov 25, 2024

    More actions

    Found that running preStops would consistently fail.

    running preStop
    [K8S PreStop Hook] Start Failover.
    [K8S PreStop Hook] Got redis role.
    [K8S PreStop Hook] This node is currently master, we trigger a failover.
    [K8S PreStop Hook] Failover failed
    
    finished preStop
    

    Found that the Sentinel container had shut down before the command could be executed on the localhost., so it kept getting a failover failed. Changed the sentinel preStop to add in a 10 sec delay to keep it alive while this happened and it seems to work every time now.

              lifecycle:
                preStop:
                  exec:
                    command:
                      - /bin/sh
                      - '-c'
                      - >-
                        sleep 10
    

    While this "might" work, it "may not" be consistent, suggest taking a look at my solution instead here: #207 (comment)

    👍React with 👍1Reacted by tschirmer
17. SCLogo commented on Jul 29, 2025

    

    SCLogo

    on Jul 29, 2025

    Contributor

    More actions

    hello all,
    what if we just simply recheck the splitbrain ? It looks the script gets wrong master if you restart the master 1 sec before splitbrain check starts.

    I added the following changes to my fix-split-brain.sh

        while true; do
            sleep 60
            identify_master
            if [ "$MASTER" = "$ANNOUNCE_IP" ]; then
                redis_role
                if [ "$ROLE" != "master" ]; then
                    echo "waiting for redis to become master"
                    sleep 15
                    identify_master
                    redis_role
                    echo "Redis role is $ROLE, expected role is master. No need to reinitialize."
                    if [ "$ROLE" != "master" ]; then
                        echo "Redis role is $ROLE, expected role is master, reinitializing"
                        reinit
                    fi
                fi
            elif [ "${MASTER}" ]; then
                identify_redis_master
                if [ "$REDIS_MASTER" != "$MASTER" ]; then
                    echo "Redis master and local master are not the same. waiting."
                    sleep 15
                    identify_master
                    identify_redis_master
                    echo "Redis master is ${MASTER}, expected master is ${REDIS_MASTER}. No need to reinitialize."
                    if [ "${REDIS_MASTER}" != "${MASTER}" ]; then
                        echo "Redis master is ${MASTER}, expected master is ${REDIS_MASTER}, reinitializing"
                        reinit
                    fi
                fi
            fi
        done
18. mhkarimi1383 commented on Jul 29, 2025

    

    mhkarimi1383

    on Jul 29, 2025

    Contributor

    More actions

    Checkout https://github.com/ParminCloud/Charts/tree/master/charts/redis-ha
    we have changed sentinel to be separated from redis-servers
19. DandyDeveloper commented on Jul 29, 2025

    

    DandyDeveloper

    on Jul 29, 2025

    Owner

    More actions

    I'd welcome PRs for these fixes.

    Even spitting Sentinel out, if we make it a conditional in the chart, we can support it.

    @mhkarimi1383 @SCLogo
20. SCLogo commented on Jul 29, 2025

    

    SCLogo

    on Jul 29, 2025

    Contributor

    More actions

    @mhkarimi1383 that is a good solution too, but we don't want to run sentinel in separate pods, that's why we did this change.
    @DandyDeveloper I am going to clean the code, template and will create a PR then.
    Thanks

    😄React with 😄1Reacted by Muhammed Hussain Karimi
21. DandyDeveloper commented on Sep 13, 2025

    

    DandyDeveloper

    on Sep 13, 2025

    Owner

    More actions

    Closing as the retry mechanism MAY resolve this entirely.
22. DandyDeveloper

    closed this as completedon Sep 13, 2025