IPVS transmitting packets to incorrect backends

We are using ipvs for L4 loadbalancing which transmits packets to L7 backends on ipip tunnel mode.

There are three ipvs systems configured with source hashing for persistence. Sometimes, ipvs is transmitting the packets to incorrect backends.

For example, ipvs 1 recieves packet from client 1.1.1.1 and it sends the packet to backend realserver 1, the same client’s next packet is received by ipvs 2 which sends it to the backend realserver 2. Now, backend 2 has no idea of this packet because the connection was actually initiated with realserver 1, thus the realserver 2 ends the connection with an RST packet.

This happens not only with a particular client, all the clients are having the same behavior.

to my understanding, all L4 ipvs should pick the same real server because of the source hashing algorithm.

I built a same setup in lab, but couldn’t reproduce it. The setup that has issue is production, hence I cannot do any huge changes to it for debugging purpose.

Keepalived is used to manage the ipvs.

Any directions on how to debug this issue with minimal impact will be really helpful.

PS – I know source hashing isn’t very consistent, but the packets being sent to wrong real servers are too high. We have other clusters where we have never seen this issue.

IPVS 1

Prot LocalAddress:Port               Conns   InPkts  OutPkts  InBytes OutBytes
  -> RemoteAddress:Port
TCP  x.x.x.26:443              401541  3620595        0  458422K        0
  -> x.x.x.1:443                401234  3618511        0  458239K        0
  -> x.x.x.2:443                    15      126        0    12341        0
  -> x.x.x.3:443                    35      213        0    20832        0
  -> x.x.x.4:443                    16      113        0    10980        0
  -> x.x.x.5:443                    19      132        0    12113        0
  -> x.x.x.6:443                    18      140        0    13616        0
  -> x.x.x.7:443                    12       97        0     9262        0
  -> x.x.x.8:443                    19      120        0    10448        0
  -> x.x.x.9:443                   164     1083        0    88618        0
  -> x.x.x.15:443                    9       60        0     5498        0


IPVS 2

  Prot LocalAddress:Port               Conns   InPkts  OutPkts  InBytes OutBytes
  -> RemoteAddress:Port
TCP  x.x.x.26:443              402903  3626029        0  459621K        0
  -> x.x.x.1:443                    12       56        0     4150        0
  -> x.x.x.2:443                    21      132        0    12967        0
  -> x.x.x.3:443                   168     1084        0    89908        0
  -> x.x.x.4:443                    14      122        0    11005        0
  -> x.x.x.5:443                    12       79        0     7045        0
  -> x.x.x.6:443                402584  3623968        0  459444K        0
  -> x.x.x.7:443                    29      146        0    12899        0
  -> x.x.x.8:443                    22      190        0    17336        0
  -> x.x.x.9:443                    10       66        0     6049        0
  -> x.x.x.15:443                   31      186        0    15724        0

Thanks !