AnswerBun.com

IPVS transmitting packets to incorrect backends

Server Fault Asked by Vignesh SP on August 1, 2020

We are using ipvs for L4 loadbalancing which transmits packets to L7 backends on ipip tunnel mode.

There are three ipvs systems configured with source hashing for persistence. Sometimes, ipvs is transmitting the packets to incorrect backends.

For example, ipvs 1 recieves packet from client 1.1.1.1 and it sends the packet to backend realserver 1, the same client’s next packet is received by ipvs 2 which sends it to the backend realserver 2. Now, backend 2 has no idea of this packet because the connection was actually initiated with realserver 1, thus the realserver 2 ends the connection with an RST packet.

This happens not only with a particular client, all the clients are having the same behavior.

to my understanding, all L4 ipvs should pick the same real server because of the source hashing algorithm.

I built a same setup in lab, but couldn’t reproduce it. The setup that has issue is production, hence I cannot do any huge changes to it for debugging purpose.

Keepalived is used to manage the ipvs.

Any directions on how to debug this issue with minimal impact will be really helpful.

PS – I know source hashing isn’t very consistent, but the packets being sent to wrong real servers are too high. We have other clusters where we have never seen this issue.

IPVS 1

Prot LocalAddress:Port               Conns   InPkts  OutPkts  InBytes OutBytes
  -> RemoteAddress:Port
TCP  x.x.x.26:443              401541  3620595        0  458422K        0
  -> x.x.x.1:443                401234  3618511        0  458239K        0
  -> x.x.x.2:443                    15      126        0    12341        0
  -> x.x.x.3:443                    35      213        0    20832        0
  -> x.x.x.4:443                    16      113        0    10980        0
  -> x.x.x.5:443                    19      132        0    12113        0
  -> x.x.x.6:443                    18      140        0    13616        0
  -> x.x.x.7:443                    12       97        0     9262        0
  -> x.x.x.8:443                    19      120        0    10448        0
  -> x.x.x.9:443                   164     1083        0    88618        0
  -> x.x.x.15:443                    9       60        0     5498        0


IPVS 2

  Prot LocalAddress:Port               Conns   InPkts  OutPkts  InBytes OutBytes
  -> RemoteAddress:Port
TCP  x.x.x.26:443              402903  3626029        0  459621K        0
  -> x.x.x.1:443                    12       56        0     4150        0
  -> x.x.x.2:443                    21      132        0    12967        0
  -> x.x.x.3:443                   168     1084        0    89908        0
  -> x.x.x.4:443                    14      122        0    11005        0
  -> x.x.x.5:443                    12       79        0     7045        0
  -> x.x.x.6:443                402584  3623968        0  459444K        0
  -> x.x.x.7:443                    29      146        0    12899        0
  -> x.x.x.8:443                    22      190        0    17336        0
  -> x.x.x.9:443                    10       66        0     6049        0
  -> x.x.x.15:443                   31      186        0    15724        0

Thanks !

Add your own answers!

Related Questions

DNS hung on old resolver

1  Asked on December 30, 2021 by hokie1999

 

Determine outbound https requests underlying TLS version

0  Asked on December 30, 2021 by davehogan

   

How to download all files from an FTP directory?

0  Asked on December 29, 2021

   

Default SMTP functionality if one of multiple addresses is invalid

1  Asked on December 29, 2021 by freefaller

 

Redirect SSH traffic through GRE tunnel

1  Asked on December 27, 2021 by user158720

       

start-stop-daemon not working as expected

1  Asked on December 27, 2021 by mhingston

     

Properly manage iptables rules on Docker host

1  Asked on December 27, 2021 by ural

         

AWS ECS: Unable to place task

1  Asked on December 27, 2021 by jonathan-coustick

     

Ask a Question

Get help from others!

© 2022 AnswerBun.com. All rights reserved. Sites we Love: PCI Database, MenuIva, UKBizDB, Menu Kuliner, Sharing RPP