Dealing with linux ARP flux when running multiple ethernet interfaces on the same physical and logical network.
Linux has a feature where if you have multiple network interfaces on the same wire and on the same IP network that arp will always reply with the MAC address of the first interface, even though that interface is not assigned the IP.
For example if you had 4 interfaces with the following IP and netmasks
192.168.1.1 255.255.255.0 192.168.1.2 255.255.255.0 192.168.1.3 255.255.255.0 192.168.1.4 255.255.255.0
If you were to send data to IP address 192.168.1.3 you would expect the data to go to the ethernet interface that has been assigned 192.168.1.3. However in linux this does not happen by default. What happens by default is that interface with the IP address 192.168.1.1 gives out its MAC address to all ARP queries for all of the IP addresses, not just its own. This cuts your actual bandwidth from 4x interface speed to 1x interface speed even though you have 4 interfaces.
Place the following in /etc/sysctl.conf to address this issue assing as many interfaces as you have.
# Deal with arp flux net.ipv4.conf.all.arp_ignore=1 net.ipv4.conf.eth0.arp_ignore=1 net.ipv4.conf.eth1.arp_ignore=1 net.ipv4.conf.all.arp_announce=2 net.ipv4.conf.eth0.arp_announce=2 net.ipv4.conf.eth1.arp_announce=2