Guide: Spreading bandwidth across two nics - Bonding, CentOS 6.4 (Final)

[Cemtey]

TipsAndTricks/BondingInterfaces - CentOS Wiki

cat /etc/redhat-release
CentOS release 6.4 (Final)

vi /etc/ips

194.204.54.42:255.255.255.248:194.204.54.47
194.204.54.43:255.255.255.248:194.204.54.47
194.204.54.44:255.255.255.248:194.204.54.47
194.204.54.45:255.255.255.248:194.204.54.47

vi /etc/resolv.conf

nameserver 8.8.4.4
nameserver 8.8.8.8
nameserver 4.2.2.1
nameserver 4.2.2.2
nameserver 4.2.2.3
nameserver 4.2.2.4

vi /etc/modprobe.d/bonding.conf

alias bond0 bonding
options bond0 miimon=80 mode=6

vi /etc/sysconfig/network-scripts/ifcfg-lo

DEVICE=lo
IPADDR=127.0.0.1
NETMASK=255.0.0.0
NETWORK=127.0.0.0
# If you're having problems with gated making 127.0.0.0/8 a martian,
# you can change this to something else (255.255.255.255, for example)
BROADCAST=127.255.255.255
ONBOOT=yes
NAME=loopback

vi /etc/sysconfig/network-scripts/ifcfg-bond0

DEVICE=bond0
IPADDR=194.204.54.41
NETMASK=255.255.255.248
NETWORK=194.204.54.40
BROADCAST=194.204.54.47
GATEWAY=194.204.54.46
ONBOOT=yes
BOOTPROTO=none
USERCTL=no

vi /etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE=eth0
ONBOOT=yes
BOOTPROTO=none
USERCTL=no
MASTER=bond0
SLAVE=yes

vi /etc/sysconfig/network-scripts/ifcfg-eth1

DEVICE=eth1
ONNBOOT=yes
BOOTPROTO=none
USERCTL=no
MASTER=bond0
SLAVE=yes

cat /proc/net/bonding/bond0
service network restart
service ipaliases restart

Following is done in WHM

Home » Server Configuration » Basic cPanel & WHM Setup

The ethernet device onto which new IP addresses are added. Devices detected with at least one IP are listed in the drop-down, or you may specify another device.

Examples: eth0:3, eth1, eth2 venet0:0 (virtuozzo servers)

Pick bond0 from dropdown list.
It should be currently only one there.
Save Changes

Home » Account Functions » Change Site’s IP Address
Put main shared IP to all dedicated IPs temporarily to be able to remove them from interfaces.

Home » IP Functions » Rebuild the IP Address Pool
Click Proceed to mark previously taken dedicated IP addresses as Free IPs

Home » IP Functions » Show or Delete Current IP Addresses
You have list of IP addresses as

194.204.54.41 bond0
194.204.54.42 eth0:cp1
194.204.54.43 eth0:cp2
194.204.54.44 eth0:cp3
194.204.54.45 eth0:cp4
...

Delete eth0:cp1, eth0:cp2.. etc because they are attached to wrong interface.

Home » IP Functions » Add a New IP Address

194.204.54.40/29

Your IP addresses here

255.255.255.248

Your subnet mask here, use subnet-calculator.com if not sure.

Go check back at
Home » IP Functions » Show or Delete Current IP Addresses
You shuld now have list of IP addresses as

194.204.54.41 bond0
194.204.54.42 bond0:cp1
194.204.54.43 bond0:cp2
194.204.54.44 bond0:cp3
194.204.54.45 bond0:cp4
...

Home » IP Functions » Change a Site’s IP Address
Change back sites dedicated IP addresses and your done.

Just for checking reasons or reboot is rec
service network restart
service ipaliases restart

Additional info on mode,

mode=1 (active-backup)

Active-backup policy: Only one slave in the bond is active. A different slave becomes active if, and only if, the active slave fails. The bond's MAC address is externally visible on only one port (network adapter) to avoid confusing the switch. This mode provides fault tolerance. The primary option affects the behavior of this mode.

mode=2 (balance-xor)

XOR policy: Transmit based on [(source MAC address XOR'd with destination MAC address) modulo slave count]. This selects the same slave for each destination MAC address. This mode provides load balancing and fault tolerance.

mode=3 (broadcast)

Broadcast policy: transmits everything on all slave interfaces. This mode provides fault tolerance.

mode=4 (802.3ad)

IEEE 802.3ad Dynamic link aggregation. Creates aggregation groups that share the same speed and duplex settings. Utilizes all slaves in the active aggregator according to the 802.3ad specification.

Pre-requisites:
Ethtool support in the base drivers for retrieving the speed and duplex of each slave.
A switch that supports IEEE 802.3ad Dynamic link aggregation. Most switches will require some type of configuration to enable 802.3ad mode.

mode=5 (balance-tlb)

Adaptive transmit load balancing: channel bonding that does not require any special switch support. The outgoing traffic is distributed according to the current load (computed relative to the speed) on each slave. Incoming traffic is received by the current slave. If the receiving slave fails, another slave takes over the MAC address of the failed receiving slave.

Prerequisite: Ethtool support in the base drivers for retrieving the speed of each slave.

mode=6 (balance-alb)

Adaptive load balancing: includes balance-tlb plus receive load balancing (rlb) for IPV4 traffic, and does not require any special switch support. The receive load balancing is achieved by ARP negotiation. The bonding driver intercepts the ARP Replies sent by the local system on their way out and overwrites the source hardware address with the unique hardware address of one of the slaves in the bond such that different peers use different hardware addresses for the server.

Additional info on miimon,

Specifies the MII link monitoring frequency in milliseconds. This determines how often the link state of each slave is inspected for link failures. A value of zero disables MII link monitoring. A value of 100 is a good starting point.

miimon=100 means the link is going to be checked every 100 milliseconds

 

[cPanelMichael]

Hello

Thank you for taking the time to share this information.

 

[Cemtey]

[SOLUTION] Troubleshooting to NTP deleting interface:

Phenomenon: Main bonded ipaddress crash (randomly between 1-24 hours) and ipaliases stay online.

Description: Message in logfile about NTPD deleting interface happens.

Solution,
Swap ETH0 to be secondary NIC and ETH1 to be main or other way around if you have currently ETH1 as main and ETH0 secondary.

So previous configuration is correct but if such issue comes up try following which should solve the case.

First of make sure which of NICs bond0 is using with,
ifconfig -a bond0

Check for MAC address and then run,
ifconfig -a eth1
ifconfig -a eth0

Make note which one is currently in use.

grep 194.204.54.41 /var/log/messages

Jul 15 18:19:58 est-et ntpd[4367]: Deleting interface #5 bond0, 194.204.54.41#123, interface stats: received=220, sent=223, dropped=0, active_time=9935 secs
Jul 15 18:20:02 est-et ntpd[4367]: Listening on interface #11 bond0, 194.204.54.41#123 Enabled
Jul 15 18:22:25 est-et named[9584]: no longer listening on 194.204.54.41#53
Jul 15 18:24:17 est-et named[2081]: listening on IPv4 interface bond0, 194.204.54.41#53
Jul 15 18:24:29 est-et ntpd[4352]: Listening on interface #5 bond0, 194.204.54.41#123 Enabled
Jul 15 18:32:09 est-et ntpd[4352]: Deleting interface #5 bond0, 194.204.54.41#123, interface stats: received=22, sent=22, dropped=1, active_time=460 secs
Jul 15 18:32:13 est-et ntpd[4352]: Listening on interface #11 bond0, 194.204.54.41#123 Enabled

grep 194.204.54.41 /var/log/messages

Jul 15 18:19:58 est-et ntpd[4367]: Deleting interface #5 bond0, 194.204.54.41#123, interface stats: received=220, sent=223, dropped=0, active_time=9935 secs
Jul 15 18:20:02 est-et ntpd[4367]: Listening on interface #11 bond0, 194.204.54.41#123 Enabled
Jul 15 18:22:25 est-et named[9584]: no longer listening on 194.204.54.41#53
Jul 15 18:24:17 est-et named[2081]: listening on IPv4 interface bond0, 194.204.54.41#53
Jul 15 18:24:29 est-et ntpd[4352]: Listening on interface #5 bond0, 194.204.54.41#123 Enabled
Jul 15 18:32:09 est-et ntpd[4352]: Deleting interface #5 bond0, 194.204.54.41#123, interface stats: received=22, sent=22, dropped=1, active_time=460 secs
Jul 15 18:32:13 est-et ntpd[4352]: Listening on interface #11 bond0, 194.204.54.41#123 Enabled

vi /etc/udev/rules.d/70-persistent-net.rules

# PCI device 0x8086:0x1096 (e1000e) (custom name provided by external tool)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:e0:81:b3:20:7c", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

# PCI device 0x8086:0x1096 (e1000e) (custom name provided by external tool)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:e0:81:b3:20:7d", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"

Note: Find out what NIC you have, currently this one is (e1000e)

vi /etc/modprobe.d/bonding.conf

alias bond0 bonding
options bond0 miimon=100 mode=6
alias eth1 e1000e
alias eth0 e1000e

Note: Type alias eth0 after alias eth1 if you had eth0 in use before.

vi /etc/sysconfig/network-scripts/ifcfg-bond0

DEVICE=bond0
IPADDR=194.204.54.41
NETMASK=255.255.255.248
#NETWORK=194.204.54.40
BROADCAST=194.204.54.47
GATEWAY=194.204.54.46
ONBOOT=yes
BOOTPROTO=none
USERCTL=no

vi /etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE=eth0
HWADDR=00:e0:81:b3:20:7c
ONBOOT=yes
BOOTPROTO=none
USERCTL=no
MASTER=bond0
SLAVE=yes

vi /etc/sysconfig/network-scripts/ifcfg-eth1

DEVICE=eth1
HWADDR=00:e0:81:b3:20:7d
ONNBOOT=yes
BOOTPROTO=none
USERCTL=no
MASTER=bond0
SLAVE=yes

service network restart
service ipaliases restart

Just to check that bond0 now runs on eth1 instead of eth0 by default, looking at MAC addresses.
ifconfig -a bond0
ifconfig -a eth1
ifconfig -a eth0

 

[Cemtey]

Additional info:

cat /proc/net/bonding/bond0

Bonding Mode: adaptive load balancing
Primary Slave: none
Currently Active Slave: eth0
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth0
MII Status: up
Speed: 100 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:e0:81:b3:20:7c
Slave queue ID: 0

Slave Interface: eth1
MII Status: up
Speed: 100 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:e0:81:b3:20:7d
Slave queue ID: 0

Following would change eth0/eth1 to be main device (temporarily, until network restart)

cat /proc/net/bonding/bond0
echo -eth1 > /sys/class/net/bond0/bonding/slaves
echo +eth1 > /sys/class/net/bond0/bonding/slaves

echo -eth0 > /sys/class/net/bond0/bonding/slaves
echo +eth0 > /sys/class/net/bond0/bonding/slaves

Following command will change bond0 MAC address without having to reboot
ifconfig bond0 hw ether 00:e0:81:b3:20:7d

Just markup what eth0 and eth1 I have as MAC addresses

HWADDR=00:e0:81:b3:20:7c
HWADDR=00:e0:81:b3:20:7d

ifconfig -a bond0
ifconfig -a eth0
ifconfig -a eth1

Testing, but as result it is not firm solution.
service network restart
service ipaliases restart

Adding primary value into bond0 configuration
/etc/sysconfig/network-scripts/ifcfg-bond0

GATEWAY
BONDING_OPTS="mode=6 miimon=100 primary=eth1"
ONBOOT
ifconfig bond0 hw ether 00:e0:81:b3:20:7d

service network restart
service ipaliases restart

Checking if it works
cat /proc/net/bonding/bond0

Bonding Mode: adaptive load balancing
Primary Slave: eth1 (primary_reselect always)
Currently Active Slave: eth1
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth0
MII Status: up
Speed: 100 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:e0:81:b3:20:7c
Slave queue ID: 0

Slave Interface: eth1
MII Status: up
Speed: 100 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:e0:81:b3:20:7d
Slave queue ID: 0

ifconfig -a bond0
ifconfig -a eth0
ifconfig -a eth1

Second test,
service network restart
service ipaliases restart

Perfect!