Guide: Spreading bandwidth across two nics - Bonding, CentOS 6.4 (Final)

Cemtey

Well-Known Member
Aug 19, 2007
69
0
56
TipsAndTricks/BondingInterfaces - CentOS Wiki

cat /etc/redhat-release
CentOS release 6.4 (Final)

vi /etc/ips
Code:
194.204.54.42:255.255.255.248:194.204.54.47
194.204.54.43:255.255.255.248:194.204.54.47
194.204.54.44:255.255.255.248:194.204.54.47
194.204.54.45:255.255.255.248:194.204.54.47
vi /etc/resolv.conf
Code:
nameserver 8.8.4.4
nameserver 8.8.8.8
nameserver 4.2.2.1
nameserver 4.2.2.2
nameserver 4.2.2.3
nameserver 4.2.2.4
vi /etc/modprobe.d/bonding.conf
Code:
alias bond0 bonding
options bond0 miimon=80 mode=6
vi /etc/sysconfig/network-scripts/ifcfg-lo
Code:
DEVICE=lo
IPADDR=127.0.0.1
NETMASK=255.0.0.0
NETWORK=127.0.0.0
# If you're having problems with gated making 127.0.0.0/8 a martian,
# you can change this to something else (255.255.255.255, for example)
BROADCAST=127.255.255.255
ONBOOT=yes
NAME=loopback
vi /etc/sysconfig/network-scripts/ifcfg-bond0
Code:
DEVICE=bond0
IPADDR=194.204.54.41
NETMASK=255.255.255.248
NETWORK=194.204.54.40
BROADCAST=194.204.54.47
GATEWAY=194.204.54.46
ONBOOT=yes
BOOTPROTO=none
USERCTL=no
vi /etc/sysconfig/network-scripts/ifcfg-eth0
Code:
DEVICE=eth0
ONBOOT=yes
BOOTPROTO=none
USERCTL=no
MASTER=bond0
SLAVE=yes
vi /etc/sysconfig/network-scripts/ifcfg-eth1
Code:
DEVICE=eth1
ONNBOOT=yes
BOOTPROTO=none
USERCTL=no
MASTER=bond0
SLAVE=yes
cat /proc/net/bonding/bond0
service network restart
service ipaliases restart

Following is done in WHM

Home » Server Configuration » Basic cPanel & WHM Setup
The ethernet device onto which new IP addresses are added. Devices detected with at least one IP are listed in the drop-down, or you may specify another device.

Examples: eth0:3, eth1, eth2 venet0:0 (virtuozzo servers)
Pick bond0 from dropdown list.
It should be currently only one there.
Save Changes

Home » Account Functions » Change Site’s IP Address
Put main shared IP to all dedicated IPs temporarily to be able to remove them from interfaces.

Home » IP Functions » Rebuild the IP Address Pool
Click Proceed to mark previously taken dedicated IP addresses as Free IPs

Home » IP Functions » Show or Delete Current IP Addresses
You have list of IP addresses as
Code:
194.204.54.41 bond0
194.204.54.42 eth0:cp1
194.204.54.43 eth0:cp2
194.204.54.44 eth0:cp3
194.204.54.45 eth0:cp4
...
Delete eth0:cp1, eth0:cp2.. etc because they are attached to wrong interface.

Home » IP Functions » Add a New IP Address
Code:
194.204.54.40/29
Your IP addresses here
Code:
255.255.255.248
Your subnet mask here, use subnet-calculator.com if not sure.

Go check back at
Home » IP Functions » Show or Delete Current IP Addresses
You shuld now have list of IP addresses as
Code:
194.204.54.41 bond0
194.204.54.42 bond0:cp1
194.204.54.43 bond0:cp2
194.204.54.44 bond0:cp3
194.204.54.45 bond0:cp4
...
Home » IP Functions » Change a Site’s IP Address
Change back sites dedicated IP addresses and your done.

Just for checking reasons or reboot is rec
service network restart
service ipaliases restart

Additional info on mode,
mode=1 (active-backup)

Active-backup policy: Only one slave in the bond is active. A different slave becomes active if, and only if, the active slave fails. The bond's MAC address is externally visible on only one port (network adapter) to avoid confusing the switch. This mode provides fault tolerance. The primary option affects the behavior of this mode.

mode=2 (balance-xor)

XOR policy: Transmit based on [(source MAC address XOR'd with destination MAC address) modulo slave count]. This selects the same slave for each destination MAC address. This mode provides load balancing and fault tolerance.

mode=3 (broadcast)

Broadcast policy: transmits everything on all slave interfaces. This mode provides fault tolerance.

mode=4 (802.3ad)

IEEE 802.3ad Dynamic link aggregation. Creates aggregation groups that share the same speed and duplex settings. Utilizes all slaves in the active aggregator according to the 802.3ad specification.

Pre-requisites:
Ethtool support in the base drivers for retrieving the speed and duplex of each slave.
A switch that supports IEEE 802.3ad Dynamic link aggregation. Most switches will require some type of configuration to enable 802.3ad mode.

mode=5 (balance-tlb)

Adaptive transmit load balancing: channel bonding that does not require any special switch support. The outgoing traffic is distributed according to the current load (computed relative to the speed) on each slave. Incoming traffic is received by the current slave. If the receiving slave fails, another slave takes over the MAC address of the failed receiving slave.

Prerequisite: Ethtool support in the base drivers for retrieving the speed of each slave.

mode=6 (balance-alb)

Adaptive load balancing: includes balance-tlb plus receive load balancing (rlb) for IPV4 traffic, and does not require any special switch support. The receive load balancing is achieved by ARP negotiation. The bonding driver intercepts the ARP Replies sent by the local system on their way out and overwrites the source hardware address with the unique hardware address of one of the slaves in the bond such that different peers use different hardware addresses for the server.
Additional info on miimon,
Specifies the MII link monitoring frequency in milliseconds. This determines how often the link state of each slave is inspected for link failures. A value of zero disables MII link monitoring. A value of 100 is a good starting point.

miimon=100 means the link is going to be checked every 100 milliseconds
 

Cemtey

Well-Known Member
Aug 19, 2007
69
0
56
[SOLUTION] Troubleshooting to NTP deleting interface:

Phenomenon: Main bonded ipaddress crash (randomly between 1-24 hours) and ipaliases stay online.

Description: Message in logfile about NTPD deleting interface happens.

Solution,
Swap ETH0 to be secondary NIC and ETH1 to be main or other way around if you have currently ETH1 as main and ETH0 secondary.

So previous configuration is correct but if such issue comes up try following which should solve the case.

First of make sure which of NICs bond0 is using with,
ifconfig -a bond0

Check for MAC address and then run,
ifconfig -a eth1
ifconfig -a eth0

Make note which one is currently in use.

Code:
grep 194.204.54.41 /var/log/messages
Code:
Jul 15 18:19:58 est-et ntpd[4367]: Deleting interface #5 bond0, 194.204.54.41#123, interface stats: received=220, sent=223, dropped=0, active_time=9935 secs
Jul 15 18:20:02 est-et ntpd[4367]: Listening on interface #11 bond0, 194.204.54.41#123 Enabled
Jul 15 18:22:25 est-et named[9584]: no longer listening on 194.204.54.41#53
Jul 15 18:24:17 est-et named[2081]: listening on IPv4 interface bond0, 194.204.54.41#53
Jul 15 18:24:29 est-et ntpd[4352]: Listening on interface #5 bond0, 194.204.54.41#123 Enabled
Jul 15 18:32:09 est-et ntpd[4352]: Deleting interface #5 bond0, 194.204.54.41#123, interface stats: received=22, sent=22, dropped=1, active_time=460 secs
Jul 15 18:32:13 est-et ntpd[4352]: Listening on interface #11 bond0, 194.204.54.41#123 Enabled
Code:
grep 194.204.54.41 /var/log/messages
Code:
Jul 15 18:19:58 est-et ntpd[4367]: Deleting interface #5 bond0, 194.204.54.41#123, interface stats: received=220, sent=223, dropped=0, active_time=9935 secs
Jul 15 18:20:02 est-et ntpd[4367]: Listening on interface #11 bond0, 194.204.54.41#123 Enabled
Jul 15 18:22:25 est-et named[9584]: no longer listening on 194.204.54.41#53
Jul 15 18:24:17 est-et named[2081]: listening on IPv4 interface bond0, 194.204.54.41#53
Jul 15 18:24:29 est-et ntpd[4352]: Listening on interface #5 bond0, 194.204.54.41#123 Enabled
Jul 15 18:32:09 est-et ntpd[4352]: Deleting interface #5 bond0, 194.204.54.41#123, interface stats: received=22, sent=22, dropped=1, active_time=460 secs
Jul 15 18:32:13 est-et ntpd[4352]: Listening on interface #11 bond0, 194.204.54.41#123 Enabled
vi /etc/udev/rules.d/70-persistent-net.rules
Code:
# PCI device 0x8086:0x1096 (e1000e) (custom name provided by external tool)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:e0:81:b3:20:7c", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

# PCI device 0x8086:0x1096 (e1000e) (custom name provided by external tool)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:e0:81:b3:20:7d", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"
Note: Find out what NIC you have, currently this one is (e1000e)

vi /etc/modprobe.d/bonding.conf
Code:
alias bond0 bonding
options bond0 miimon=100 mode=6
alias eth1 e1000e
alias eth0 e1000e
Note: Type alias eth0 after alias eth1 if you had eth0 in use before.

vi /etc/sysconfig/network-scripts/ifcfg-bond0
Code:
DEVICE=bond0
IPADDR=194.204.54.41
NETMASK=255.255.255.248
#NETWORK=194.204.54.40
BROADCAST=194.204.54.47
GATEWAY=194.204.54.46
ONBOOT=yes
BOOTPROTO=none
USERCTL=no
vi /etc/sysconfig/network-scripts/ifcfg-eth0
Code:
DEVICE=eth0
HWADDR=00:e0:81:b3:20:7c
ONBOOT=yes
BOOTPROTO=none
USERCTL=no
MASTER=bond0
SLAVE=yes
vi /etc/sysconfig/network-scripts/ifcfg-eth1
Code:
DEVICE=eth1
HWADDR=00:e0:81:b3:20:7d
ONNBOOT=yes
BOOTPROTO=none
USERCTL=no
MASTER=bond0
SLAVE=yes
service network restart
service ipaliases restart

Just to check that bond0 now runs on eth1 instead of eth0 by default, looking at MAC addresses.
ifconfig -a bond0
ifconfig -a eth1
ifconfig -a eth0
 
Last edited:

Cemtey

Well-Known Member
Aug 19, 2007
69
0
56
Additional info:

cat /proc/net/bonding/bond0
Code:
Bonding Mode: adaptive load balancing
Primary Slave: none
Currently Active Slave: eth0
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth0
MII Status: up
Speed: 100 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:e0:81:b3:20:7c
Slave queue ID: 0

Slave Interface: eth1
MII Status: up
Speed: 100 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:e0:81:b3:20:7d
Slave queue ID: 0
Following would change eth0/eth1 to be main device (temporarily, until network restart)

cat /proc/net/bonding/bond0
echo -eth1 > /sys/class/net/bond0/bonding/slaves
echo +eth1 > /sys/class/net/bond0/bonding/slaves

echo -eth0 > /sys/class/net/bond0/bonding/slaves
echo +eth0 > /sys/class/net/bond0/bonding/slaves


Following command will change bond0 MAC address without having to reboot
ifconfig bond0 hw ether 00:e0:81:b3:20:7d

Just markup what eth0 and eth1 I have as MAC addresses
Code:
HWADDR=00:e0:81:b3:20:7c
HWADDR=00:e0:81:b3:20:7d
ifconfig -a bond0
ifconfig -a eth0
ifconfig -a eth1


Testing, but as result it is not firm solution.
service network restart
service ipaliases restart


Adding primary value into bond0 configuration
/etc/sysconfig/network-scripts/ifcfg-bond0
Code:
GATEWAY
BONDING_OPTS="mode=6 miimon=100 primary=eth1"
ONBOOT
ifconfig bond0 hw ether 00:e0:81:b3:20:7d
service network restart
service ipaliases restart


Checking if it works
cat /proc/net/bonding/bond0
Code:
Bonding Mode: adaptive load balancing
Primary Slave: eth1 (primary_reselect always)
Currently Active Slave: eth1
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth0
MII Status: up
Speed: 100 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:e0:81:b3:20:7c
Slave queue ID: 0

Slave Interface: eth1
MII Status: up
Speed: 100 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:e0:81:b3:20:7d
Slave queue ID: 0
ifconfig -a bond0
ifconfig -a eth0
ifconfig -a eth1


Second test,
service network restart
service ipaliases restart


Perfect!
 
Last edited: