![[Monkey]](../pics/monkey_wht.jpg)
![[Network Diagram]](svg/hc-ha-lb-eg.png)
This topology enables the maximum throughput through the network as return traffic no longer has to travel through an Linux-Director. This topology builds on the existing routing infrastructure of a network by allowing existing internal routing policies to dictate the flow of outbound traffic. For a more detailed explanation of the function of each component please see the overview.
The documentation that follows assumes that all nodes on the network are set up with correct interfaces and routes for each network they are connected to as per the diagram above.
The Linux-Directors must be able to route traffic to the real-servers. Specifically in addition to correctly configuring the interfaces and routes you must enable IPV4 forwarding. This is done by adding configuration of net.ipv4.ip_forward to /etc/sysctl.conf as follows:
# Enables packet forwarding net.ipv4.ip_forward = 1For these changes to take effect the sysctl command may be used:
/sbin/sysctl -p net.ipv4.ip_forward = 1
Heartbeat runs on the two Linux-Directors and handles bringing up the interface for the virtual address. This is the address to which end-users should connect, and is typically advertised using DNS.
To configure heartbeat /etc/ha.d/ha.cf, /etc/ha.d/haresources and /etc/ha.d/authkeys must be installed. The node names in /etc/ha.d/ha.cf
chmod 600 /etc/ha.d/authkeysThe configuration files supplied expect that the linux-directors are connected via eth0 and by a null modem connected to /dev/ttyS0. This may be modified but it is highly recommended that heartbeat be run over at least two links.
The monitoring of real-servers, and their insertion and removal from the pool of servers available is controlled by ldirectord which is run by heartbeat. To configure ldirectord /etc/ha.d/ldirectord.cf must be installed. Information on customising this file can be found in the ldirectord(8) man page.
Ldirectord is run by heartbeat. On supported versions of Red Hat and Fedora to ensure that heartbeat starts up (on run-levels 2, 3, 4 and 5) and that ldirectord does not start on reboot up the chkconfig command is used:
/sbin/chkconfig --level 2345 heartbeat on /sbin/chkconfig --del ldirectord
On Debian the update-rc.d command is used:
/usr/sbin/update-rc.d heartbeat start 75 2 3 4 5 . stop 05 0 1 6 . /usr/sbin/update-rc.d -f ldirectord removeTo ensure that ldirectord is not running and start heartbeat with the new configuration run:
/etc/init.d/ldirectord stop /etc/init.d/heartbeat start
After a few moments heartbeat should bring add the virtual IP address on the master linux-director. This can be verified using the ip command.
ip addr sh eth0
4: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 00:50:56:4f:30:07 brd ff:ff:ff:ff:ff:ff
inet 192.168.6.2/24 brd 192.168.6.255 scope global eth0
inet 192.168.6.240/24 brd 192.168.6.255 scope global secondary eth0
Heartbeat should also run ldirectord to configure LVS on this node. To check that ldirectord is running use:
/usr/sbin/ldirectord ldirectord.cf statusTo inspect the current LVS kernel table the ipvsadm command may be used. A sample invocation follows showing that ldirectord found all servers to be available. Note that the FTP virtual service on port 21 is persistent when using direct routing (DR).
/sbin/ipvsadm -L -n Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 192.168.6.240:443 rr -> 192.168.6.4:443 Route 1 0 0 -> 192.168.6.5:443 Route 1 0 0 TCP 192.168.6.240:80 rr -> 192.168.6.4:80 Route 1 0 0 -> 192.168.6.5:80 Route 1 0 0 TCP 192.168.6.240:21 rr persistent 600 -> 192.168.6.4:21 Route 1 0 0 -> 192.168.6.5:21 Route 1 0 0
As well as adding the virtual ipaddresses and running ldirectord, heartbeat should also run the LVS synchronisation daemon. The master daemon should be running on the current master linux-director, and the backup daemon on the other linux-director. You can verify this by inspecting the status of LVSSyncDaemonSwap resource.
# On the master linux-director /etc/ha.d/resource.d/LVSSyncDaemonSwap master status master running (ipvs_syncmaster pid: 4530) # On the stand-by linux-director /etc/ha.d/resource.d/LVSSyncDaemonSwap master status master stopped (ipvs_syncbackup pid: 3895)
Failover can be tested by stopping heartbeat on the master linux-director which should cause the hot-standby linux-director to become the master.
/etc/init.d/heartbeat stop
Heartbeat and Ldirectord log debugging and status information to /var/log/messages using syslog. These logs should be inspected if problems occur. Please see notes on logging to ensure that all logs are written to disk for debugging purposes.
The real-servers need to be configured to see traffic for the virtual IP address 192.168.6.240 as local. This can be done by making use of an IP alias on the loopback device. But before doing this, ARP Advertisments need to be restricted so that the real servers do not advertise the virtual IP address, which should only be advertised by the active linux-director.
On Debian, the arp_ignore and arp_announce sysctl flags are used to ensure that the VIP is not announced via ARP on the real-servers, eth0 should be set to only use its addresses for ARP. This is done by adding the following lines to /etc/sysctl.conf. The eth0 lines below should be replicated for any devices that respond to ARP.
# Enable configuration of arp_ignore option net.ipv4.conf.all.arp_ignore = 1 # When an arp request is received on eth0, only respond if that address is # configured on eth0. In particular, do not respond if the address is # configured on lo net.ipv4.conf.eth0.arp_ignore = 1 # Ditto for eth1, add for all ARPing interfaces #net.ipv4.conf.eth1.arp_ignore = 1 # Enable configuration of arp_announce option net.ipv4.conf.all.arp_announce = 2 # When making an ARP request sent through eth0 Always use an address that # is configured on eth0 as the source address of the ARP request. If this # is not set, and packets are being sent out eth0 for an address that is on # lo, and an arp request is required, then the address on lo will be used. # As the source IP address of arp requests is entered into the ARP cache on # the destination, it has the effect of announcing this address. This is # not desirable in this case as adresses on lo on the real-servers should # be announced only by the linux-director. net.ipv4.conf.eth0.arp_announce = 2 # Ditto for eth1, add for all ARPing interfaces #net.ipv4.conf.eth1.arp_announce = 2
The Red Hat Enterprise Linux kernel does not currently have the arp_ignore and arp_announce flags. This is partly because adding them will break the kernel ABI. If that occurs for another reason, then they may be added, but in the mean time a arptables can be used to obtain mich the same effect. Be for running the commands below, make sure that the the arptables_jf package version >=0.0.7, supplied by Red Hat Enterprise Linux is installed.
# Flush existing rules /etc/init.d/arptables_jf stop Flushing all chains: [ OK ] Removing user defined chains: [ OK ] Resetting built-in chains to the default ACCEPT policy: [ OK ] # Drop ARP requests to and from the virtual IP address /usr/sbin/arptables-noarp-addr 192.168.6.240 start # This the above command produces the following rules, # dropping incoming ARP requests for the VIP (192.168.6.240) # and mangles outgoing ARP announcements on each ARPing interface (eth0) # See arptables-noarp-addr(8) for more details #/sbin/arptables -A IN -j DROP -d 192.168.6.240 #/sbin/arptables -A OUT -j mangle -o eth0 -s 192.168.6.240 --mangle-ip-s 192.168.6.2 # Save the rules /etc/init.d/arptables_jf save Saving current rules to /etc/sysconfig/arptables: [ OK ] # Make sure rules are activated on reboot (at run levels 2, 3, 4 and 5) /sbin/chkconfig --level 2345 arptables_jf on # Activate the rules /etc/init.d/arptables_jf start Flushing all current rules and user defined chains: [ OK ] Clearing all current rules and user defined chains: [ OK ] Applying arptables firewall rules: [ OK ]
You can inspect the arptables rules, and examine many hits they have had, by using the arptables command.
/sbin/arptables -L -v -n
Chain IN (policy ACCEPT 229 packets, 6412 bytes)
pkts bytes target in out source-ip destination-ip source-hw destination-hw hlen op hrd pro
1 28 DROP * * 0.0.0.0/0 192.168.6.240 00/00 00/00 any 0000/0000 0000/0000 0000/0000
Chain OUT (policy ACCEPT 40 packets, 1120 bytes)
pkts bytes target in out source-ip destination-ip source-hw destination-hw hlen op hrd pro
0 0 mangle * eth0 192.168.6.240 0.0.0.0/0 00/00 00/00 any 0000/0000 0000/0000 0000/0000 --mangle-ip-s 192.168.6.5
Now that ARP Adviertisments of the virtual IP address have been controlled, it can be added as an alias to the loopback interface to allow the real-server to accept packets for the virutal service.
You can verify that the interface came up using the ifconfig command. The following output has been truncated to only show information for lo:0.