This topology combines the function of Linux Directors and Real Servers into the same hosts, so that high availability and load balancing can be achived with just two nodes. For a more detailed explanation of the function of each component please see the overview.
The documentation that follows assumes that all nodes on the network are set up with correct interfaces and routes for each network they are connected to as per the diagram above
Linux-Directors / Real-Servers
The linux-directors / real-servers need to be configured to see traffic for the virtual IP address 192.168.6.240 as local. This can be done by making use of an IP alias on the loopback device. But before doing this, ARP Advertisments need to be restricted so that the real servers do not advertise the virtual IP address, which should only be advertised by the active linux-director.
On Debian, the arp_ignore and arp_announce sysctl flags are used to ensure that the VIP is not announced via ARP on the real-servers, eth0 should be set to only use its addresses for ARP. This is done by adding the following lines to /etc/sysctl.conf. The eth0 lines below should be replicated for any devices that respond to ARP.
# Enable configuration of arp_ignore option net.ipv4.conf.all.arp_ignore = 1 # When an arp request is received on eth0, only respond if that address is # configured on eth0. In particular, do not respond if the address is # configured on lo net.ipv4.conf.eth0.arp_ignore = 1 # Ditto for eth1, add for all ARPing interfaces #net.ipv4.conf.eth1.arp_ignore = 1 # Enable configuration of arp_announce option net.ipv4.conf.all.arp_announce = 2 # When making an ARP request sent through eth0 Always use an address that # is configured on eth0 as the source address of the ARP request. If this # is not set, and packets are being sent out eth0 for an address that is on # lo, and an arp request is required, then the address on lo will be used. # As the source IP address of arp requests is entered into the ARP cache on # the destination, it has the effect of announcing this address. This is # not desirable in this case as adresses on lo on the real-servers should # be announced only by the linux-director. net.ipv4.conf.eth0.arp_announce = 2 # Ditto for eth1, add for all ARPing interfaces #net.ipv4.conf.eth1.arp_announce = 2
The Red Hat Enterprise Linux kernel does not currently have the arp_ignore and arp_announce flags. This is partly because adding them will break the kernel ABI. If that occurs for another reason, then they may be added, but in the mean time a arptables can be used to obtain mich the same effect. Be for running the commands below, make sure that the the arptables_jf package version >=0.0.7, supplied by Red Hat Enterprise Linux is installed.
# Flush existing rules /etc/init.d/arptables_jf stop Flushing all chains: [ OK ] Removing user defined chains: [ OK ] Resetting built-in chains to the default ACCEPT policy: [ OK ] # Drop ARP requests to and from the virtual IP address /usr/sbin/arptables-noarp-addr 192.168.6.240 start # This the above command produces the following rules, # dropping incoming ARP requests for the VIP (192.168.6.240) # and mangles outgoing ARP announcements on each ARPing interface (eth0) # See arptables-noarp-addr(8) for more details #/sbin/arptables -A IN -j DROP -d 192.168.6.240 #/sbin/arptables -A OUT -j mangle -o eth0 -s 192.168.6.240 --mangle-ip-s 192.168.6.2 # Save the rules /etc/init.d/arptables_jf save Saving current rules to /etc/sysconfig/arptables: [ OK ] # Make sure rules are activated on reboot (at run levels 2, 3, 4 and 5) /sbin/chkconfig --level 2345 arptables_jf on # Activate the rules /etc/init.d/arptables_jf start Flushing all current rules and user defined chains: [ OK ] Clearing all current rules and user defined chains: [ OK ] Applying arptables firewall rules: [ OK ] # Make sure these rules are deactivated when the VIP is obtained, # And activated when it is released # This does not need to be done on hosts that will not act as linux-directors ( cd /etc/ha.d/rc.d && ln -s arptables-noarp-addr_takeip local_takeip; ) ( cd /etc/ha.d/rc.d && ln -s arptables-noarp-addr_giveip local_giveip; )
You can inspect the arptables rules, and examine many hits they have had, by using the arptables command.
/sbin/arptables -L -v -n Chain IN (policy ACCEPT 229 packets, 6412 bytes) pkts bytes target in out source-ip destination-ip source-hw destination-hw hlen op hrd pro 1 28 DROP * * 0.0.0.0/0 192.168.6.240 00/00 00/00 any 0000/0000 0000/0000 0000/0000 Chain OUT (policy ACCEPT 40 packets, 1120 bytes) pkts bytes target in out source-ip destination-ip source-hw destination-hw hlen op hrd pro 0 0 mangle * eth0 192.168.6.240 0.0.0.0/0 00/00 00/00 any 0000/0000 0000/0000 0000/0000 --mangle-ip-s 192.168.6.2
Now that ARP Adviertisments of the virtual IP address have been controlled, it can be added as an alias to the loopback interface to allow the real-server to accept packets for the virutal service.
The file /etc/sysconfig/network-scripts/ifcfg-lo:0 configures the lo:0 interface, an example follows:
DEVICE=lo:0 IPADDR=192.168.6.240 NETMASK=255.255.255.255 NETWORK=192.168.0.0 BROADCAST=192.168.0.255 ONBOOT=yes NAME=loopbackTo bring up the IP alias the ifup command is used:
The lo:0 interface can be configured by adding an entry to /etc/network/interfaces, an example showing the lo and lo:0 interfaces follows.
auto lo iface lo inet loopback auto lo:0 iface lo:0 inet static address 192.168.6.240 netmask 255.255.255.255 pre-up sysctl -p > /dev/nullTo bring up the IP alias the ifup command is used:
ip addr sh lo 1: lo:Note that a netmask of 255.255.255.255 on the lo:0 interface indicates that this interface will only accept traffic 192.168.6.240.
mtu 16436 qdisc noqueue link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo inet 192.168.6.240/32 scope global lo:0
The linux-directors / real-servers need to be set up so that their default route is set to the gateway router's address on the server network and not an address on one of the linux-directors.
The gateway is configured as part of the interface configuation in /etc/network/interfaces. This example interface configuration in /etc/network/interfaces shows how to set a gateway of 192.168.6.254 for eth0.
auto eth0 iface eth0 inet static address 192.168.6.4 netmask 255.255.255.0 gateway 192.168.6.254
To effect this change networking should be restarted:
/etc/init.d/networking restart Reconfiguring network interfaces: done.
Using the simple case of a single gateway with an address on the server network of 192.168.6.254 this can be done by editing the GATEWAY specified in the interface configuration for the interface connected to 192.168.6.0/24. In the case of eth0, this is, /etc/sysconfig/network-scripts/ifcfg-eth0. A sample of this file follows:
DEVICE=eth0 ONBOOT=yes BOOTPROTO=static IPADDR=192.168.6.5 NETMASK=255.255.255.0 GATEWAY=192.168.6.254
To effect this networking should be restarted:
/etc/init.d/network restart Shutting down interface eth0 [ OK ] Setting network parameters [ OK ] Bringing up interface lo [ OK ] Bringing up interface eth0 [ OK ]
This change can be verified using the ip command.
/sbin/ip route show 0/0 default via 192.168.6.254 dev eth0
The linux-directors / real-servers should be configured to run the underlying services for their respective virtual services. For instance, an HTTP daemon, such as Apache should be configured on each Real Server if an HTTP virtual service has been set up. In addition the the request URLs as specified in /etc/ha.d/ldirectord.cf should be present and contain the receive string.
The linux-directors / real-servers must be able to route traffic to the real-servers. Specifically in addition to correctly configuring the interfaces and routes you must enable IPV4 forwarding. This is done by adding configuration of net.ipv4.ip_forward to /etc/sysctl.conf as follows:
# Enables packet forwarding net.ipv4.ip_forward = 1For these changes to take effect the sysctl command may be used:
/sbin/sysctl -p net.ipv4.ip_forward = 1
Heartbeat runs on the two Linux-Directors and handles bringing up the interface for the virtual address. This is the address to which end-users should connect, and is typically advertised using DNS.
To configure heartbeat /etc/ha.d/ha.cf, /etc/ha.d/haresources and /etc/ha.d/authkeys must be installed. The node names in /etc/ha.d/ha.cf and /etc/ha.d/haresources have to be set according to the output of the uname -n command on each linux-director. The key ultramonkey in /etc/ha.d/authkeys, should be modified to something confidential to the site. The /etc/ha.d/authkeys must be mode 600, this can be done using the chmod command.
chmod 600 /etc/ha.d/authkeysThe configuration files supplied expect that the linux-directors are connected via eth0 and by a null modem connected to /dev/ttyS0. This may be modified but it is highly recommended that heartbeat be run over at least two links.
The monitoring of real-servers, and their insertion and removal from the pool of servers available is controlled by ldirectord which is run by heartbeat. To configure ldirectord /etc/ha.d/ldirectord.cf must be installed. Information on customising this file can be found in the ldirectord(8) man page.
Ldirectord is run by heartbeat. On supported versions of Red Hat and Fedora to ensure that heartbeat starts up (on run-levels 2, 3, 4 and 5) and that ldirectord does not start on reboot up the chkconfig command is used:
/sbin/chkconfig --level 2345 heartbeat on /sbin/chkconfig --del ldirectord
On Debian the update-rc.d command is used:
/usr/sbin/update-rc.d heartbeat start 75 2 3 4 5 . stop 05 0 1 6 . /usr/sbin/update-rc.d -f ldirectord removeTo ensure that ldirectord is not running and start heartbeat with the new configuration run:
/etc/init.d/ldirectord stop /etc/init.d/heartbeat start
After a few moments heartbeat should bring add the virtual IP address on eth0 and remove it from lo the master linux-director. This can be verified using the ip command.
ip addr sh 1: lo: <LOOPBACK,UP> mtu 16436 qdisc noqueue link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 brd 127.255.255.255 scope host lo 4: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000 link/ether 00:50:56:4f:30:07 brd ff:ff:ff:ff:ff:ff inet 192.168.6.2/24 brd 192.168.6.255 scope global eth0 inet 192.168.6.240/24 brd 192.168.6.255 scope global secondary eth0
Heartbeat should also run ldirectord to configure LVS on this node. To check that ldirectord is running use:
/usr/sbin/ldirectord ldirectord.cf statusTo inspect the current LVS kernel table the ipvsadm command may be used. A sample invocation follows showing that ldirectord found all servers to be available. Note that the FTP virtual service on port 21 is persistent when using direct routing (DR).
/sbin/ipvsadm -L -n Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 192.168.6.240:443 rr -> 192.168.6.4:443 Route 1 0 0 -> 192.168.6.5:443 Route 1 0 0 TCP 192.168.6.240:80 rr -> 192.168.6.4:80 Route 1 0 0 -> 192.168.6.5:80 Route 1 0 0 TCP 192.168.6.240:21 rr persistent 600 -> 192.168.6.4:21 Route 1 0 0 -> 192.168.6.5:21 Route 1 0 0
As well as adding the virtual ipaddresses and running ldirectord, heartbeat should also run the LVS synchronisation daemon. The master daemon should be running on the current master linux-director, and the backup daemon on the other linux-director. You can verify this by inspecting the status of LVSSyncDaemonSwap resource.
# On the master linux-director /etc/ha.d/resource.d/LVSSyncDaemonSwap master status master running (ipvs_syncmaster pid: 4530) # On the stand-by linux-director /etc/ha.d/resource.d/LVSSyncDaemonSwap master status master stopped (ipvs_syncbackup pid: 3895)
Failover can be tested by stopping heartbeat on the master linux-director which should cause the hot-standby linux-director to become the master.
Heartbeat and Ldirectord log debugging and status information to /var/log/messages using syslog. These logs should be inspected if problems occur. Please see notes on logging to ensure that all logs are written to disk for debugging purposes.
Additional real servers should be confugred as per above with the omission of configuring packet forwarding and heartbeat.
Copyright © 2000-2005,
Last Updated: Sat Mar 4 16:33:56 2006 +0900
Debian is a registered trademark of Software in the Public Interest, Inc.
Red Hat, the Red Hat Shadowman logo and Fedora are registered trademarks of Red Hat, Inc.
Red Hat may also be refered to as RedHat on this site.
Linux is a registered trademark of Linus Torvalds.
All other trademarks are the property of their respective owners.