![[Monkey]](../pics/monkey_wht.jpg)
![[Network Diagram]](svg/ha-lb-eg.png)
This topology provides a highly available and load balanced service. A minimum of 4 nodes is required for this topology, and real-servers can be added to the network as additional capacity is required. For a more detailed explanation of the function of each component please see the overview.
The documentation that follows assumes that all nodes on the network are set up with correct interfaces and routes for each network they are connected to as per the diagram above. The return path for packets must be through the active linux-director. In most cases this will mean that the the default route should be set to the internal virtual address as described in the real-servers section below.
IPV4 forwarding must be enabled. This is done by adding configuration of net.ipv4.ip_forward to /etc/sysctl.conf as follows:
# Enables packet forwarding net.ipv4.ip_forward = 1For these changes to take effect the sysctl command may be used:
/sbin/sysctl -p net.ipv4.ip_forward = 1
Heartbeat runs on the two linux-directors and ensures that the virtual IP address are present on the currently active linux-director. The external virtual IP address is the address taht end-users should connect, and is typically advertised using DNS. The internal virtual IP address is generally the default route of the real-servers.
To configure heartbeat /etc/ha.d/ha.cf, /etc/ha.d/haresources and /etc/ha.d/authkeys must be installed. The node names in /etc/ha.d/ha.cf
chmod 600 /etc/ha.d/authkeysThe configuration files supplied expect that the linux-directors are connected via eth0 and eth1. This may be modified but it is highly recommended that heartbeat be run over at least two links.
The monitoring of real-servers, and their insertion and removal from the pool of servers available is controlled by ldirectord which is run by heartbeat. To configure ldirectord /etc/ha.d/ldirectord.cf must be installed. Information on customising this file can be found in the ldirectord(8) man page.
Ldirectord is run by heartbeat. On supported versions of Red Hat and Fedora to ensure that heartbeat starts up (on run-levels 2, 3, 4 and 5) and that ldirectord does not start on reboot up the chkconfig command is used:
/sbin/chkconfig --level 2345 heartbeat on /sbin/chkconfig --del ldirectord
On Debian the update-rc.d command is used:
/usr/sbin/update-rc.d heartbeat start 75 2 3 4 5 . stop 05 0 1 6 . /usr/sbin/update-rc.d -f ldirectord removeTo ensure that ldirectord is not running and start heartbeat with the new configuration run:
/etc/init.d/ldirectord stop /etc/init.d/heartbeat start
After a few moments heartbeat should bring add the virtual IP address on the master linux-director. This can be verified using the ip command.
ip addr sh
1: lo: <LOOPBACK,UP> mtu 16436 qdisc noqueue
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
4: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 00:50:56:4f:30:07 brd ff:ff:ff:ff:ff:ff
inet 192.168.6.2/24 brd 192.168.6.255 scope global eth0
inet 192.168.6.240/24 brd 192.168.6.255 scope global secondary eth0
5: eth1: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 00:50:56:c1:49:c8 brd ff:ff:ff:ff:ff:ff
inet 192.168.7.2/24 brd 192.168.7.255 scope global eth1
inet 192.168.7.240/24 brd 192.168.7.255 scope global secondary eth1
Heartbeat should also run ldirectord to configure LVS on this node. To
check that ldirectord is running use:
/etc/ha.d/resource.d/ldirectord ldirectord.cf status ldirectord for ldirectord.cf is running with pid: 1617To inspect the current LVS kernel table the ipvsadm command may be used. A sample invocation follows showing that ldirectord found all servers to be available:
/sbin/ipvsadm -L -n IP Virtual Server version 1.0.11 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 192.168.6.240:443 rr -> 192.168.7.4:443 Masq 1 0 0 -> 192.168.7.5:443 Masq 1 0 0 TCP 192.168.6.240:80 rr -> 192.168.7.4:80 Masq 1 0 0 -> 192.168.7.5:80 Masq 1 0 0 TCP 192.168.7.240:21 rr -> 192.168.6.4:21 Masq 1 0 0 -> 192.168.6.5:21 Masq 1 0 0
As well as adding the virtual ipaddresses and running ldirectord, heartbeat should also run the LVS synchronisation daemon. The master daemon should be running on the current master linux-director, and the backup daemon on the other linux-director. You can verify this by inspecting the status of LVSSyncDaemonSwap resource.
# On the master linux-director /etc/ha.d/resource.d/LVSSyncDaemonSwap master status master running (ipvs_syncmaster pid: 4530) # On the stand-by linux-director /etc/ha.d/resource.d/LVSSyncDaemonSwap master status master stopped (ipvs_syncbackup pid: 3895)
Failover can be tested by stopping heartbeat on the master linux-director which should cause the hot-standby linux-director to become the master.
/etc/init.d/heartbeat stop
Heartbeat and Ldirectord log debugging and status information to /var/log/messages using syslog. These logs should be inspected if problems occur. Please see notes on logging to ensure that all logs are written to disk for debugging purposes.
NAT (masq) is being used as the forwarding mechanism by LVS, as per the ldirectord.cf. LVS will handle NAT of incoming, load-balanced connections. However, you may also want to NAT outgoing connections that originate from the real-servers. This can be done by the linux-director as follows.
Nat can be configured using the iptables init script.
# Flush existing rules in the nat table /etc/init.d/iptables stop Resetting built-in chains to the default ACCEPT policy: [ OK ] # Masquerade for 192.168.7.0/24 bound for any host /sbin/iptables -t nat -A POSTROUTING -j MASQUERADE -s 192.168.7.0/24 # Log all packets that attempt to be forwarded # Useful for Debugging. Questionable for Production #/sbin/iptables -t nat -A POSTROUTING -j LOG # Save the rules /etc/init.d/iptables save Saving current rules to /etc/sysconfig/iptables: [ OK ] # Make sure rules are activated on reboot (at run levels 2, 3, 4 and 5) /sbin/chkconfig --level 2345 iptables on # Activate the rules /etc/init.d/iptables start Flushing all current rules and user defined chains: [ OK ] Clearing all current rules and user defined chains: [ OK ] Applying iptables firewall rules: [ OK ]
On Debian masqurading can be configured as part of the interface configuration for the interface that is on Real Servers' network. This example entry in /etc/network/interfaces Sets up masquerading on eth1.
auto eth1 iface eth1 inet static address 192.168.7.4 netmask 255.255.255.0 up iptables -t nat -A POSTROUTING -j MASQUERADE -s 192.168.7.0/24 down iptables -t nat -D POSTROUTING -j MASQUERADE -s 192.168.7.0/24
For this change to take palce restart networking.
/etc/init.d/networking restart
To verify the masquerading rules the iptables command may be used. The following example has been truncated to show only the POSTROUTING chain.
/sbin/iptables -t nat -L -n Chain POSTROUTING (policy ACCEPT) target prot opt source destination MASQUERADE all -- 192.168.7.0/24 0.0.0.0/0
If an FTP Virtual Service is to be used then the ip_vs_ftp kernel module needs to be used. This may be done by running the following command.
/sbin/modprobe ip_vs_ftp
If this module is needed it is important to ensure that it is inserted into the kernel at boot time.
On Debian append the following line to /etc/modules
ip_vs_ftp
On Red Hat Enterprise Linux appended the above modprobe command to /etc/rc.local.
The modules presence in the kernel may be checked using lsmod. The output of the following command has been truncated to only show the ip_vs_ftp module.
/sbin/lsmod Module Size Used by ip_vs_ftp 3232 0
The real-servers should be configured to run the underlying services for their respective virtual services. For instance, an HTTP daemon, such as Apache [offsite] should be configured on each Real Server if an HTTP virtual service has been set up. In addition the the request URLs as specified in /etc/ha.d/ldirectord.cf should be present and contain the receive string.
As connections are forwarded to the real-servers using NAT it is important that the return path for these connections passes through the linux-director. This is so that NAT process can be reversed, else the return packet received by the end-user will be from the real-server and not the linux-director and thus the connection will be dropped. This is usually achieved, by having the internal IP address of the linux-director, in this example 192.168.7.1, as the default gateway for the real-servers.
Using the simple case of a single gateway with an address on the server network of 192.168.7.1 this can be done by editing the GATEWAY specified in the network configuration file, /etc/sysconfig/network.
NETWORKING=yes HOSTNAME=b1.lab.ultramonkey.org GATEWAY=192.168.7.240
To effect this networking should be restarted:
/etc/init.d/network restart Shutting down interface eth0 [ OK ] Setting network parameters [ OK ] Bringing up interface lo [ OK ] Bringing up interface eth0 [ OK ]
The gateway is configured as part of the interface configuation in /etc/network/interfaces This example interface configuration in /etc/network/interfaces shows how to set a gateway of 192.168.7.1 for eth0.
auto eth0 iface eth0 inet static address 192.168.7.4 netmask 255.255.255.0 gateway 192.168.7.240
To effect this change networking should be restarted:
/etc/init.d/networking restart Reconfiguring network interfaces: done.
/sbin/ip route show 0/0 default via 192.168.7.240 dev eth0
Copyright © 2000-2005,
Horms
Last Updated: Sat Mar 4 16:33:56 2006 +0900
Debian is a registered trademark of Software in the Public Interest, Inc.
Red Hat, the Red Hat Shadowman logo and Fedora are
registered trademarks of Red Hat, Inc.
Red Hat may also be refered to as RedHat on this site.
Linux is a registered trademark of Linus Torvalds.
All other trademarks are the property of their respective owners.