This topology provides a highly available and load balanced service. A minimum of 4 nodes is required for this topology, and real servers can be added to the network as additional capacity is required. For a more detailed explanation of the function of each component please see the overview.
The documentation that follows assumes that all nodes on the network are set up with correct interfaces and routes for each network they are connected to as per the diagram above. The return path for packets must be through the active Linux Director. In most cases this will mean that the the default route should be set to the internal virtual address as described in the real servers section below.
The Linux Directors must be able to route traffic from the external network to the server network and vice versa. Specifically, in addition to correctly configuring the interfaces and routes you must enable IPV4 forwarding. This is done by modifying the line containing net.ipv4.ip_forward in /etc/sysctl.conf. An example /etc/sysctl.conf follows:
# Enables packet forwarding net.ipv4.ip_forward = 1 # Enables source route verification net.ipv4.conf.default.rp_filter = 1 # Disables the magic-sysrq key kernel.sysrq = 0For these changes to take effect the sysctl command may be used:
/sbin/sysctl -p net.ipv4.ip_forward = 1 net.ipv4.conf.default.rp_filter = 1 kernel.sysrq = 0Heartbeat runs on the two Linux Directors and handles bringing up the interface for the virtual address. This is the address to which end users should connect, and is typically advertised using DNS. and /etc/ha.d/haresources have to be set according to the output of the uname -n command on each linux director. The key ultramonkey in /etc/ha.d/authkeys, should be modified to something confidential to the site. The /etc/ha.d/authkeys must be mode 600, this can be done using the chmod command.
chmod 600 /etc/ha.d/authkeysThe configuration files supplied expect that the linux directors are connected via eth0, eth1 and by a null modem connected to /dev/ttyS0. This may be modified but it is highly recommended that heartbeat be run over at least two links.
The monitoring of real servers, and their insertion and removal from the pool of servers available is controlled by ldirectord which is run by heartbeat. To configure ldirectord /etc/ha.d/ldirectord.cf must be installed. Information on customising this file can be found in the ldirectord(8) man page.
Ldirectord is run by heartbeat. On supported versions of Red Hat and Fedora to ensure that heartbeat starts up (on run-levels 2, 3, 4 and 5) and that ldirectord does not start on reboot up the chkconfig command is used:
/sbin/chkconfig --level 2345 heartbeat on /sbin/chkconfig --del ldirectord
On Debian the update-rc.d command is used:
/usr/sbin/update-rc.d heartbeat start 2 3 4 5 . /usr/sbin/update-rc.d -f ldirectord removeTo ensure that ldirectord is not running and start heartbeat with the new configuration run:
/etc/init.d/ldirectord stop /etc/init.d/heartbeat start
After a few moments heartbeat should bring up an IP alias for the virtual address on the master linux director. This can be verified using the ifconfig command. The output of the following command has been truncated to only show the eth0:0 and eth1:0 interfaces. Depending on the setup of the host it is possible that heartbeat will use difference interfaces.
/sbin/ifconfig eth0:0 Link encap:Ethernet HWaddr 00:D0:B7:BE:6B:CF inet addr:192.168.6.240 Bcast:192.168.6.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 Interrupt:17 Base address:0xef00 eth1:0 Link encap:Ethernet HWaddr 00:90:27:74:84:ED inet addr:192.168.7.340 Bcast:192.168.7.355 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 Interrupt:18 Base address:0xee80Heartbeat should also run ldirectord to configure LVS on this node. To check that ldirectord is running use:
/usr/sbin/ldirectord ldirectord.cf status ldirectord for ldirectord.cf is running with pid: 30314To inspect the current LVS kernel table the ipvsadm command may be used. A sample invocation follows showing that ldirectord found all servers to be available:
/sbin/ipvsadm -L -n IP Virtual Server version 0.9.16 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 192.168.7.340:443 rr -> 192.168.6.4:443 Masq 1 0 0 -> 192.168.6.5:443 Masq 1 0 0 TCP 192.168.7.340:80 rr -> 192.168.6.4:80 Masq 1 0 0 -> 192.168.6.5:80 Masq 1 0 0 TCP 192.168.7.340:21 rr -> 192.168.6.4:21 Masq 1 0 0 -> 192.168.6.5:21 Masq 1 0 0The other server should become the stand-by and stopping heartbeat on the master with the following command should effect a fail over:
/etc/init.d/heartbeat stopHeartbeat and Ldirectord log debugging and status information to /var/log/messages using syslog. These logs should be inspected if problems occur. Please see notes on logging to ensure that all logs are written to disk for debugging purposes.
As masquerading is being used as the forwarding mechanism by LVS, as per the ldirectord.cf, the linux director must masquerade for the real servers.
On supported versions of Red Hat and Fedora masquerading can be configured using the iptables init script.
# Flush existing rules in the nat table /etc/init.d/iptables stop Resetting built-in chains to the default ACCEPT policy: [ OK ] # Masquerade for 192.168.6.0/24 bound for any host /sbin/iptables -t nat -A POSTROUTING -j MASQUERADE -s 192.168.6.0/24 # Log all packets that attempt to be forwarded # Useful for Debugging. Questionable for Production #/sbin/iptables -t nat -A POSTROUTING -j LOG # Save the rules /etc/init.d/iptables save Saving current rules to /etc/sysconfig/iptables: [ OK ] # Make sure rules are activated on reboot (at run levels 2, 3, 4 and 5) /sbin/chkconfig --level 2345 iptables on # Activate the rules /etc/init.d/iptables start Flushing all current rules and user defined chains: [ OK ] Clearing all current rules and user defined chains: [ OK ] Applying iptables firewall rules: [ OK ]
On Debian masqurading can be configured as part of the interface configuration for the interface that is on Real Servers' network. This example entry in /etc/network/interfaces Sets up masquerading on eth1.
auto eth1 iface eth1 inet static address 192.168.6.4 netmask 255.255.255.0 up iptables -t nat -A POSTROUTING -j MASQUERADE -s 192.168.6.0/24 down iptables -t nat -D POSTROUTING -j MASQUERADE -s 192.168.6.0/24
For this change to take palce restart networking.
To verify the masquerading rules the iptables command may be used. The following example has been truncated to show only the POSTROUTING chain.
/sbin/iptables -t nat -L -n Chain POSTROUTING (policy ACCEPT) target prot opt source destination MASQUERADE all -- 192.168.6.0/24 0.0.0.0/0
If an FTP Virtual Service is to be used then the ip_vs_ftp kernel module needs to be used. This may be done by running the following command.
If this module is needed it is important to ensure that it is inserted into the kernel at boot time.
On supported versions of Red Hat and Fedora appended the above modprobe command to /etc/rc.local.
The modules presence in the kernel may be checked using lsmod. The output of the following command has been truncated to only show the ip_vs_ftp module.
/sbin/lsmod Module Size Used by ip_vs_ftp 3232 0
The Real Servers should be configured to run the underlying services for their respective virtual services. For instance, an HTTP daemon, such as Apache should be configured on each Real Server if an HTTP virtual service has been set up. In addition the the request URLs as specified in /etc/ha.d/ldirectord.cf should be present and contain the receive string.
As connections are forwarded to the real servers using NAT it is important that the return path for these connections passes through the linux director. This is so that NAT process can be reversed, else the return packet received by the end user will be from the real server and not the linux director and thus the connection will be dropped. This is usually achieved, by having the internal IP address of the linux director, in this example 192.168.6.240, as the default gateway for the real servers.
Using the simple case of a single gateway with an address on the server network of 192.168.6.240 this can be done by editing the GATEWAY specified in the interface configuration for the interface connected to 192.168.6.0/24. In the case of eth0, this is, /etc/sysconfig/network-scripts/ifcfg-eth0. A sample of this file follows:
DEVICE=eth0 ONBOOT=yes BOOTPROTO=static IPADDR=192.168.6.4 NETMASK=255.255.255.0 GATEWAY=192.168.6.240
To effect this networking should be restarted:
/etc/init.d/network restart Shutting down interface eth0 [ OK ] Setting network parameters [ OK ] Bringing up interface lo [ OK ] Bringing up interface eth0 [ OK ]
auto eth0 iface eth0 inet static address 192.168.6.4 netmask 255.255.255.0 gateway 192.168.6.240
To effect this change networking should be restarted:
/etc/init.d/networking restart Reconfiguring network interfaces: done.
/sbin/ip route show 0/0 default via 192.168.6.240 dev eth0
Copyright © 2000-2005,
Last Updated: Tue May 17 17:37:24 2005 +0900
Debian is a registered trademark of Software in the Public Interest, Inc.
Red Hat, the Red Hat Shadowman logo and Fedora are registered trademarks of Red Hat, Inc.
Red Hat may also be refered to as RedHat on this site.
Linux is a registered trademark of Linus Torvalds.
All other trademarks are the property of their respective owners.