This topology enables the maximum throughput through the network as return traffic no longer has to travel through an Linux Director. This topology builds on the existing routing infrastructure of a network by allowing existing internal routing policies to dictate the flow of outbound traffic. For a more detailed explanation of the function of each component please see the overview.
The documentation that follows assumes that all nodes on the network are set up with correct interfaces and routes for each network they are connected to as per the diagram above.
The Linux Directors must be able to route traffic to the real servers. Specifically in addition to correctly configuring the interfaces and routes you must enable IPV4 forwarding. This is done by modifying the line containing net.ipv4.ip_forward in /etc/sysctl.conf. An example /etc/sysctl.conf follows:
# Enables packet forwarding net.ipv4.ip_forward = 1 # Enables source route verification net.ipv4.conf.default.rp_filter = 1For these changes to take effect the sysctl command may be used:
/sbin/sysctl -p net.ipv4.ip_forward = 1 net.ipv4.conf.default.rp_filter = 1
Heartbeat runs on the two Linux Directors and handles bringing up the interface for the virtual address. This is the address to which end users should connect, and is typically advertised using DNS.and /etc/ha.d/haresources have to be set according to the output of the uname -n command on each linux director. The key ultramonkey in /etc/ha.d/authkeys, should be modified to something confidential to the site. The /etc/ha.d/authkeys must be mode 600, this can be done using the chmod command.
chmod 600 /etc/ha.d/authkeysThe configuration files supplied expect that the linux directors are connected via eth0 and by a null modem connected to /dev/ttyS0. This may be modified but it is highly recommended that heartbeat be run over at least two links.
The monitoring of real servers, and their insertion and removal from the pool of servers available is controlled by ldirectord which is run by heartbeat. To configure ldirectord /etc/ha.d/ldirectord.cf must be installed. Information on customising this file can be found in the ldirectord(8) man page.
Ldirectord is run by heartbeat. On supported versions of Red Hat and Fedora to ensure that heartbeat starts up (on run-levels 2, 3, 4 and 5) and that ldirectord does not start on reboot up the chkconfig command is used:
/sbin/chkconfig --level 2345 heartbeat on /sbin/chkconfig --del ldirectord
On Debian the update-rc.d command is used:
/usr/sbin/update-rc.d heartbeat start 2 3 4 5 . /usr/sbin/update-rc.d -f ldirectord removeTo ensure that ldirectord is not running and start heartbeat with the new configuration run:
/etc/init.d/ldirectord stop /etc/init.d/heartbeat start
After a few moments heartbeat should bring up an IP alias for the virtual address on the first real server that heartbeat is started on, the master. This can be verified using the ifconfig command. Heartbeat should also run ldirectord to configure LVS on this node. To check that ldirectord is running use:
/usr/sbin/ldirectord ldirectord.cf statusTo inspect the current LVS kernel table the ipvsadm command may be used. A sample invocation follows showing that ldirectord found all servers to be available. Note that the FTP virtual service on port 21 is persistent when using direct routing (DR).
/sbin/ipvsadm -L -n Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 188.8.131.52:443 rr -> 184.108.40.206:443 Route 1 0 0 -> 220.127.116.11:443 Route 1 0 0 -> 18.104.22.168:443 Route 1 0 0 TCP 22.214.171.124:80 rr -> 126.96.36.199:80 Route 1 0 0 -> 188.8.131.52:80 Route 1 0 0 -> 184.108.40.206:80 Route 1 0 0 TCP 220.127.116.11:21 rr persistent 600 -> 18.104.22.168:21 Route 1 0 0 -> 22.214.171.124:21 Route 1 0 0 -> 126.96.36.199:21 Route 1 0 0The other server should become the standby and stopping heartbeat on the master with the following command should effect a fail over:
/etc/init.d/heartbeat stopHeartbeat and Ldirectord log debugging and status information to /var/log/messages using syslog. These logs should be inspected if problems occur. Please see notes on logging to ensure that all logs are written to disk for debugging purposes.
The Real Servers should be configured to run the underlying services for their respective virtual services. For instance, an HTTP daemon, such as Apache should be configured on each Real Server if an HTTP virtual service has been set up. In addition the the request URLs as specified in /etc/ha.d/ldirectord.cf should be present and contain the receive string.
The real servers need to be configured to see traffic for the virtual address 192.168.6.240 as local. This can be done by making use of an IP alias on the loopback device.
The file /etc/sysconfig/network-scripts/ifcfg-lo:0 configures the lo:0 interface, an example follows:
DEVICE=lo:0 IPADDR=192.168.6.240 NETMASK=255.255.255.255 NETWORK=192.168.0.0 BROADCAST=192.168.0.255 ONBOOT=yes NAME=loopback
To avoid unnecessary warnings you should ensure that the gateway device is set to the ethernet interface. The following line from /etc/sysconfig/network sets the gateway device to eth0.
GATEWAYDEV=eth0To bring up the IP alias the ifup command is used:
The lo:0 interface can be configured by adding an entry to /etc/network/interfaces, an example showing the lo and lo:0 interfaces follows.
auto lo iface lo inet loopback auto lo:0 iface lo:0 inet static address 192.168.6.240 netmask 255.255.255.255 up sysctl -p > /dev/nullTo bring up the IP alias the ifup command is used:
/sbin/ifconfig lo:0 Link encap:Local Loopback inet addr:192.168.0.0 Mask:255.255.255.0 UP LOOPBACK RUNNING MTU:3924 Metric:1Note: The netmask of 255.255.255.255 on the lo:0 interface indicates that this interface will only accept traffic 192.168.6.240.
To ensure that the loopback device does not answer ARP requests the interface should be made hidden. This is done by adding the following lines to /etc/sysctl.conf.
# Enable configuration of hidden devices net.ipv4.conf.all.hidden = 1 # Make the loopback device hidden net.ipv4.conf.lo.hidden = 1For these changes to take effect the sysctl may be used:
Due the the ordering of boot-up on Debian and, supported versions of Red Hat and Fedora, sysctl is called before interfaces are brought up. This means that entries in /etc/sysctl.conf that act on the loopback interface will not take effect and interfaces will not be hidden on reboot as desired.
To resolve this problem add the following to /etc/rc.d/rc.local:
if [ -x /sbin/sysctl -a -e /etc/sysctl.conf ]; then echo -n "Hiding loopback interface" /sbin/sysctl -p /etc/sysctl.conf > /dev/null echo fi
On Debian this is resolved using the up in the interface configuration for lo:0 above which will be called after the lo:0 interfce is brought up. No futher configuration is required for sysctl on Debian.
The real servers need to be set up so that their default route is set to the gateway router's address on the server network and not an address on one of the linux directors.
Using the simple case of a single gateway with an address on the server network of 192.168.6.1 this can be done by editing the GATEWAY specified in the interface configuration for the interface connected to 192.168.6.0/24. In the case of eth0, this is, /etc/sysconfig/network-scripts/ifcfg-eth0. A sample of this file follows:
DEVICE=eth0 ONBOOT=yes BOOTPROTO=static IPADDR=192.168.6.2 NETMASK=255.255.255.0 GATEWAY=192.168.6.1
To effect this networking should be restarted:
/etc/init.d/network restart Shutting down interface eth0 [ OK ] Setting network parameters [ OK ] Bringing up interface lo [ OK ] Bringing up interface eth0 [ OK ]
auto eth0 iface eth0 inet static address 192.168.6.2 netmask 255.255.255.0 gateway 192.168.6.1
To effect this change networking should be restarted:
/etc/init.d/networking restart Reconfiguring network interfaces: done.
/sbin/ip route show 0/0 default via 192.168.6.1 dev eth0
Copyright © 2000-2005,
Last Updated: Tue May 17 17:37:24 2005 +0900
Debian is a registered trademark of Software in the Public Interest, Inc.
Red Hat, the Red Hat Shadowman logo and Fedora are registered trademarks of Red Hat, Inc.
Red Hat may also be refered to as RedHat on this site.
Linux is a registered trademark of Linus Torvalds.
All other trademarks are the property of their respective owners.