Ultra Monkey: Topologies: Streamline High Availability and Load Balancing

Ultra Monkey: High Availability and Load Balancing Solution for Linux

Topologies

Streamline High Availability and Load Balancing

[Network Diagram]

This topology combines the function of Linux Directors and Real Servers into the same hosts, so that high availability and load balancing can be achived with just two nodes. For a more detailed explanation of the function of each component please see the overview.

The documentation that follows assumes that all nodes on the network are set up with correct interfaces and routes for each network they are connected to as per the diagram above

Linux Directors / Real Servers
The servers should be configured to run the underlying services for the virtual services. For instance, an HTTP daemon, such as Apache should be configured on each server if an HTTP virtual service has been set up. In addition the the request URLs as specified in /etc/ha.d/ldirectord.cf should be present and contain the receive string.
The servers need to be configured to see traffic for the virtual addresses in 192.168.6.240 as local, even if they are not acting ast the linux director. This can be done by making use of an IP alias on the loopback device.
- Supported Versions of Red Hat and Fedora
  The file /etc/sysconfig/network-scripts/ifcfg-lo:0 configures the lo:0 interface, an example follows:
```
DEVICE=lo:0
IPADDR=192.168.6.240
NETMASK=255.255.255.255
NETWORK=192.168.0.0
BROADCAST=192.168.0.255
ONBOOT=yes
NAME=loopback
```
  To avoid unnecessary warnings you should ensure that the gateway device is set to the ethernet interface. The following line from /etc/sysconfig/network sets the gateway device to eth0.
```
GATEWAYDEV=eth0
```
  To bring up the IP alias the ifup command is used:
```
/sbin/ifup lo
```
- Debian
  The lo:0 interface can be configured by adding an entry to /etc/network/interfaces, an example showing the lo and lo:0 interfaces follows.
```
auto lo
iface lo inet loopback

auto lo:0
iface lo:0 inet static
  address 192.168.6.240
  netmask 255.255.255.255
  up sysctl -p > /dev/null
```
  To bring up the IP alias the ifup command is used:
```
/sbin/ifup lo:0
```
You can verify that the interface came up using the ifconfig command. The following output has been truncated to only show information for lo:0.
```
/sbin/ifconfig
lo:0      Link encap:Local Loopback  
          inet addr:192.168.0.0  Mask:255.255.255.0
	  UP LOOPBACK RUNNING  MTU:3924  Metric:1
```
Note: The netmask of 255.255.255.255 on the lo:0 interface indicates that this interface will only accept traffic 192.168.6.240.
To ensure that the loopback device does not answer ARP requests the interface should be made hidden. This is done by adding the following lines to /etc/sysctl.conf.
```
# Enable configuration of hidden devices
net.ipv4.conf.all.hidden = 1
# Make the loopback device hidden
net.ipv4.conf.lo.hidden = 1
```
For these changes to take effect the sysctl may be used:
```
/sbin/sysctl -p
```
Due the the ordering of boot-up on Debian and supported versions of Red Hat and Fedora sysctl is called before interfaces are brought up. This means that entries in /etc/sysctl.conf that act on the loopback interface will not take effect and interfaces will not be hidden on reboot as desired.
- Supported Versions of Red Hat and Fedora
  To resolve this problem add the following to /etc/rc.d/rc.local:
```
if [ -x /sbin/sysctl -a -e /etc/sysctl.conf ]; then
  echo -n "Hiding loopback interface"
  /sbin/sysctl -p /etc/sysctl.conf > /dev/null
  echo
fi
```
- Debian
  On Debian this is resolved using the up in the interface configuration for lo:0 above which will be called after the lo:0 interfce is brought up. No futher configuration is required for sysctl on Debian.
Heartbeat runs on the two Linux Directors / Real Servers and handles bringing up the interface for the virtual address. This is the address to which end users should connect, and is typically advertised using DNS.
To configure heartbeat /etc/ha.d/ha.cf, /etc/ha.d/haresources and /etc/ha.d/authkeys must be installed. The node names in /etc/ha.d/ha.cf and /etc/ha.d/haresources have to be set according to the output of the uname -n command on each linux director. The key ultramonkey in /etc/ha.d/authkeys, should be modified to something confidential to the site. The /etc/ha.d/authkeys must be mode 600, this can be done using the chmod command.
```
chmod 600 /etc/ha.d/authkeys
```
The configuration files supplied expect that the linux directors are connected via eth0 and by a null modem connected to /dev/ttyS0. This may be modified but it is highly recommended that heartbeat be run over at least two links.
The monitoring of real servers, and their insertion and removal from the pool of servers available is controlled by ldirectord which is run by heartbeat. To configure ldirectord /etc/ha.d/ldirectord.cf must be installed. Information on customising this file can be found in the ldirectord(8) man page.
Ldirectord is run by heartbeat. On supported versions of Red Hat and Fedora, to ensure that heartbeat starts up (on run-levels 2, 3, 4 and 5) and that ldirectord does not start on reboot up the chkconfig command is used:
```
/sbin/chkconfig --level 2345 heartbeat on
/sbin/chkconfig --del ldirectord
```
On Debian the update-rc.d command is used:
```
/usr/sbin/update-rc.d heartbeat start 2 3 4 5 .
/usr/sbin/update-rc.d -f ldirectord remove
```
To ensure that ldirectord is not running and start heartbeat with the new configuration run:
```
/etc/init.d/ldirectord stop
/etc/init.d/heartbeat start
```
After a few moments heartbeat should bring up an IP alias for the virtual address on the first real server that heartbeat is started on, the master. This can be verified using the ifconfig command. Heartbeat should also run ldirectord to configure LVS on this node. To check that ldirectord is running use:
```
/usr/sbin/ldirectord ldirectord.cf status
```
To inspect the current LVS kernel table the ipvsadm command may be used. A sample invocation follows showing that ldirectord found all servers to be available. Note that the FTP virtual service on port 21 is persistent when using direct routing (DR).
```
/sbin/ipvsadm -L -n
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port          Forward Weight ActiveConn InActConn
TCP  192.28.6.240:443 rr
  -> 192.28.6.2:443              Route   1      0          0         
  -> 192.28.6.3:443              Route   1      0          0         
  -> 192.28.6.6:443              Route   1      0          0         
TCP  192.28.6.240:80 rr
  -> 192.28.6.2:80               Route   1      0          0         
  -> 192.28.6.3:80               Route   1      0          0         
  -> 192.28.6.6:80               Route   1      0          0         
TCP  192.28.6.240:21 rr persistent 600
  -> 192.28.6.2:21               Route   1      0          0         
  -> 192.28.6.3:21               Route   1      0          0        
  -> 192.28.6.6:21               Route   1      0          0        
```
The other server should become the standby and stopping heartbeat on the master with the following command should effect a fail over:
```
/etc/init.d/heartbeat stop
```
Heartbeat and Ldirectord log debugging and status information to /var/log/messages using syslog. These logs should be inspected if problems occur. Please see notes on logging to ensure that all logs are written to disk for debugging purposes.
The linux directors / real servers need to be set up so that their default route is set to the gateway router's address on the network and not an address on one of the linux directors.
- Supported Versions of Red Hat and Fedora
  Using the simple case of a single gateway with an address on the server network of 192.168.6.1 this can be done by editing the GATEWAY specified in the interface configuration for the interface connected to 192.168.6.0/24. In the case of eth0, this is, /etc/sysconfig/network-scripts/ifcfg-eth0. A sample of this file follows:
```
DEVICE=eth0
ONBOOT=yes
BOOTPROTO=static
IPADDR=192.168.6.2
NETMASK=255.255.255.0
GATEWAY=192.168.6.1
```
  To effect this networking should be restarted:
```
/etc/init.d/network restart
Shutting down interface eth0                               [  OK  ] 
Setting network parameters                                 [  OK  ]
Bringing up interface lo                                   [  OK  ]
Bringing up interface eth0                                 [  OK  ]
```
- Debian The gateway is configured as part of the interface configuation in /etc/network/interfaces This example interface configuration in /etc/network/interfaces shows how to set a gateway of 192.168.6.1 for eth0.
```
auto eth0
iface eth0 inet static
   address 192.168.6.2
   netmask 255.255.255.0
   gateway 192.168.6.1
```
  To effect this change networking should be restarted:
```
/etc/init.d/networking restart
Reconfiguring network interfaces: done.
```
This can be verified using the ip command.
```
/sbin/ip route show 0/0
default via 192.168.6.1 dev eth0
```

Copyright © 2000-2005, Horms
Last Updated: Tue May 17 17:37:24 2005 +0900

Debian is a registered trademark of Software in the Public Interest, Inc.
Red Hat, the Red Hat Shadowman logo and Fedora are registered trademarks of Red Hat, Inc.
Red Hat may also be refered to as RedHat on this site.
Linux is a registered trademark of Linus Torvalds.
All other trademarks are the property of their respective owners.