This topology provides a highly available service with minimal hardware requirements. For a more detailed explanation of the function of each component please see the overview.
The documentation that follows assumes that all nodes on the network are set up with correct interfaces and routes for each network they are connected to as per the diagram above. The return path for packets must be through the linux-director. In most cases this will mean that the the default route should be set to the linux-director.
The linux-director must be able to route traffic from the external network to the server network and vice versa. Specifically, in addition to correctly configuring the interfaces and routes IPV4 forwarding must be enabled. This is done by adding configuration of net.ipv4.ip_forward to /etc/sysctl.conf as follows:
# Enables packet forwarding net.ipv4.ip_forward = 1For these changes to take effect the sysctl command may be used:
/sbin/sysctl -p net.ipv4.ip_forward = 1
The monitoring of real-servers, and their insertion and removal from the pool of servers available is controlled by ldirectord. To configure ldirectord /etc/ha.d/ldirectord.cf must be installed. Information on customising this file can be found in the ldirectord(8) man page.
On Debian the update-rc.d command is used:
/usr/sbin/update-rc.d heartbeat start 75 2 3 4 5 . stop 05 0 1 6 . /usr/sbin/update-rc.d -f heartbeat remove
Red Hat Enterprise Linux to ensure that ldirectord starts up (on run-levels 2, 3, 4 and 5) and that heartbeat does not start on reboot up the chkconfig command is used:
/sbin/chkconfig --level 2345 ldirectord on /sbin/chkconfig --del heartbeat
To ensure that heartbeat is not running and start ldirectord with the new configuration run:
/etc/init.d/heartbeat stop /etc/init.d/ldirectord start
Ldirectord logs debugging and status information to /var/log/messages using syslog. These logs should be inspected if problems occur. Please see notes on logging to ensure that all logs are written to disk for debugging purposes.
The current Linux Virtual Server kernel table may be displayed using the ipvsadm command. A sample invocation follows showing that ldirectord found all servers to be available:
ipvsadm -L -n IP Virtual Server version 1.0.11 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 192.168.6.240:21 rr persistent 600 -> 192.168.7.4:21 Masq 1 0 0 -> 192.168.7.5:21 Masq 1 0 0 TCP 192.168.6.240:80 rr -> 192.168.7.4:80 Masq 1 0 0 -> 192.168.7.5:80 Masq 1 0 0 TCP 192.168.6.240:443 rr -> 192.168.7.4:443 Masq 1 0 0 -> 192.168.7.5:443 Masq 1 0 0
NAT (masq) is being used as the forwarding mechanism by LVS, as per the ldirectord.cf. LVS will handle NAT of incoming, load-balanced connections. However, you may also want to NAT outgoing connections that originatel from the real-servers. This can be done by the linux-director as follows.
Nat can be configured using the iptables init script.
# Flush existing rules in the nat table /etc/init.d/iptables stop Resetting built-in chains to the default ACCEPT policy: [ OK ] # Masquerade for 192.168.7.0/24 bound for any host /sbin/iptables -t nat -A POSTROUTING -j MASQUERADE -s 192.168.7.0/24 # Log all packets that attempt to be forwarded # Useful for Debugging. Questionable for Production #/sbin/iptables -t nat -A POSTROUTING -j LOG # Save the rules /etc/init.d/iptables save Saving current rules to /etc/sysconfig/iptables: [ OK ] # Make sure rules are activated on reboot (at run levels 2, 3, 4 and 5) /sbin/chkconfig --level 2345 iptables on # Activate the rules /etc/init.d/iptables start Flushing all current rules and user defined chains: [ OK ] Clearing all current rules and user defined chains: [ OK ] Applying iptables firewall rules: [ OK ]
On Debian masqurading can be configured as part of the interface configuration for the interface that is on Real Servers' network. This example entry in /etc/network/interfaces Sets up masquerading on eth1.
auto eth1 iface eth1 inet static address 192.168.7.4 netmask 255.255.255.0 up iptables -t nat -A POSTROUTING -j MASQUERADE -s 192.168.7.0/24 down iptables -t nat -D POSTROUTING -j MASQUERADE -s 192.168.7.0/24
For this change to take palce restart networking.
To verify the masquerading rules, and inspect how many hits they have had, the iptables command may be used.
/sbin/iptables -t nat -L POSTROUTING -n -v Chain POSTROUTING (policy ACCEPT) pkts bytes target prot opt source destination 25957 1592K MASQUERADE all -- 192.168.7.0/24 0.0.0.0/0
If an FTP Virtual Service is to be used then the ip_vs_ftp kernel module needs to be used. This may be done by running the following command.
If this module is needed it is important to ensure that it is inserted into the kernel at boot time.
On Debian append the following line to /etc/modules
On Red Hat Enterprise Linux appended the above modprobe command to /etc/rc.local.
The modules presence in the kernel may be checked using lsmod. The output of the following command has been truncated to only show the ip_vs_ftp module.
/sbin/lsmod Module Size Used by ip_vs_ftp 3232 0
The real-servers should be configured to run the underlying services for their respective virtual services. For instance, an HTTP daemon, such as Apache [offsite] should be configured on each Real Server if an HTTP virtual service has been set up. In addition the the request URLs as specified in /etc/ha.d/ldirectord.cf should be present and contain the receive string.
As connections are forwarded to the real-servers using NAT it is important that the return path for these connections passes through the linux-director. This is so that NAT process can be reversed, else the return packet received by the end-user will be from the real-server and not the linux-director and thus the connection will be dropped. This is usually achieved, by having the internal IP address of the linux-director, in this example 192.168.7.1, as the default gateway for the real-servers.
Using the simple case of a single gateway with an address on the server network of 192.168.7.1 this can be done by editing the GATEWAY specified in the network configuration file, /etc/sysconfig/network.
NETWORKING=yes HOSTNAME=b1.lab.ultramonkey.org GATEWAY=192.168.7.240
To effect this networking should be restarted:
/etc/init.d/network restart Shutting down interface eth0 [ OK ] Setting network parameters [ OK ] Bringing up interface lo [ OK ] Bringing up interface eth0 [ OK ]
The gateway is configured as part of the interface configuation in /etc/network/interfaces This example interface configuration in /etc/network/interfaces shows how to set a gateway of 192.168.7.1 for eth0.
auto eth0 iface eth0 inet static address 192.168.7.4 netmask 255.255.255.0 gateway 192.168.7.240
To effect this change networking should be restarted:
/etc/init.d/networking restart Reconfiguring network interfaces: done.
/sbin/ip route show 0/0 default via 192.168.7.240 dev eth0
Copyright © 2000-2005,
Last Updated: Sat Mar 4 16:33:56 2006 +0900
Debian is a registered trademark of Software in the Public Interest, Inc.
Red Hat, the Red Hat Shadowman logo and Fedora are registered trademarks of Red Hat, Inc.
Red Hat may also be refered to as RedHat on this site.
Linux is a registered trademark of Linus Torvalds.
All other trademarks are the property of their respective owners.