Ultra Monkey: The Linux Virtual Server

Ultra Monkey: High Availability and Load Balancing Solution for Linux

English | Japanese
Top | About | Mirrors | History | Contacts
News Archive | Papers | Ultra Monkey L7 [offsite]

Ultra Monkey 3 : Download | Installation | Configuration || Linux Virtual Server | Linux-HA | Ldirectord

The Linux Virtual Server

The Linux Virtual Server [offsite] (LVS) enables TCP/IP and UDP/IP connections to be load balanced This mechanism of connection control is often referred to as Layer 4 Switching as information available at layer 3 of the ISO seven layer protocol stack is used to make load balancing decisions. That is IP address and port information is used.

Linux-Director

The host that LVS runs on is refered to as the Linux-Director. And the service that connections are recieved for is refered to as the virtual service, as it is a service that may be provided by many physical servers. The Linux Director essentially acts as a specialised router that forwards packets from end-users to real-servers - hosts running the applications that actually process the requests.

When packets are received for a virtual service by the linux-director, the scheduling algorithm decides which which real-server to send the packet to. Once this decision is made subsequent packets to for the same connection will be sent to the same real server. Thus the integrity of the connection is maintained. Optionally, a service can be marked as persistent. When this is done all subsequent connections from a given end-user will be forwarded to the same real-server until a timeout elapses. This is useful for applications such as FTP where end-users should consistently be directed to the same real-server to ensure integrity of data connections.

Virtual Services

On the Linux Director a virtual service is defined by either an IP Address, port and protocol, or a Firewall-Mark.

IP Address, Port and Protocol: A virtual server may be specified by:
- An IP Address: The IP address that end users will use to access the service.
- A port: The port that end users will connect to.
- A protocol. Either UDP or TCP.
Firewall-Mark: Packets may be marked with a 32-bit unsigned value using ipchains(1). The Linux Virtual Server is able to match use this mark to designate packets destined for a virtual service and route them accordingly. This is particularly useful if a large number of contiguous IP based virtual services are required with the same real servers. Or to group persistancy between different ports. For instance to ensure that a given end user is sent to the same real server for both HTTP and HTTPS.

Scheduling Algorithm

The virtual service is assigned a scheduling algorithm to use to allocate incoming connections to the real-servers. There are many different sheduling algorithms available, and new ones are added from time to time. For an up to date list, and more detailed description, please see the ipvsadm(8) man page.

Least-Connection (lc): Allocate connections to the real-server with the least number of connections.
Weighted Least-Connection (wlc):
Weighted version of Least-Connection.
Round-Robin (rr): Place the real-servers in a circular list and allocate connections to each real-server in turn.
Weighted Round-Robin (wrr): Weighted version of rount-robin.
Locality-Bassed Least-Connection (lblc): Try to assign connections addressed to the same IP address to the same real-server. This is frequently used in conjunction with transparent http proxy services.
Locality-Based Least-Connection with Replication (lblcr): Variation of Locality-Bassed Least-Connection that allows a pool of servers for a given destination IP address to be maintained in sutiations of high load.
Destination-Hashing (dh): Use a static hash of the destination IP address to allocate connections.
Source-Hashing (sh): Similar to Destination-Hashing, but the source IP address is hashed.
Shortest Expected Delay (sed): Allocate connections to the server that will service the request with the shortest expected delay.
Never Queue (nq): Allocate a connections to a idle real-servers if there are any, else use the Shortest Expected Delay altgorithm.

Packet Forwarding

The Linux Virtual Server has three different ways of forwarding packets: Network Address Translation (NAT), IP-IP encapsulation or tunnelling and Direct Routing.

Direct Routing: Packets from end users are forwarded directly to the real server. The IP packet is not modified, so the real servers must be configured to accept traffic for the virtual server's IP address. This can be done using a dummy interface, or packet filtering to redirect traffic addressed to the virtual server's IP address to a local port. The real server may send replies directly back to the end user. That is if a host based layer 4 switch is used, it may not be in the return path.
IP-IP Encapsulation: IP-IP Encapsulation or Tunnelling enables packets addressed to an IP address to be redirected to another address, possibly on a different network. In the context of layer 4 switching the behaviour is very similar to that of direct routing, except that when packets are forwarded they are encapsulated in an IP packet, rather than just manipulating the ethernet frame. The main advantage of using tunnelling is that real servers can be on a different networks.
Network Address Translation: Network Address Translation or NAT is a method of manipulating the source and/or destination port and/or address of a packet to map networks. The most common use of this is IP Masquerading that is often used to enable RFC 1918 private networks to access the internet. In the context of layer 4 switching, packets are received from end users and the destination port and IP address are changed to that of the chosen real server. Return packets pass through the layer 4 switching device at which time the mapping is undone so the end user sees replies from the expected source.

Connection Synchronisation

In order for LVS to forward all packets for the same connection to the same real server a small ammount of state is maintained for each connection. In a nuthsell, the source and destination ip address and port of the connection, and the real-server that is being used. If persistance is in use then the information to allow an end-user to reconnect to the same real-server is kept in a similar fashion. If two linux-directors are being used in an active-standby set-up, such as the any of the High Availability and Load Balancing topologies, then this connection infromation needs to be synchronised, so existing connections can continue if the active linux-director fails and the standby becomes active. This is done by an in-kernel synchronisation daemon that is part of LVS. The daemon on the active linux-director periodically sends out the neccessary connection information for the active connections using multicast UDP. A corresponding daemon on the standby linux director uses this information to seed the local connection table.

Real Servers

Real servers are the ultimate destination of a connection made to a virtual-service. They run the applications that handle requests from end-users. A virtual service's real-servers and are defined by an IP address, a port and optionally a weight if supported by the scheduling algorithm that is in effect. In general, the port of the real-servers must be the same as that of the virtual service. If firewall-mark virtual services are used then the port that an incoming connection is received on will be the port that the connection is forward to on the real-server, that is the port specification for the real-erver is not used. The health of the real-servers may be monitored using ldirectord.

Version

The version of LVS used by Ultra Monkey 3 depends on the kernel in use. The 2.4.27-8 kernel for Debian Sarge and the 2.4.21-27.0.2.EL kernel for Red Hat Enterprise Linux 3 have been tested and work with the configurations described.

Copyright © 2000-2005, Horms
Last Updated: Sat Mar 4 16:33:58 2006 +0900

Debian is a registered trademark of Software in the Public Interest, Inc.
Red Hat, the Red Hat Shadowman logo and Fedora are registered trademarks of Red Hat, Inc.
Red Hat may also be refered to as RedHat on this site.
Linux is a registered trademark of Linus Torvalds.
All other trademarks are the property of their respective owners.