Topologies

Firewall Marks

Firewall marks provide a powerful mechanism to group services together . This topology provides is intended as an example of how fwmarks can be used. To give the example some grounding it is baed on the high availability load balancing topology. However, it is applicable to any of the Load Balancing topologies discussed.

This setup will load balance traffic sent to any IP address in the range 192.168.0.0-192.168.0.255 (192.168.0.0/24) on any UDP or TCP port to the real severs.

A minimum of 4 nodes is required for this setup. Real Servers may be added to the network as additional capacity is required. For a more detailed explanation of the function of each component please see the overview of the high availability load balancing topology.

The documentation that follows assumes that all nodes on the network are set up with correct interfaces and routes for each network they are connected to as per the diagram above. The return path for packets must be through the active Linux Director. In almost all cases this will mean that the the default route for the real servers should be set to the internal virtual address as described in the real servers section for a single virtual service.

Before proceding, you need to follow the configuration steps provided for high availability load balancing or whichever topology you wish to use firewall marks in conjunction with.

• Network Preparation

  For this configuration to work any gateway routers on the external network need to route traffic for the virtual services network, 192.168.0.0/24 via the external virtual address, 192.168.7.340. Details on configuring this for a Linux router can be found in the documentation for highly availability: network of virtual services. Be aware that this example forwards traffic to 192.168.6.240 and not 192.168.7.340. For non-linux routers please refer to the vendor-supplied documentation.

• Linux Directors

  Please follow the steps for configuring a linux directors for a highly available, load balanced single virtual service before continuing.

  The configuration file for ldirectord, /etc/ha.d/ldirectord.cf needs to be replaced with so that the virtual addresses, 192.168.0.0/24 will be managed by LVS. For this change to take effect restart heartbeat.

  
  /etc/init.d/heartbeat restart
  
  

  As per the single virtual address example, current LVS kernel table may be inspected using the ipvsadm command. A sample invocation follows showing that ldirectord found all servers to be available:

  
  ipvsadm -L -n
  IP Virtual Server version 0.9.11 (size=4096)
  Prot LocalAddress:Port Scheduler Flags
    -> RemoteAddress:Port          Forward Weight ActiveConn InActConn
  FWM  1 rr
    -> 192.168.6.3:80              Route   1      0          0
    -> 192.168.6.2:80              Route   1      0          0
  
  

  The ipchains rules must be updated to allow forwarding of traffic to the network of virtual services.

  • Supported Versions of Red Hat and Fedora

    On supported versions of Red Hat and Fedora this can be configured using the iptables init script.

    
    # Flush existing rules in the nat table
    /etc/init.d/iptables stop
    Resetting built-in chains to the default ACCEPT policy:          [  OK  ]
    
    # Masquerade for 192.168.6.0/24 bound for any host
    /sbin/iptables -t nat -A POSTROUTING -j MASQUERADE -s 192.168.6.0/24
    
    # Set fwmark 1 on packets for destined for virtual service addresses
    /sbin/iptables -t mangle -A PREROUTING -d 192.168.0.0/24 -j MARK --set-mark 1
    
    # Log all packets that attempt to be forwarded
    # Useful for Debugging. Questionable for Production
    #/sbin/iptables -t nat -A POSTROUTING -j LOG
    
    # Save the rules
    /etc/init.d/iptables save
    Saving current rules to /etc/sysconfig/iptables:                 [  OK  ]
    
    # Make sure rules are activated on reboot (at run levels 2, 3, 4 and 5)
    /sbin/chkconfig --level 2345 iptables on
    
    # Activate the rules
    /etc/init.d/iptables start
    Flushing all current rules and user defined chains:              [  OK  ]
    Clearing all current rules and user defined chains:              [  OK  ]
    Applying iptables firewall rules:                                [  OK  ]
    
    

  • Debian

    On Debian masqurading can be configured as part of the interface configuration for the interface that is on Real Servers' network. This example entry in /etc/network/interfaces Sets up masquerading on eth1.

    
    auto eth1
    iface eth1 inet static
       address 192.168.6.4
       netmask 255.255.255.0
       up iptables -t nat -A POSTROUTING -j MASQUERADE -s 192.168.6.0/24
       up iptables -t mangle -A PREROUTING -d 192.168.0.0/24 -j MARK --set-mark 1
       down iptables -t nat -D POSTROUTING -j MASQUERADE -s 192.168.6.0/24
       down iptables -t mangle -D PREROUTING -d 192.168.0.0/24 -j MARK --set-mark 1
    
    

    For this change to take palce restart networking.

    
    /etc/init.d/networking restart
    
    

  To verify the masquerading rules the iptables command may be used. The following example has been truncated to show only the POSTROUTING chain in the nat table and the PREROUTING chain in the mangle table..

  
  /sbin/iptables -t nat -L -n
  Chain POSTROUTING (policy ACCEPT)
  target     prot opt source               destination         
  MASQUERADE  all  --  192.168.6.0/24       0.0.0.0/0      
  
  /sbin/iptables -t mangle -L -n
  Chain PREROUTING (policy ACCEPT)
  target     prot opt source               destination         
  MARK       all  --  0.0.0.0/0            192.168.0.0/24     MARK set 0x1 
  
  

  If an FTP Virtual Service is to be used then the ip_vs_ftp kernel module needs to be used. This may be done by running the following command.

  
  /sbin/modprobe ip_vs_ftp
  
  

  If this module is needed it is important to ensure that it is inserted into the kernel at boot time.

  • Supported Versions of Red Hat and Fedora

    On supported versions of Red Hat and Fedora appended the above modprobe command to /etc/rc.local.

  • Debian On Debian append the following line to /etc/modules

    
    ip_vs_ftp
    
    

  The modules presence in the kernel may be checked using lsmod. The output of the following command has been truncated to only show the ip_vs_ftp module.

  
  /sbin/lsmod
  Module                  Size  Used by
  ip_vs_ftp               3232   0 
  
  

• Real Servers

  Please follow the steps for configuring real servers for a highly availability and load balancing. One this is done, no additional configuration is required on real servers for this setup.

Additional Notes

The configuration above establishes one virtual service which covers all ports on both UDP and TCP for all IP addresses in the range 192.168.0.0 - 192.168.0.255 (192.168.0.0/24). At the heart of this is the following iptables rule:

/sbin/iptables -t mangle -A PREROUTING -d 192.168.0.0/24 -j MARK --set-mark 1

By changing the packets that this rule matches, the scope of the virtual service is changed. Multiple rules may be used to add matches. As long as packets belloning to a connection are marked with the fwmark (in this case 1, as per --set-mark 1), that corresponds to an LVS virtual service, the connection will be load balanced. For example to set up a virtual service that only covers TCP ports 80 and 443 (HTTP and HTTPS) for the hosts in 192.168.0.0/28 use:

/sbin/iptables -t mangle -A PREROUTING -p tcp -d 192.168.0.0/28 --dport 80 -j MARK --set-mark 1
/sbin/iptables -t mangle -A PREROUTING -p tcp -d 192.168.0.0/28 --dport 443 -j MARK --set-mark 1

As well as providing a convenient way to reduce complexity by grouping services by address and/or ports. The grouping also provides a convenient way to couple persistancy together. If persistancy was set for the fwmark virual service 1 in the /etc/ha.d/ldirectord.cf and the configuration for ports 80 and 443 immediately above was used, then an end user connecting to port 80 and then folloing a link to port 443 would end up on the same real&nsbp;server for both connections.

It is also of note that multiple fwmark services may be used by speficying different firewall marks. We have used 1, but any number in the range 1-65535 is valid. For example, there could be two different virtual services defined in /etc/ha.d/ldirectord.cf for firewall marks 1 and 2. These may have different pools of real servers or other unique attributes. The following iptables rules are show how to match using two different firewall marks.

/sbin/iptables -t mangle -A PREROUTING -d 192.168.0.0/24 -j MARK --set-mark 1
/sbin/iptables -t mangle -A PREROUTING -d 192.168.0.0/24 -j MARK --set-mark 2

Copyright © 2000-2005, Horms
Last Updated: Tue May 17 17:37:24 2005 +0900

Debian is a registered trademark of Software in the Public Interest, Inc.
Red Hat, the Red Hat Shadowman logo and Fedora are registered trademarks of Red Hat, Inc.
Red Hat may also be refered to as RedHat on this site.
Linux is a registered trademark of Linus Torvalds.
All other trademarks are the property of their respective owners.