August 1, 2010 Hatem

Two nodes Load balance and Failover with keepalived and Ubuntu Server 10.04 x64

In an ideal system architecture using load balancers in separate nodes is preferred, however it’s also possible to have your load balancers in the same nodes with your applications. I have used in this architecture the same hardware as the previous Master/Master MySQL cluster, including Ubuntu server 10.04 x64, Apache2 as web server, two nodes HP DL380G6 with 3 hard disks 15K in RAID5 and connected to a SAN storage via Fiber. For load balancing and failover I used keepalived and LVS, and you can use heartbeat to get your cluster running.

First you will need to set at least two IPs (10.10.0.1 and 10.10.0.2) for your servers, and one virtual (10.10.0.3) shared between the two servers, you will have for the first interface :

# The primary network interface
auto eth0
iface eth0 inet static
address 10.10.0.1
netmask 255.255.255.0
network 10.10.0.0
broadcast 10.10.0.255
gateway 10.10.0.250
auto eth0:0
iface eth0:0 inet static
address 10.10.0.3
netmask 255.255.255.0
network 10.10.0.0
broadcast 10.10.0.255

and the second interface :

# The primary network interface
auto eth0
iface eth0 inet static
address 10.10.0.2
netmask 255.255.255.0
network 10.10.0.0
broadcast 10.10.0.255
gateway 10.10.0.250
auto eth0:0
iface eth0:0 inet static
address 10.10.0.3
netmask 255.255.255.0
network 10.10.0.0
broadcast 10.10.0.255

Then we can start by installing keepalived (v1.1.17 is available in Ubuntu repositories)

sudo apt-get install keepalived

You will have to create two configuration files for the first node 10.10.0.1 (Master) and second node 10.10.0.2 (Backup). So we add in the master node :

usr01@server01:~$ sudo nano /etc/keepalived/keepalived.conf
# Keepalived Configuration File
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 10
priority 200
virtual_ipaddress {
10.10.0.3/24
}
notify_master "/etc/keepalived/notify.sh del 10.10.0.3"
notify_backup "/etc/keepalived/notify.sh add 10.10.0.3"
notify_fault "/etc/keepalived/notify.sh add 10.10.0.3"
}
virtual_server 10.10.0.3 80 {
delay_loop 30
lb_algo rr
lb_kind DR
persistence_timeout 50
protocol TCP
real_server 10.10.0.1 80 {
weight 100
HTTP_GET {
url {
path /index.php
digest d41d8cd98f00b204e9800998ecf8427e
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 2
}
}
real_server 10.10.0.2 80 {
weight 100
HTTP_GET {
url {
path /index.php
digest d41d8cd98f00b204e9800998ecf8427e
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 2
}
}
}

And in the backup node :

usr01@server02:~$ cat /etc/keepalived/keepalived.conf
# Keepalived Configuration File
vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 10
priority 100
virtual_ipaddress {
10.10.0.3/24
}
notify_master "/etc/keepalived/notify.sh del 10.10.0.3"
notify_backup "/etc/keepalived/notify.sh add 10.10.0.3"
notify_fault "/etc/keepalived/notify.sh add 10.10.0.3"
}
virtual_server 10.10.0.3 80 {
delay_loop 30
lb_algo rr
lb_kind DR
persistence_timeout 50
protocol TCP
real_server 10.10.0.1 80 {
weight 100
HTTP_GET {
url {
path /check.txt
digest d41d8cd98f00b204e9800998ecf8427e
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 2
}
}
real_server 10.10.0.2 80 {
weight 100
HTTP_GET {
url {
path /check.txt
digest d41d8cd98f00b204e9800998ecf8427e
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 2
}
}
}

The hash is created using, notice that you can add exception so apache don’t log check.txt requests.

usr01@server01:~$ genhash -s 10.10.0.1 -p 80 -u /check.txt
MD5SUM = d41d8cd98f00b204e9800998ecf8427e
usr01@server01:~$ genhash -s 10.10.0.2 -p 80 -u /check.txt
MD5SUM = d41d8cd98f00b204e9800998ecf8427e

Also in both nodes we have to add a small utility to notify (/etc/keepalived/notify.sh) :


#!/bin/bash
VIP="$2"
case "$1" in
add)
/sbin/iptables -A PREROUTING -t nat -d $VIP -p tcp -j REDIRECT
;;
del)
/sbin/iptables -D PREROUTING -t nat -d $VIP -p tcp -j REDIRECT
;;
*)
echo "Usage: $0 {add|del} ipaddress"
exit 1
esac
exit 0

Launch keepalived on the two nodes :

sudo /etc/init.d/keepalived start

Now we need to enable ip_forward on the two nodes permanently

net.ipv4.ip_forward = 1

restart network on the two nodes

sudo /etc/init.d/networking restart

And we can check that load balancing is working correctly on Master :

usr01@server02:~$ sudo ipvsadm -L -n
[sudo] password for usr01:
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.10.0.3:80 rr persistent 50
-> 10.10.0.1:80 Local 100 0 0
-> 10.10.0.2:80 Route 100 0 0

Also on Backup server

usr01@server02:~$ sudo ipvsadm -L -n
[sudo] password for usr01:
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.10.0.3:80 rr persistent 50
-> 10.10.0.1:80 Route 100 0 0
-> 10.10.0.2:80 Local 100 0 0

We are almost done, we only need to add a preroute rule on the backup node manually to get started :

usr01@server02$ iptables -A PREROUTING -t nat -d 10.10.0.3 -p tcp -j REDIRECT
usr01@server02$ iptables -t nat --list
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
REDIRECT tcp -- anywhere 10.10.0.3
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination

usr01@server02$sudo iptables -t nat -A PREROUTING -p tcp --dport 80 -j DNAT --to-destination 10.10.0.3:80
usr01@server02$ iptables -t nat --list
target prot opt source destination
DNAT tcp -- anywhere anywhere tcp dpt:www to:10.10.0.3:80

Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

Chain POSTROUTING (policy ACCEPT)
target prot opt source destination

That’s all.

Now you can connect to http://10.10.0.3 and you can notice load distributed between two nodes internally. In case one of the nodes fail, it will takes few seconds until the backup server notice the failure and update its iptables prerouting rule. When apache service goes down, you will notice that request on port 80 will be automatically redirected to second node.

As I have mentioned in the beginning, failover control cannot goes without downtime in such architecture, but it still great to distribute load if you are limited in hardware.

Finally, it will be much easier (even faster) to load balance using Round Robin DNS from active directory for example, if you can manage to monitor failed service or node, however this architecture remain better on failover even with a short downtime.

Update 2017-11-14 : I had an issue with iptables REDIRECT which was not redirecting to virtual IP anymore, replace it with DNAT fixes the issue.

, , , , , ,

(HBY) Consultancy