Friday, September 26, 2008

Load Balancing two ISPs

Today I finally managed to use my two ISPs together on my desktop, combining both bandwidths into one big (almost 3Mbit!) pipe.

My setup:
  1. ADSL modem 1Mb down, 320Kb up
  2. GSM modem 2Mb down, 512Kb up
Configuring both at the same time is simple, but then we have two default gateways and our packets always go out through the first one found (or with the lower cost as defined in the 'metric' parameter.)

So how can we divide our packets over both links?

Googling around I found several suggestions to use iptables.
The general idea is:
  • use -m statistic in a chain to choose packets to use on of two links (either the 'nth'-method or the 'average'-method
  • set a mark on the packet
  • use an 'ip rule' to select a routing table for mark 1, mark 2, etc.
That sounded like a perfect solution. This way I could really balance my two links like 40%/60% or whatever.

But it didn't work...

My desktop is not a router, so I have to treat the packets in the OUTPUT chain, where routing has already taken place. The above-described method works on Linux routers treating the PREROUTING chain in iptables, where we can mark a package before routing.

So I studied IP ROUTE and IP RULES a bit more, browsing through the fantastic Linux Advanced Routing & Traffic Control site.

I discovered that we can use 'nexthop' to 'hop' between several routes.
After experimenting a bit I wrote the following script:
# bal_local Load-balance internet connection over two local links
# Version: 1.0.0 - Fri, Sep 26, 2008
# Author: Niels Horn

# Set devices:
DEV1=${1-eth0} # default eth0
DEV2=${2-ppp0} # default ppp0

# Get IP addresses of our devices:
ip1=`ifconfig $DEV1 | grep inet | awk '{ print $2 }' | awk -F: '{ print $2 }'`
ip2=`ifconfig $DEV2 | grep inet | awk '{ print $2 }' | awk -F: '{ print $2 }'`

# Get default gateway for our devices:
gw1=`route -n | grep $DEV1 | grep '^' | awk '{ print $2 }'`
gw2=`route -n | grep $DEV2 | grep '^' | awk '{ print $2 }'`

echo "$DEV1: IP=$ip1 GW=$gw1"
echo "$DEV2: IP=$ip2 GW=$gw2"

### Definition of routes ###

# Check if tables exists, if not -> create them:
if [ -z "`cat /etc/iproute2/rt_tables | grep '^251'`" ] ; then
   echo "251 rt_dev1" >> /etc/iproute2/rt_tables
if [ -z "`cat /etc/iproute2/rt_tables | grep '^252'`" ] ; then
   echo "252 rt_dev2" >> /etc/iproute2/rt_tables

# Define routing tables:
ip route add default via $gw1 table rt_dev1
ip route add default via $gw2 table rt_dev2

# Create rules:
ip rule add from $ip1 table rt_dev1
ip rule add from $ip2 table rt_dev2

# If we already have a 'nexthop' route, delete it:
if [ ! -z "`ip route show table main | grep 'nexthop'`" ] ; then
   ip route del default scope global

# Balance links based on routes:
ip route add default scope global nexthop via $gw1 dev $DEV1 weight 1 \
   nexthop via $gw2 dev $DEV2 weight 1

# Flush cache table:
ip route flush cache

# All done...

You can download the script here from my homepage.

This is not the perfect solution, as routes are cached, so once you connected to an external site, it will continue to use the linkt hat was originally selected.
So an FTP download won't benefit from this solution, but torrent downloads will, as they use several parallel connections.

I tested the result and managed to download using BitTorrent with the incredible speed of 250KBytes/sec:

Labels: ,


Anonymous Anonymous said...

great post, thanks
but what is that utility you are watching interface bandwith with?

December 8, 2008 at 8:24 PM  
Blogger holson said...

I used 'iptraf' as it comes with Slackware. It's a nice tool that works from the console.

December 9, 2008 at 9:02 AM  
Anonymous Anonymous said...

Niels, interesting article. I am experimenting with Opera Unite. I have 2 wwan accounts (Verizon). How do I serve up a web page thru both connections simultaneously?

July 4, 2009 at 8:22 PM  
Blogger holson said...

Sorry for the delay in responding - I have been busy with some other projects.
If I understand correctly, you want your server to be accessible from both internet connections?
This is not impossible, but then the load-balancing is not done on your side.
When a user accesses your web page like "http://yourserver/page.html" the address of your server is translated by the DNS server to an IP address.
Since two internet connections means two external IP addresses, you cannot have incoming requests from both connections.
Professional site resolve this at the DNS level.
Try for instance "dig".
If you repeat the command several times, you can get different answers for the IP address...

Hope this answered your question at least partially!

July 23, 2009 at 9:18 PM  
Anonymous gmbastos said...

Hello, Niels.

Thank you for this great post. :)

But, even though you are far more experienced than me in scripting, I dare to tell you that awk can make that IP address extraction in one run (I'm referring to your load balance script):

# Get IP addresses of our devices:
ip1=`ifconfig $DEV1 | awk '/inet addr:/ { split($2,inetline,":"); print inetline[2]}'`
ip2=`ifconfig $DEV2 | awk '/inet addr:/ { split($2,inetline,":"); print inetline[2]}'`

As you work with heavy-weight servers, perhaps this can make some difference (for a SOHO user speed and footprint improvements are unnoticeable).

October 12, 2009 at 4:08 AM  
Blogger niels.horn said...

@gmbastos: You're right, it can be done in a shorter line. Like they say "there are many ways to Rome". :)
I kept the script as simple & clear as possible to make it easier to understand. But everyone is free to improve it like you did.
Awk if an extremely powerful tool and I sometimes think I only use like 10% of its power...

October 12, 2009 at 9:12 AM  

Post a Comment

Subscribe to Post Comments [Atom]

<< Home