Traffic Shaping with cbq

Since I am on the subject of networking in Linux, here is another post about some nice things we can do: shaping our traffic so that we can limit how much of our bandwidth we use for each type of traffic.

Just as a small reminder: I am a long-time Slackware user (since 1996) and I only test my configurations on this distribution. I have used other Linux 'flavors' in the past but know much less about them.
Most thing I will post here will work though on other systems, but don't shoot me if they do not.

I started using cbq for traffic shaping on my local network because of the following situation:
I use rsync to copy some files I cannot afford to loose from my desktop to my wife's and v.v. I use crontab to automatically do this at certain hours.
Rsync is a wonderful protocol that only copies files that have changed, saving time and bandwidth.
But sometimes many files are changed or added, and then the whole bandwith of my local network is used, slowing down other traffic.
At these times even browsing the internet can become very slow, just because I am backing up some folders of new digital pictures.

Rsync has its own '--bwlimit' option, but I wanted a better, more structured solution. And this solution is cbq.

Basically configuring up cbq is done in three steps:

1) Setting up cbq
cbq is actually a script that can be found in the documentation of iproute2 in Slackware. We have to copy it to /sbin and make it executable:

cp /usr/doc/iproute2-2.6.16-060323/examples/cbq.init-v0.7.3 /sbin/cbq
chmod +x /sbin/cbq

cbq expects its configuration files in /etc/sysconfig/cbq
If this directory does not exist, create it:

mkdir /etc/sysconfig/cbq

2) Creating the rules-file
cbq reads files in /etc/sysconfig/cbq with the following names:
cbq-nnnn.yyy where:

• nnnn: is a hexadecimal number from 0002 to ffff
• yyy: is the name of your network interface, like eth0, eth1, etc

In my case, the network interface for my local network is eth1, so I created "cbq-0002.eth1"
Here is the contents of my file:

DEVICE=eth1,100Mbit,10Mbit
RATE=5000Kbit
WEIGHT=500Kbit
PRIO=5
RULE=192.168.1.110:873,192.168.1.0/24
BOUNDED=no
ISOLATED=no

Some explanations:

• DEVICE: the interface you want to limit, with its real speed and its weight (1/10 of the max. speed)
• RATE: the bandwith you want to offer for this particular application / port / address
• WEIGHT: 1/10 of the RATE
• PRIO: Priority setting. 5 is default
• RULE: source,destination --> in my case 192.168.1.110 is my desktop, 873 is the port rsync uses
• BOUNDED: Default no, used if you have other filters
• ISOLATED: 'no' means that the rate can be used by other traffic if not in use

3) Starting the bandwidth limiting
Use cbq compile to prepare the new filters or after you alter your cbq-nnnn.yyy files.
Then use cbq start to start your traffic-shaping!
To always start cbq, include it in your rc.local script.

You can monitor your results with iptraf or wireshark.

More information can be found using "man tc-cbq".

Load Balancing two ISPs

Today I finally managed to use my two ISPs together on my desktop, combining both bandwidths into one big (almost 3Mbit!) pipe.

My setup:

1. ADSL modem 1Mb down, 320Kb up
2. GSM modem 2Mb down, 512Kb up
   

Configuring both at the same time is simple, but then we have two default gateways and our packets always go out through the first one found (or with the lower cost as defined in the 'metric' parameter.)

So how can we divide our packets over both links?

Googling around I found several suggestions to use iptables.
The general idea is:

• use -m statistic in a chain to choose packets to use on of two links (either the 'nth'-method or the 'average'-method
• set a mark on the packet
• use an 'ip rule' to select a routing table for mark 1, mark 2, etc.
  

That sounded like a perfect solution. This way I could really balance my two links like 40%/60% or whatever.

But it didn't work...

My desktop is not a router, so I have to treat the packets in the OUTPUT chain, where routing has already taken place. The above-described method works on Linux routers treating the PREROUTING chain in iptables, where we can mark a package before routing.

So I studied IP ROUTE and IP RULES a bit more, browsing through the fantastic Linux Advanced Routing & Traffic Control site.

I discovered that we can use 'nexthop' to 'hop' between several routes.
After experimenting a bit I wrote the following script:

#!/bin/bash
#
# bal_local        Load-balance internet connection over two local links
#
# Version:         1.0.0 - Fri, Sep 26, 2008
#
# Author:          Niels Horn 
#

# Set devices:
DEV1=${1-eth0}    # default eth0
DEV2=${2-ppp0}    # default ppp0

# Get IP addresses of our devices:
ip1=`ifconfig $DEV1 | grep inet | awk '{ print $2 }' | awk -F: '{ print $2 }'`
ip2=`ifconfig $DEV2 | grep inet | awk '{ print $2 }' | awk -F: '{ print $2 }'`

# Get default gateway for our devices:
gw1=`route -n | grep $DEV1 | grep '^0.0.0.0' | awk '{ print $2 }'`
gw2=`route -n | grep $DEV2 | grep '^0.0.0.0' | awk '{ print $2 }'`

echo "$DEV1: IP=$ip1 GW=$gw1"
echo "$DEV2: IP=$ip2 GW=$gw2"

### Definition of routes ###

# Check if tables exists, if not -> create them:
if [ -z "`cat /etc/iproute2/rt_tables | grep '^251'`" ] ; then
   echo "251    rt_dev1" >> /etc/iproute2/rt_tables
fi
if [ -z "`cat /etc/iproute2/rt_tables | grep '^252'`" ] ; then
   echo "252    rt_dev2" >> /etc/iproute2/rt_tables
fi

# Define routing tables:
ip route add default via $gw1 table rt_dev1
ip route add default via $gw2 table rt_dev2

# Create rules:
ip rule add from $ip1 table rt_dev1
ip rule add from $ip2 table rt_dev2

# If we already have a 'nexthop' route, delete it:
if [ ! -z "`ip route show table main | grep 'nexthop'`" ] ; then
   ip route del default scope global
fi

# Balance links based on routes:
ip route add default scope global nexthop via $gw1 dev $DEV1 weight 1 \
   nexthop via $gw2 dev $DEV2 weight 1

# Flush cache table:
ip route flush cache

# All done...

You can download the script here from my homepage.

This is not the perfect solution, as routes are cached, so once you connected to an external site, it will continue to use the linkt hat was originally selected.
So an FTP download won't benefit from this solution, but torrent downloads will, as they use several parallel connections.

I tested the result and managed to download using BitTorrent with the incredible speed of 250KBytes/sec: