Using trickle and Bacula in a multi-homed environment

About

Here is a brief tutorial on how I have set up the powerful Bacula backup tool on a server supporting a split-backup environment:

1. Daily backups are made on NIC 1 to a local network storage facility (hosted by a Bacula backup server)
2. On-demand backups are pulled via NIC 2 across a public network channel - e.g. VPN link - to a complete off-site storage facility (hosted by another Bacula backup server)

An extra limitation exists in the fact that the on-demand backups suffer a metering charge, one that based on a maximum allowed throughput per time unit, say 10 mbits/s. It is therefore important that any Bacula backup run stays well below that traffic threshold. Unless you are running an Entreprise version of Bacula, it will be impossible to instruct Bacula to throttle its communication between client (bacula-fd) and server (bacula-dir). This is where the nifty Trickle software comes into play.

From its man page:

trickle is a user space bandwidth manager. Currently, trickle supports the shaping of any SOCK_STREAM (see socket(2)) connection established via the socket(2) interface. Furthermore, trickle will not work with statically linked executables, nor with setuid(2) executables. trickle is highly configurable; download and upload rates can be set separately, or in an aggregate fashion.*

By running the Bacula file daemon (aka client) under supervision of trickle, it is possible to limit the client daemon’s bandwith to a certain level.

For example (from /etc/init.d/bacula-fd.conf):

# /usr/bin/trickle -s -d 500 -u 500 /usr/sbin/bacula-fd $2 ${FD_OPTIONS} -c /etc/bacula/bacula-fd.conf

which would limit Bacula I/O to 500KB/s (+/- 4mbit/s). This way we will never exceed the 10mbit/s threshold, at least not by just running a Bacula backup. Next, I wanted to make sure that only the external backup was throttled by trickle. Limiting the throughput on the internal network does not make any sense and would only artificially prolong any backup run. The clue in achieving this is to have two separate bacula client (file daemons) running: one bonded to NIC1 powering the local backups; the other bonded to NIC 2 and powering the external backups. In order to support two separate bacula client daemons, we need two separate configuration files:

• /etc/bacula/bacula-fd-internal.conf: the configuration file that will drive the backups across the internal network
• /etc/bacula/bacula-fd.conf: the (standard) configuration file that will drive the backups across the external network

Before all: stop the currently running bacula-fd process and make a copy of the original bacula-fd.conf configuration file:

# /etc/init.d/bacula-fd stop
# cp -p /etc/bacula/bacula-fd.conf /etc/bacula/bacula-fd-internal.conf

The bacula-fd.internal.conf configuration file needs a directive telling the internal bacula-fd process to bond to NIC 1, e.g.:

FDport = 9102
FDAddress = backup-master.internal

where backup.master is the locally resolvable DNS name of the internal Bacula backup server. You may prefer to use the IP address instead of a DNS name. Additionally, we want to be sure that this Bacula client connects only to the Bacula server (directory) on the local network. For this, a dedicated Director resource needs to be configured, e.g.:

Director { Name = backup-master.internal.server
Password = [cryptohash] }

Choose a different working & PID directory to avoid conflicts between both client daemons (do not forget to pre-create these directories!):

WorkingDirectory = /var/bacula-internal
PidDirectory = /var/run/bacula-internal

The original bacula-fd.conf configuration file requires a similar change:

FDport = 9102
FDAddress = backup-master.mydomain.com

where backup.master is the externally resolvable DNS name of our external Bacula backup server. You may prefer to use the IP address instead of a DNS name.

Like the internal Bacula client, we also want to be sure that this Bacula client connects only to the Bacula server (directory) on the external network. For this, a dedicated Director resource needs to be configured, e.g.:

Director { Name = backup-master.external.server
Password = [cryptohash] }

These steps complete the setup of the Bacula application itself. The final to-do is to take care of the 2 processes starting and stopping independently and the internal backup client having its traffic throttled by trickle.

Note 1: the following guidelines are based on system init scripts as found on RHEL/CentOS distrubutions. If you are are running any other flavour of GNU Linux/Unix, then your mileage may vary. First, we undouble the standard bacula-fd daemon program in /usr/sbin by creating a hard link to the original binary:

# ln /usr/sbin/bacula-fd /usr/sbin/bacula-fd-internal
# ls -l /usr/bin/bacula-fd*
-rwxr-xr-- 2 root root 1354748 Apr 14 2007 bacula-fd
-rwxr-xr-- 2 root root 1354748 Apr 14 2007 bacula-fd-internal

Note 2: Using a soft (symbolic) link will not work in this case, as it will cause both bacula client processes to be stopped at the same time by either of the /etc/init.d scripts. Secondly, we undouble the standard bacula-fd init scripts in /etc/init.d:

# cp -p /etc/init.d/bacula-fd.conf /etc/init.d/bacula-fd.internal.conf

Update the /etc/init.d/bacula-fd.internal.conf to include references to the new configuration file and binary:

# grep internal bacula-fd-internal
daemon /usr/sbin/bacula-fd-internal $2 ${FD_OPTIONS} -c /etc/bacula/bacula-fd-internal.conf
[ $RETVAL -eq 0 ] && touch /var/lock/subsys/bacula-fd-internal
killproc /usr/sbin/bacula-fd-internal
[ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/bacula-fd-internal
status /usr/sbin/bacula-fd-internal

Update the /etc/init.d/bacula-fd.conf to include the trickle command:

daemon /usr/bin/trickle -s -d 500 -u 500 /usr/sbin/bacula-fd $2 ${FD_OPTIONS} -c /etc/bacula/bacula-fd.conf

Restart both Bacula client daemons and verify their correct operation by issuing a

sta client

check from their respective Bacula directors. If you run into problems then verify whether the Bacula client daemons are actually bonded to the right network card:

# netstat -an | grep 9102
tcp 0 0 223.223.203.203:9102 0.0.0.0:* LISTEN
tcp 0 0 10.10.0.102:9102 0.0.0.0:* LISTEN

You may also need to adjust your firewall settings. Again YMMV.

Note: You can check the actual bandwidth usage by Bacula if you install a real-time bandwidth measuring tool like bwm-ng. Here is a sample output of bwm-ng:

bwm-ng v0.5 (probing every 0.500s), press 'h' for help input: libstatgrab type: rate - iface Rx Tx Total
===========================================================================
lo: 0.00 KB/s 0.00 KB/s 0.00 KB/s eth0: 4.30 KB/s 141.51 KB/s 145.81 KB/s eth1: 0.00 KB/s 0.00 KB/s 0.00 KB/s