Friday, 24 May 2013

Holland Backup Manager

Part 1 - Installing Holland Backup Manager

I spoke at Percona Live Conference and Expo 2013 about backups. Part of the talk focussed on the backup products in the ecosystem that will help you make a backup of your MySQL data. This follow-up article touches on one of the frameworks from my talk, the Holland Backup Manager. I was able to have a chat with some of the guys on the Rackspace booth about Holland and had some questions regarding features answered.

Holland is a backup framework focussing mostly on MySQL backups but it is pluggable so you can write add backup providers to extend it to your own needs. Using the framework you're able to configure and deploy backup jobs of varying scope to multiple machines. The framework which was originally developed at Rackspace, currently supports mysqldump, lvm, xtrabackup and pgdump (Postgres) also sqllite. The latest version is number 1.0.8 which brings a few nice new features with it. I'll talk about the most significant of these in the third of the series.
I want to show some installation, configuration and then use of the framework. I'm a fan of this project and if you've got the need to standardise whilst being flexible then Holland could be the solution you've been looking for.

Firstly, head over to github to grab the files.
I've been using vagrant to provide a quick VM environment. This OS I'm using here is a CentOS 6.4 64bit host running in Virtualbox. All dependencies for Holland should be available on the popular package managers but if you're using something other then debian (apt-get) or redhat based (yum) then you might require some research elsewhere to meet the requirements of Holland.

Download the zip archive from github (the extension was lost in wget but it is a .zip file)
[vagrant@node1 ~]$ wget ... Saving to: `master' 100%[==================================================>] 602,186 572K/s in 1.0s (572 KB/s) - `master' saved [602186/602186]

So then proceed to unzip your archive
[vagrant@node1 ~]$ unzip ./master
Archive: ./master
creating: holland-master/
inflating: holland-master/CHANGES.txt
inflating: holland-master/INSTALL
inflating: holland-master/LICENSE
inflating: holland-master/README
creating: holland-master/config/
inflating: holland-master/config/README

ProTip: Take some time to go through the various Holland directories. Throughout the directories there README and INSTALL files that will explain more about the accompanying files.

[vagrant@node1 holland-master]$ ls -l
total 56
-rw-rw-r-- 1 vagrant vagrant 7064 May 13 17:45 CHANGES.txt
drwxrwxr-x 4 vagrant vagrant 4096 May 13 17:45 config
drwxrwxr-x 3 vagrant vagrant 4096 May 13 17:45 contrib
drwxrwxr-x 4 vagrant vagrant 4096 May 13 17:45 docs
drwxrwxr-x 7 vagrant vagrant 4096 May 13 17:45 holland
-rw-rw-r-- 1 vagrant vagrant 1782 May 13 17:45 INSTALL
-rw-rw-r-- 1 vagrant vagrant 3290 May 13 17:45 LICENSE
drwxrwxr-x 13 vagrant vagrant 4096 May 13 17:45 plugins
-rw-rw-r-- 1 vagrant vagrant 514 May 13 17:45 README
-rw-rw-r-- 1 vagrant vagrant 62 May 13 17:45 setup.cfg
-rw-rw-r-- 1 vagrant vagrant 1441 May 13 17:45
drwxrwxr-x 2 vagrant vagrant 4096 May 13 17:45 tests
-rwxr-xr-x 1 vagrant vagrant 1782 May 13 17:45

Before going any further lets take care of the dependencies. They're easily met using yum/apt or if you want to compile them yourself then go right ahead;
  - python-setuptools
  - mysqldb (python's mysql connector)

Now, lets take care of some directories that need to exists. These are configurable later, but for simplicity sake we'll roll forward with the Holland defaults. Your environment is likely to be different especially if you're mounting remote storage to store the backup files you make.

[vagrant@node1 config]$ sudo mkdir -p /etc/holland [vagrant@node1 plugins]$ sudo mkdir -p /var/log/holland [vagrant@node1 plugins]$ sudo mkdir -p /var/spool/holland

Now the directories we've created will only be writable to root since we needed elevated privileges to make them. You might choose to chmod the directories if you're going to use a specific backup user to run your backup jobs. Now we're ready to run the python setup file from the root of the unarchived directory
[vagrant@node1 holland-master]$ sudo python ./ install
running install
running bdist_egg
running egg_info
creating holland.egg-info
writing holland.egg-info/PKG-INFO
Installing holland script to /usr/bin

Installed /usr/lib/python2.6/site-packages/holland-1.0.8-py2.6.egg
Processing dependencies for holland==1.0.8
Finished processing dependencies for holland==1.0.8

Without any errors in the previous output it's safe to assume we're installed.
[vagrant@node1 config]$ holland --version
Holland Backup v1.0.8
Copyright (c) 2008-2010 Rackspace US, Inc.
More info available at

Lets review what is in the plugin directory. These are the providers. Some are core libraries for the use of particular providers. There's also an example and random directory which we will reference in the next article on config.

[vagrant@node1 plugins]$ ls -w1

There are a couple of essential installs we need to take care of before we're ready to tackle the plugins for the backup tools. These are the holland.lib.common & holland.lib.mysql. For each you need to run the file from within the directories. The output should look something like the following.

[vagrant@node1]$ cd holland.lib.common/
[vagrant@node1 holland.lib.common]$ sudo python install
running install
Adding holland.lib.common 1.0.8dev to easy-install.pth file

Installed /usr/lib/python2.6/site-packages/holland.lib.common-1.0.8dev-py2.6.egg
Processing dependencies for holland.lib.common==1.0.8dev
Finished processing dependencies for holland.lib.common==1.0.8dev

[vagrant@node1 holland.lib.common]$ cd ../holland.lib.mysql/
[vagrant@node1 holland.lib.mysql]$ sudo python install
running install
Adding holland.lib.mysql 1.0.8dev to easy-install.pth file

Installed /usr/lib/python2.6/site-packages/holland.lib.mysql-1.0.8dev-py2.6.egg
Processing dependencies for holland.lib.mysql==1.0.8dev
Finished processing dependencies for holland.lib.mysql==1.0.8dev

Now, I'm mainly interested in mysqldump and xtrabackup but you might want to use the mysql_lvm. I don't think there are many takers for mysqlhotcopy these days. The same steps apply for each provider you want to install on each machine.
[vagrant@node1 holland.backup.mysqldump]$ sudo python install running install ... Adding holland.backup.mysqldump 1.0.8dev to easy-install.pth file Installed /usr/lib/python2.6/site-packages/holland.backup.mysqldump-1.0.8dev-py2.6.egg Processing dependencies for holland.backup.mysqldump==1.0.8dev Finished processing dependencies for holland.backup.mysqldump==1.0.8dev

and then finally from the xtrabackup directory;
[vagrant@node1 plugins]$ cd holland.backup.xtrabackup/ [vagrant@node1 holland.backup.xtrabackup]$ sudo python install running install ... Adding holland.backup.xtrabackup 1.0.8dev to easy-install.pth file Installed /usr/lib/python2.6/site-packages/holland.backup.xtrabackup-1.0.8dev-py2.6.egg Processing dependencies for holland.backup.xtrabackup==1.0.8dev Finished processing dependencies for holland.backup.xtrabackup==1.0.8dev

So in summary it's the library files for the plugins;
  - holland.lib.common
  - holland.lib.mysql
  - holland.lib.lvm (depending if you're going to use it)

In part two I'll be giving an overview of the config files and then part three will be some cookbook-like config for the various providers.