From Bazaar to Sandbox in 5 moves

Now that MySQL has switched its development to Bazaar, we can play a bit with the new tools. One thing that I have tried immediately upon adoption of the new RCS is to create a test bench for multiple MySQL versions by combining Bazaar and MySQL Sandbox.

If you are into bleeding edge testing, you may like to know how to get a new MySQL server up and running in a sandbox from a source tree.

First move - make the repository

This is a zero time consuming move, and you will have to do this only once. But this move will save you heaps of time in future. By creating a shared repository, you are telling Bazaar to avoid redundant copies of files (actually Bazaar is much smarter than this, but as a simple explanation it will do).

$ bzr init-repo shared
$ cd shared/

Second move - get the source

Now we're ready to get the first copy of our source. The first branch we get is the current tree, 5.1. Depending on the speed of your network connection, it will take between 20 and 40 minutes.

$ time bzr branch lp:mysql-server/5.1  51

lp stands for launchpad. mysql-server is the project name, and 5.1 is the version. 51 is the directory we are creating on the local disk. You can call it whatever you want. I prefer short names.

This operation may be tricky. The current Bazaar client (1.5) is not verbose. It may actually appear to be doing nothing for very long time. It is actually importing code. Don't be discouraged and bear with me on this. It will pay off in the end. After a while you will see a progress bar and eventually it will display the result.

Branched 2655 revision(s).                                                                                                   

real 19m14.110s
user 7m22.883s
sys 1m21.372s

In my laptop, it took less than 20 minutes, but it could be much slower, depending on connection type and intensity of net traffic.

Let's have a look at the size of what we have downloaded. The hidden directory .bzr contains the biggest chunk.

$ du -sh .bzr/ */
498M .bzr/
129M 51/

Now for the juicy part. We want to branch also MySQL 5.0. You may be ready for yet another 20 minutes and 600 more MB of data to be added, but you are in for a good surprise:

$ time bzr branch lp:mysql-server/5.0 50
Branched 2634 revision(s).                                                                                                   

real 0m42.205s
user 0m28.100s
sys 0m4.757s

Less than one minute! And what about storage?

$ du -sh .bzr/ */
498M .bzr/
100M 50/
129M 51/

As you see, the new code added just the difference between 5.1 and 5.0, so it did not even had to get it from the net because it mostly got the previous version from the internal history (we'll come back to this point in a later post).
A similar experience awaits you when branching the newest version, 6.0. A bit slower than 5.0 (because it has to get something more from the net), but still quite fast compared to the full tree we got in the first run.

$ time bzr branch lp:mysql-server/6.0 60
Branched 2664 revision(s).                                                                                                   

real 4m45.383s
user 1m39.727s
sys 0m20.776s

$ du -sh .bzr/ */
566M .bzr/
100M 50/
129M 51/
146M 60/

Third move - export the code

We have now on disk the code for three versions. We could just go inside each directory and build it, but that would not be clean. So in this move we just export the code to a build directory that we create for this task.

$ mkdir ~/install/build

$ ls
50    51    60

Using a shell loop, we export each tree to a build directory. It takes about 30 seconds for each tree.

$ for R in 50 51 60 ; do echo $R ; cd $R ; bzr export ~/install/build/$R ; cd .. ; done
50
51
60

Fourth move - build the code

We are finally ready to build. Each build can take quite long, depending on your box. In my laptop, it takes about 40 minutes per build.

In the ./BUILD directory in each source tree there are several building scripts. Choose the one that suits your purposes, eventually disabling the parts that you don't need (e.g. cluster).

$ cd ~/install/build

$ for R in 50 51 60 ; do \
  cd $R ; \
  ./BUILD/your_favorite_script && ./scripts/make_binary_distribution ; \
  cd .. ; \
  done

After each build, this loop runs make_binary_distribution, which creates a binary tarball, right what you need for MySQL Sandbox.

Fifth move - sandbox it!

Now, let's use our favorite installation tool to use these new binaries.

Download the sandbox. Make sure that you got at least version 1.21.

$ export SANDBOX_HOME=$HOME/sandboxes
$ cd ~/install/mysql_sandbox-1.21

For each version, you invoke the sandbox installer as

./express_install full/path/to/tarball_name.tar.gz

Here we go

$ ./express_install.pl ~/install/build/50/mysql-5.0.66-darwin9.3.0-i386.tar.gz
unpacking /Users/gmax/install/build/50/mysql-5.0.66-darwin9.3.0-i386.tar.gz
Executing ./install.pl --basedir=/Users/gmax/install/build/50/5.0.66 \
 --sandbox_directory=msb_5_0_66 \
 --install_version=5.0 \
 --sandbox_port=5066 \
 --no_ver_after_name
  The MySQL Sandbox,  version 1.21 09-Jun-2008
  (C) 2006,2007,2008 Giuseppe Maxia, MySQL AB
installing with the following parameters:
home_directory                 = /Users/gmax/sandboxes
sandbox_directory              = msb_5_0_66
sandbox_port                   = 5066
datadir_from                   = script
install_version                = 5.0
basedir                        = /Users/gmax/install/build/50/5.0.66
[...]
loading grants
sandbox server started
installation options saved to current_options.conf.
To repeat this installation with the same options,
use ./install.pl --conf_file=current_options.conf
----------------------------------------
Your sandbox server was installed in /Users/gmax/sandboxes/msb_5_0_66

$ ./express_install.pl ~/install/build/51/mysql-5.1.26-rc-darwin9.3.0-i386.tar.gz
unpacking /Users/gmax/install/build/51/mysql-5.1.26-rc-darwin9.3.0-i386.tar.gz
Executing ./install.pl --basedir=/Users/gmax/install/build/51/5.1.26 \
 --sandbox_directory=msb_5_1_26 \
 --install_version=5.1 \
 --sandbox_port=5126 \
 --no_ver_after_name
  The MySQL Sandbox,  version 1.21 09-Jun-2008
  (C) 2006,2007,2008 Giuseppe Maxia, MySQL AB
installing with the following parameters:
home_directory                 = /Users/gmax/sandboxes
sandbox_directory              = msb_5_1_26
sandbox_port                   = 5126
datadir_from                   = script
install_version                = 5.1
basedir                        = /Users/gmax/install/build/51/5.1.26
my_file                        =
[ ... ]
sandbox server started
installation options saved to current_options.conf.
To repeat this installation with the same options,
use ./install.pl --conf_file=current_options.conf
----------------------------------------
Your sandbox server was installed in /Users/gmax/sandboxes/msb_5_1_26

$ ./express_install.pl ~/install/build/60/mysql-6.0.6-alpha-darwin9.3.0-i386.tar.gz
unpacking /Users/gmax/install/build/60/mysql-6.0.6-alpha-darwin9.3.0-i386.tar.gz
Executing ./install.pl --basedir=/Users/gmax/install/build/60/6.0.6 \
 --sandbox_directory=msb_6_0_6 \
 --install_version=6.0 \
 --sandbox_port=6060 \
 --no_ver_after_name
  The MySQL Sandbox,  version 1.21 09-Jun-2008
  (C) 2006,2007,2008 Giuseppe Maxia, MySQL AB
installing with the following parameters:
home_directory                 = /Users/gmax/sandboxes
sandbox_directory              = msb_6_0_6
sandbox_port                   = 6060
datadir_from                   = script
install_version                = 6.0
basedir                        = /Users/gmax/install/build/60/6.0.6
[ ... ]
sandbox server started
installation options saved to current_options.conf.
To repeat this installation with the same options,
use ./install.pl --conf_file=current_options.conf
----------------------------------------
Your sandbox server was installed in /Users/gmax/sandboxes/msb_6_0_6

At the end of the exercise, you have three sandboxes with the three major MySQL versions, ready to use. Enjoy!