Wednesday, June 25, 2008

Scalable architectures with MySQL Proxy

MySQL community, mark your calendars!. On July 8th, 2008, there is a Webinar on Designing scalable architectures with MySQL Proxy.
This is not the usual marketing sponsored webinar. Although we love to show off, this is not a "look-how-good-we-are" presentation. This is a community driven event, where a community member, using only MySQL Proxy and some creativity, solved his production problems.

This is a real story of a community member who used open source software to build a customized scalable architecture to suit his purposes. Isn't it a good story?
I won't steal his thunder and tell you in advance what was the problem about. I will introduce the general concepts about Proxy, and then our guest John Loehrer will tell hist story in full colors.
Don't miss this one!

Friday, June 20, 2008

MySQL Sandbox - planning for 2.0

Taking advantage of Launchpad environment, I started paving the way for version 2.0 of MySQL Sandbox.
I opened a new branch for version 2.0, which currently holds 1.99.1, and I listed the necessary tasks using the built-in blueprints feature.
MySQL Sandbox
So now I have a list of tasks to keep track of the development.
  • implement test suite (have a good prototype - I am getting to it)
  • replace sleep with check loop (thinking - some analysis required)
  • create function library (good progress)
  • script to kill unresponsive servers (good progress)
  • remote sandbox installer (thinking - hard to implement without being intrusive)
  • allow url for tarball (thinking - needs external components)
  • define circular replication (thinking - easily doable)
  • refactoring for internationalization (thinking - easily doable - long task)
The above features should all go to version 2, although the list is still open. No feature freeze has been called. It's still alpha.
I am unsure about the remote installer. I have implemented such a feature a few years ago, and it turned out to be quite intrusive (the choice was between being intrusive and fragile). I may consider making it a plugin or moving it to a later version. Still thinking.

The development model is open. If you use the Sandbox and want to submit a patch or have comments on its development plans, please contribute!

Thursday, June 19, 2008

From Bazaar to Sandbox in 5 moves


Now that MySQL has switched its development to Bazaar, we can play a bit with the new tools. One thing that I have tried immediately upon adoption of the new RCS is to create a test bench for multiple MySQL versions by combining Bazaar and MySQL Sandbox.

If you are into bleeding edge testing, you may like to know how to get a new MySQL server up and running in a sandbox from a source tree.

First move - make the repository

This is a zero time consuming move, and you will have to do this only once. But this move will save you heaps of time in future. By creating a shared repository, you are telling Bazaar to avoid redundant copies of files (actually Bazaar is much smarter than this, but as a simple explanation it will do).
$ bzr init-repo shared
$ cd shared/

Second move - get the source

Now we're ready to get the first copy of our source. The first branch we get is the current tree, 5.1. Depending on the speed of your network connection, it will take between 20 and 40 minutes.
$ time bzr branch lp:mysql-server/5.1  51
lp stands for launchpad. mysql-server is the project name, and 5.1 is the version. 51 is the directory we are creating on the local disk. You can call it whatever you want. I prefer short names.

This operation may be tricky. The current Bazaar client (1.5) is not verbose. It may actually appear to be doing nothing for very long time. It is actually importing code. Don't be discouraged and bear with me on this. It will pay off in the end. After a while you will see a progress bar and eventually it will display the result.
Branched 2655 revision(s).                                                                                                   
real 19m14.110s
user 7m22.883s
sys 1m21.372s
In my laptop, it took less than 20 minutes, but it could be much slower, depending on connection type and intensity of net traffic.

Let's have a look at the size of what we have downloaded. The hidden directory .bzr contains the biggest chunk.
$ du -sh .bzr/ */
498M .bzr/
129M 51/
Now for the juicy part. We want to branch also MySQL 5.0. You may be ready for yet another 20 minutes and 600 more MB of data to be added, but you are in for a good surprise:
$ time bzr branch lp:mysql-server/5.0 50
Branched 2634 revision(s).
real 0m42.205s
user 0m28.100s
sys 0m4.757s
Less than one minute! And what about storage?
$ du -sh .bzr/ */
498M .bzr/
100M 50/
129M 51/
As you see, the new code added just the difference between 5.1 and 5.0, so it did not even had to get it from the net because it mostly got the previous version from the internal history (we'll come back to this point in a later post).
A similar experience awaits you when branching the newest version, 6.0. A bit slower than 5.0 (because it has to get something more from the net), but still quite fast compared to the full tree we got in the first run.
$ time bzr branch lp:mysql-server/6.0 60
Branched 2664 revision(s).
real 4m45.383s
user 1m39.727s
sys 0m20.776s
$ du -sh .bzr/ */
566M .bzr/
100M 50/
129M 51/
146M 60/

Third move - export the code

We have now on disk the code for three versions. We could just go inside each directory and build it, but that would not be clean. So in this move we just export the code to a build directory that we create for this task.
$ mkdir ~/install/build
$ ls
50 51 60
Using a shell loop, we export each tree to a build directory. It takes about 30 seconds for each tree.
$ for R in 50 51 60 ; do echo $R ; cd $R ; bzr export ~/install/build/$R ; cd .. ; done
50
51
60

Fourth move - build the code

We are finally ready to build. Each build can take quite long, depending on your box. In my laptop, it takes about 40 minutes per build.

In the ./BUILD directory in each source tree there are several building scripts. Choose the one that suits your purposes, eventually disabling the parts that you don't need (e.g. cluster).
$ cd ~/install/build
$ for R in 50 51 60 ; do \
cd $R ; \
./BUILD/your_favorite_script && ./scripts/make_binary_distribution ; \
cd .. ; \
done
After each build, this loop runs make_binary_distribution, which creates a binary tarball, right what you need for MySQL Sandbox.

Fifth move - sandbox it!

Now, let's use our favorite installation tool to use these new binaries.

Download the sandbox. Make sure that you got at least version 1.21.
$ export SANDBOX_HOME=$HOME/sandboxes
$ cd ~/install/mysql_sandbox-1.21
For each version, you invoke the sandbox installer as
./express_install full/path/to/tarball_name.tar.gz
Here we go
$ ./express_install.pl ~/install/build/50/mysql-5.0.66-darwin9.3.0-i386.tar.gz
unpacking /Users/gmax/install/build/50/mysql-5.0.66-darwin9.3.0-i386.tar.gz
Executing ./install.pl --basedir=/Users/gmax/install/build/50/5.0.66 \
--sandbox_directory=msb_5_0_66 \
--install_version=5.0 \
--sandbox_port=5066 \
--no_ver_after_name
The MySQL Sandbox, version 1.21 09-Jun-2008
(C) 2006,2007,2008 Giuseppe Maxia, MySQL AB
installing with the following parameters:
home_directory = /Users/gmax/sandboxes
sandbox_directory = msb_5_0_66
sandbox_port = 5066
datadir_from = script
install_version = 5.0
basedir = /Users/gmax/install/build/50/5.0.66
[...]
loading grants
sandbox server started
installation options saved to current_options.conf.
To repeat this installation with the same options,
use ./install.pl --conf_file=current_options.conf
----------------------------------------
Your sandbox server was installed in /Users/gmax/sandboxes/msb_5_0_66

$ ./express_install.pl ~/install/build/51/mysql-5.1.26-rc-darwin9.3.0-i386.tar.gz
unpacking /Users/gmax/install/build/51/mysql-5.1.26-rc-darwin9.3.0-i386.tar.gz
Executing ./install.pl --basedir=/Users/gmax/install/build/51/5.1.26 \
--sandbox_directory=msb_5_1_26 \
--install_version=5.1 \
--sandbox_port=5126 \
--no_ver_after_name
The MySQL Sandbox, version 1.21 09-Jun-2008
(C) 2006,2007,2008 Giuseppe Maxia, MySQL AB
installing with the following parameters:
home_directory = /Users/gmax/sandboxes
sandbox_directory = msb_5_1_26
sandbox_port = 5126
datadir_from = script
install_version = 5.1
basedir = /Users/gmax/install/build/51/5.1.26
my_file =
[ ... ]
sandbox server started
installation options saved to current_options.conf.
To repeat this installation with the same options,
use ./install.pl --conf_file=current_options.conf
----------------------------------------
Your sandbox server was installed in /Users/gmax/sandboxes/msb_5_1_26
$ ./express_install.pl ~/install/build/60/mysql-6.0.6-alpha-darwin9.3.0-i386.tar.gz
unpacking /Users/gmax/install/build/60/mysql-6.0.6-alpha-darwin9.3.0-i386.tar.gz
Executing ./install.pl --basedir=/Users/gmax/install/build/60/6.0.6 \
--sandbox_directory=msb_6_0_6 \
--install_version=6.0 \
--sandbox_port=6060 \
--no_ver_after_name
The MySQL Sandbox, version 1.21 09-Jun-2008
(C) 2006,2007,2008 Giuseppe Maxia, MySQL AB
installing with the following parameters:
home_directory = /Users/gmax/sandboxes
sandbox_directory = msb_6_0_6
sandbox_port = 6060
datadir_from = script
install_version = 6.0
basedir = /Users/gmax/install/build/60/6.0.6
[ ... ]
sandbox server started
installation options saved to current_options.conf.
To repeat this installation with the same options,
use ./install.pl --conf_file=current_options.conf
----------------------------------------
Your sandbox server was installed in /Users/gmax/sandboxes/msb_6_0_6
At the end of the exercise, you have three sandboxes with the three major MySQL versions, ready to use. Enjoy!

Monday, June 16, 2008

Sun Tech day in Cagliari

The event had an ominous start. Apparently, nobody among the geeks who organized it had paid any close attention to the date to see if it had any significance, and nobody noticed that it was one of the dates when Italian football team was playing a game in the European championship!
I was reminded of a similar case, during FrOSCon 2006, which was held during the World cup. Lenz gave a speech in front of 5 people, all friends and colleagues! So we were afraid that we could face a similar debacle.
Our fears were swept away when the event started, and the room was filled in to capacity by a very dedicated audience, welcomed by professor Giulio Concas, our generous host, who also introduced a project on software quality metrics.
Domenico Minchella
Domenico Minchella, a passionate Solaris ambassador, made a convincing demonstration of ZFS crash recovery. He is never afraid of picking a fight with the audience, and he turns every flamebait into advocacy.
Stefano Sanna and his Sun SPOT toys
Stefano Sanna presented what he called "the pitiful case of a grown up man who has to use Sun SPOT to justify with his wife why he is still playing with Lego toys!".
Stefano Sanna and his Sun SPOT toys
The toy performed as expected, and Stefano has more excuses to continue promoting the Sun SPOT.
Fabrizio Gianneschi
Also Fabrizio Gianneschi had a toy to play with. In addition to the news of Java technology advancement, he played with a Java powered pen that speaks and replays what you write.
throwing dolphins at the audience
After the last session, where I had to warn the impatient audience that the dolphins from the desk would be distributed only at the very end, Massimiliano Dessì and yours truly performed the by now celebrated "throw them the dolphins" stunt, which was greatly appreciated. With our delight, most of the audience stayed after the session, to enjoy technical chatting and refreshments.

stupid things you can do to kill your business

This summary is not available. Please click here to view the post.

Sunday, June 15, 2008

MySQL Sandbox 1.22 and the joys of developing on launchpad

I have recently moved MySQL Sandbox to launchpad and Bazaar.
MySQL Sandboxlaunchpad

Working with Launchpad is really a joy. It is much more than providing a web interface to Bazaar version control. It also includes provisions for bug reporting and handling, writing blueprints for your roadmap, creating distribution files, and manage your packages if you want to release them in Ubuntu.
Using the above tools, I implemented a couple of new features to MySQL Sandbox, and I started to work on a roadmap for the next version, which is now taking shape.
The latest version (1.22) includes the ability of using a default option file ($HOME/.msandboxrc), a safer usage of the "--force" option, and several bug fixes.

Tuesday, June 10, 2008

MySQL Sandbox 1.21 - matrioska and launchpad

MySQL Sandbox has moved to a new home. It is now hosted at Launchpad, a friendly environment where developers can work together.

What's new with MySQL Sandbox? The matrioskas in the image are a clue. Version 1.21 introduces the concept of SANDBOX_HOME, which previously was only a path under which store the sandboxes. With time, when I was using more and more sandboxes, I realized that I wanted to take control of all the sandboxes at once, and issue global commands to all of them.
Starting with this version, the sandbox installer recognizes an environmental variable named $SANDBOX_HOME, and builds the sandboxes under that path.
In addition to grouping all the sandboxes in a place, the installer creates a few scripts that allow you to take control of all the active sandboxes.
Let's say that you have a single sandbox of MySQL 5.0.62, a replication system of 7 5.1.24 servers, and a multiple installation of 5 6.0.5 servers.
Prior to this version, if you wanted to stop each sandbox, you should have to enter each sandbox directory and run "./stop" or "./stop_all". With the sandbox outer container, you just have to run "./stop_all" from one place.
You can also execute the same command in each server in all your sandboxes, by means of the "./use_all" script.
If you want to learn more, on Thursday, June 12, 2008 at 15:00 CEST, I will host a MySQL University session on this very subject.

Thursday, June 05, 2008

Which OS for your MySQL server?

On MySQL developers zone there is a quick poll on which operating system you use.

The combined results of all Linux flavors accounts for 50% of the answers so far. What surprised me is the 6% of mac OSX used as a server. My own experience with macs is just as desktop or laptop. In my very personal view, it feels like a waste to use such a beautiful thing like a Mac as a server. Linux, as Neal Stephenson put it in In The Beginning Was The Command Line is like a tank. Efficient but not appealing, compared to Macs, which are described as stylish cars.
So, if business is a battle, the Linux tank seems your best bet to get out alive when the smoke settles. The funny thing is that Solaris is an even sturdier tank than Linux, and it should deserve more attention. I guess that, being a recent contender in the open source arena, it will have to pay the price of the learning curve. A price that Linux has paid long time ago and now is reaping the fruit.
If you disagree, have your say!

Monday, June 02, 2008

BoF at OSCON


I have just been informed that in addition to a regular session, I will host two Birds Of a Feather sessions at OSCON 2008.

The call for papers is still open. Any suggestions for more entries?

Firefox 3 is coming


Firefox has been my steady web companion since version 1.0 (and Mozilla before that). For some time now I wanted to try Firefox 3. I was reluctant to abandon Firefox 2, because of the unavailability of most add-ons. I was especially missing the Google Bar, which I have been using on a regular basis.
However, a few days ago I found a replacement. GoogleBar Lite is almost like the original, minus some feature that I wasn't using anyway.
So I have been using Firefox 3 for a while, and I noticed better performance, and no trouble. The additional features are cool, like the Most Visited pull down menu or the location bar integrated with history and search.
Firefox enthusiasts want to set a download world record. I don't think it's necessary, but certainly will be pleasant to achieve it. I pledged.

Sunday, June 01, 2008

Can you see advertising?

While preparing a presentation for Rome University, I took many snapshots of MySQL web site. I asked for review, and Colin pointed at the advertising that was in most every page.
Now that he mentioned it, yes. I saw the advertising. But when I was working with the live page, taking the screenshots, adjusting them with the Gimp, and inserting them into the presentation, I did not notice them at all.
I am blind to ads.
I must be the worst nightmare for advertisers. I have pop-up blockers and javascript filters in my browser, so don't see many of them, but the remaining ones are like a ink stain on the page. My brain registers the presence of an alien presence, and quickly instructs my senses to ignore it. If someone tells me that there is an ad in a given page, I have to look at the page twice, to put the ad into focus.
Why does this happen?
I guess that, after so much time spent using the web, I am trained to spot where the content is, and I filter off automatically all the distracting bits.
If the ad contains an interesting image, I may register that, and even remember it sometimes, but I have no idea of what they were selling with that picture.

For instance, I saw this image in the Singapore subway. I took a picture because it was cute. I can't remember what was advertised.
Now, if this filtering happens due to my frequent usage of the web, I wonder how many users have developed the same self defense and evolved into a Homo Sapiens Non Publicitarius, a particular breed of human being who has been saturated by ads and can't see them anymore.
Anyone from the same mutation out there?