Bug or not bug? A Falcon mystery solved

Bug#33184 looked like a tough nut to crack. The initial report was quite clear. Some tests failed on Mac OS X.

I initially assumed that our testing boxes did not include the platform I was using, and informed Hakan. He tested the suite on his box, and reported success.
Too bad. Being able to reproduce a bug is the first step towards its fix. So we exchanged notes on how we compiled the sources. It turned out that I was using the optimized build, while Hakan was using a build with debug options.
I tried with debug options, and indeed, the test suite passed. However, since users are likely to adopt the optimized build, the bug was still confirmed.
More time passed. Hakan made some changes, and the test suite passed in his box, but failed in mine.
Finally, Hakan found the problem. The test suite failed because of the default gcc compiler provided by Apple. Mac OS X 10.4 and 10.5 offer gcc 4.0.1 with Xcode. Apparently, as Hakan found out, there is a bug in the exception handling of Apple's gcc.
After upgrading to gcc 4.2.2, the test suite passed without errors.
Good job, Hakan!

Chasing elusive bugs

Since the announcement of Maria, I have been trying to use the same crashing test with Falcon, to establish conclusively if its crash recovery features are as reliable as I hope.

I tried the same test used for Maria, and after a crash, Falcon did not recover nicely. In fact, the server crashed on restart. I have not been able to repeat that behavior on Linux. Actually, I did, on a remote server that is not available to me at the moment, using an earlier revision, and I am waiting until I am back home to do some more testing with the latest tree.
However, I managed to repeat the problem with Mac OS X.
I run this script on a 6.0.5 server:

set storage_engine=falcon;
drop table if exists t1;
create table t1 (id int, b longblob) ;
insert into t1 values (1, repeat('a',1000000));
insert into t1 select * from t1;
insert into t1 select * from t1;
insert into t1 select * from t1;
insert into t1 select * from t1;
insert into t1 select * from t1;
insert into t1 select * from t1;
insert into t1 select * from t1;
insert into t1 select * from t1;
insert into t1 select * from t1;

When the script reaches the last line (has loaded 128 records and it's inserting the next batch), I stop the server with a vicious kill -9.
Upon restarting, the system looks under control, and I run the command that was previously interrupted:

insert into t1 select * from t1;

Here, the server crashes. I have repeated the above steps several times, and it crashes consistently on Mac OS X. On Linux, which I have on a virtual machine only (I am still on the road!) I can't repeat the crash with the current revision.
So, this is a bug, no questions about that, but not a serious one, unless I manage to prove otherwise.
Perhaps this bug is related to Bug#33517. We'll see.
If anyone has noted something similar, please let me know.

Finding Falcon hidden bugs - with a little help from my shell

There is a song that keeps popping on my mind when I test something difficult. Usually, when I can't get the result I am seeking with normal operations, and I have to resort to some trick, I hear Joe Cocker singing With a little help from my friends. My best friends when I test are bash, vim, and perl.
Today, I was testing Falcon on my Intel based MacBook. The last Falcon release does not compile on Macs, but it has been recently patched, and today with my delight it built smoothly.
Having got the binaries, I ran the test suite, because that's the fastest way of finding out if something is broken. Soon enough, something nasty popped up. Test "falcon_bug_29040" failed with timeout. Thus, I switched to full bug finding mode, and ran the test again.
Odd. When run alone, it succeeds.
I restart the test suite. Again, it fails, but it is executed after some 40 different tests. Apparently, there is a conflict. Either a previous test is not cleaning up after itself, or there is yet another bug to be filed.
Then the task is: how do I know which of the previous tests make it fail?
The solution is to run the tests before "falcon_bug_28040" one at the time, with the suspect one immediately after. When I get a failure, I know who the culprit is.
With a little help from my shell, the task is easy:

for T in t/falcon_bug_2[234567]*.test ; \
  do \ 
    TEST=`basename $T .test` ; \
    echo $T $TEST ; 
    ./mtr --skip-ndb --skip-im $TEST falcon_bug_28040 ; \
done

The tests before the one under examination are all falcon_bug_XXXXX, where XXXX ranges from 22089 to 27997. So I ask the shell to single out the tests in that range. For each one, I get the base name (without extension), and I pass it to the test runner. Total time to find the culprit: three minutes.
It took me much more to submit the bug report and to write this account of the facts!