Tuesday, March 06, 2012

Tungsten 2.0.5 with more power and ease of use

Tungsten Replicator 2.0.5 was released this week-end. The release notes have quite a long list of bug fixes. Thanks to all the ones who have submitted bug reports, and fixes! There are a couple of new features as well. The replicator includes now a slave prefetch service. Unlike parallel replication, this feature works fine with a single database, and provides performance improvements that in many cases solve the slave lagging problems. This was a bitch of a feature to get right. Many have tried it, many have experienced various degrees of success, and several failures. We started with the bold assertiveness of the brave after an exciting talk at Percona Live in October, and I was sorry to report one bad performance result after the other for a few months, until finally the tide turned, and the good results started showing up, and improving! The key to success was the realization that the prefetch is hard to set up and tune right, but also the need for multiple threads that do the pre-fetching efficiently. Since we had already an efficient engine that we use for parallel replication, the final design started bearing fruits at the end of January, and became definitely good and reliable in February. The other noteworthy improvements were made in the installer. Thanks to the many users who have tried it and reported usability issues, we have made the Tungsten Replicator installation a much better experience, and a powerful tool. The best proof of the installer maturity is that the prefetch installation required little work to be implemented and it worked flawlessly at the first attempt! Other improvements in the tools include trepctl and thl better understanding of their environment. They no longer require a service name if there is only one installed in a given host, and they provide more instrumentation for parallel replication, pre-fetching, and for the processing of huge transactions (quite common when dealing with RBR). This version was also the first with Oracle to MySQL support. This is not open source, however. As this feature requires substantial investments, it is not possible to release it as the rest of the replicator. But the list of goodies is not over yet. The feature that probably more than anything else has been used in the past months has been the star schema topology, which is something that was probably possible in 2.0.4, but nobody had tried it before.
Tungsten topologies
We are not stopping here, however. The investment in the installer has given us the know-how necessary to improve and simplify the installation of our flagship product (Tungsten Enterprise) which is about to ship with similar usability enhancements. We have plans to enhance multiple master replication and management, we are developing powerful parallel processing administration tools, and we are also trying to simplify the powerful filters that Tungsten provides. There are more open source releases to discuss, but these will require more than one article to be described conveniently. We have released more tools in the Tungsten Toolbox project. A better Tungsten Sandbox, capable of installing every technology, and some more ancillary tools for Tungsten. I will come back to those in the near future. Much as I like coding, I also like talking about the cool things that we have made. And, another thing that kept me busy and happy: Continuent and SkySQl are now partners. This has given me quite a lot of work, since we had deliver training to SkySQL field operatives. It was a beautiful experience (teaching to a class of advanced users always is) also because most of the attendees were my former colleagues at MySQL AB. The future looks good. More to come.

12 comments:

George said...

Congrats, hoping that the latest release works with MariaDB 5.2.10 as well with bug fix to issue #256

heads up your link to Tungsten toolbox is broken, missing semi-colon

http//code.google.com/p/tungsten-toolbox/

Giuseppe Maxia said...

George,
Thanks. Fixed the typo. As for the MariaDB bug, it's high in my list.

Tim said...

With the start topology, would it be possible to apply a filter that controls what tables "fan in" and another filter that controls what tables "fan out"?

Thanks,

Tim

Giuseppe Maxia said...

Tim,
Sure. The star topology is made of separate services. In every spoke of the star there are two services: one is the master of the data originated on that spoke, and one is the slave of the master service in the hub.
In the hub there is a master service (data out) and one slave service for each spoke (data in).
You can set a filter at either end of each service.
If you filter at the master end, the data is not set in circle at all. If you set the filter at the slave end, you can decide which slaves get that data and which ones don't.

Tim said...

Nice. I already have a "fan in" topology working. Can I simply add the "fan out" slave services to the mix or should I do a clean install? (I'm still in the testing phase anyway.)

Also, I don't remember seeing a "star" recipe in the cookbook. Are there any examples to look at?

Tim

Giuseppe Maxia said...

Tim,
To set a star you need to add "-a --svc-allow-any-remote-service=true" in the slave service on every spoke, but not on the services on the hub.

We are improving the architecture in 2.0.6 (need to reduce the overall traffic) . As soon as we are satisfied with it, we will publish a recipe.

Tim said...

Thank you. I'll give it a try.

Alan Grey said...

I've been playing around with Tungsten Replicator using an all-master topology with 4 masters.

From what I gather, this would be an excellent solution except for one major glaring problem.

create table syntax seems to go into an infinite replication cycle due to log-slave-updates needing to be on.

I checked the tungsten replicator issue list and this seems to be a known issue
http://code.google.com/p/tungsten-replicator/issues/detail?id=12
(first reported some 18 months ago)


From my perspective, this is a massive issue that makes all-master topologies too unwieldy for commercial use.

Alan Grey said...

Wasn't trying to sound harsh. Just the end of a long day...

I'm sure I'm probably missing something simple, but I'm concerned about having to set sql-bin-log=0 before every create statement on every server will be a long running pain.

Giuseppe Maxia said...

Alan,
Issue#12 was fixed long time ago. It was fixed by the same fix for issue#268, but we forgot to update that issue.

So, multi-master operations are quite safe and strong in the latest Tungsten builds.

Alan Grey said...

Thanks for the update, I appreciate it!

The problem is, I have encountered the infinite loop on create table statements problem in 2.0.5, so something must have gotten stuffed up. I had thought it curious that it only happened with one set of table creates.

I'll work on a steps to replicate and post it as a bug report as soon as I can

Alan Grey said...

With more investigation, looks like it is a problem with 'DROP TABLE IF EXISTS'. Same sort of problem as issue 12 was fixing.

Created issue 388.