Showing posts with label linux. Show all posts
Showing posts with label linux. Show all posts

Wednesday, May 04, 2016

Docker for Mac beta and MySQL - First impressions

Using Docker for development is a great way of ensuring that what you develop will be the same that you deploy in production. This is true for almost everything. If you develop on Linux, the above statement holds. If you develop on a different operating system (OSX or Windows) there are several restrictions.

I showed one of those issues in a recent article (MySQL and Docker on a Mac: networking oddity.) When you want to export a port from a service running in the container, the exported port is not available in your mac, but in the virtual machine that runs Docker services. This happens with any application that listens to a port.

The second limitation I found affects only MySQL, and it is related to using volumes. The proper way of achieving data persistence with containers is through volumes, i.e. telling the container to run the data directory in a virtual path that refers to some safe place in the host computer. That can't be done on a Mac, because the host computer is a virtual machine, and even though Docker can access a folder in your Mac, the server installation fails for lack of permissions.

Both the above restrictions are lifted if you use the beta release of Docker for Mac and Windows. It's a private beta: you need to apply and wait to be given an operational token, but once you are in, you notice the differences between the beta and the "old" Docker-Toolbox:

Docker beta

  • The Docker app is a native app, which you install by copying its icon to the /Application folder;
  • You don't need Virtualbox or VMware Fusion. It comes with its own lightweight VM based on xhyve.
  • There is no need to run docker-machine start xxx and eval $(docker-machine env xxx). The new app is fully integrated with the OS.
  • Ports exported from a container are available in your Mac.
  • You can keep both the Docker Toolbox and the new Docker app in the same host, provided that you don't run them both in the same terminal session.

Back to our claim of lifted limitations: let's try a full installation on a Mac as we would do it on Linux.

$ docker run --name mybox -e MYSQL_ROOT_PASSWORD=secret -d \
    -v ~/docker/mysql/single:/var/lib/mysql \
    -p 5000:3306 mysql/mysql-server
72ca99918076ff0e5702514311cc706ffcc27f98917f211e98ed187dfda3b47b

$ ls  ~/docker/mysql/single/
auto.cnf    client-key.pem  ibdata1     mysql.sock.lock    server-cert.pem
ca-key.pem  ib_buffer_pool  ibtmp1      performance_schema server-key.pem
ca.pem      ib_logfile0     mysql       private_key.pem    sys
client-cert.pem             ib_logfile1 mysql.sock         public_key.pem

We create a MySQL server container with the internal port 3306 exposed to the external port 5000, and the data directory running in the host directory $HOME/docker/mysql/single. It seems that the data directory was created correctly. Now we use a MySQL client on the Mac to connect to the container, using port 5000 on the local network address (Note: there is NO database server running on my mac. Only in the container).

$ sudo netstat -atn  |grep LISTEN | grep 5000
tcp4       0      0  *.5000                 *.*                    LISTEN

$ ~/opt/mysql/5.7.12/bin/mysql -h 127.0.0.1  -u root -psecret -P 5000
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.7.12 MySQL Community Server (GPL)

Copyright (c) 2000, 2016, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> create schema hello_docker;
Query OK, 1 row affected (0.01 sec)

mysql> use hello_docker
Database changed
mysql> create table i_am_here(id int);
Query OK, 0 rows affected (0.03 sec)

mysql> exit
Bye

$ ls  ~/docker/mysql/single/hello_docker/
db.opt        i_am_here.frm    i_am_here.ibd

This is full integration! Using a Mac client we connected to the container, where we created a table, which then appeared inside the data directory in the Mac host!

It's still early to say if this beta is ready for more serious work, but the first impressions are really good!

Tuesday, September 11, 2012

My speaking engagements - Q4 2012

After a long pause in the speaking game, I am back.

It's since April that I haven't been on stage, and it is now time to resume my public duties.

  • I will speak at MySQL Connect in San Francisco, just at the start of Oracle Open World, with a talk on MySQL High Availability: Power and Usability. It is about the cool technology that is keeping me busy here at Continuent, which can make life really easy for DBAs. This talk will be a demo fest. If you are attending MySQL Connect, you should see it!
  • A happy return for me. On October 27th I will talk about open source databases and the pleasures of command line operations at Linux Day in Cagliari, my hometown. Since I speak more in California than in my own backyard, I am happy that this year I managed to get a spot here.
  • The company will have a team meeting in Nopvember (Barcelona, here we come!) and from there I will fly to Bulgaria, where I am speaking at the Bulgarian Oracle User Group conference. Here I will have two talks, one about MySQL for business, and the other is "MySQL High Availability for the masses".
  • A few days later, again on the road, in London, for Percona Live, with a talk on MySQL High Availability: Power, Magic, and Usability. It is again about our core products, with some high technology fun involved. I will show how our tools can test the software, spot the mistakes, fix the cluster, and even build a step-by-step demo.
See you around. Look for me carefully, though. I may look differently from how I have been depicted so far.

Tuesday, February 16, 2010

Linux MySQL distros meeting in Brussels

When I saw Shlomi's post on why not to use apt-get or yum for MySQL, I thought immediately that his conclusions are quite reasonable. What you get from the Linux distributions is not the same thing that you find in the official MySQL downloads page. Now, whether you value more the completeness of the server or the ease of administration through the distribution installation tools, it's up to you and your business goals. We at the MySQL team have organized a meeting with the Linux distributions with the intent of finding out which differences and problems we may have with each other, and to solve them by improving communication. What follows is a summary of what happened in Brussels during the meeting.

Linux Distro MySQL packagers meeting

Summary


Linux distributions ship MySQL products (Server, GUI Tools, connectors, Cluster) with different criteria and different grade of maturity, according to their own goals.
Due to lack of communication and policy conflicts, the distros almost always ship outdated versions of MySQL server and MySQL Cluster. The lag between the shipped version and the latest product shipped by MySQL ranges from a few months to several years.
By mutual understanding, the distros will now try to ship recent versions of Cluster (7.x) in a separate package.

Participants


(see some more pictures from the meeting).

Giuseppe Maxia, MySQL Community Team Lead, Italy
Tomas Ulin, MySQL VP Engineering, Sweden
Harmut Holzgraefe, MySQL Support, Germany
Lars Heill, MySQL Build, Norway (Trondheim)
Joro Kodinov, MySQL Engineering / 5.1, Bulgaria
Oden Eriksson, Mandriva, Sweden
Mathias Gug, Canonical/Ubuntu, Canada
Robin H. Johnson, Gentoo Linux, Canada
Michal Hrušecký, Novell/openSUSE, Czech Republic
Geir Høydalsvik, MySQL QA, Norway (Trondheim)
Norbert Tretkowski, Debian Linux, Germany
Kaj Arnö, MySQL VP Community Relations, Germany

Main issues from the distributions:

d1. Security bugs are invisible until MySQL releases a fix. They would like to get visibility of the bug report, to become aware of the problem and eventually help fixing it. We are looking into this matter.

d2. Due to lack of communication, the distros were running the test suite with different parameters. Gentoo packages the server with UTF-8 as default character set, and this causes several tests to fail. Our QA team is looking into it.

d3. Bug databases are different for each distros. They usually solve problems on their own, or send the issue upstream (to the MySQL team at Sun, now Oracle) when it is a legitimate bug.

d4. Debian and Ubuntu don't apply all our patches to the server that they ship. They only apply security bugs and fix for bugs that don't introduce new or changed functionality. This is, IMO, mostly a matter of terminology, since the new functionality is only added as a side effect of fixing a bug. For example, when we fixed Bug#49222: Mark RAND() as unsafe, there is a change in functionality. Now RAND() is logged in ROW format, as it should have been in the first place. It is indeed a new functionality, but as a user I would rather have this bug fix in my server, than adhering to the strict rules of no changes.

d5. GUI tools are still shipped as current although they aren't actively supported, with patches provided by OpenSuse.

d6. While we provide specifications for .rpm packages, we don't do that for .deb ones. Debian/Ubuntu ask if we can include them in our code.

Main issues from the MySQL team:

m1. Cluster packages are outdated. Mainly for miscommunication, some distros are building the cluster binaries with the server package, thus shipping quite old and non-functional cluster binaries. After an explanation on the Cluster roadmap, the distros agreed to ship 7.x binaries from now on. We agreed that we will modify the build scripts in the server to avoid compiling the cluster binaries unintentionally.

m2. MySQL Workbench is not included in the stable releases. There are two reasons: it is not GA yet, and its source includes non-GPL code (for Windows and Mac) that needs to be removed before being used by Debian and derivatives. Moreover, Debian communicates that some DBAs don't like the idea of deploying a design tool for daily database administration.

Conclusions


All in all, I feel that this meeting was a success. We achieved a lot during the proceedings, solving problems ranging from simple communication mismatches to neglected bugs. And meeting in person with the ones who deal actively with MySQL in the Linux distros is quite a rewarding experience.
From a technical standpoint, I hadn't realized that every distribution is shipping a different server. That is quite a challenge for the common users who may need to choose between versions in several sites. However, this meeting has also shown me that all the participants have very high quality standards, and the difference in shipped versions is mostly due to the peculiarity of many shipping calendars.
Thanks to all the participants. We fixed many issues, and we had lots of fun at the same time. We should do that more often!

Thursday, September 24, 2009

Jeremy's article on MySQL Sandbox in Linux Magazine


Jeremy Zawodny and MySQL Sandbox

Jeremy Zawodny of Craiglist has written a kind article about MySQL Sandbox.
The article, MySQL Sandbox: Treat MySQL Instances like Virtual Machines, is a practical test of MySQL Sandbox with usage examples and warm appreciation.
Thanks, Jeremy!

The article was published in July but I noticed it only today. I guess I should pay more attention to my favorite topics when I travel.

Thursday, June 05, 2008

Which OS for your MySQL server?

On MySQL developers zone there is a quick poll on which operating system you use.

The combined results of all Linux flavors accounts for 50% of the answers so far. What surprised me is the 6% of mac OSX used as a server. My own experience with macs is just as desktop or laptop. In my very personal view, it feels like a waste to use such a beautiful thing like a Mac as a server. Linux, as Neal Stephenson put it in In The Beginning Was The Command Line is like a tank. Efficient but not appealing, compared to Macs, which are described as stylish cars.
So, if business is a battle, the Linux tank seems your best bet to get out alive when the smoke settles. The funny thing is that Solaris is an even sturdier tank than Linux, and it should deserve more attention. I guess that, being a recent contender in the open source arena, it will have to pay the price of the learning curve. A price that Linux has paid long time ago and now is reaping the fruit.
If you disagree, have your say!

Tuesday, May 27, 2008

Virtual squares - Taking virtualization to new limits


During the Italian Free Software Conference in Trento, I attended an amazing presentation on virtual components.
Renzo Davoli, professor at Bologna University and hacker, has stretched the concept of virtualization to new limits. Virtual Square is a project aiming at developing more dynamic implementations of virtual entities, which eventually get separated from the concept of operating system and root privileges.

The coolest aspect of all this project is the virtualization of single elements like a disk drive, a net port, a file system, without root privileges, and with no impact on other users.
Virtualizing single elements makes life easier for demanding users, and more quite for their neighbors, who won't be affected by massive reduction of overall resources as it happen with normal virtualization of operating systems.
Think of the applications: for example, it would be easier to establish dedicated quotas for database and web server users, with better security and easier maintenance and without creating new OS users. MySQL, with its limited interface for user resources, would surely benefit from this system. There is a lot of potential in this idea. Let's hope it is pushed a bit farther than the academic circles.

Friday, February 08, 2008

From open source to open country

During the last day of the Linux Conference Australia, I and Colin had the choice between joining the crowd for the open day, or meeting a semi-private invitation from Greg Lehey a well known icon in the open source world. He organized a hackers barbecue in his very isolated place, and we could not resist. In his original invitation, Greg said that he lives "down the road", without specifying how long the road is. It turned out that it's a 90 minutes drive, which is an indication of the relative sense of distance that you get in Australia.

In the Australian bush, Greg leads a quiet life with his wife, his horses and dogs, and a few not so distant friends, dealing occasionally with stray kangaroos invading his fields, and brewing open source beer.
The barbecue met everybody's expectations, and so did Greg's famous home brewed beer, thus stimulating that sort of hacker talk that only good food and good company can generate. The subjects ranged from the origin of the hacker's word to the openness of languages, touching virtualization, databases, operating systems, embedded libraries, and user interfaces in the process.

After lunch, we were given a demonstration of the damages kangaroos can do, along with a visual recognition of their dumpings. The peek of the visit, however, came with the open source brewery, which is controlled by a FreeBSD box, in action since 2001.

On our way back, we stopped at a wildlife center, where we had close encounters with kangaroos and koalas, before reaching back to the civilized Melbourne, where Colin gave me a crash course on "how to have fun on weekends," but I won't disclose any of its contents. Suffice it to say that Colin approaches this matter with rare professionalism, and if you need advice, you must contact him directly.

Tuesday, January 29, 2008

Slides from MySQL Proxy presentation at LCA 2008


The MySQL Proxy presentation at Linux Conf Australia 2008 was done just before lunch.
As announced, here are the presentation slides
The penguin on the above image is actually painted on my room's window glass.

Monday, January 28, 2008

Attending Linux Conference Australia - Melbourne 2008

The world tour proceeds. After Sydney, I am now in Melbourne, to attend the Linux Conf Au, hosted at Melbourne University. I am actually lodged at the charming Trinity College.

I have already met several known people and many more new people. Currently, I am getting ready for my presentation on MySQL Proxy at the MySQL mini-conf, and I have also a surprise presentation that will happen during the lighting talks.
Curious? You should be! See you tomorrow afternoon for the surprise talk!

Friday, January 25, 2008

Session on MySQL Proxy at the Linux Conf Australia


It's getting closer.
On Tuesday late morning, I will hold a session on MySQL Proxy during the MySQL MiniConf at Linux Conf Australia.
It will be an introduction to MySQL Proxy with some live demo.
If you are in Melbourne, come along!

Thursday, January 10, 2008

World tour - step one


I am about to embark for my longest trip ever.
I will board a plane on Saturday in my hometown in Sardinia, Italy, heading to Munich. A comfortable two hours flight, to be ended with a business meeting and a traditional Munich dinner, both provided by Kaj Arnö.
The easy part of the trip ends here. From Munich, I will go to Orlando, USA, to the Company Meeting.
After the meeting, a short stopover in Los Angeles, and then the loooong jump to Sydney, which I will be wisiting for the first time. And another first time will be in Melbourne, a few days later, where I will attend the Linux Conf Au, and also present a session about MySQL Proxy during the MySQL MiniConf.

That is not the end of it, of course. I will be just half world away from home. Then I will go to Singapore for a vacation, and then back th Europe.

More travel news to come in the next days!

Saturday, September 15, 2007

How to hijack a Linux + MySQL server

I hope I got your attention with the title. The politically correct term should be "how to redirect MySQL traffic on Linux", but you probably wouldn't have noticed.
The scenario is this. You want to switch servers without need for the clients to change connection parameters.
Why would you want to do this? Some reasons:
  • You want to test a new server version, which you installed to listen to port 13306, and you want to do it without modifying the applications parameters;
  • You ant to use MySQL Proxy on port 4040 to log the traffic for a while without restarting the server, in order to debug an inexplicable problem;
Whatever the reason, you can do it.

Here is how you can proceed on Linux.

Set a iptables rule to redirect port 3306 to port 13306 or to port 4040:
sudo iptables -t nat -I PREROUTING \
-s ! 127.0.0.1 -p tcp \
--dport 3306 -j \
REDIRECT --to-ports 13306
Caveat: This will only work if your clients are connecting from a separate machine. On the same server, this rule spares the local IP address 127.0.0.1.

To undo the redirection, repeat the above command with -D instead of -I.