tag:blogger.com,1999:blog-169599462024-03-07T08:14:13.670+01:00The Data CharmerData seem sometimes to have their own life and will, and they refuse to behave as we wish. <br>
Then, you need a firm hand to tame the wild data and turn them into quiet and obeying pets.Giuseppe Maxiahttp://www.blogger.com/profile/15801583338057324813noreply@blogger.comBlogger568125tag:blogger.com,1999:blog-16959946.post-13155894353594955472019-08-25T22:51:00.000+02:002019-08-25T22:51:04.422+02:00One upgrade to rule them all<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgg2Vp9KrzZQJPgXuJ6mXKNmmsZQ8_XjpH6QuMWOLUuK8xip5Z7bFXECZxX9GE4s_ej03vekEOq5Wzge_uQRJDK7I8KZKz-iPw_gkddOYhTxGmfUEKj1r3MiWkYC7cylvFeW17n/s1600/2019-08-25+22.44.19.jpg" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgg2Vp9KrzZQJPgXuJ6mXKNmmsZQ8_XjpH6QuMWOLUuK8xip5Z7bFXECZxX9GE4s_ej03vekEOq5Wzge_uQRJDK7I8KZKz-iPw_gkddOYhTxGmfUEKj1r3MiWkYC7cylvFeW17n/s320/2019-08-25+22.44.19.jpg" width="320" height="240" data-original-width="1600" data-original-height="1200" /></a><br />
<p>Up to now, the way of updating dbdeployer was the same as <a href="https://github.com/datacharmer/dbdeployer#Installation">installing it for the first time</a>, i.e. looking at the releases page, downloading the binaries for your operating system, unpacking it and finally replacing the existing binaries.<br/><br />
This is not the procedure I follow, however, as for me <em>updating</em> means just compile the latest version I have just finished coding. For this reason, when <a href="https://github.com/sjmudd">Simon Mudd</a> mentioned to me that dbdeployer should update itself over the Internet, I didn’t immediately grasp the concept. But then he talked to me again, and he even coded a sample script that does what he suggests: look for the latest release, download it, and replace the current executable with the latest one.</p><p>The only problem with that approach is that dbdeployer should then be distributed with one binary and one update script (and we should make sure that more tools such as <em>curl</em> and <em>gunzip</em> are available in the host computer), while I have pushed the concept that the binary executable should be the only thing needed. Thus I spent a few hours converting the script ideas into dbdeployer internal workings, and given that it had already all the components in place (such as downloading files from the internet and unpacking them) it was done quickly. As a result, dbdeployer can now look online for newer versions of itself, and get its own replacement painlessly.</p><p>The new functionality is <code>dbdeployer update</code>, which works almost silently to do what you expect, if the version online is higher than the local one, and protects you against accidentally fetching an outdated version or binaries from the wrong operating system.</p><p>This is the last update that you need to do manually. After that, the next ones can be fetched directly with dbdeployer itself.</p><p>The new command has several options, mostly added for the advanced user:</p><pre><code>$ dbdeployer update -h
Updates dbdeployer in place using the latest version (or one of your choice)
Usage:
dbdeployer update [version] [flags]
Examples:
$ dbdeployer update
# gets the latest release, overwrites current dbdeployer binaries
$ dbdeployer update --dry-run
# shows what it will do, but does not do it
$ dbdeployer update --new-path=$PWD
# downloads the latest executable into the current directory
$ dbdeployer update v1.34.0 --force-old-version
# downloads dbdeployer 1.34.0 and replace the current one
# (WARNING: a version older than 1.36.0 won't support updating)
Flags:
--OS string Gets the executable for this Operating system
--docs Gets the docs version of the executable
--dry-run Show what would happen, but don't execute it
--force-old-version Force download of older version
-h, --help help for update
--new-path string Download updated dbdeployer into a different path
--verbose Gives more info</code></pre><br />
<br />
<p>The option that would also benefit casual users is <code>--dry-run</code>, as it shows which release it would get the executable from, without actually changing anything.</p><br />
<p>You may ask: why would I want to update using an older version? Although at first sight is not a sound idea, there are at least two reasons for that:</p><br />
<ol><li>there was a bug in the latest release. It isn’t an uncommon occurrence, and much as I would like to claim my code is bug free, I know that’s a pipe dream. So, if a bug is found that affects your workflow, a safe course of action is getting the previous release.</li>
<li>From time to time, I publish experimental releases out of a branch. You may want to try them out, and then go back to the comfort of the latest non-experimental one.</li>
</ol><br />
<p>If you don’t know what the releases contain, you can look them up without using a browser, with the command <code>dbdeployer info releases</code>. For example:</p><pre><code>$ dbdeployer info releases latest
--------------------------------------------------------------------------------
Remote version: v1.36.1
Release: dbdeployer 1.36.1
Date: 2019-08-18T13:23:53Z
## BUG FIXES
- Fix a bug in 'dbdeployer update': it only worked with --verbose enabled
- Fix output of 'dbdeployer update': it now confirms that the update has
happened
--------------------------------------------------------------------------------
dbdeployer-1.36.1-docs.linux.tar.gz (5.9 MB)
dbdeployer-1.36.1-docs.osx.tar.gz (6.1 MB)
dbdeployer-1.36.1.linux.tar.gz (5.8 MB)
dbdeployer-1.36.1.osx.tar.gz (5.9 MB)
</code></pre><br />
<p>There is another feature that complements <em>dbdeployer update</em>: it’s the ability of setting up <a href="https://github.com/datacharmer/dbdeployer#Command-line-completion">command line completion for dbdeployer</a>.</p><pre><code>$ dbdeployer defaults enable-bash-completion -h
Enables bash completion using either a local copy of dbdeployer_completion.sh or a remote one
Usage:
dbdeployer defaults enable-bash-completion [flags]
Flags:
--completion-file string Use this file as completion
-h, --help help for enable-bash-completion
--remote Download dbdeployer_completion.sh from GitHub
--remote-url string Where to downloads dbdeployer_completion.sh from (default "https://raw.githubusercontent.com/datacharmer/dbdeployer/master/docs/dbdeployer_completion.sh")
--run-it Run the command instead of just showing it
</code></pre><br />
<p>This command retrieves the latest completion file, and helps you to install it in the right position. By default, it gets the file and doesn’t change anything in your <code>bash_completion</code> directory: it tells you what commands to run. You can let it do the copying (which may require <code>sudo</code> access) using <code>--run-it</code>.</p>Giuseppe Maxiahttp://www.blogger.com/profile/15801583338057324813noreply@blogger.com0tag:blogger.com,1999:blog-16959946.post-39544191208118442832019-06-30T13:34:00.000+02:002019-06-30T22:33:47.701+02:00From an empty box to MySQL custom replication in 3 minutes<p>Starting with version 1.32.0, <a href="https://github.com/datacharmer/dbdeployer">dbdeployer</a> has the ability of downloading a selection of MySQL tarballs from several sources.<br />
<br />
This means that, when working in an empty box, you can populate it with database servers using<br />
<br />
dbdeployer.</p><p>The “empty box” mentioned in the title is not really empty. It’s a Linux (or MacOS) host that is able to run a MySQL server. As such, it needs to have at least the prerequisites to run MySQL server (such as the <code>libnuma</code> and <code>libaio</code> packages), and a <code>bash</code> shell to run the scripts created by dbdeployer.</p><p>To try the thrill of an empty box that quickly becomes a working environment, we can use a docker image <code>datacharmer/mysql-sb-base</code> that I have created for this purpose.</p><pre><code>$ docker pull datacharmer/mysql-sb-base
Using default tag: latest
latest: Pulling from datacharmer/mysql-sb-base
6b98dfc16071: Pull complete
4001a1209541: Pull complete
6319fc68c576: Pull complete
b24603670dc3: Pull complete
97f170c87c6f: Pull complete
b78c78fcfc94: Pull complete
379084573ce7: Pull complete
0afd193b699a: Pull complete
dfb4eecd399a: Pull complete
Digest: sha256:492c38b8662d393436141de5b3a9ad5b3994a7b095610b43896033fd740523ef
Status: Downloaded newer image for datacharmer/mysql-sb-base:latest</code></pre><p>We can start a container from this image, and we won’t need anything else from the host computer.</p><pre><code>$ docker run -ti --hostname dbtest datacharmer/mysql-sb-base bash
msandbox@dbtest:~$</code></pre><p>The container runs as a regular user. Given that dbdeployer is designed specifically to run without root access (although it <em>can</em> run as root), this is the perfect scenario.</p><p>dbdeployer is already installed, but mysql is not.</p><pre><code>msandbox@dbtest:~$ dbdeployer --version
dbdeployer version 1.34.0
msandbox@dbtest:~$ mysql
bash: mysql: command not found</code></pre><p>Thus, we start getting our software from the locations that dbdeployer knows.</p><pre><code>$ dbdeployer downloads list
Available tarballs
name OS version flavor size minimal
-------------------------------------------------------- ------- --------- -------- -------- ---------
tidb-master-linux-amd64.tar.gz Linux 3.0.0 tidb 26 MB
mysql-8.0.16-linux-glibc2.12-x86_64.tar.xz Linux 8.0.16 mysql 461 MB
mysql-8.0.16-linux-x86_64-minimal.tar.xz Linux 8.0.16 mysql 44 MB Y
mysql-5.7.26-linux-glibc2.12-x86_64.tar.gz Linux 5.7.26 mysql 645 MB
mysql-5.6.44-linux-glibc2.12-x86_64.tar.gz Linux 5.6.44 mysql 329 MB
mysql-5.5.62-linux-glibc2.12-x86_64.tar.gz Linux 5.5.62 mysql 199 MB
mysql-8.0.15-linux-glibc2.12-x86_64.tar.xz Linux 8.0.15 mysql 376 MB
mysql-8.0.13-linux-glibc2.12-x86_64.tar.xz Linux 8.0.13 mysql 394 MB
mysql-5.7.25-linux-glibc2.12-x86_64.tar.gz Linux 5.7.25 mysql 645 MB
mysql-5.6.43-linux-glibc2.12-x86_64.tar.gz Linux 5.6.43 mysql 329 MB
mysql-5.5.61-linux-glibc2.12-x86_64.tar.gz Linux 5.5.61 mysql 199 MB
mysql-5.1.73-linux-x86_64-glibc23.tar.gz Linux 5.1.73 mysql 134 MB
mysql-5.0.96.tar.xz Linux 5.0.96 mysql 5.5 MB Y
mysql-5.1.72.tar.xz Linux 5.1.72 mysql 10 MB Y
mysql-5.5.61.tar.xz Linux 5.5.61 mysql 6.6 MB Y
mysql-5.5.62.tar.xz Linux 5.5.62 mysql 6.6 MB Y
mysql-5.6.43.tar.xz Linux 5.6.43 mysql 9.0 MB Y
mysql-5.6.44.tar.xz Linux 5.6.44 mysql 9.1 MB Y
mysql-5.7.25.tar.xz Linux 5.7.25 mysql 23 MB Y
mysql-5.7.26.tar.xz Linux 5.7.26 mysql 23 MB Y
mysql-5.0.96-linux-x86_64-glibc23.tar.gz Linux 5.0.96 mysql 127 MB
mysql-4.1.22.tar.xz Linux 4.1.22 mysql 4.6 MB Y
mysql-cluster-gpl-7.6.10-linux-glibc2.12-x86_64.tar.gz Linux 7.6.10 ndb 916 MB
mysql-cluster-8.0.16-dmr-linux-glibc2.12-x86_64.tar.gz Linux 8.0.16 ndb 1.1 GB
</code></pre><p>The above command shows all the tarballs that are available for the current operating system. You see that in addition to vanilla MySQL, there are also <a href="https://datacharmer.blogspot.com/2019/03/dbdeployer-community-part-3-mysql.html">NDB</a> and <a href="https://datacharmer.blogspot.com/2019/03/dbdeployer-community-part-1-tidb.html">TiDB</a> packages.</p><p>We start by getting the latest MySQL version using the command <code>get-unpack</code> that is available since version 1.33.0. This command downloads the tarball, compares the checksum, and unpacks it into the expected place.</p><pre><code>$ dbdeployer downloads get-unpack mysql-8.0.16-linux-x86_64-minimal.tar.xz
Downloading mysql-8.0.16-linux-x86_64-minimal.tar.xz
.... 44 MB
File /home/msandbox/mysql-8.0.16-linux-x86_64-minimal.tar.xz downloaded
Checksum matches
Unpacking tarball mysql-8.0.16-linux-x86_64-minimal.tar.xz to $HOME/opt/mysql/8.0.16
.........100.........200.219
Renaming directory /home/msandbox/opt/mysql/mysql-8.0.16-linux-x86_64-minimal to /home/msandbox/opt/mysql/8.0.16
</code></pre><p>The same operation for 5.7 gives us the second version available.</p><pre><code>+ dbdeployer downloads get-unpack mysql-5.7.26.tar.xz
Downloading mysql-5.7.26.tar.xz
.. 23 MB
File /home/msandbox/mysql-5.7.26.tar.xz downloaded
Checksum matches
Unpacking tarball mysql-5.7.26.tar.xz to $HOME/opt/mysql/5.7.26
.........99
Renaming directory /home/msandbox/opt/mysql/mysql-5.7.26 to /home/msandbox/opt/mysql/5.7.26</code></pre><p>Now there are two versions that can be used for operations.</p><pre><code>$ dbdeployer versions
Basedir: /home/msandbox/opt/mysql
5.7.26 8.0.16</code></pre><p>And we are going to deploy one sandbox from each version, because we want to put them in replication.</p><pre><code>$ dbdeployer deploy single 5.7.26 --master
Creating directory /home/msandbox/sandboxes
Database installed in $HOME/sandboxes/msb_5_7_26
run 'dbdeployer usage single' for basic instructions'
. sandbox server started
$ dbdeployer deploy single 8.0.16 --master
Database installed in $HOME/sandboxes/msb_8_0_16
run 'dbdeployer usage single' for basic instructions'
.. sandbox server started
$ dbdeployer sandboxes --full-info
.------------.--------.---------.---------------.--------.-------.--------.
| name | type | version | ports | flavor | nodes | locked |
+------------+--------+---------+---------------+--------+-------+--------+
| msb_5_7_26 | single | 5.7.26 | [5726 ] | mysql | 0 | |
| msb_8_0_16 | single | 8.0.16 | [8016 18016 ] | mysql | 0 | |
'------------'--------'---------'---------------'--------'-------'--------'</code></pre><p>This are our active assets. The sandboxes are independent, but each sandbox has the ability of becoming the receiver of replication. In this case we want to replicate from version 5.7 to version 8.0, as it is always recommended to replicate from earlier to later version.</p><pre><code>$ ~/sandboxes/msb_8_0_16/replicate_from msb_5_7_26
Connecting to /home/msandbox/sandboxes/msb_5_7_26
--------------
CHANGE MASTER TO master_host="127.0.0.1",
master_port=5726,
master_user="rsandbox",
master_password="rsandbox"
, master_log_file="mysql-bin.000001", master_log_pos=4089
--------------
--------------
start slave
--------------
Master_Log_File: mysql-bin.000001
Read_Master_Log_Pos: 4089
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Exec_Master_Log_Pos: 4089
Retrieved_Gtid_Set:
Executed_Gtid_Set:
Auto_Position: 0
</code></pre><br />
<br />
<p>Replication is active. We can now quickly check that it is working:</p><pre><code>$ ~/sandboxes/msb_5_7_26/use -e 'create table test.t1(id int not null primary key, msg1 varchar(50), msg2 varchar(50)) default charset=utf8mb4'
$ ~/home/msandbox/sandboxes/msb_5_7_26/use -e 'insert into test.t1 values (1, @@version, @@server_uuid)'</code></pre><br />
<br />
<p>We create a table in 5.7, taking care of using a character set that agrees with 8.0 defaults (we could also use <code>utf8</code>, but this is the one that presents less potential problems. We fill the table with server specific information (its version and UUID).</p><br />
<p>Now we can check that the slave is working</p><br />
<pre><code>$ ~/sandboxes/msb_8_0_16/use -e 'SHOW SLAVE STATUS\G' | grep 'Running\|Master_\|Log_'
Master_Host: 127.0.0.1
Master_User: rsandbox
Master_Port: 5726
Master_Log_File: mysql-bin.000001
Read_Master_Log_Pos: 4636
Relay_Log_File: mysql-relay.000002
Relay_Log_Pos: 868
Relay_Master_Log_File: mysql-bin.000001
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Exec_Master_Log_Pos: 4636
Relay_Log_Space: 1072
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Master_SSL_Verify_Server_Cert: No
Master_Server_Id: 5726
Master_UUID: 00005726-0000-0000-0000-000000005726
Master_Info_File: mysql.slave_master_info
Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
Master_Retry_Count: 86400
Master_Bind:
Master_SSL_Crl:
Master_SSL_Crlpath:
Master_TLS_Version:
Master_public_key_path:</code></pre><br />
<p>And finally we retrieve from the 8.0 slave the data that was created in 5.7</p><pre><code>$ ~/sandboxes/msb_8_0_16/use -e 'show tables from test'
+----------------+
| Tables_in_test |
+----------------+
| t1 |
+----------------+
$ ~/sandboxes/msb_8_0_16/use -e 'select * from test.t1'
+----+------------+--------------------------------------+
| id | msg1 | msg2 |
+----+------------+--------------------------------------+
| 1 | 5.7.26-log | 00005726-0000-0000-0000-000000005726 |
+----+------------+--------------------------------------+</code></pre><p>QED.</p>Giuseppe Maxiahttp://www.blogger.com/profile/15801583338057324813noreply@blogger.com4tag:blogger.com,1999:blog-16959946.post-82496503974301659272019-04-02T09:30:00.000+02:002019-04-02T15:45:14.930+02:00dbdeployer cookbook - Advanced techniques<p>In the <a href="https://datacharmer.blogspot.com/2019/03/dbdeployer-cookbook-usability-by-example.html">previous post about the dbdeployer recipes</a> we saw the basics of using the <code>cookbook</code> command and the simpler tutorials that the recipes offer.</p><p>Here we will see some more advanced techniques, and more demanding examples.</p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh-BLHFXSTw5F7ep2gVM5MpsoissZuX6DvwO3QRAjqRVP7q1TeMKJd-LpPoV5RTEDzJY9a_Ikv1ygrP1F8PE0a4qtaZl-BSvNjo1YPAerzfc8zbFwui-ZnAwSoJit-mscmVEtIy/s1600/dbdeployer-cookbook-replication.png" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh-BLHFXSTw5F7ep2gVM5MpsoissZuX6DvwO3QRAjqRVP7q1TeMKJd-LpPoV5RTEDzJY9a_Ikv1ygrP1F8PE0a4qtaZl-BSvNjo1YPAerzfc8zbFwui-ZnAwSoJit-mscmVEtIy/s320/dbdeployer-cookbook-replication.png" width="320" height="293" data-original-width="1196" data-original-height="1094" /></a><br />
<p>We saw that the recipe for a single deployment would get a <code>NOTFOUND</code> when no versions were available, or the highest MySQL version when one was found.</p><pre><code>$ dbdeployer cookbook show single | grep version=
version=$1
[ -z "$version" ] && version=8.0.16</code></pre><p>But what if we want the latest Percona Server or MariaDB for this recipe? One solution would be to run the script with an argument, but we can ask dbdeployer to find the most recent version for a given flavor and use it in our recipe:</p><pre><code>$ dbdeployer cookbook show single --flavor=percona | grep version=
version=$1
[ -z "$version" ] && version=ps8.0.15
$ dbdeployer cookbook show single --flavor=pxc | grep version=
version=$1
[ -z "$version" ] && version=pxc5.7.25
$ dbdeployer cookbook show single --flavor=mariadb | grep version=
version=$1
[ -z "$version" ] && version=ma10.4.3</code></pre><br />
<p>This works for all the recipes that don’t require a given flavor. When one is indicated (see <code>dbdeployer cookbook list</code>) you can override it using <code>--flavor</code>, but do that at your own risk. Running the <code>ndb</code> recipe using <code>pxc</code> flavor won’t produce anything usable.</p><br />
<h2 id="Replication-between-sandboxes">Replication between sandboxes</h2><p>When I proposed <a href="https://datacharmer.blogspot.com/2019/03/dbdeployer-community-part-3-mysql.html">dbdeployer support for NDB</a>, the immediate reaction was that this was good to test cluster-to-cluster replication. Although I did plenty of such topologies in one of my previous jobs, I had limited experience replicating between single or composite sandboxes. Thus, I started thinking about how to do it. In the old MySQL-Sandbox, I had an option <code>--slaveof</code> that allowed a single sandbox to replicate from an existing one. I did not implement the same thing in dbdeployer, because that solution looked limited, and only useful in a few scenarios.<br />
<br />
I wanted something more dynamic, and initially I thought of creating a grandiose scheme, involving custom templates and user-defined fillers. While I may end up doing that some day, I quickly realized that it was overkill for this purpose, and that the sandboxes had already all the information needed to replicate from and to every other sandbox. I just had to expose the data in such a way that it can be used to plug one sandbox to the other.<br />
<br />
Now every sandbox has a script named <code>replicate_from</code>, and a companion script called <code>metadata</code>. Using a combination of the two (in fact, <code>replicate_from</code> on the would-be replica calls <code>metadata</code> from the donor) we can quickly define the replication command needed for most situations.</p><br />
<h3 id="Replication-between-single-sandboxes">Replication between single sandboxes</h3><p>Before we tackle the most complex one, let’s demonstrate that the system works with a simple case.</p><p>There is a recipe named <code>replication_between_single</code> that creates a file named, aptly, <code>./recipes/replication-between-single.sh</code>.</p><p>If you run it, you will see something similar to the following:</p><pre><code>$ ./recipes/replication-between-single.sh 5.7.25
+ dbdeployer deploy single 5.7.25 --master --gtid --sandbox-directory=msb_5_7_25_1 --port-as-server-id
Database installed in $HOME/sandboxes/msb_5_7_25_1
run 'dbdeployer usage single' for basic instructions'
. sandbox server started
0
+ dbdeployer deploy single 5.7.25 --master --gtid --sandbox-directory=msb_5_7_25_2 --port-as-server-id
Database installed in $HOME/sandboxes/msb_5_7_25_2
run 'dbdeployer usage single' for basic instructions'
. sandbox server started
0
+ dbdeployer sandboxes --full-info
.--------------.--------.---------.---------------.--------.-------.--------.
| name | type | version | ports | flavor | nodes | locked |
+--------------+--------+---------+---------------+--------+-------+--------+
| msb_5_7_25_1 | single | 5.7.25 | [5725 ] | mysql | 0 | |
| msb_5_7_25_2 | single | 5.7.25 | [5726 ] | mysql | 0 | |
'--------------'--------'---------'---------------'--------'-------'--------'
0
+ $HOME/sandboxes/msb_5_7_25_1/replicate_from msb_5_7_25_2
Connecting to $HOME/sandboxes/msb_5_7_25_2
--------------
CHANGE MASTER TO master_host="127.0.0.1",
master_port=5726,
master_user="rsandbox",
master_password="rsandbox"
, master_log_file="mysql-bin.000001", master_log_pos=4089
--------------
--------------
start slave
--------------
Master_Log_File: mysql-bin.000001
Read_Master_Log_Pos: 4089
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Exec_Master_Log_Pos: 4089
Retrieved_Gtid_Set:
Executed_Gtid_Set: 00005725-0000-0000-0000-000000005725:1-16
Auto_Position: 0
0
# Inserting data in msb_5_7_25_2
+ $HOME/sandboxes/msb_5_7_25_2/use -e 'create table if not exists test.t1 (id int not null primary key, server_id int )'
+ $HOME/sandboxes/msb_5_7_25_2/use -e 'insert into test.t1 values (1, @@server_id)'
# Retrieving data from msb_5_7_25_1
+ $HOME/sandboxes/msb_5_7_25_1/use -e 'select *, @@port from test.t1'
+----+-----------+--------+
| id | server_id | @@port |
+----+-----------+--------+
| 1 | 5726 | 5725 |
+----+-----------+--------+</code></pre><br />
<p>The script deploys two sandboxes of the chosen version, using different directory names (dbdeployer takes care of choosing a free port) and then starts replication between the two using <code>$SANDBOX1/replicate_from $SANDBOX2</code>. Then a quick test shows that the data created in a sandbox can be retrieved in the other.</p><br />
<h3 id="Replication-between-group-replication-clusters">Replication between group replication clusters</h3><p>The method used to replicate between two group replications is similar to the one seen for single sandboxes. The script <code>replicate_from</code> on the group top directory delegates the replication task to its first node, which points to the second group.</p><pre><code>$ ./recipes/replication-between-groups.sh 5.7.25
+ dbdeployer deploy replication 5.7.25 --topology=group --concurrent --port-as-server-id --sandbox-directory=group_5_7_25_1
[...]
+ dbdeployer deploy replication 5.7.25 --topology=group --concurrent --port-as-server-id --sandbox-directory=group_5_7_25_2
[...]
+ dbdeployer sandboxes --full-info
.----------------.---------------------.---------.----------------------------------------.--------.-------.--------.
| name | type | version | ports | flavor | nodes | locked |
+----------------+---------------------+---------+----------------------------------------+--------+-------+--------+
| group_5_7_25_1 | group-multi-primary | 5.7.25 | [20226 20351 20227 20352 20228 20353 ] | mysql | 3 | |
| group_5_7_25_2 | group-multi-primary | 5.7.25 | [20229 20354 20230 20355 20231 20356 ] | mysql | 3 | |
'----------------'---------------------'---------'----------------------------------------'--------'-------'--------'
0
+ $HOME/sandboxes/group_5_7_25_1/replicate_from group_5_7_25_2
Connecting to $HOME/sandboxes/group_5_7_25_2/node1
--------------
CHANGE MASTER TO master_host="127.0.0.1",
master_port=20229,
master_user="rsandbox",
master_password="rsandbox"
, master_log_file="mysql-bin.000001", master_log_pos=1082
--------------
--------------
start slave
--------------
Master_Log_File: mysql-bin.000001
Read_Master_Log_Pos: 1082
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Exec_Master_Log_Pos: 1082
Retrieved_Gtid_Set:
Executed_Gtid_Set: 00020225-bbbb-cccc-dddd-eeeeeeeeeeee:1-3
Auto_Position: 0
0
# Inserting data in group_5_7_25_2 node1
+ $HOME/sandboxes/group_5_7_25_2/n1 -e 'create table if not exists test.t1 (id int not null primary key, server_id int )'
+ $HOME/sandboxes/group_5_7_25_2/n1 -e 'insert into test.t1 values (1, @@server_id)'
# Retrieving data from one of group_5_7_25_1 nodes
# At this point, the data was replicated twice
+ $HOME/sandboxes/group_5_7_25_1/n2 -e 'select *, @@port from test.t1'
+----+-----------+--------+
| id | server_id | @@port |
+----+-----------+--------+
| 1 | 20229 | 20227 |
+----+-----------+--------+</code></pre><br />
<p>The interesting thing about this recipe is that the sandboxes are created using the option <code>--port-as-server-id</code>. While it was used also in the replication between single sandboxes as an excess of caution, in this recipe, and in all the recipes involving compound sandboxes, it is a necessity, as the replication would fail if primary and replica servers have the same <code>server_id</code>.</p><p>All the work is done by the <code>replicate_from</code> script, which knows how to check whether the target is a single sandbox or a composite one, and where to find the primary server.</p><p>Using a similar method, we can run more recipes on the same tune.</p><br />
<h3 id="Replication-between-different-things">Replication between different things</h3><p>I won’t reproduce the output of all recipes here. I will just mention what every recipe needs to prepare to ensure a positive outcome.</p><ul><li><strong>Replication between NDB clusters</strong>. Nothing special here, except making sure to use a MySQL Cluster tarball. If you don’t dbdeployer will detect it and refuse the installation. For the rest, it’s like replication between groups.</li>
<li><strong>Replication between master/slave</strong>. This is a bit trickier, because the replication data comes to a master, and if we want to propagate to its slaves we need to activate <code>log-slave-update</code>. The recipe shows how to do it.</li>
<li><strong>Replication between group and master/slave</strong>. In addition to the trick mentioned in the previous recipe, we need to make sure that the master/slave deployment is using GTID.</li>
<li><strong>Replication between master/slave and group</strong>. See the previous one.</li>
<li><strong>Replication between group and single (and vice versa)</strong>. We just need to make sure the single sandbox has GTID enabled.</li>
</ul><br />
<h3 id="Replication-between-different-versions">Replication between different versions</h3><p>This is a simple recipe that comes from a <a href="https://github.com/datacharmer/dbdeployer/issues/28">feature request</a>. All you need to do is make sure that the version on the master is lower than the one on the slaves. The recipe script <code>replication-multi-versions.sh</code>, looks for tarballs of 5.6, 5.7, and 8.0, but you can start it using three versions that you’d like. For example:</p><p><code>./recipes/replication-multi-versions.sh 5.7.23 5.7.24 5.7.25</code></p><p>The first version will be used as the master.</p><br />
<h3 id="Circular-replication">Circular replication</h3><p>I didn’t want to do this, as I consider ring replication to be weak and difficult to handle. I stated that much in the <a href="https://github.com/datacharmer/dbdeployer/issues/47">feature request</a> and in the <a href="https://github.com/datacharmer/dbdeployer/blob/master/docs/features.md">list of dbdeployer features</a>. But then I saw that with the latest enhancements it was so easy, that I had to at least make a recipe for it. And then you have it. <code>recipes/circular-replication.sh</code> does what it promises, but the burden of maintenance is still on the user’s shoulders. I suggest looking at it, and then forgetting it.</p><br />
<h2 id="Upgrade-from-MySQL-5.5-to-8.0-(through-5.6-and-5.7)">Upgrade from MySQL 5.5 to 8.0 (through 5.6 and 5.7)</h2><p>This is one of the most advanced recipes. To enjoy it, you need to have expanded tarballs from 5.5, 5.6, 5.7, and 8.0.<br />
<br />
Provided that you do, running this script will do the following:</p><ol><li>deploy MySQL 5.5</li>
<li>Create a table <code>upgrade_log</code> and insert some data.</li>
<li>deploy MySQL 5.6</li>
<li>run <code>mysql_upgrade</code> (through dbdeployer)</li>
<li>Add data to the log table</li>
<li>deploy MySQL 5.7</li>
<li>run <code>mysql_upgrade</code> again</li>
<li>add data to the log table</li>
<li>deploy MySQL 8.0</li>
<li>run <code>mysql_upgrade</code> for the last time</li>
<li>Show the data from the table</li>
</ol><p>Here’s a full transcript of the operation. It’s interesting to see how the upgrade procedure has changed from older versions to current ones.</p><br />
<pre><code>$ ./recipes/upgrade.sh
# ****************************************************************************
# Upgrading from 5.5.53 to 5.6.41
# ****************************************************************************
+ dbdeployer deploy single 5.5.53 --master
Database installed in $HOME/sandboxes/msb_5_5_53
run 'dbdeployer usage single' for basic instructions'
.. sandbox server started
0
+ dbdeployer deploy single 5.6.41 --master
Database installed in $HOME/sandboxes/msb_5_6_41
run 'dbdeployer usage single' for basic instructions'
. sandbox server started
0
+ $HOME/sandboxes/msb_5_5_53/use -e 'CREATE TABLE IF NOT EXISTS test.upgrade_log(id int not null auto_increment primary key, server_id int, vers varchar(50), urole varchar(20), ts timestamp)'
+ $HOME/sandboxes/msb_5_5_53/use -e 'INSERT INTO test.upgrade_log (server_id, vers, urole) VALUES (@@server_id, @@version, '\''original'\'')'
+ dbdeployer admin upgrade msb_5_5_53 msb_5_6_41
stop $HOME/sandboxes/msb_5_5_53
stop $HOME/sandboxes/msb_5_6_41
Data directory msb_5_5_53/data moved to msb_5_6_41/data
. sandbox server started
Looking for 'mysql' as: $HOME/opt/mysql/5.6.41/bin/mysql
Looking for 'mysqlcheck' as: $HOME/opt/mysql/5.6.41/bin/mysqlcheck
Running 'mysqlcheck' with connection arguments: '--port=5641' '--socket=/var/folders/rz/cn7hvgzd1dl5y23l378dsf_c0000gn/T/mysql_sandbox5641.sock'
Running 'mysqlcheck' with connection arguments: '--port=5641' '--socket=/var/folders/rz/cn7hvgzd1dl5y23l378dsf_c0000gn/T/mysql_sandbox5641.sock'
mysql.columns_priv OK
mysql.db OK
mysql.event OK
mysql.func OK
mysql.general_log OK
mysql.help_category OK
mysql.help_keyword OK
mysql.help_relation OK
mysql.help_topic OK
mysql.host OK
mysql.ndb_binlog_index OK
mysql.plugin OK
mysql.proc OK
mysql.procs_priv OK
mysql.proxies_priv OK
mysql.servers OK
mysql.slow_log OK
mysql.tables_priv OK
mysql.time_zone OK
mysql.time_zone_leap_second OK
mysql.time_zone_name OK
mysql.time_zone_transition OK
mysql.time_zone_transition_type OK
mysql.user OK
Running 'mysql_fix_privilege_tables'...
Running 'mysqlcheck' with connection arguments: '--port=5641' '--socket=/var/folders/rz/cn7hvgzd1dl5y23l378dsf_c0000gn/T/mysql_sandbox5641.sock'
Running 'mysqlcheck' with connection arguments: '--port=5641' '--socket=/var/folders/rz/cn7hvgzd1dl5y23l378dsf_c0000gn/T/mysql_sandbox5641.sock'
test.upgrade_log OK
OK
The data directory from msb_5_6_41/data is preserved in msb_5_6_41/data-msb_5_6_41
The data directory from msb_5_5_53/data is now used in msb_5_6_41/data
msb_5_5_53 is not operational and can be deleted
+ dbdeployer delete msb_5_5_53
List of deployed sandboxes:
$HOME/sandboxes/msb_5_5_53
Running $HOME/sandboxes/msb_5_5_53/stop
Running rm -rf $HOME/sandboxes/msb_5_5_53
Directory $HOME/sandboxes/msb_5_5_53 deleted
+ $HOME/sandboxes/msb_5_6_41/use -e 'INSERT INTO test.upgrade_log (server_id, vers, urole) VALUES (@@server_id, @@version, '\''upgraded'\'')'
+ $HOME/sandboxes/msb_5_6_41/use -e 'SELECT * FROM test.upgrade_log'
+----+-----------+------------+----------+---------------------+
| id | server_id | vers | urole | ts |
+----+-----------+------------+----------+---------------------+
| 1 | 5553 | 5.5.53-log | original | 2019-04-01 20:27:38 |
| 2 | 5641 | 5.6.41-log | upgraded | 2019-04-01 20:27:46 |
+----+-----------+------------+----------+---------------------+
# ****************************************************************************
# The upgraded database is now upgrading from 5.6.41 to 5.7.25
# ****************************************************************************
+ dbdeployer deploy single 5.7.25 --master
Database installed in $HOME/sandboxes/msb_5_7_25
run 'dbdeployer usage single' for basic instructions'
. sandbox server started
0
+ $HOME/sandboxes/msb_5_6_41/use -e 'CREATE TABLE IF NOT EXISTS test.upgrade_log(id int not null auto_increment primary key, server_id int, vers varchar(50), urole varchar(20), ts timestamp)'
+ $HOME/sandboxes/msb_5_6_41/use -e 'INSERT INTO test.upgrade_log (server_id, vers, urole) VALUES (@@server_id, @@version, '\''original'\'')'
+ dbdeployer admin upgrade msb_5_6_41 msb_5_7_25
stop $HOME/sandboxes/msb_5_6_41
stop $HOME/sandboxes/msb_5_7_25
Data directory msb_5_6_41/data moved to msb_5_7_25/data
.. sandbox server started
Checking if update is needed.
Checking server version.
Running queries to upgrade MySQL server.
Checking system database.
mysql.columns_priv OK
mysql.db OK
mysql.engine_cost OK
mysql.event OK
mysql.func OK
mysql.general_log OK
mysql.gtid_executed OK
mysql.help_category OK
mysql.help_keyword OK
mysql.help_relation OK
mysql.help_topic OK
mysql.host OK
mysql.innodb_index_stats OK
mysql.innodb_table_stats OK
mysql.ndb_binlog_index OK
mysql.plugin OK
mysql.proc OK
mysql.procs_priv OK
mysql.proxies_priv OK
mysql.server_cost OK
mysql.servers OK
mysql.slave_master_info OK
mysql.slave_relay_log_info OK
mysql.slave_worker_info OK
mysql.slow_log OK
mysql.tables_priv OK
mysql.time_zone OK
mysql.time_zone_leap_second OK
mysql.time_zone_name OK
mysql.time_zone_transition OK
mysql.time_zone_transition_type OK
mysql.user OK
Upgrading the sys schema.
Checking databases.
sys.sys_config OK
test.upgrade_log
error : Table rebuild required. Please do "ALTER TABLE `upgrade_log` FORCE" or dump/reload to fix it!
Repairing tables
`test`.`upgrade_log`
Running : ALTER TABLE `test`.`upgrade_log` FORCE
status : OK
Upgrade process completed successfully.
Checking if update is needed.
The data directory from msb_5_7_25/data is preserved in msb_5_7_25/data-msb_5_7_25
The data directory from msb_5_6_41/data is now used in msb_5_7_25/data
msb_5_6_41 is not operational and can be deleted
+ dbdeployer delete msb_5_6_41
List of deployed sandboxes:
$HOME/sandboxes/msb_5_6_41
Running $HOME/sandboxes/msb_5_6_41/stop
Running rm -rf $HOME/sandboxes/msb_5_6_41
Directory $HOME/sandboxes/msb_5_6_41 deleted
+ $HOME/sandboxes/msb_5_7_25/use -e 'INSERT INTO test.upgrade_log (server_id, vers, urole) VALUES (@@server_id, @@version, '\''upgraded'\'')'
+ $HOME/sandboxes/msb_5_7_25/use -e 'SELECT * FROM test.upgrade_log'
+----+-----------+------------+----------+---------------------+
| id | server_id | vers | urole | ts |
+----+-----------+------------+----------+---------------------+
| 1 | 5553 | 5.5.53-log | original | 2019-04-01 20:27:38 |
| 2 | 5641 | 5.6.41-log | upgraded | 2019-04-01 20:27:46 |
| 3 | 5641 | 5.6.41-log | original | 2019-04-01 20:27:51 |
| 4 | 5725 | 5.7.25-log | upgraded | 2019-04-01 20:28:01 |
+----+-----------+------------+----------+---------------------+
# ****************************************************************************
# The further upgraded database is now upgrading from 5.7.25 to 8.0.15
# ****************************************************************************
+ dbdeployer deploy single 8.0.15 --master
Database installed in $HOME/sandboxes/msb_8_0_15
run 'dbdeployer usage single' for basic instructions'
.. sandbox server started
0
+ $HOME/sandboxes/msb_5_7_25/use -e 'CREATE TABLE IF NOT EXISTS test.upgrade_log(id int not null auto_increment primary key, server_id int, vers varchar(50), urole varchar(20), ts timestamp)'
+ $HOME/sandboxes/msb_5_7_25/use -e 'INSERT INTO test.upgrade_log (server_id, vers, urole) VALUES (@@server_id, @@version, '\''original'\'')'
+ dbdeployer admin upgrade msb_5_7_25 msb_8_0_15
stop $HOME/sandboxes/msb_5_7_25
Attempting normal termination --- kill -15 10357
stop $HOME/sandboxes/msb_8_0_15
Data directory msb_5_7_25/data moved to msb_8_0_15/data
... sandbox server started
Checking if update is needed.
Checking server version.
Running queries to upgrade MySQL server.
Upgrading system table data.
Checking system database.
mysql.columns_priv OK
mysql.component OK
mysql.db OK
mysql.default_roles OK
mysql.engine_cost OK
mysql.func OK
mysql.general_log OK
mysql.global_grants OK
mysql.gtid_executed OK
mysql.help_category OK
mysql.help_keyword OK
mysql.help_relation OK
mysql.help_topic OK
mysql.host OK
mysql.innodb_index_stats OK
mysql.innodb_table_stats OK
mysql.ndb_binlog_index OK
mysql.password_history OK
mysql.plugin OK
mysql.procs_priv OK
mysql.proxies_priv OK
mysql.role_edges OK
mysql.server_cost OK
mysql.servers OK
mysql.slave_master_info OK
mysql.slave_relay_log_info OK
mysql.slave_worker_info OK
mysql.slow_log OK
mysql.tables_priv OK
mysql.time_zone OK
mysql.time_zone_leap_second OK
mysql.time_zone_name OK
mysql.time_zone_transition OK
mysql.time_zone_transition_type OK
mysql.user OK
Found outdated sys schema version 1.5.1.
Upgrading the sys schema.
Checking databases.
sys.sys_config OK
test.upgrade_log OK
Upgrade process completed successfully.
Checking if update is needed.
The data directory from msb_8_0_15/data is preserved in msb_8_0_15/data-msb_8_0_15
The data directory from msb_5_7_25/data is now used in msb_8_0_15/data
msb_5_7_25 is not operational and can be deleted
+ dbdeployer delete msb_5_7_25
List of deployed sandboxes:
$HOME/sandboxes/msb_5_7_25
Running $HOME/sandboxes/msb_5_7_25/stop
Running rm -rf $HOME/sandboxes/msb_5_7_25
Directory $HOME/sandboxes/msb_5_7_25 deleted
+ $HOME/sandboxes/msb_8_0_15/use -e 'INSERT INTO test.upgrade_log (server_id, vers, urole) VALUES (@@server_id, @@version, '\''upgraded'\'')'
+ $HOME/sandboxes/msb_8_0_15/use -e 'SELECT * FROM test.upgrade_log'
+----+-----------+------------+----------+---------------------+
| id | server_id | vers | urole | ts |
+----+-----------+------------+----------+---------------------+
| 1 | 5553 | 5.5.53-log | original | 2019-04-01 20:27:38 |
| 2 | 5641 | 5.6.41-log | upgraded | 2019-04-01 20:27:46 |
| 3 | 5641 | 5.6.41-log | original | 2019-04-01 20:27:51 |
| 4 | 5725 | 5.7.25-log | upgraded | 2019-04-01 20:28:01 |
| 5 | 5725 | 5.7.25-log | original | 2019-04-01 20:28:07 |
| 6 | 8015 | 8.0.15 | upgraded | 2019-04-01 20:28:20 |
+----+-----------+------------+----------+---------------------+</code></pre><br />
<h2 id="What-else-can-we-do?">What else can we do?</h2><p>The replication recipes seen so far use the same principles. The method used in these recipes doesn’t work for all-masters and fan-in replication, because mixing named channels and nameless ones is not allowed. Also, there are things that don’t respond to replication commands at all, like <a href="https://datacharmer.blogspot.com/2019/03/dbdeployer-community-part-1-tidb.html">TiDB</a>. But it should be easy to enhance the current scripts (or to add some more specialized ones) that will include also these exceptions. Given the recent wave of collaboration, I expect it will happen relatively soon.</p>Giuseppe Maxiahttp://www.blogger.com/profile/15801583338057324813noreply@blogger.com0tag:blogger.com,1999:blog-16959946.post-50473396897136783462019-03-29T14:12:00.000+01:002019-03-29T14:12:28.887+01:00dbdeployer cookbook - usability by example<p>When I designed dbdeployer, I wanted to eliminate most of the issues that the old MySQL-Sandbox had:</p><ul><li>dependencies during installation</li>
<li>mistaken tarballs</li>
<li>clarity of syntax</li>
<li>features (un)awareness.</li>
</ul><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjes97KCQaJA6qXu8OA-gc3KA5_arqNNak16dxsxEWQ04-bDHkKgzKYSmXh0OitiR2Vko8eJoEN5TfXUJ4p1HpwPd3y_Uz4phur8Qbj1hyphenhyphensW1MFSD6xnEFpFuCiuoe2_aS6XqzM/s1600/dbdeployer_cookbook.png" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjes97KCQaJA6qXu8OA-gc3KA5_arqNNak16dxsxEWQ04-bDHkKgzKYSmXh0OitiR2Vko8eJoEN5TfXUJ4p1HpwPd3y_Uz4phur8Qbj1hyphenhyphensW1MFSD6xnEFpFuCiuoe2_aS6XqzM/s320/dbdeployer_cookbook.png" width="320" height="279" data-original-width="1262" data-original-height="1100" /></a><br />
<br />
<p><strong>Dependencies during installation</strong> did go away right from the start, as the dbdeployer executable is ready to be used without additional components. The only dependency is to have a host that can run MySQL. There is little dbdeployer can do about detecting whether or not your system can run MySQL. It depends on which version and flavor of MySQL you are running. It should not be a big deal as I assume that anyone in need of dbdeployer has already the necessary knowledge about MySQL prerequisites. This is not always the case, but the issue goes beyond dbdeployer’s scope.</p><br />
<p><strong>Mistaken tarballs</strong> are a common problem for users who have never seen a binaries tarball. Here dbdeployer can help, up to a point, to guide the user. It recognizes most cases where you are trying to use a source tarball or the wrong operating system tarball. It still does not recognize when you try to run a sandbox for a Linux 64bits out of a 32bit tarball, but to be honest I haven’t tried to solve this problem yet. There are still cases where users are a great risk of picking the wrong tarball (Percona Server download page is a minefield and the one for MariaDB is not picnic either) but I feel that I have given dbdeployer users a big help on this count.</p><br />
<p><strong>Clarity of syntax</strong> is probably the biggest issue with the previous tool. It’s a consequence of the tool being developed over several years, slowly adapting to changing circumstances. I made dbdeployer clearer from the beginning, when I decided to avoid piling up many actions as the consequence of the same command. In dbdeployer, you need to unpack the tarball explicitly before running a deployment, and this gives dbdeployer users the visibility on the architecture that eluded many MySQL-Sandbox users. The architecture of dbdeployer is such that adding new features, commands, and options is easy and fits within an easily detectable paradigm. Thus, the operations are easier to spot and use.</p><br />
<p><strong>Features awareness</strong> is still a problem. There is a lengthy description of everything dbdeployer can do, but, as often happens with even the best tools, <strong>users don’t read manuals</strong>.<br/><br />
The biggest obstacle about reading manuals is that dbdeployer executable is installed without any need to take the <a href="https://github.com/datacharmer/dbdeployer/blob/master/README.md">README</a> file along. If you need it, you should go online and read it, and given that dbdeployer is built to be used mostly without internet connection, there are cases when you want to know how to do something, but you can’t get the manual right away.<br/><br />
There is the tool integrated help (<code>dbdeployer [command] -h</code>), which gives you a lot of information, but this tells you how to do something that you know already exists, not what you can do in general.<br/><br />
To help on this count, I added a collection of samples (the <strong>cookbook</strong>) that were initially released in a directory of the <a href="https://github.com/datacharmer/dbdeployer">GitHub project</a>, but then the sample scripts suffered of the same invisibility that plagues the README file. There is one more problem: when you tried using the generic cookbook scripts (now removed) you had to use the same environment as I did when defining them, or they would fail.<br/><br />
The current solution is to include cookbook files right within the tool, using templates (same as for regular sandbox scripts) with the double advantage that the scripts are available anywhere the dbdeployer executable is, and the scripts are adapted to the environment, since dbdeployer knows how to search for available binaries and can suggest the best parameters for the scripts.</p><br />
<h2 id="Introducing-dbdeployer-cookbook">Introducing dbdeployer cookbook</h2><p>The command <code>dbdeployer cookbook</code> (with aliases <code>recipes</code> or <code>samples</code>) has three subcommands:</p><ul><li><code>list</code> displays a list of available samples.</li>
<li><code>show</code> displays the contents of a recipe.</li>
<li><code>create</code> (with alias <code>make</code>) will build the recipe script.</li>
</ul><br />
<p>We should try the list first:</p><pre><code>$ dbdeployer cookbook list
.----------------------------------.-------------------------------------.--------------------------------------------------------------------.--------.
| recipe | script name | description | needed |
| | | | flavor |
+----------------------------------+-------------------------------------+--------------------------------------------------------------------+--------+
| all-masters | all-masters-deployment.sh | Creation of an all-masters replication sandbox | mysql |
| delete | delete-sandboxes.sh | Delete all deployed sandboxes | |
| fan-in | fan-in-deployment.sh | Creation of a fan-in (many masters, one slave) replication sandbox | mysql |
| group-multi | group-multi-primary-deployment.sh | Creation of a multi-primary group replication sandbox | mysql |
| group-single | group-single-primary-deployment.sh | Creation of a single-primary group replication sandbox | mysql |
| master-slave | master-slave-deployment.sh | Creation of a master/slave replication sandbox | |
| ndb | ndb-deployment.sh | Shows deployment with ndb | ndb |
| prerequisites | prerequisites.sh | Shows dbdeployer prerequisites and how to make them | |
| pxc | pxc-deployment.sh | Shows deployment with pxc | pxc |
| remote | remote.sh | Shows how to get a remote MySQL tarball | |
| replication-restart | repl-operations-restart.sh | Show how to restart sandboxes with custom options | |
| replication-operations | repl-operations.sh | Show how to run operations in a replication sandbox | |
| replication_between_groups | replication-between-groups.sh | Shows how to run replication between two group replications | mysql |
| replication_between_master_slave | replication-between-master-slave.sh | Shows how to run replication between two master/slave replications | |
| replication_between_ndb | replication-between-ndb.sh | Shows how to run replication between two NDB clusters | ndb |
| show | show-sandboxes.sh | Show deployed sandboxes | |
| single | single-deployment.sh | Creation of a single sandbox | |
| single-reinstall | single-reinstall.sh | Re-installs a single sandbox | |
| tidb | tidb-deployment.sh | Shows deployment and some operations with TiDB | tidb |
| upgrade | upgrade.sh | Shows a complete upgrade example from 5.5 to 8.0 | mysql |
'----------------------------------'-------------------------------------'--------------------------------------------------------------------'--------'</code></pre><br />
<p>The recipe that seems to be the simplest one is <code>single</code>. We can try to see what is in there:</p><br />
<pre><code>$ dbdeployer cookbook show single
#!/bin/bash
[...]
# Generated by dbdeployer 1.26.0 using template single on Fri Mar 29 12:27:53 UTC 2019
cd $(dirname $0)
source cookbook_include.sh
version=$1
[ -z "$version" ] && version=<b>NOTFOUND_mysql</b>
check_version $version
if [ -n "$(dbdeployer sandboxes | grep 'single\s*'$version)" ]
then
echo "single version $version is already installed"
else
header "Deploying a single sandbox for version $version"
run dbdeployer deploy single $version
fi</code></pre><br />
<p>What looks odd is the line that says <strong>NOTFOUND</strong>. If we try creating that script and then running it, it won’t work, and rightfully so.</p><br />
<p>There is a recipe named <code>prerequisites</code> that could probably help us.</p><pre><code>$ dbdeployer cookbook create prerequisites
recipes/prerequisites.sh created</code></pre><br />
<p>So, now, we have a starting point. Let’s run it:</p><br />
<pre><code>$ ./recipes/prerequisites.sh
# ****************************************************************************
# Creating Sandbox binary directory ($HOME/opt/mysql)
# ****************************************************************************
## HOW TO GET binaries for dbdeployer
# FOR REGULAR MYSQL
# run the commands:
1. dbdeployer remote list
2. dbdeployer remote get mysql-5.7.25
3. dbdeployer unpack mysql-5.7.25.tar.xz
4. dbdeployer versions
# FOR MySQL forks, MySQL Cluster, PXC:
# 1. Get the binaries from the maker download pages
# 2. run the command
dbdeployer unpack FlavorName-X.X.XX-OS.tar.gz --prefix=FlavorName
3. dbdeployer versions</code></pre><br />
<p>The first thing we see is that the sandbox binary directory was created, and then we see a series of steps to fill it in.</p><p>Let’s try:</p><pre><code>$ dbdeployer remote list
Files available in https://raw.githubusercontent.com/datacharmer/mysql-docker-minimal/master/dbdata/available.json
5.5 -> [mysql-5.5.61 mysql-5.5.62]
5.6 -> [mysql-5.6.41 mysql-5.6.43]
5.7 -> [mysql-5.7.24 mysql-5.7.25]
8.0 -> [mysql-8.0.13 mysql-8.0.15]
4.1 -> [mysql-4.1.22]
5.0 -> [mysql-5.0.15 mysql-5.0.96]
5.1 -> [mysql-5.1.72]
$ dbdeployer remote get mysql-5.7.25
File /home/msandbox/mysql-5.7.25.tar.xz downloaded
$ dbdeployer unpack mysql-5.7.25.tar.xz
Unpacking tarball mysql-5.7.25.tar.xz to $HOME/opt/mysql/5.7.25
[...]
Renaming directory $HOME/opt/mysql/mysql-5.7.25 to $HOME/opt/mysql/5.7.25
$ dbdeployer versions
Basedir: /home/msandbox/opt/mysql
5.7.25</code></pre><br />
<p>If we repeat the <code>show</code> command now, we get a different result:</p><br />
<pre><code>$ dbdeployer cookbook show single
#!/bin/bash
[...]
# Generated by dbdeployer 1.26.0 using template single on Fri Mar 29 12:37:26 UTC 2019
cd $(dirname $0)
source cookbook_include.sh
version=$1
[ -z "$version" ] && version=<b>5.7.25</b>
check_version $version
if [ -n "$(dbdeployer sandboxes | grep 'single\s*'$version)" ]
then
echo "single version $version is already installed"
else
header "Deploying a single sandbox for version $version"
run dbdeployer deploy single $version
fi</code></pre><br />
<p>There! instead of the <em>NOTFOUND</em> we saw before, it now shows the version that we just downloaded. If we repeat the same procedure (<code>remote list</code>, <code>remote get</code>, <code>unpack</code>) for MySQL 8.0.15, we would see <code>8.0.15</code> as the recommended version.</p><br />
<p>Now we can create the <code>single</code> recipe. Or even better, since we want to try several ones, we can create all of them.</p><br />
<pre><code>$ dbdeployer cookbook create all
recipes/replication-between-master-slave.sh created
recipes/single-reinstall.sh created
recipes/fan-in-deployment.sh created
recipes/group-multi-primary-deployment.sh created
recipes/repl-operations.sh created
recipes/tidb-deployment.sh created
recipes/remote.sh created
recipes/upgrade.sh created
recipes/ndb-deployment.sh created
recipes/cookbook_include.sh created
recipes/master-slave-deployment.sh created
recipes/prerequisites.sh created
recipes/replication-between-groups.sh created
recipes/replication-between-ndb.sh created
recipes/pxc-deployment.sh created
recipes/single-deployment.sh created
recipes/show-sandboxes.sh created
recipes/delete-sandboxes.sh created
recipes/all-masters-deployment.sh created
recipes/group-single-primary-deployment.sh created
recipes/repl-operations-restart.sh created</code></pre><br />
<p>Now it’s time to try one:</p><br />
<pre><code>msandbox@505969e46289:~$ ./recipes/single-deployment.sh
# ****************************************************************************
# Deploying a single sandbox for version 5.7.25
# ****************************************************************************
+ dbdeployer deploy single 5.7.25
Creating directory /home/msandbox/sandboxes
Database installed in $HOME/sandboxes/msb_5_7_25
run 'dbdeployer usage single' for basic instructions'
. sandbox server started
0</code></pre><br />
<p>Looking at the list, we see a <code>single-reinstall</code> recipe. If we run it, we will get a mini tutorial on how to use a single sandbox:</p><br />
<pre><code>$ ./recipes/single-deployment.sh
# ****************************************************************************
# Deploying a single sandbox for version 5.7.25
# ****************************************************************************
+ dbdeployer deploy single 5.7.25
Creating directory $HOME/sandboxes
Database installed in $HOME/sandboxes/msb_5_7_25
run 'dbdeployer usage single' for basic instructions'
. sandbox server started
0
msandbox@505969e46289:~$ ./recipes/show-sandboxes.sh
+ dbdeployer sandboxes --full-info
.------------.--------.---------.---------.--------.-------.--------.
| name | type | version | ports | flavor | nodes | locked |
+------------+--------+---------+---------+--------+-------+--------+
| msb_5_7_25 | single | 5.7.25 | [5725 ] | mysql | 0 | |
'------------'--------'---------'---------'--------'-------'--------'
0
msandbox@505969e46289:~$ ./recipes/single-reinstall.sh
# ****************************************************************************
# Deploying the same sandbox again, with different parameters
# We need to use --force, as we are overwriting an existing sandbox
# Incidentally, the new deployment will run a query before and after the grants
# ****************************************************************************
+ dbdeployer deploy single 5.7.25 '--pre-grants-sql=select host, user from mysql.user' '--post-grants-sql=select host, user from mysql.user' --force
Overwriting directory $HOME/sandboxes/msb_5_7_25
stop $HOME/sandboxes/msb_5_7_25
Database installed in $HOME/sandboxes/msb_5_7_25
run 'dbdeployer usage single' for basic instructions'
. sandbox server started
+-----------+---------------+
| host | user |
+-----------+---------------+
| localhost | mysql.session |
| localhost | mysql.sys |
| localhost | root |
+-----------+---------------+
+-----------+---------------+
| host | user |
+-----------+---------------+
| 127.% | msandbox |
| 127.% | msandbox_ro |
| 127.% | msandbox_rw |
| 127.% | rsandbox |
| localhost | msandbox |
| localhost | msandbox_ro |
| localhost | msandbox_rw |
| localhost | mysql.session |
| localhost | mysql.sys |
| localhost | root |
+-----------+---------------+
# ****************************************************************************
# Deploying the same sandbox with a different directory.
# No --force is necessary, as dbdeployer will choose a different port
# ****************************************************************************
+ dbdeployer deploy single 5.7.25 --sandbox-directory=msb_5_7_25_new
Database installed in $HOME/sandboxes/msb_5_7_25_new
run 'dbdeployer usage single' for basic instructions'
. sandbox server started
0
+ dbdeployer sandboxes --header
name type version ports
---------------------------- -------- --------- ---------
msb_5_7_25 : single 5.7.25 [5725 ]
msb_5_7_25_new : single 5.7.25 [5726 ]
0
# ****************************************************************************
# Removing the second sandbox
# ****************************************************************************
+ dbdeployer delete msb_5_7_25_new
List of deployed sandboxes:
$HOME/sandboxes/msb_5_7_25_new
Running $HOME/sandboxes/msb_5_7_25_new/stop
stop $HOME/sandboxes/msb_5_7_25_new
Running rm -rf $HOME/sandboxes/msb_5_7_25_new
Directory $HOME/sandboxes/msb_5_7_25_new deleted
0</code></pre><br />
<p>This script teaches us the basics of starting and restarting a sandbox, with useful twists as running an SQL command before granting privileges.</p><br />
<p>There is a similar tutorial for replication operations, but we’ll have a look at a slightly different one.</p><br />
<pre><code>$ ./recipes/master-slave-deployment.sh
+ dbdeployer deploy replication 5.7.25 --concurrent
$HOME/sandboxes/rsandbox_5_7_25/initialize_slaves
initializing slave 1
initializing slave 2
Replication directory installed in $HOME/sandboxes/rsandbox_5_7_25
run 'dbdeployer usage multiple' for basic instructions'
$ ./recipes/repl-operations.sh
# ****************************************************************************
# Running a simple command with the master in the sandbox.
# Notice the usage of the '-e', as if we were using the 'mysql' client
# ****************************************************************************
+ $HOME/sandboxes/rsandbox_5_7_25/m -e 'SHOW MASTER STATUS'
+------------------+----------+--------------+------------------+-------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+------------------+----------+--------------+------------------+-------------------+
| mysql-bin.000001 | 4089 | | | |
+------------------+----------+--------------+------------------+-------------------+
# ****************************************************************************
# Creating a table in the master
# ****************************************************************************
+ $HOME/sandboxes/rsandbox_5_7_25/m -e 'DROP TABLE IF EXISTS test.t1'
+ $HOME/sandboxes/rsandbox_5_7_25/m -e 'CREATE TABLE test.t1(id int not null primary key)'
# ****************************************************************************
# Inserting 3 lines into the new table
# ****************************************************************************
+ $HOME/sandboxes/rsandbox_5_7_25/m -e 'INSERT INTO test.t1 VALUES(1)'
+ $HOME/sandboxes/rsandbox_5_7_25/m -e 'INSERT INTO test.t1 VALUES(2)'
+ $HOME/sandboxes/rsandbox_5_7_25/m -e 'INSERT INTO test.t1 VALUES(3)'
# ****************************************************************************
# Getting the table contents from one slave
# ****************************************************************************
+ $HOME/sandboxes/rsandbox_5_7_25/s1 -e 'SELECT * FROM test.t1'
+----+
| id |
+----+
| 1 |
| 2 |
| 3 |
+----+
# ****************************************************************************
# Getting the table count from all nodes (NOTE: no '-e' is needed)
# $HOME/sandboxes/rsandbox_5_7_25/use_all 'SELECT COUNT(*) FROM test.t1'
# master
COUNT(*)
3
# server: 1
COUNT(*)
3
# server: 2
COUNT(*)
3
# ****************************************************************************
# Checking the status of all slaves
# ****************************************************************************
+ $HOME/sandboxes/rsandbox_5_7_25/check_slaves
master
port 19226 - server_id 100
File: mysql-bin.000001
Position: 5213
Executed_Gtid_Set:
slave1
port 19227 - server_id 200
Master_Log_File: mysql-bin.000001
Read_Master_Log_Pos: 5213
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Exec_Master_Log_Pos: 5213
Retrieved_Gtid_Set:
Executed_Gtid_Set:
Auto_Position: 0
slave2
port 19228 - server_id 300
Master_Log_File: mysql-bin.000001
Read_Master_Log_Pos: 5213
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Exec_Master_Log_Pos: 5213
Retrieved_Gtid_Set:
Executed_Gtid_Set:
Auto_Position: 0
0
# ****************************************************************************
# Running a multiple query in all slaves
# ****************************************************************************
+ $HOME/sandboxes/rsandbox_5_7_25/use_all_slaves 'STOP SLAVE; SET GLOBAL slave_parallel_workers=3; START SLAVE;show processlist '
# server: 1
Id User Host db Command Time State Info
11 msandbox localhost NULL Query 0 starting show processlist
12 system user NULL Connect 0 Checking master version NULL
13 system user NULL Connect 0 System lock NULL
14 system user NULL Connect 0 Waiting for an event from Coordinator NULL
15 system user NULL Connect 0 Waiting for an event from Coordinator NULL
16 system user NULL Connect 0 Waiting for an event from Coordinator NULL
# server: 2
Id User Host db Command Time State Info
10 msandbox localhost NULL Query 0 starting show processlist
11 system user NULL Connect 0 Checking master version NULL
12 system user NULL Connect 0 System lock NULL
13 system user NULL Connect 0 Waiting for an event from Coordinator NULL
14 system user NULL Connect 0 Waiting for an event from Coordinator NULL
15 system user NULL Connect 0 Waiting for an event from Coordinator NULL</code></pre><br />
<br />
<p>By studying the commands mentioned in these samples, you will become proficient in dbdeployer components, allowing you to use it for advanced testing operations.</p>Giuseppe Maxiahttp://www.blogger.com/profile/15801583338057324813noreply@blogger.com0tag:blogger.com,1999:blog-16959946.post-84665943075140808922019-03-19T00:58:00.000+01:002019-03-19T00:58:28.879+01:00dbdeployer community: Part 3 - MySQL Cluster (NDB)<p>I remember wanting to create <a href="https://dev.mysql.com/doc/refman/5.7/en/mysql-cluster.html">MySQL Cluster</a> sandboxes several years ago. By then, however, MySQL-Sandbox technology was not flexible enough to allow an easy inclusion, and the cluster software itself was not as easy to install as it is today. Thus, I kept postponing the implementation, until I started working with <a href="https://github.com/datacharmer/dbdeployer">dbdeployer</a>.</p><p>I included the skeleton of support for MySQL Cluster since the beginning (by keeping a range of ports dedicated for this technology, but I didn’t do anything until June 2018, when I made public my intentions to add support for NDB in dbdeployer with <a href="https://github.com/datacharmer/dbdeployer/issues/20">issue #20 (Add support for MySQL Cluster)</a>). The issue had just a bare idea, but I needed help from someone, as my expertise with NDB was limited, and outdated.</p><p>Help came in November, when <a href="https://github.com/dveeden">Daniël van Eeden</a> started giving me bits of information on how to put together a cluster sandbox. I still resisted forcing my hand at the implementation, because by then I had realised that my method of checking the database server version to know whether it supported a given feature was inadequate to support anything other than vanilla MySQL or fully complaint forks.</p><p>The game changer was the <a href="https://datacharmer.blogspot.com/2019/03/dbdeployer-community-part-1-tidb.html">cooperation with TiDB</a> that opened the way for <a href="https://datacharmer.blogspot.com/2019/03/dbdeployer-community-part-2-percona.html">supporting Percona XtraDB Cluster</a>. Even though these technologies are way different from MySQL Cluster, they forced me to improve dbdeployer’s code, making it more flexible, easier to enhance.</p><p>When I finally decided to start working on NDB, it took me only a few days to implement it, because I had all the pieces ready for this technology to become part of dbdeployer.</p><p>Following Däniel’s instructions, I had a prototype working, which I submitted to <code>#dbdeployer</code> channel on <a href="https://mysqlcommunity.slack.com">MySQL community slack</a>. In that channel, I got help again from Däniel van Eeden, and then <a href="https://twitter.com/lefred">Frédéric Descamps</a> summoned two more experts (<a href="https://twitter.com/wwwtedw">Ted Wennmark</a> and Bernd Ocklin), who gave me feedback, advice, and some quick lessons on how the cluster should work, which allowed me to publish a release (<a href="https://github.com/datacharmer/dbdeployer/releases/tag/v1.23.0">dbdeployer 1.23.0</a>) this past week-end.</p><p>The implementation has some peculiarities for both users of dbdeployer and MySQL Cluster. For the ones used to dbdeployer, the biggest change is that we are deploying two entities, of which the main one is an NDB cluster, with its own directories and processes, while the MySQL servers are just the visible part of the cluster, but are, in fact, only cluster clients. Still, the cluster works smoothly in dbdeployer paradigm: the cluster is deployed (like Group replication or PXC) as a replication topology, and as such we can run the standard replication test and expect to get the same result that we would see when checking another multi-source deployment.</p><p>For people used to NDB, though, seeing NDB as “replication” feels odd, because the cluster is seeing as a distinct entity, and replication is when we transfer data between two clusters. If we were developing a dedicated tool for NDB clusters, this is probably what we would have done, but since we want dbdeployer integration, we must play by the general rules of the tool, where “single” is a stand-alone server instance, and we can’t have <code>dbdeployer deploy single --topology=ndb</code>, because single instance don’t have a topology, which is a property of a group of entities. Therefore, the price to pay for dbdeployer support accepting to see a MySQL cluster deployment as replication.</p><p>Now that we have covered all the philosophical angle, it’s time to show an example. Unlike <a href="https://datacharmer.blogspot.com/2019/03/dbdeployer-community-part-2-percona.html">PXC</a>, which is requires Linux, MySQL Cluster can also run on MacOS, which makes my testing much easier.</p><p>The first step to run a cluster in dbdeployer is to download a tarball from <a href="https://dev.mysql.com/downloads/cluster/">dev.mysql.com/downloads/cluster</a>, and then expand it in our usual directory (<code>$HOME/opt/mysql</code>):</p><pre><code>$ dbdeployer unpack --prefix=ndb --flavor=ndb \
~/Downloads/mysql-cluster-gpl-7.6.9-macos10.14-x86_64.tar.gz
Unpacking tarball $HOME/Downloads/mysql-cluster-gpl-7.6.9-macos10.14-x86_64.tar.gz to $HOME/opt/mysql/ndb7.6.9
[...]
Renaming directory $HOME/opt/mysql/mysql-cluster-gpl-7.6.9-macos10.14-x86_64 to $HOME/opt/mysql/ndb7.6.9
</code></pre><p>We can repeat the same operation for MySQL Cluster 8.0.14, and in the end we will two expanded tarballs named <code>ndb7.6.9</code> and <code>ndb8.0.14</code>. With this we can install a few clusters in the same host:</p><pre><code>$ dbdeployer deploy replication ndb7.6 --topology=ndb --concurrent
# ndb7.6 => ndb7.6.9
$HOME/sandboxes/ndb_msb_ndb7_6_9/initialize_nodes
MySQL Cluster Management Server mysql-5.7.25 ndb-7.6.9
2019-03-18 23:47:15 [ndbd] INFO -- Angel connected to 'localhost:20900'
2019-03-18 23:47:16 [ndbd] INFO -- Angel allocated nodeid: 2
2019-03-18 23:47:16 [ndbd] INFO -- Angel connected to 'localhost:20900'
2019-03-18 23:47:16 [ndbd] INFO -- Angel allocated nodeid: 3
executing 'start' on node 1
................ sandbox server started
executing 'start' on node 2
.. sandbox server started
executing 'start' on node 3
.. sandbox server started
NDB cluster directory installed in $HOME/sandboxes/ndb_msb_ndb7_6_9
run 'dbdeployer usage multiple' for basic instructions'
$ dbdeployer deploy replication ndb8.0 --topology=ndb --concurrent
# ndb8.0 => ndb8.0.14
$HOME/sandboxes/ndb_msb_ndb8_0_14/initialize_nodes
MySQL Cluster Management Server mysql-8.0.14 ndb-8.0.14-dmr
2019-03-18 23:45:53 [ndbd] INFO -- Angel connected to 'localhost:21400'
2019-03-18 23:45:53 [ndbd] INFO -- Angel allocated nodeid: 2
2019-03-18 23:45:53 [ndbd] INFO -- Angel connected to 'localhost:21400'
2019-03-18 23:45:53 [ndbd] INFO -- Angel allocated nodeid: 3
executing 'start' on node 1
........ sandbox server started
executing 'start' on node 2
... sandbox server started
executing 'start' on node 3
.. sandbox server started
NDB cluster directory installed in $HOME/sandboxes/ndb_msb_ndb8_0_14
run 'dbdeployer usage multiple' for basic instructions'
</code></pre><p>If we look at the sandbox directories, we will see a few more subdirectories than we usually have with other topologies. For example:</p><pre><code> ndb_conf # cluster configuration
ndbnode1 # management node (1)
ndbnode2 # data node (2)
ndbnode3 # data node (3)
node1 # MySQL node 1
node2 # MySQL node 2
node3 # MySQL node 3
</code></pre><p>The clusters are well framed into dbdeployer’s architecture, and they respond to standard commands like any other sandbox:</p><pre><code>$ dbdeployer sandboxes --full-info
.-------------------.------.-----------.----------------------------------------------.--------.-------.--------.
| name | type | version | ports | flavor | nodes | locked |
+-------------------+------+-----------+----------------------------------------------+--------+-------+--------+
| ndb_msb_ndb7_6_9 | ndb | ndb7.6.9 | [20900 27510 27511 27512 ] | ndb | 3 | |
| ndb_msb_ndb8_0_14 | ndb | ndb8.0.14 | [21400 28415 38415 28416 38416 28417 38417 ] | ndb | 3 | |
'-------------------'------'-----------'----------------------------------------------'--------'-------'--------'
$ dbdeployer global status
# Running "status_all" on ndb_msb_ndb7_6_9
MULTIPLE /Users/gmax/sandboxes/ndb_msb_ndb7_6_9
node1 : node1 on - port 27510 (27510)
node2 : node2 on - port 27511 (27511)
node3 : node3 on - port 27512 (27512)
# Running "status_all" on ndb_msb_ndb8_0_14
MULTIPLE /Users/gmax/sandboxes/ndb_msb_ndb8_0_14
node1 : node1 on - port 28415 (28415)
node2 : node2 on - port 28416 (28416)
node3 : node3 on - port 28417 (28417)
$ dbdeployer global test-replication
# Running "test_replication" on ndb_msb_ndb7_6_9
# master 1
# master 2
# master 3
# slave 1
ok - '3' == '3' - Slaves received tables from all masters
# slave 2
ok - '3' == '3' - Slaves received tables from all masters
# slave 3
ok - '3' == '3' - Slaves received tables from all masters
# pass: 3
# fail: 0
# Running "test_replication" on ndb_msb_ndb8_0_14
# master 1
# master 2
# master 3
# slave 1
ok - '3' == '3' - Slaves received tables from all masters
# slave 2
ok - '3' == '3' - Slaves received tables from all masters
# slave 3
ok - '3' == '3' - Slaves received tables from all masters
# pass: 3
# fail: 0
</code></pre><p>Like other topologies, also the NDB cluster has a script that shows the status of the nodes:</p><pre><code>$ ~/sandboxes/ndb_msb_ndb7_6_9/check_nodes
+---------+-----------+---------------+--------+---------+-------------+-------------------+
| node_id | node_type | node_hostname | uptime | status | start_phase | config_generation |
+---------+-----------+---------------+--------+---------+-------------+-------------------+
| 2 | NDB | localhost | 58 | STARTED | 0 | 1 |
| 3 | NDB | localhost | 58 | STARTED | 0 | 1 |
| 1 | MGM | localhost | NULL | NULL | NULL | NULL |
| 4 | API | localhost | NULL | NULL | NULL | NULL |
| 5 | API | localhost | NULL | NULL | NULL | NULL |
| 6 | API | localhost | NULL | NULL | NULL | NULL |
| 7 | API | localhost | NULL | NULL | NULL | NULL |
+---------+-----------+---------------+--------+---------+-------------+-------------------+
Connected to Management Server at: localhost:20900
Cluster Configuration
---------------------
[ndbd(NDB)] 2 node(s)
id=2 @127.0.0.1 (mysql-5.7.25 ndb-7.6.9, Nodegroup: 0, *)
id=3 @127.0.0.1 (mysql-5.7.25 ndb-7.6.9, Nodegroup: 0)
[ndb_mgmd(MGM)] 1 node(s)
id=1 @127.0.0.1 (mysql-5.7.25 ndb-7.6.9)
[mysqld(API)] 4 node(s)
id=4 @127.0.0.1 (mysql-5.7.25 ndb-7.6.9)
id=5 @127.0.0.1 (mysql-5.7.25 ndb-7.6.9)
id=6 @127.0.0.1 (mysql-5.7.25 ndb-7.6.9)
id=7 (not connected, accepting connect from localhost)
</code></pre><p>It is possible that we will need more iterations to make the deployment more robust. When testing it, keep in mind that this deployment is only for testing, and it won’t probably have all the performance that you may find in a well deployed production cluster. Still, compared to other topologies, the replication tests performed faster than I expected.</p>Giuseppe Maxiahttp://www.blogger.com/profile/15801583338057324813noreply@blogger.com0tag:blogger.com,1999:blog-16959946.post-67197718947249560582019-03-08T11:00:00.000+01:002019-03-08T11:00:06.587+01:00dbdeployer community - Part 2: Percona XtraDB Cluster<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjEtlphIZ5jS8aWzdBSD3RAlTGuqUbF-HLBW2B035B_UBhNqDVV5L1e00m3tBksP3dIC6IECuWDtCQGnoCSPUfMViraYMwnXQUWBrJvoOQDKz04CQRRYIAYYEuINMN_tpkZ01CA/s1600/dbdeployer_pxc.png" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjEtlphIZ5jS8aWzdBSD3RAlTGuqUbF-HLBW2B035B_UBhNqDVV5L1e00m3tBksP3dIC6IECuWDtCQGnoCSPUfMViraYMwnXQUWBrJvoOQDKz04CQRRYIAYYEuINMN_tpkZ01CA/s320/dbdeployer_pxc.png" width="320" height="245" data-original-width="1248" data-original-height="956" /></a><p>This was not on the radar. I have never been proficient in Galera clusters and related technologies, and thus I hadn’t given much thought to <a href="https://www.percona.com/software/mysql-database/percona-xtradb-cluster">Percona Xtradb Cluster</a> (PXC), until <a href="https://twitter.com/ask_dba">Alkin</a> approached me at FOSDEM, and proposed to extend dbdeployer features to support PXC. He mentioned that many support engineers at Percona use <a href="[dbdeployer](https://github.com/datacharmer/dbdeployer">dbdeployer</a>) on a daily basis and that the addition of PXC would be welcome.</p><p>I could not follow up much during the conference, but we agreed on making a proof-of-concept in an indirect way: if several nodes of PXC can run in the same host using shell scripts, dbdeployer could reproduce that behavior.</p><p>A few weeks later, when dbdeployer had already been enhanced with <a href="https://github.com/datacharmer/dbdeployer/issues/50">flavors and capabilities</a>, I got the script that can deploy several nodes in the same host. It’s a simplification of the ones used in <a href="https://github.com/Percona-QA/percona-qa/tree/master/pxc-tests">Percona PXC tests</a>, which got me started.</p><p>I followed a method similar to the one I used for MySQL Group replication. The technology is similar, although the MySQL Team used a different approach for the installation. The basic principle is that the cluster needs two ports per node: in addition to the regular MySQL port, there is a communication port (SST or Snapshot State Transfer port) that is needed to exchange cluster data. Using this information, and following the sample in the script, I could produce a prototype that surprisingly worked at the first try!</p><p>The cluster did deploy, and the replication test, which comes free of charge when you implement a replication-type sandbox using standard templates, worked flawlessly.</p><p>Then I hooked the deployment method into dbdeployer concurrency engine, which is able to deploy several nodes at once. Here I hit the first problem. In PXC, the nodes are not equal at startup. The first node needs to be initialised without other nodes addresses, and it becomes the reference for other nodes to join the cluster. If I provided complete references for all nodes (as I do for MySQL Group Replication,) it didn’t work.</p><p>After some talk with Percona engineers on Slack, I figured out that the nodes can be deployed together, and the second and third node will just wait for the first one to come online and then join. That worked in principle, or when I deployed sequentially, but not when they are deployed all at once. Fortunately, dbdeployer has several ways of enabling debugging output, and after a few unsuccessful attempts I got the reason: PXC initialisation happens using <code>rsync</code> on port 4444. When the nodes are started sequentially, the receiving node takes control of port 4444 without conflicts, gets the job done and releases the port. When we deploy all nodes at once, there is a race for the possession of the synchronisation port, and a random node will win it, leaving the others waiting forever.<br/><br />
Thus, I modified the installation to allocate a different rsync port for each node, and after that the concurrent installation worked as well.</p><p>The last obstacle was the discovery that there is yet another port (IST, or Incremental State Transfer port), which is always one number bigger than the SST port. Thus, if the SST port is, say, 5555, the IST port is set to 5556. This means that, unlike other dbdeployer clusters, I can’t set port numbers incrementally, but I need to set them with an interval. I did that, and the cluster came with a default allocation of four ports per node (MySQL, rsync, SST, IST). If we also enable MySQLX, which comes includes as PXC binaries are based on MySQL 5.7, we would set 5 ports per node, and a majestic 15 ports for a three-node cluster.</p><p>Anyway, the support for Percona XtraDB Cluster is available in <a href="https://github.com/datacharmer/dbdeployer/releases/tag/v1.21.0">dbdeployer 1.21.0</a>. Let’s see a sample session to use the new functionality.</p><pre><code>$ $ dbdeployer --version
dbdeployer version 1.21.0
$ dbdeployer unpack --prefix=pxc ~/downloads/Percona-XtraDB-Cluster-5.7.25-rel28-31.35.1.Linux.x86_64.ssl100.tar.gz
[...]
Renaming directory $HOME/opt/mysql/Percona-XtraDB-Cluster-5.7.25-rel28-31.35.1.Linux.x86_64.ssl100 to $HOME/opt/mysql/pxc5.7.25
</code></pre><p>Before trying the cluster, it would be a good practice to make sure that your system can deploy a single node.</p><pre><code>$ dbdeployer deploy single pxc5.7.25
Database installed in $HOME/sandboxes/msb_pxc5_7_25
run 'dbdeployer usage single' for basic instructions'
. sandbox server started
$ $HOME/sandboxes/msb_pxc5_7_25/test_sb
[...]
# Tests : 11
# PASS : 11
# fail : 0
$ dbdeployer delete msb_pxc5_7_25
[...]
</code></pre><p>And now for the real test:</p><pre><code>$ dbdeployer deploy replication --topology=pxc pxc5.7.25
Installing and starting node 1
. sandbox server started
Installing and starting node 2
...... sandbox server started
Installing and starting node 3
..... sandbox server started
Replication directory installed in $HOME/sandboxes/pxc_msb_pxc5_7_25
run 'dbdeployer usage multiple' for basic instructions'
</code></pre><p>We should now see all the allocated ports.</p><pre><code>$ dbdeployer sandboxes --header
name type version ports
---------------------------- ------------------------ ----------- ----------------------------------------------------------------------------
pxc_msb_pxc5_7_25 : Percona-Xtradb-Cluster pxc5.7.25 [26226 26352 26353 26364 26227 26354 26355 26365 26228 26356 26357 26366 ]
</code></pre><p>If we want more detail, we can look at the sandbox description file:</p><pre><code>$ cat $HOME/sandboxes/pxc_msb_pxc5_7_25/sbdescription.json
{
"basedir": "$HOME/opt/mysql/pxc5.7.25",
"type": "Percona-Xtradb-Cluster",
"version": "pxc5.7.25",
"flavor": "pxc",
"port": [
26226,
26352,
26353,
26364,
26227,
26354,
26355,
26365,
26228,
26356,
26357,
26366
],
"nodes": 3,
"node_num": 0,
"dbdeployer-version": "1.21.0",
"timestamp": "Thu Mar 7 17:20:03 CET 2019",
"command-line": "dbdeployer deploy replication --topology=pxc pxc5.7.25"
}
</code></pre><p>Now we can run the replication test. Given that we have a cluster where all nodes are masters, the test will create a table in each node, and read the result in each slave (again, each node):</p><pre><code>$ $HOME/sandboxes/pxc_msb_pxc5_7_25/test_replication
# master 1
# master 2
# master 3
# slave 1
ok - '3' == '3' - Slaves received tables from all masters
# slave 2
ok - '3' == '3' - Slaves received tables from all masters
# slave 3
ok - '3' == '3' - Slaves received tables from all masters
# pass: 3
# fail: 0
</code></pre><p>It’s a simple test, but it tells us that the cluster is fully functional.</p><p>Thanks to Alkin and other Percona engineers who have tested the prototype in real time.</p>Giuseppe Maxiahttp://www.blogger.com/profile/15801583338057324813noreply@blogger.com1tag:blogger.com,1999:blog-16959946.post-86260769685202498912019-03-06T22:52:00.000+01:002019-03-06T22:52:55.526+01:00dbdeployer community - Part 1: TiDB<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiZQ9CzY3m8yz635IoOo8WkzKNEFcQcuGifPZp-0kjIt7QtvXn3gzZuSF86JJ-QzN0vWvFhA4BKwFXx7d92k7p_rLfbli01IjDaACvpKaS2VkFk6XAgCIVyFV7TSjgv_luzHsQt/s1600/dbdeployer_black_on_white_tidb.png" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiZQ9CzY3m8yz635IoOo8WkzKNEFcQcuGifPZp-0kjIt7QtvXn3gzZuSF86JJ-QzN0vWvFhA4BKwFXx7d92k7p_rLfbli01IjDaACvpKaS2VkFk6XAgCIVyFV7TSjgv_luzHsQt/s320/dbdeployer_black_on_white_tidb.png" width="320" height="260" data-original-width="742" data-original-height="604" /></a><p>After a conference, when I take stock of what I have learned, I usually realise that the best achievements are the result of interacting with other attendees during the breaks, rather than simply listening to the lectures. It might be because I follow closely the blogosphere and thus the lectures have few surprises in store for me, or perhaps because many geeks take the conference as an excuse to refresh dormant friendships, catch up with technical gossip, and ask their friends some questions that were too sensitive to be discussed over Twitter and have been waiting for a chance of an in-person meeting to see the light of the day.</p><p>I surely had some of such questions, and I took advantage of the conference to ask them. As it often happens, I got satisfactory responses, but the latest FOSDEM conference was different than usual, because I got the best experience from the questions that others did ask me.</p><p>As it turned out, others were waiting for a chance to discuss things over coffee or food, and I saw that my pet project (<a href="https://github.com/datacharmer/dbdeployer">dbdeployer</a>) is a lot more popular than I thought, and it is being used silently in several environments. It should not be surprising if you read several MySQL reports on bugs at <a href="https://bugs.mysql.com/search.php?search_for=msandbox&status=All&severity=all&limit=All">bugs.mysql.com</a> where it is common the usage of sandboxes to reproduce user issues. Anyway, I got some praise, some requests, a few ideas for improvements, advance notice of an incoming graphical interface, and a few concrete collaboration proposals.</p><p>One of such proposals came from <a href="http://www.tocker.ca/">Morgan Tocker</a>, who suggested enhancing dbdeployer to support <a href="https://pingcap.com/docs/overview/">TiDB</a>. At first, it seemed uninteresting, as TiDB is designed to be distributed, and installing just a component didn’t immediately look useful. However, Morgan pointed out that it could be used as a tool to test compatibility with existing applications, and as such it could gain much more value than I initially thought. We decided to try a quick hackathon to make a proof of concept.</p><p>It was a great pleasure to figure out, in just over one hour of close interaction, that dbdeployer <a href="https://datacharmer.blogspot.com/2018/02/meet-dbdeployer-new-sandbox-maker.html">design for flexibility</a> was up to the task. We managed to make TiDB work with dbdeployer simply by exporting, editing, and re-loading a few templates.</p><p>The exercise showed strengths and limitations in both projects. We agreed that dbdeployer had to lose some assumptions (such as “I am working with a MySQL server”) and become able to recognise which flavor of MySQL-lookalike we are dealing with. At the same time, we noted that TiDB is not compatible when it comes to deployment and bootstrap: it is so simple and straightforward that its initialisation doesn’t fit in the complex operation that is a MySQL server warm-up.</p><p>Pleased with the initial success, we kept in touch and, after dbdeployer acquired the ability of <a href="https://github.com/datacharmer/dbdeployer/issues/50">telling one flavor from another</a>, we put together the various pieces to <a href="https://github.com/datacharmer/dbdeployer/issues/54">make dbdeployer recognise and install TiDB</a>. We found and fixed several bugs in both project, and finally released <a href="https://github.com/datacharmer/dbdeployer/releases/tag/v1.19.0">dbdeployer 1.19.0</a>, which can use a TiDB server transparently.</p><p>What does <em>transparently</em> mean? It means that tests for TiDB deployment can run alongside tests for other MySQL servers, and the sandbox scripts (such as <code>start</code>, <code>stop</code>, <code>use</code>, <code>status</code>, and <code>test_sb</code>) work as expected and produce a compatible output. Thus, there is a TiDB test running together with another dozen MySQL versions.</p><p>Now, if you want, you can <a href="http://www.tocker.ca/tidb-now-supported-in-dbdeployer-1-19-0.html">evaluate TiDB in your computer</a> without installing the full stack. It won’t be as fast as the real thing: what is installed as a single node is a slower emulation of the real database, but it is enough to give you an idea of what queries you can and cannot run in TiDB, and perhaps try to see if your application could run on TiDB at all.</p><p>The collaboration with TiDB was especially useful because the changes needed to smooth the TiDB integration have made made dbdeployer better suited to add support for more not-quite-mysql servers, such as the one that we’ll see in the next post.</p><p>But before reaching that point, here’s an example of TiDB deployment on Linux:</p><pre><code>$ wget https://download.pingcap.org/tidb-master-linux-amd64.tar.gz
[...]
2019-02-24 04:46:26 (2.26 MB/s) - 'tidb-master-linux-amd64.tar.gz' saved [16304317/16304317]
$ dbdeployer unpack tidb-master-linux-amd64.tar.gz --unpack-version=3.0.0 --prefix=tidb
Unpacking tarball tidb-master-linux-amd64.tar.gz to $HOME/opt/mysql/tidb3.0.0
1
Renaming directory /home/msandbox/opt/mysql/tidb-master-linux-amd64 to /home/msandbox/opt/mysql/tidb3.0.0
</code></pre><br />
TiDB tarballs doesn't come with a client. We need to use one from MYSQL 5.7. Rather than downloading the huge tarball from MySQL site, we can get a smaller one from a GitHub repository, using dbdeployer itself (NB: this reduced tarball is only for Linux)<br />
<br />
<pre><code>
$ dbdeployer remote list
Files available in https://raw.githubusercontent.com/datacharmer/mysql-docker-minimal/master/dbdata/available.json
5.7 -> [mysql-5.7.24 mysql-5.7.25]
8.0 -> [mysql-8.0.13 mysql-8.0.15]
4.1 -> [mysql-4.1.22]
5.0 -> [mysql-5.0.15 mysql-5.0.96]
5.1 -> [mysql-5.1.72]
5.5 -> [mysql-5.5.61 mysql-5.5.62]
5.6 -> [mysql-5.6.41 mysql-5.6.43]
$ dbdeployer remote get mysql-5.7.25
File /home/msandbox/mysql-5.7.25.tar.xz downloaded
$ dbdeployer unpack mysql-5.7.25.tar.xz
[...]
Renaming directory /home/msandbox/opt/mysql/mysql-5.7.25 to /home/msandbox/opt/mysql/5.7.25
</code></pre><p>Now we are ready to install TiDB:</p><pre><code>$ dbdeployer deploy single tidb3.0.0 --client-from=5.7.25
Creating directory /home/msandbox/sandboxes
Database installed in $HOME/sandboxes/msb_tidb3_0_0
run 'dbdeployer usage single' for basic instructions'
. sandbox server started
</code></pre><p>Once installed, a TiDB sandbox behaves like a MySQL sandbox.</p><pre><code>$ $HOME/sandboxes/msb_tidb3_0_0/use
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.7.10-TiDB-v3.0.0-beta-111-g266ff4b6f MySQL Community Server (Apache License 2.0)
Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql [localhost:3000] {msandbox} ((none)) >
</code></pre>Giuseppe Maxiahttp://www.blogger.com/profile/15801583338057324813noreply@blogger.com1tag:blogger.com,1999:blog-16959946.post-71513028880863916252018-04-20T23:57:00.000+02:002018-04-20T23:57:37.691+02:00MySQL adjustment bureau<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiaWMMNdn-CKMq8Rk0KJ3WOp1lngGwkPIb7Azs-STVMHo0BUA2XdNedbHjrDL5m2uHWq_xK5lmSPB20Ye9l-xOVbd1u3foTDwfiTxbPg5govHR-o6xjYHzQ5XIb8l4MZEu1c4NU/s1600/hack1_no_bg.jpg" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiaWMMNdn-CKMq8Rk0KJ3WOp1lngGwkPIb7Azs-STVMHo0BUA2XdNedbHjrDL5m2uHWq_xK5lmSPB20Ye9l-xOVbd1u3foTDwfiTxbPg5govHR-o6xjYHzQ5XIb8l4MZEu1c4NU/s320/hack1_no_bg.jpg" width="249" height="320" data-original-width="489" data-original-height="629" /></a><br />
<p>When maintainng any piece of software, we usually deal with two kind of actions:</p><ul><li>bug fixing,</li>
<li>new features. </li>
</ul><h3 id="toc_1">bugs and features</h3><p>A bug happens when there is an error in the software, which does not behave according to the documentation or the specifications. In short, it's a breech of contract between the software maintainer and the users. The promise, i.e. the software API that was published at every major version, is broken, and the software must be reconciled with the expectations and <u>fixed</u>, so that it behaves again as the documentation says. When we fix a bug in this way, we increment the <em>revision</em> number of the software version (e.g. 1.0.0 to 1.0.1. See <a href="https://semver.org/">semantic versioning</a>).</p><p>New features, in turn, can be of two types:</p><ul><li>backward compatible enhancements, which add value to the software without breaking the existing functionality. This is the kind of change that requires an increment of the <em>minor</em> indicator in the version (for example: 1.1.15 to 1.2.0.)</li>
<li>Incompatible changes that break the existing behavior and require users to change their workflow. This kind of change requires bumping up the <em>major</em> number in the version (as in 2.1.3 to 3.0.0.)</li>
</ul><h3 id="toc_2">Not a bug, nor a feature, but an adjustment.</h3><p>The above concepts seem simple enough: you either fix something that's broken or add new functionality. </p><p>However, when maintaining a tool that has the purpose of helping users to deal with another software (as it is the case of <a href="https://www.dbdeployer.com">dbdeployer</a> that helps users to deploy MySQL databases) there is yet another category of changes that don't fall into the standard categories: it's what happens when the software being helped (MySQL) changes its behavior, which would break the normal functioning of the helping tool, giving the maintainer a difficult choice:</p><ul><li>shall I modify the tool's interface to adapt to the new behavior, breaking existing procedures?</li>
<li>or shall I adapt the tool's functioning behind the scenes to keep the interface unchanged?</li>
</ul><p>My philosophy with dbdeployer (and MySQL-Sandbox before it) is to preserve the tool's interface, so that users don't have to change existing procedures. I call this kind of changes <strong>adjustments</strong>, because they are not bugs, as they are not a consequence of a coding error, and not a feature, as the intervention is not a conscious decision to add new functionality, but an emergency operation to preserve the status quo. You can think of this category as a capricious change in specifications, which so often happens to software developers, with the difference that the one changing the specs is not the user, but a third party who doesn't know, or care, about our goal of preserving the API integrity.</p><p>For example, from MySQL 8.0.3 to 8.0.4 there was a change in the default authentication plugin. Instead of <code>mysql_native_password</code>, MySQL 8.0.4 uses <code>caching_sha2_password</code>. The immediate side effect for MySQL-Sandbox and dbdeployer was that replication doesn't work out of the box. A possible solution would be to force the old authentication plugin, but this would not allow users to test the new one. Since the main reason to use a tool like dbdeployer is to experiment with new releases safely, I had to keep the default behavior. Thus, I left the default plugin in place, and changed the way the replication works. It's <a href="http://lefred.be/content/mysql-8-0-more-about-new-authentication-plugin-and-replication/">an ugly workaround</a> actually, but allows users to see the new behavior without losing existing functionality.<br />
To complete the adjustment, I added a new option <code>--native-auth-plugin</code>, which would deploy using the old <code>mysql_native_password</code>. In total, the adjustment consists of a behind-the-scenes change, almost undetectable by users, and a new option to keep using the familiar authentication if users want it.</p><p>From the point of view of semantic versioning, this kind of change is a backward-compatible modification of the API, which warrants an increase of the <em>minor</em> number of the version.</p><p>Another example: when MySQL went <a href="https://dev.mysql.com/doc/relnotes/mysql/8.0/en/news-8-0-11.html">from 8.0.4 to 8.0.11</a>, it introduced a deal breaker change: the X Plugin is now loaded by default. This is easy for users of <a href="https://dev.mysql.com/doc/refman/8.0/en/document-store.html">MySQL as a document store</a>, as they don't need to enable the plugin manually, but bad news for anyone else, as the server is opening a port and a socket that many users may not choose to open voluntarily. What's worse, when installing more sandboxes of version 8.0.11 in the same host (for example in replication), one will succeed in reserving the plugin port and socket, while the others will have the error log populated with surprising errors about a socket being already in use.</p><p><a href="https://github.com/datacharmer/dbdeployer/releases/tag/1.3.0">The solution</a> is similar to the previous one. When dbdeployer detect MySQL 8.0.11 or newer, it adds options to customize the <code>mysqlx</code> plugin port and socket, thus allowing a frictionless deployment where the new functionality is available to the brave experimenters. At the same time, I added a new option (<code>--disable-mysqlx</code>) for the ones who really don't want an extra port and socket in their servers, not even for testing.</p><p>These adjustment are usually costly additions. While the added code is not that much, they require extra tests, which are often complex and require more time to write and execute them. The process to add an <strong>adjustment</strong> goes mostly like this:</p><ul><li>I dedicate my morning walk to think about the fix. Sometimes the fix requires several walks, while I decide the less intrusive solution.</li>
<li>If the walk has been fruitful, writing the code requires just a few minutes. If I missed something, I iterate.</li>
<li>Then the more difficult part: writing meaningful tests that prove that the adjustment is correct and it doesn't introduce side effects in any MySQL version. And of course the option that reintroduces the old behavior must be tested too.</li>
<li>A positive side effect of this exercise is that often I realize that I was missing a test for an important behavior and then I write down that as well. The test suite included 6,000+ tests 1 month ago, and now it has almost doubled.</li>
</ul>Giuseppe Maxiahttp://www.blogger.com/profile/15801583338057324813noreply@blogger.com2tag:blogger.com,1999:blog-16959946.post-70842677687867990122018-04-03T05:30:00.000+02:002018-04-03T06:05:25.230+02:00Test MySQL 8.0 right in your computer<p><a href="https://dev.mysql.com/doc/refman/8.0/en/mysql-nutshell.html">MySQL 8.0</a> GA is right around the corner. I don't have precise information about its release, as I don't work at Oracle. If I did, I would probably know, but I couldn't tell when the release is scheduled to appear because of company policies. I can, however, speculate and infer, based of my experience with previous releases. My personal assessment is that the release will appear <em>before</em> <a href="https://www.percona.com/live/18/schedule/day-1-grid">9:00am PT on April 24, 2018</a>. The "<em>before</em>" can be anything from a few minutes to one week in advance.<br />
Then, again, it may not happen at all if someone finds an atrocious bug that needs to be fixed asap.</p><p>Either way, users are keen on testing the new release in its current state of release candidate. Here I show a few methods that allow you to have a taste of the new goodies without waiting for the triumphal (keynote) announcement.</p><br />
<h3 id="toc_1">1. Docker containers</h3><p>If you are a <a href="https://www.docker.com/what-docker">docker</a> user, using a container to test MySQL is a no brainer. Unlike virtual machines or standalone servers, a docker container comes ready to use, with nothing to configure. All you need to do is pulling the right image. As with every docker images, you pull once and then use as many times as you need.</p><p>There are two reliable images that contain the latest MySQL. One is called <code>mysql:8.0</code> and is tagged as <em>official</em>, which means that it is released by the Docker maintenance team. The other one, which is released by the MySQL team, is called <code>mysql/mysql-server:8.0</code>.</p><div><pre><code class="language-none">$ docker pull mysql:8.0
8.0: Pulling from library/mysql
Digest: sha256:7004063f8bd0c7bade8d1c526b9b8f5188c8288f411d76ee4ba83131e00c6f02
Status: Downloaded newer image for mysql:8.0
$ docker pull mysql/mysql-server:8.0
8.0: Pulling from mysql/mysql-server
Digest: sha256:e81d95f788adb04a4d2fa5f6f7e9283ca0f6360fb518efe65af5a7377a4ec282
Status: Downloaded newer image for mysql/mysql-server:8.0</code></pre></div><p>The <code>mysql</code> image is based on Debian, while the original package, as you would expect, is based on Oracle Linux.</p><p>Let's see how to run MySQL in a container.</p><div><pre><code class="language-none">$ docker run --name official -e MYSQL_ROOT_PASSWORD=secret -d mysql:8.0
60ec307578a139f5083ded07e94d737690d287b1b95093878675983a5cc40174
$ docker run --name original -e MYSQL_ROOT_PASSWORD=secret \
-d mysql/mysql-server:8.0
0c93bb4a97ffa53232a69732d3ae45413a443e38fa43ad6fdc4057168cba42d2</code></pre></div><p>With the above commands we get two containers, one for the <em>official</em> image and one for the <em>original</em> one.<br />
We can't use them straight away, though. We need to wait for the servers to be ready. An easy method to verify the status of the server is looking at docker logs:</p><div><pre><code class="language-none">$ docker logs original --tail 1
2018-04-01T21:23:30.395461Z 0 [System] [MY-010931] /usr/sbin/mysqld: ready for connections. Version: '8.0.4-rc-log' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL).
$ docker logs original --tail 1
2018-04-01T21:23:30.395461Z 0 [System] [MY-010931] /usr/sbin/mysqld: ready for connections. Version: '8.0.4-rc-log' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL).</code></pre></div><p>Here, after about 10 seconds, both containers are ready to use. We can now access the servers. One easy method is through <code>docker exec</code></p><div><pre><code class="language-none">$ docker exec -ti original mysql -psecret
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 15
Server version: 8.0.4-rc-log MySQL Community Server (GPL)
Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql></code></pre></div><p>A similar command would allow us to access the other container.</p><p>If you want to try replication, more work is needed. In these articles you will find more details on Docker operations, and examples of advanced deployments:</p><ul><li><a href="http://datacharmer.blogspot.com/2015/10/mysql-docker-operations-part-1-getting.html">MySQL-Docker operations. - Part 1: Getting started with MySQL in Docker.</a></li>
<li><a href="http://datacharmer.blogspot.com/2015/11/mysql-docker-operations-part-2.html">MySQL-Docker operations. - Part 2: Customizing MySQL in Docker.</a></li>
<li><a href="http://datacharmer.blogspot.com/2015/11/mysql-docker-operations-part-3-mysql.html">MySQL-Docker operations. - Part 3: MySQL replication in Docker.</a></li>
<li><a href="https://datacharmer.blogspot.com/2016/02/a-safer-mysql-box-in-docker.html">a proposal for better MySQL security</a></li>
<li><a href="https://mysqlrelease.com/2017/08/docker-secrets-and-mysql-password-management/">another security proposal</a></li>
<li><a href="https://severalnines.com/blog/mysql-docker-containers-understanding-basics">MySQL docker containers: understanding the basics</a></li>
</ul><br />
<h3 id="toc_2">2. Sandboxes</h3><p>A sandboxed database is deployed in a non-dedicated box, with its configuration altered in such a way that it will run independently from other similar deployment and even from databases running in the main space. <br />
The granddaddy of the sandbox deployer was <a href="http://mysqlsandbox.net">MySQL-Sandbox</a>, which has recently evolved into the more powerful and easier to use <a href="https://www.dbdeployer.com">dbdeployer</a>.<br />
You can use MySQL-Sandbox to test a MySQL 8.0 tarball on MacOS</p><div><pre><code class="language-none">$ make_sandbox --export_binaries mysql-8.0.4-rc-macos10.13-x86_64.tar.gz</code></pre></div><p>This command unpacks the tarball into <code>$HOME/opt/mysql</code> and deploys the database in <code>$HOME/sandboxes/msb_8_0_4</code>.<br />
Until recently, the same command would work on Linux without modifications. In MySQL 8.0.4, though, the tarball organization for Linux has changed. There are symbolic links for SSL libraries inside the <code>./bin</code> directory. Those symlinks are not extracted by default, but only if you use the option <code>--keep-directory-symlink</code> when opening the tarball. MySQL-Sandbox doesn't do it, also because this option is not standard to every version of <code>tar</code>. </p><p>Thus, if you want to use the old MySQL-Sandbox, you need to run the extraction manually.</p><div><pre><code class="language-none">$ cd $HOME/opt/mysql
$ tar -xzf --keep-directory-symlink /tmp/mysql-8.0.4-rc-linux-glibc2.12-x86_64.tar.gz
$ mv mysql-8.0.4-rc-linux-glibc2.12-x86_64 8.0.4
$ make_sandbox 8.0.4</code></pre></div><p>I don't recommend the above procedure, for either Linux or MacOS. The main reason, in addition to the manual operations involved, is that MySQL-Sandbox is not going to be updated for the time being. Instead, you should use <a href="https://www.dbdeployer.com">dbdeployer</a>, which has all the main features of MySQL-Sandbox and a lot of new ones. Here's the equivalent procedure:</p><div><pre><code class="language-none">$ dbdeployer unpack /tmp/mysql-8.0.4-rc-linux-glibc2.12-x86_64.tar.gz
$ dbdeployer deploy single 8.0.4
Database installed in $HOME/sandboxes/msb_8_0_4
run 'dbdeployer usage single' for basic instructions'
. sandbox server started</code></pre></div><p>dbdeployer uses a different method to initialize the database server, which at the same time makes the initialization more visible and avoids the problem of the phantom SSL libraries.</p><p><b>Note:</b> Tarballs for recent MySQL versions are really big. MySQL 8.0.4 binaries expand to 1.9 GB. If storage is an issue, you should get the tarballs from <a href="https://github.com/datacharmer/mysql-docker-minimal/tree/master/dbdata">a collection of minimised tarballs</a> (Linux only) for most MySQL versions. For now, it's maintained by me, but I hope that the <a href="http://lefred.be/">the MySQL team</a> will release something similar.</p><p>Once you have deployed a sandbox with MySQL 8.0, using it is easy:</p><div><pre><code class="language-none">$ cd $HOME/sandboxes/msb_8_0_4
$ ./use
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 8
Server version: 8.0.4-rc-log MySQL Community Server (GPL)
Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql [localhost] {msandbox} ((none)) ></code></pre></div><p>dbdeployer creates several shortcuts for the most common commands to use the database. <code>./use</code> is the most common, and provides access to the MySQL client with all the necessary options needed to use it correctly. For more information on what is available, run</p><div><pre><code class="language-none">$ dbdeployer usage single</code></pre></div><p>This functionality would be enough to decide for a sandbox as your preferred method for testing. However, it this is only a tiny portion of what you can do with dbdeployer in your own computer. With a single command, you can test master/slave replication, multi-primary group replication, single primary group replication, fan-in, and all-masters topologies.</p><p>You can try the following commands:</p><div><pre><code class="language-none">$ dbdeployer deploy single 8.0.4
$ dbdeployer deploy replication 8.0.4
$ dbdeployer deploy replication 8.0.4 --topology=group
$ dbdeployer deploy replication 8.0.4 --topology=group --single-primary
$ dbdeployer deploy replication 8.0.4 --topology=all-masters
$ dbdeployer deploy replication 8.0.4 --topology=fan-in</code></pre></div><p>If you have enough RAM, all these deployments will survive in parallel. <br />
In my desktop, I can run:</p><div><pre><code class="language-none">$ dbdeployer sandboxes --header
name type version ports
---------------- ------- ------- -----
all_masters_msb_8_0_4 : all-masters 8.0.4 [15001 15002 15003]
fan_in_msb_8_0_4 : fan-in 8.0.4 [14001 14002 14003]
group_msb_8_0_4 : group-multi-primary 8.0.4 [20009 20134 20010 20135 20011 20136]
group_sp_msb_8_0_4 : group-single-primary 8.0.4 [21405 21530 21406 21531 21407 21532]
msb_8_0_4 : single 8.0.4 [8004]
rsandbox_8_0_4 : master-slave 8.0.4 [19009 19010 19011]</code></pre></div><p>When <a href="https://mysqlrelease.com/2018/03/mysql-8-0-it-goes-to-11/">MySQL 8.0.11</a> is released, you can replace "8.0.4" with "8.0.11" and get a similar result.</p><p>BTW, you have seen that deploying replication sandboxes may take a long time. You may try adding <code>--concurrent</code> to each command, and enjoy a <a href="https://datacharmer.blogspot.com/2018/03/concurrent-sandbox-deployment.html">notable speed increase</a>.</p><p>What else can you do with the sandboxes you have just deployed? Plenty! For a complete list, have a look at the <a href="https://github.com/datacharmer/dbdeployer">online documentation</a>. But for the moment, you may try this:</p><div><pre><code class="language-none">$ dbdeployer global status
$ dbdeployer global test
$ dbdeployer global test-replication</code></pre></div><br />
<h3 id="toc_3">3. Other methods</h3><p>Besides the methods that I recommend, there are others that you could use, but I won't advise about them as there are <a href="http://lefred.be/">more qualified ones</a> for that.</p><ul><li><strong>Standalone server</strong>. If you have the luxury of having one or more standalone servers sitting in a lab, by all means go for it. Just follow <a href="https://dev.mysql.com/doc/refman/8.0/en/installing.html">the instructions</a> about installing MySQL on your lucky server. Be advised, though, that depending on the method you choose and the version of your operating system, you may face compatibility issues (.rpm or .deb dependencies).</li>
<li> <strong>Virtual machines</strong>. VMs share with standalone servers the same ease of installation (and the same dependency issues), only a bit slower. They are convenient, as you can use them to test in conditions that more closely resemble production settings, and if you use a configuration server such as Puppet or Ansible, your task of testing the new version could be greatly simplified. The instructions for the virtual machines are <a href="https://dev.mysql.com/doc/refman/8.0/en/installing.html">the same seen for standalone servers</a>.</li>
</ul>Giuseppe Maxiahttp://www.blogger.com/profile/15801583338057324813noreply@blogger.com1tag:blogger.com,1999:blog-16959946.post-67603275347482857742018-04-02T12:49:00.000+02:002018-04-02T12:49:07.723+02:00dbdeployer GA and semantic versioning<p><a href="https://www.dbdeployer.com">dbdeployer</a> went into release candidate status a few weeks ago. Since then, I added no new features, but a lot of tests. The test suite now runs 3,000+ tests on MacOS and a bit more on Linux, for a grand total of 6,000+ tests that need to run at least twice: once with concurrency enabled and once without. I know that testing can't prove the absence of bugs, but I am satisfied with the results, since all this grinding has allowed me to find several bugs and fix them. </p><p>In this framework, I felt that dbdeployer could exit candidate status and get to version 1.0. This happened on March 26th. An immediate side effect of this change is that from this point on, dbdeployer must adhere to the <a href="https://semver.org/">semantic versioning</a> principles:</p><p>A version number is made of Major, Minor, and Revision. When changes are applied, the following happens:</p><ul><li>Backward-compatible bug fixes increment the <strong>Revision</strong> number (e.g. 1.0.0 to 1.0.1)</li>
<li>Backward-compatible new features increment the <strong>Minor</strong> number (1.0.1 to 1.1.0)</li>
<li>Backward incompatible changes (either features or bug fixes that break compatibility with the API) increment the <strong>Major</strong> number (1.15.9 to 2.0.0)</li>
</ul><p>The starting API is defined in <a href="https://github.com/datacharmer/dbdeployer/blob/master/docs/API-1.0.md">API-1.0.md</a>, which was generated manually.<br />
The file <a href="https://github.com/datacharmer/dbdeployer/blob/master/docs/API-1.1.md">API-1.1.md</a> contains the same API definition, but was generated automatically and can be used to better compare the initial API with further version.</p><p>So the app went from 1.0 to 1.1 in less than one week. In obedience to semantic versioning principles, if a new backward-compatible feature is added, the minor number of the version increases. What does <strong>backward-compatible</strong> mean? It means that commands, procedures, and workflows that were working with the previous version will also work with the current one. It's just that the new release will have more capabilities. In this case, the added feature is the ability of having environment variables <code>HOME</code> and <code>PWD</code> recognized and properly expanded in the configuration file. It's nothing very exciting, but changing the minor number gives the user a hint of what to expect from the new release.</p><p>Let's give a few examples:</p><ul><li>Version goes from 1.0.0 to 1.0.1: It means that there are only bug fixes, and you should expect to use it without modifications.</li>
<li>Version goes from 1.0.1 to 1.1.0: You should be able to use dbdeployer just as before, but you should check the release notes to see what's new, because there are new functionalities that might be useful to you.</li>
<li>Version goes from 1.3.15 to 2.0.0: Danger! A major number bumped up means that something has changed in the API, which is now partially or totally incompatible with the previous release. Your workflow may break, and you must check the release notes and the documentation to learn how to use the new version.</li>
</ul><p>This is different from other applications. For example, the MySQL server uses version numbers with hard to predict meaning:</p><ul><li>MySQL 5.1, 5.5, 5.6, and 5.7 should be, in fact, major version number changes, not minor ones. Each one of them introduces incompatible changes that require careful review of the novelties.</li>
<li>Within the same version (such as MySQL 5.7) there are a lot of compatible and incompatible changes, although the minor number stays the same.</li>
</ul><p>The plan with dbdeployer is to use the version number as a manifest, to give users an immediate feeling of what to expect. Rather than changing minor or major number only when the developers think there is some juicy new thing of which they can be proud, the version number will tell whether users should worry about compatibility or not.</p><p>In my general development plan, you are more likely to see versions like "1.25.16" than version "2.0," meaning that I will try to keep the current API valid as much as possible. A major version change will signify that a new feature could not fit in the current infrastructure and a new one would be needed.</p><p>You can draw your own conclusions here. A semantic versioning paradigm is unlikely to be adopted by most software vendors, because version numbers are often marketing gimmicks, and they can charge you more convincingly for a version 6.0 than for version 1.34.<br />
Free software, OTOH, can do this. My goal with dbdeployer is to help the MySQL community, and I will achieve that goal more easily if my releases can be adopted without fear of incompatibility.</p><br />
Giuseppe Maxiahttp://www.blogger.com/profile/15801583338057324813noreply@blogger.com0tag:blogger.com,1999:blog-16959946.post-81240661854156244552018-03-12T21:22:00.000+01:002018-03-12T21:22:49.955+01:00dbdeployer release candidate<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEivxvjwagoLKp85DNS1jPq1HE7u9IxfW6G0ZEyow2LDC5naWi5Lm-amu6E1hJRm6mtgVjTH55ohcNy7p1ve-Yy8Bnl78GMiVXwlJTgk_mcRq5BSCHIlqt6uSCuu9-YWddXINOJ4/s1600/np_launching_1342404_000000.png" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEivxvjwagoLKp85DNS1jPq1HE7u9IxfW6G0ZEyow2LDC5naWi5Lm-amu6E1hJRm6mtgVjTH55ohcNy7p1ve-Yy8Bnl78GMiVXwlJTgk_mcRq5BSCHIlqt6uSCuu9-YWddXINOJ4/s320/np_launching_1342404_000000.png" width="320" height="320" data-original-width="300" data-original-height="300" /></a><br />
<p>The latest release of <a href="https://github.com/datacharmer/dbdeployer/releases/tag/0.3.1">dbdeployer</a> is possibly the last one with a leading 0. If no serious bugs are found in the next two weeks, the next release will bear a glorious 1.0.</p><h3 id="toc_0">Latest news</h3><p>The decision to get out of the stream of pre-releases that were published until now comes because I have implemented <a href="https://github.com/datacharmer/dbdeployer/blob/master/docs/features.md">all the features that I wanted to add</a>: mainly, all the ones that I wished to add to <a href="https://mysqlsandbox.net">MySQL-Sandbox</a> but it would have been too hard:</p><ul><li><a href="http://datacharmer.blogspot.com/2018/03/concurrent-sandbox-deployment.html">concurrent deployment and deletion</a></li>
<li>group replication;</li>
<li>multi-source replication;</li>
<li><a href="http://datacharmer.blogspot.com/2018/03/customizing-dbdeployer.html">Full customisation</a> </li>
<li>Centralised administration.</li>
<li>Container-based tests;</li>
<li>Mock tests that can run without any real database server.</li>
</ul><p>The latest addition is the ability of running multi-source topologies. Now we can run four topologies: </p><ul><li><strong>master-slave</strong> is the default topology. It will install one master and two slaves. More slaves can be added with the option <code>--nodes</code>.</li>
<li><strong>group</strong> will deploy three peer nodes in group replication. If you want to use a single primary deployment, add the option <strong><code>--single-primary</code></strong>. Available for MySQL 5.7 and later.</li>
<li><strong>fan-in</strong> is the opposite of master-slave. Here we have one slave and several masters. This topology requires MySQL 5.7 or higher.<br />
<strong>all-masters</strong> is a special case of fan-in, where all nodes are masters and are also slaves of all nodes.</li>
</ul><p>It is possible to tune the flow of data in multi-source topologies. The default for fan-in is three nodes, where 1 and 2 are masters, and 2 are slaves. You can change the predefined settings by providing the list of components:</p><div><pre><code class="language-none">$ dbdeployer deploy replication \
--topology=fan-in \
--nodes=5 \
--master-list="1 2 3" \
--slave-list="4 5" \
8.0.4 \
--concurrent</code></pre></div><p>In the above example, we get 5 nodes instead of 3. The first three are master (<code>--master-list="1 2 3"</code>) and the last two are slaves (<code>--slave-list="4 5"</code>) which will receive data from all the masters. There is a test automatically generated to test replication flow. In our case it shows the following:</p><div><pre><code class="language-none">$ ~/sandboxes/fan_in_msb_8_0_4/test_replication
# master 1
# master 2
# master 3
# slave 4
ok - '3' == '3' - Slaves received tables from all masters
# slave 5
ok - '3' == '3' - Slaves received tables from all masters
# pass: 2
# fail: 0</code></pre></div><p>The first three lines show that each master has done something. In our case, each master has created a different table. Slaves in nodes 5 and 6 then count how many tables they found, and if they got the tables from all masters, the test succeeds.<br />
Note that for <em>all-masters</em> topology there is no need to specify master-list or slave-list. In fact, those lists will be auto-generated, and they will both include all deployed nodes.</p><h3 id="toc_1">What now?</h3><p>Once I make sure that the current features are reasonably safe (I will only write more tests for the next 10~15 days) I will publish the first (<em>non-pre</em>) release of dbdeployer. From that moment, I'd like to follow the recommendations of the <a href="https://semver.org">Semantic Versioning</a>:</p><ul><li>The initial version will be 1.0.0 (major, minor, revision);</li>
<li>The spects for 1.0 will be the API that needs to be maintained.</li>
<li>Bug fixes will increment the <em>revision</em> counter.</li>
<li>New features that don't break compatibility with the API will increment the <em>minor</em> counter;</li>
<li>New features or changes that break compatibility will trigger a <em>major</em> counter increment. </li>
</ul><p>Using this method will give users a better idea of what to expect. If we get a revision number increase, it is only bug fixes. An increase in the minor counter means that there are new features, but all previous features work as before. An increase in the major counter means that something will break, either because of changed interface or because of changed behavior.<br />
In practice, the tests released with 1.0.0 should run with any 1.x subsequent version. When those tests need changes to run correctly, we will need to bump up the major version.</p><p>Let's see if this method is sustainable. So far, I haven't had need to do behavioural changes, which are usually provoked by new versions of MySQL that introduce incompatible behavior (definitely MySQL does not follow the Semantic Versioning principles.) When the next version becomes available, I will see if this RC of dbdeployer can stand its ground.</p><br />
Giuseppe Maxiahttp://www.blogger.com/profile/15801583338057324813noreply@blogger.com2tag:blogger.com,1999:blog-16959946.post-63528017057332984092018-03-11T23:16:00.001+01:002018-03-11T23:16:59.530+01:00Concurrent sandbox deployment<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhpoGMO0ZBlh5K7Gq9KYWsnUO78qii7M_OsOh1SXjz3hS94QBzsAIIIyWrB4ltxIUywTo1JXYi3-F_dU2oJ2I6damr53DdHztSd6Q2unKc74Gyua-QTIVEAMqWk-H49vrCo5NFM/s1600/simultaneous.png" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhpoGMO0ZBlh5K7Gq9KYWsnUO78qii7M_OsOh1SXjz3hS94QBzsAIIIyWrB4ltxIUywTo1JXYi3-F_dU2oJ2I6damr53DdHztSd6Q2unKc74Gyua-QTIVEAMqWk-H49vrCo5NFM/s320/simultaneous.png" width="320" height="320" data-original-width="300" data-original-height="300" /></a><br />
<p>Version 0.3.0 of <a href="https://github.com/datacharmer/dbdeployer">dbdeployer</a> has gained the ability of deploying multiple sandboxes concurrently. Whenever we deploy a group of sandboxes (<em>replication</em>, <em>multiple</em>) we can use the <code>--concurrent</code> flag, telling dbdeployer that it should run operations concurrently. </p><p>What happens when a single sandbox gets deployed? There are six sets of operations:</p><ol><li>Create the sandbox directory and write down its scripts;</li>
<li>Run the initialisation script;</li>
<li>Start the database server;</li>
<li>Run the pre-grants SQL commands (if any;)</li>
<li>Load the grants;</li>
<li>Run the post-grants SQL commands (if any;)</li>
</ol><p>When several sandboxes are deployed concurrently, dbdeployer runs only the first step, and then creates a list of commands with an associated priority index. These commands are assembled for every sandbox, and then executed concurrently for every step. <br />
The sequence of events for a deployment of three sandboxes in replication would be like this:</p><ol><li>Create the sandbox skeleton for every sandbox;</li>
<li>Initialise all database servers;</li>
<li>start all the servers;</li>
<li>run the pre-grants, grants, post-grants scripts.</li>
<li>Runs the group initialisation script (start master and slaves, or setup group replication).</li>
</ol><p>Depending on the computer architecture, the server version, and the number of nodes, the speed of deployment can <strong>increase from 2 to 5 times</strong>.</p><p>Let's see an example:</p><pre><code>$ time dbdeployer deploy replication 5.7.21
[...]
real 0m13.789s
user 0m1.143s
sys 0m1.873s
$ time dbdeployer deploy replication 5.7.21 --concurrent
[...]
real 0m7.780s
user 0m1.329s
sys 0m1.811s
</code></pre><p>There is a significant speed increase. The gain rises sharply if we use an higher number of nodes.</p><pre><code>$ time dbdeployer deploy replication 5.7.21 --nodes=5
[...]
real 0m23.425s
user 0m1.923s
sys 0m3.106s
$ time dbdeployer deploy replication 5.7.21 \
--nodes=5 --concurrent
[...]
real 0m7.686s
user 0m2.248s
sys 0m2.777s
</code></pre><p>As we can see, the time for deploying 5 nodes is roughly the same used for 3 nodes. While the sequential operations take time proportionally with the number of nodes, the concurrent task stays almost constant.</p><p>Things a re a bit different for group replication, as the group initialisation (which happens after all the servers are up and running) takes more time than the simple master/slave deployment, and can't be easily reduced using the current code. </p><p>A similar optimisation happens when we delete multiple sandboxes. Here the operation is at sandbox level (1 replication cluster = 1 sandbox) not at server level, and for that reason the gain is less sharp. Still, operations are noticeably faster.</p><p>There is room for improvement, but I have seen that the total testing time for dbdeployer test suite has dropped from 26 to 15 minutes. I think it was a week end well spent.</p>Giuseppe Maxiahttp://www.blogger.com/profile/15801583338057324813noreply@blogger.com0tag:blogger.com,1999:blog-16959946.post-56579241014439821082018-03-05T08:00:00.000+01:002018-03-05T08:00:05.476+01:00Customizing dbdeployer<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjMmgh4iAXHOtibL_bAzqWqul3jn_P_jWsTWu_zGOdpIw1aXGul0579oKKYIrC8mTD7NtfO8NeMskk2Ndqr8XMSRvz_dlG-UYYCS8sWzyKV8UInIEyhDVYjgjVuiJBSavDvdFYU/s1600/db_customization.png" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjMmgh4iAXHOtibL_bAzqWqul3jn_P_jWsTWu_zGOdpIw1aXGul0579oKKYIrC8mTD7NtfO8NeMskk2Ndqr8XMSRvz_dlG-UYYCS8sWzyKV8UInIEyhDVYjgjVuiJBSavDvdFYU/s320/db_customization.png" width="320" height="313" data-original-width="648" data-original-height="634" /></a><br />
<p>As of version 0.2.1, <a href="https://github.com/datacharmer/dbdeployer">dbdeployer</a> allows users to customize composite sandboxes more than ever. This is done by manipulating the default settings, which are used to deploy the sandbox templates.</p><p>In order to appreciate the customization capabilities, let's start with a vanilla deployment, and then we have a look at the possible changes.</p><pre><code>$ <b>dbdeployer deploy replication 8.0.4</b>
Installing and starting <b>master</b>
Database installed in $HOME/sandboxes/rsandbox_8_0_4/<b>master</b>
. sandbox server started
Installing and starting <b>slave</b> 1
Database installed in $HOME/sandboxes/rsandbox_8_0_4/<b>node</b>1
. sandbox server started
Installing and starting <b>slave</b> 2
Database installed in $HOME/sandboxes/rsandbox_8_0_4/<b>node</b>2
. sandbox server started
$HOME/sandboxes/rsandbox_8_0_4/initialize_<b>slaves</b>
initializing <b>slave</b> 1
initializing <b>slave</b> 2
Replication directory installed in $HOME/sandboxes/rsandbox_8_0_4
run 'dbdeployer usage multiple' for basic instructions'
</code></pre><p>A regular replication sandbox has one <strong>master</strong> and two <strong>slaves</strong>. Each slave is inside a directory called <strong>node</strong>X.</p><p>The resulting sandbox has a directory called <em>master</em>, two <em>node</em>X directories, a shortcut for the master called <em>m</em>, and two shortcuts for the slaves called <em>s1</em> and <em>s2</em>. There are also two management scripts called <em>initialize_slaves</em> and <em>check_slaves</em>.</p><pre><code> $ ls -l ~/sandboxes/rsandbox_8_0_4/
total 152
-rwxr--r-- 1 user staff 1500 Mar 5 06:21 <b>check_slaves</b>
-rwxr--r-- 1 user staff 1160 Mar 5 06:21 clear_all
-rwxr--r-- 1 user staff 1617 Mar 5 06:21 <b>initialize_slaves</b>
-rwxr--r-- 1 user staff 806 Mar 5 06:21 <b>m</b>
drwxr-xr-x 22 user staff 748 Mar 5 06:21 <b>master</b>
-rwxr--r-- 1 user staff 806 Mar 5 06:21 n1
-rwxr--r-- 1 user staff 804 Mar 5 06:21 n2
-rwxr--r-- 1 user staff 804 Mar 5 06:21 n3
drwxr-xr-x 23 user staff 782 Mar 5 06:21 <b>node1</b>
drwxr-xr-x 23 user staff 782 Mar 5 06:21 <b>node2</b>
-rwxr--r-- 1 user staff 855 Mar 5 06:21 restart_all
-rwxr--r-- 1 user staff 804 Mar 5 06:21 <b>s1</b>
-rwxr--r-- 1 user staff 804 Mar 5 06:21 <b>s2</b>
-rw-r--r-- 1 user staff 173 Mar 5 06:21 sbdescription.json
-rwxr--r-- 1 user staff 1127 Mar 5 06:21 send_kill_all
-rwxr--r-- 1 user staff 1296 Mar 5 06:21 start_all
-rwxr--r-- 1 user staff 1680 Mar 5 06:21 status_all
-rwxr--r-- 1 user staff 1087 Mar 5 06:21 stop_all
-rwxr--r-- 1 user staff 4598 Mar 5 06:21 test_replication
-rwxr--r-- 1 user staff 1315 Mar 5 06:21 test_sb_all
-rwxr--r-- 1 user staff 1100 Mar 5 06:21 use_all
</code></pre><p>Now, let's see how we can change this. We'll start by listing the current defaults</p><pre><code class="language-json">$ dbdeployer defaults show
# <b>Internal values:</b>
{
"version": "0.2.1",
"sandbox-home": "$HOME/sandboxes",
"sandbox-binary": "$HOME/opt/mysql",
"master-slave-base-port": 11000,
"group-replication-base-port": 12000,
"group-replication-sp-base-port": 13000,
"fan-in-replication-base-port": 14000,
"all-masters-replication-base-port": 15000,
"multiple-base-port": 16000,
"group-port-delta": 125,
<b>"master-name": "master",</b>
<b>"master-abbr": "m",</b>
<b>"node-prefix": "node",</b>
<b>"slave-prefix": "slave",</b>
<b>"slave-abbr": "s",</b>
"sandbox-prefix": "msb_",
"master-slave-prefix": "rsandbox_",
"group-prefix": "group_msb_",
"group-sp-prefix": "group_sp_msb_",
"multiple-prefix": "multi_msb_",
"fan-in-prefix": "fan_in_msb_",
"all-masters-prefix": "all_masters_msb_"
}
</code></pre><p>The values that we want to change are <code>master-name</code>, <code>master-abbr</code>, <code>node-prefix</code>, <code>slave-prefix</code>, and <code>slave-abbr</code>. We can export the defaults to a file, and import them after editing the values we want to change.</p><div><pre><code class="language-none">$ dbdeployer defaults export defaults.json
# Defaults exported to file defaults.json
$ vim defaults.json
$ dbdeployer defaults import defaults.json
Defaults imported from defaults.json into $HOME/.dbdeployer/config.json</code></pre></div><p>Now dbdeployer is using the new defaults.</p><br />
<pre><code class="language-json">$ dbdeployer defaults show
# <b>Configuration file: $HOME/.dbdeployer/config.json</b>
{
"version": "0.2.1",
"sandbox-home": "/Users/gmax/sandboxes",
"sandbox-binary": "/Users/gmax/opt/mysql",
"master-slave-base-port": 11000,
"group-replication-base-port": 12000,
"group-replication-sp-base-port": 13000,
"fan-in-replication-base-port": 14000,
"all-masters-replication-base-port": 15000,
"multiple-base-port": 16000,
"group-port-delta": 125,
<b>"master-name": "primary",</b>
<b>"master-abbr": "p",</b>
<b>"node-prefix": "branch",</b>
<b>"slave-prefix": "replica",</b>
<b>"slave-abbr": "r",</b>
"sandbox-prefix": "msb_",
"master-slave-prefix": "rsandbox_",
"group-prefix": "group_msb_",
"group-sp-prefix": "group_sp_msb_",
"multiple-prefix": "multi_msb_",
"fan-in-prefix": "fan_in_msb_",
"all-masters-prefix": "all_masters_msb_"
}
</code></pre>We have now *primary* for *master*, *replica* for *slave*, *branch* for *node*, and the abbreviations for master and slave changed to *p* and *r* respectively. <br />
Let's see how these defaults can play together when we run the same command as we did before for replication. We first remove the previous deployment.<br />
<pre><code>
$ dbdeployer delete rsandbox_8_0_4
List of deployed sandboxes:
$HOME/sandboxes/rsandbox_8_0_4
Running $HOME/sandboxes/rsandbox_8_0_4/stop_all
# executing "stop" on $HOME/sandboxes/rsandbox_8_0_4
executing "stop" on slave 1
executing "stop" on slave 2
executing "stop" on master
Running rm -rf $HOME/sandboxes/rsandbox_8_0_4
Sandbox $HOME/sandboxes/rsandbox_8_0_4 deleted
</code></pre><p>The deployment command is the same as before, but the output changes:</p><pre><code>$ <b>dbdeployer deploy replication 8.0.4</b>
Installing and starting <b>primary</b>
Database installed in $HOME/sandboxes/rsandbox_8_0_4/<b>primary</b>
. sandbox server started
Installing and starting <b>replica</b> 1
Database installed in $HOME/sandboxes/rsandbox_8_0_4/<b>branch</b>1
. sandbox server started
Installing and starting <b>replica</b> 2
Database installed in $HOME/sandboxes/rsandbox_8_0_4/<b>branch</b>2
.. sandbox server started
$HOME/sandboxes/rsandbox_8_0_4/initialize_<b>replicas</b>
initializing <b>replica</b> 1
initializing <b>replica</b> 2
Replication directory installed in $HOME/sandboxes/rsandbox_8_0_4
run 'dbdeployer usage multiple' for basic instructions'
</code></pre><p>This looks already as if our defaults have been adopted. Let's see the sandbox itself:</p><pre><code>$ ls -l ~/sandboxes/rsandbox_8_0_4/
total 152
drwxr-xr-x 23 user staff 782 Mar 5 06:45 <b>branch</b>1
drwxr-xr-x 23 user staff 782 Mar 5 06:45 <b>branch</b>2
-rwxr--r-- 1 user staff 1515 Mar 5 06:45 <b>check_replicas</b>
-rwxr--r-- 1 user staff 1170 Mar 5 06:45 clear_all
-rwxr--r-- 1 user staff 1629 Mar 5 06:45 <b>initialize_replicas</b>
-rwxr--r-- 1 user staff 807 Mar 5 06:45 n1
-rwxr--r-- 1 user staff 806 Mar 5 06:45 n2
-rwxr--r-- 1 user staff 806 Mar 5 06:45 n3
-rwxr--r-- 1 user staff 807 Mar 5 06:45 <b>p</b>
drwxr-xr-x 22 user staff 748 Mar 5 06:45 <b>primary</b>
-rwxr--r-- 1 user staff 806 Mar 5 06:45 <b>r1</b>
-rwxr--r-- 1 user staff 806 Mar 5 06:45 <b>r2</b>
-rwxr--r-- 1 user staff 855 Mar 5 06:45 restart_all
-rw-r--r-- 1 user staff 173 Mar 5 06:45 sbdescription.json
-rwxr--r-- 1 user staff 1137 Mar 5 06:45 send_kill_all
-rwxr--r-- 1 user staff 1308 Mar 5 06:45 start_all
-rwxr--r-- 1 user staff 1700 Mar 5 06:45 status_all
-rwxr--r-- 1 user staff 1097 Mar 5 06:45 stop_all
-rwxr--r-- 1 user staff 4613 Mar 5 06:45 test_replication
-rwxr--r-- 1 user staff 1325 Mar 5 06:45 test_sb_all
-rwxr--r-- 1 user staff 1106 Mar 5 06:45 use_all
</code></pre><p>We see that the new defaults were used and the script names have changed. But the differences are deeper than this. Also the internal values in the scripts were changed accordingly.</p><pre><code>$ ~/sandboxes/rsandbox_8_0_4/test_replication
# <b>primary</b> log: mysql-bin.000001 - Position: 14073 - Rows: 20
# Testing <b>replica</b> #1
ok - <b>replica</b> #1 acknowledged reception of transactions from <b>primary</b>
ok - <b>replica</b> #1 IO thread is running
ok - <b>replica</b> #1 SQL thread is running
ok - Table t1 found on <b>replica</b> #1
ok - Table t1 has 20 rows on #1
# Testing <b>replica</b> #2
ok - <b>replica</b> #2 acknowledged reception of transactions from <b>primary</b>
ok - <b>replica</b> #2 IO thread is running
ok - <b>replica</b> #2 SQL thread is running
ok - Table t1 found on <b>replica</b> #2
ok - Table t1 has 20 rows on #2
# Tests : 10
# failed: 0 ( 0.0%)
# PASSED: 10 (100.0%)
# exit code: 0
</code></pre><p>The test script calls the components with the names that we defined in the new defaults. Let's have a look at what the shortcuts for the master and slaves (now <em>primary</em> and <em>replicas</em>) do:</p><pre><code>$ ~/sandboxes/rsandbox_8_0_4/p
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 35
Server version: 8.0.4-rc-log MySQL Community Server (GPL)
Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
<b>primary</b> [localhost] {msandbox} ((none)) >
$ ~/sandboxes/rsandbox_8_0_4/r1
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 15
Server version: 8.0.4-rc-log MySQL Community Server (GPL)
Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
<b>replica1</b> [localhost] {msandbox} ((none)) >
</code></pre><p>Also the internal prompt has been adapted to the new naming.</p><p>Should we want to revert to the old behavior, we can just reset the defaults:</p><div><pre><code class="language-none">$ dbdeployer defaults reset
#File $HOME/.dbdeployer/config.json removed</code></pre></div><p>The current replication sandbox is left untouched, but the next one will use the default values.</p><p>If we don't want to change the defaults permanently, there is an alternative. The <code>--defaults</code> flag allows us to change defaults on-the-fly just for the command we're running. For example, we could have achieved the same result, without editing the configuration file, using this command:</p><div><pre><code class="language-none"> dbdeployer deploy replication 8.0.4 \
--defaults=master-name:primary \
--defaults=master-abbr:p \
--defaults=slave-prefix:replica \
--defaults=slave-abbr:r \
--defaults=node-prefix:branch</code></pre></div><br />
<p>The syntax for <code>--defaults</code> requires the name of the variable and the new value, separated by a colon. The flag can be used as many times as needed.</p>Giuseppe Maxiahttp://www.blogger.com/profile/15801583338057324813noreply@blogger.com0tag:blogger.com,1999:blog-16959946.post-24680070294044422172018-03-05T05:30:00.000+01:002018-03-05T05:30:12.944+01:00MySQL security for real users<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgigGRwsdgjvZM1H7eu2hWmBT-OJti1v4RnItFVQOUXTpkYoCb_qp6oyYPG73m9W7luCMIt7CMEdChyhTa46xRCk3FuNDoDkktj1N7-8MEpWRpGZhv-KkdNMQ1mzb250kdXrQa3/s1600/mysql_security.png" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgigGRwsdgjvZM1H7eu2hWmBT-OJti1v4RnItFVQOUXTpkYoCb_qp6oyYPG73m9W7luCMIt7CMEdChyhTa46xRCk3FuNDoDkktj1N7-8MEpWRpGZhv-KkdNMQ1mzb250kdXrQa3/s320/mysql_security.png" width="239" height="320" data-original-width="642" data-original-height="860" /></a><br />
<h3 id="toc_0">Security features overview</h3><p>One of <a href="http://oracle.com">Oracle</a>'s tenets is the focus on security. For this reason, when it took over the stewardship of MySQL, it started addressing the most common issues. It was not quick acting, but we have seen real progress:</p><ol><li>MySQL 5.7 has removed the anonymous accounts, which was the greatest threat to security. Because of those accounts, and the default privileges granted to them, users without any privileges could access the "test" database and do serious damage. Additionally, because of the way the privilege engine evaluates accounts, anonymous users could hijack legitimate users, by preventing them to work properly.</li>
<li>The "root" account now comes with a password defined during initialization. This is good news for security, but bad news for how the change was implemented. </li>
<li>There is a new way of setting an options file for connection credentials: the <a href="https://dev.mysql.com/doc/refman/5.7/en/mysql-config-editor.html"><code>mysql_config_editor</code></a> paired with option <a href="https://dev.mysql.com/doc/mysql-utilities/1.5/en/mysql-utils-intro-connspec-mylogin.cnf.html"><code>--login-path</code></a> allows users to store encrypted credentials for secure use. Also here, while we should rejoice for the added security, we can't help feeling that the implementation is yet again far from meeting users needs.</li>
<li>There is an useful warning (introduced in MySQL 5.6) when using a password on the command line, telling users that it is a risk. Also in this case, we have a usability issue: while users care about their production deployments and use option files to avoid using passwords on the command line, there are, nonetheless, a lot of testing scripts, used in safe environment or with non-valuable data, where a password in the command line was not an issue, and the new warning simply screws up the result of those carefully crafted tests. This change, which can't be overcome without modifying the MySQL clients code, needs users to change their existing tests to adapt to the new behavior.</li>
<li>MySQL 8 introduces roles, which simplify the accounts management. There are some minor usability issues, although in general the feature meets expectations.</li>
</ol><p>This is the scenario of the main enhancements in MySQL since 5.6. Each one of them has some usability problems, some minor, some really bad.<br />
We will first have a look at the problems mentioned above, and then examine the root cause for why they have arisen.</p><br />
<h3 id="toc_1">Usability issues</h3><p>I start by noticing that some developers in the MySQL team have been working there for many years, starting with the time when MySQL was a different database and was used really differently. <br />
In those times, managing the database meant that a human (the DBA) would run operations manually, take a look at the result, and adjust when needed. And then, when things went wrong, the same human explored the database system to find out what happened, took action, and went back to sleep.</p><p>Human-centered management leads to human problems: lazy DBA left their databases without password, using the root account, and exposing the server to uninspired attacks; they used passwords on the command line, without caring for options files (or without knowing about them.) Careless DBAs did not deal with anonymous users, leaving a dangerous backdoor in their server. </p><p>Some of the new functionalities introduced in the latest MySQL versions are aimed at this type of users: when you install MySQL, you get a message saying: your root password is ************, and the lazy DBAs have no option but to take note and use it. When they use the password on the command line, the annoying warning forces them to start using an options file or the <code>mysql_config_editor</code>.</p><p>This is all good, but the main problem here is that the DBAs of 10 years ago are on the verge of extinction. They are replaced by a new breed of DBAs who are not lazy, because they can't afford to be, and need to use dozens, hundreds, thousands of databases at once, using configuration management tools that don't require manual intervention, and actually abhor it. In the land of automation, some of the MySQL security enhancements are not seen as a solution, but as new problems.</p><p>Let's see an interesting example: docker containers. </p><p>Using Docker, <a href="https://hub.docker.com/r/mysql/mysql-server/">MySQL images</a> are deployed using a password on the command line. This is done for compatibility with the first implementation of <a href="https://hub.docker.com/_/mysql/">the image maintained by the Docker team</a>, where you deploy with this syntax:</p><div><pre><code class="language-none">docker run -e MYSQL_ROOT_PASSWORD=secret -d mysql</code></pre></div><p>The <code>MYSQL_ROOT_PASSWORD</code> is a directive that becomes an environment variable inside the container, and the server uses it during initialization. As you can imagine, this is not recommended for a secure environment. Then, what's the MySQL team recommendation? They suggest the same strategy used for manual installation: set a directive <code>MYSQL_RANDOM_ROOT_PASSWORD</code> that results in a random password being generated, then collected by the DBA and used. Alternatively, the directive <code>MYSQL_ONETIME_PASSWORD</code> will force the root user to change the password on first connection.</p><p>The above suggestions were designed with the ancient DBA still in mind, while container deployment is even more automated than VMs, and it is based on the principle of immutable objects, i.e. containers that spring up from the cloud ready to run, with no configuration needed, and especially no configuration that requires someone (or some tool) to extract a new password from a log. I proposed <a href="https://datacharmer.blogspot.com/2016/02/a-safer-mysql-box-in-docker.html">a different solution</a>, that would never show passwords on the command line and while it was implemented, but it still feels like a hack to circumvent an inadequate design.</p><p>As a result, the implementation inside the MySQL recommended Docker image uses "--initialize-insecure" to start the server. This is an implicit recognition of the bad design of the initialization feature. What was designed to overcome DBA's laziness becomes an obstacle towards automation.</p><p>We have a similar problem with <code>mysql_config_editor</code>: the tool will create a safe configuration file with credentials for multiple instances, but the password <strong>must be inserted manually</strong>. Consequently, this potentially useful feature doesn't get adopted, because it would be too difficult or impossible to automate properly.</p><p>We have seen that, of the security features that were introduced lately, only a few can be used safely in an automated environment, and all of them have at least one small usability quirk. I have talked about a confusing issue <a href="https://datacharmer.blogspot.com/2015/11/default-users-in-mysql-57.html">related to the removal of anonymous users</a> where in their eagerness of removing the vulnerability the MySQL team removed also the "test" database, which was a consequence, not the cause of the problem. And I have recently talked about <a href="http://datacharmer.blogspot.com/2017/09/revisiting-roles-in-mysql-80.html">roles usability</a> where there are still open problems, like the <a href="https://bugs.mysql.com/bug.php?id=84244">ability of telling roles from users</a> which are apparently not considered a bug by the MySQL team.</p><p>All the above considerations led me to ask: how did we get to this point? There is an active community, and feedback is offered often with plenty of detail. How come we have such an abundance of usability issues? Don't the developers spend time with users at conferences to learn what they expect? Don't they read articles and blog posts about how a new feature meets expectations? Don't they talk to customers who have adopted new features? They certainly do. Then, why the usability problems persist?</p><p>What follows is my observation and speculation on this matter.</p><br />
<h3 id="toc_2">Disconnection between MySQL developers and users community</h3><p>My experience working with system providers has put me in contact with many users. I have seen that in most cases users are very much protective of their current deployment, because it took them long time to get it right, and they don't upgrade unless they don't have another choice. I've seen users test the newer versions, realize that they would break some of their procedures, and defer the upgrade to better times that never come. I remember last year a user with a not so large set of servers was considering an upgrade to MySQL 5.6, while 5.7 had been GA for two years. The reason was a set of incompatibilities that made the upgrade too difficult.</p><p>For companies that deal with thousands of servers, the problem is similar, but exacerbated by the quantity of servers to upgrade and the need to do it without stopping operations. This latest requirement has made some users decide not to use GTID, because it required offline time for a master, and they hadn't had time enough to test the upgrade to MySQL 5.7 that would solve that problem.</p><p>For one reason or the other, many companies upgrade only <strong>two or three years</strong> after a given version became GA. And this is the main problem: until they use it in production, or at least test the version for a projected upgrade, users can't give valuable feedback, the one that is related to usage in production, and when they do, the version for which they provide feedback has been GA for long time, and can't be changed, while the next one is already close to GA, and as such will be untouchable.</p><p>The MySQL team gets feedback on a release from a handful of curious users who don't delay testing until the new version is GA, but don't provide the kind of <strong>important</strong> feedback that get the development team attention, such as deployment in production by large customers. In many cases, large customers are the ones that upgrade several years after GA, and by then their input is difficult to act upon.</p><p>We have then a curious situation, where the warnings given by the early software explorers are confirmed years later by the users to which the MySQL team listens more closely, but by then the next version of the server is already locked in a release schedule that nobody wants to alter to fix usability issues.</p><p>How can we solve this problem? Simple: listen to early software explorers and try to fix problems before GA.</p>Giuseppe Maxiahttp://www.blogger.com/profile/15801583338057324813noreply@blogger.com3tag:blogger.com,1999:blog-16959946.post-32898448190446842522018-03-01T06:00:00.000+01:002018-03-01T06:00:40.860+01:00Using MySQL 8.0: what to expect<img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh5XZ5AOKAwvZn-3pmJ_nzhSeZrJ13m5Mb2991l5tn4bcLzFOD5wIe7y1e6liRTjO1-qXK2Awqb3nAogN9zLiBl2q1iFvbVGdASprWlOzTN8VU1s5QkMXghr_Z56iboEHmKOch6/?imgmax=1600" alt="Mysql8" title="mysql8.png" border="0" width="250" /></p><p><a href="https://dev.mysql.com/doc/refman/8.0/en/mysql-nutshell.html">MySQL 8.0</a> will be GA soon (just my assumption: Oracle doesn't tell me anything about its release plans) and it's time to think about having a look at it. <br />
If this is your first try of MySQL 8, get prepared for several impacting differences from previous versions. </p><p>In this article I won't tell you what you <em>can</em> do with MySQL 8: there is plenty of material about this, including in this very blog. I will instead concentrate on differences from previous versions that users need to know if they want to avoid surprises.</p><h3 id="toc_0">Data Directory</h3><p>Let's start with an observation of the data directory. <br />
After a standard installation, without any additional options, I see the following:</p><h4 id="toc_1">Files that I expected to see</h4><div><pre><code class="language-none">auto.cnf
ib_buffer_pool
ib_logfile0
ib_logfile1
ibdata1
ibtmp1
(dir) mysql
(dir) performance_schema
(dir) sys</code></pre></div><p>These files are also present in 5.7. </p><h4 id="toc_2">Files that are new in 8.0</h4><div><pre><code class="language-none">binlog.000001
binlog.index</code></pre></div><p><em>log-bin</em> is ON by default. You need to remember this if you are using a MySQL server for a benchmark test that used to run without binary logs. </p><div><pre><code class="language-none">ca-key.pem
ca.pem
client-cert.pem
client-key.pem
private_key.pem
public_key.pem
server-cert.pem
server-key.pem</code></pre></div><p>Now the MySQL generates all the certificates needed to run connections securely. This will greatly simplify your task when setting up a new instance.</p><div><pre><code class="language-none">mysql.ibd</code></pre></div><p>This was completely unexpected! The <code>mysql</code> database has now its own tablespace. This is probably due to the new <a href="http://datacharmer.blogspot.com/2017/04/revisiting-hidden-mysql-80-data.html">Data Dictionary</a>, which is implemented in InnoDB. You will notice that all the InnoDB tables in MySQL use this tablespace, not only dictionary tables. This will help keeping administrative data separate from operational data in the rest of the server.</p><div><pre><code class="language-none">undo_001
undo_002</code></pre></div><p>The undo logs have now their own tablespace by default.</p><h3 id="toc_3">Global variables</h3><p>There are a lot of changes in global variables. Here's the list of what will impact your work when you use MySQL 8.0 for the first time:</p><div><pre><code class="language-none">character_set_client utf8mb4
character_set_connection utf8mb4
character_set_database utf8mb4
character_set_results utf8mb4
character_set_server utf8mb4</code></pre></div><p>All character sets are now <code>utf8mb4</code>. In MySQL 5.7, the default values are a mix of <code>utf8</code> and <code>latin1</code>.</p><div><pre><code class="language-none">default_authentication_plugin caching_sha2_password</code></pre></div><p>This is huge. Using this plugin, passwords are stored <a href="https://dev.mysql.com/doc/refman/8.0/en/caching-sha2-pluggable-authentication.html">in a different way</a>, which guarantees more security, but will probably break several workflows among the users. The bad thing about this change implementation is that this password format contains characters that don't display well on screen, and you can see garbled output when inspecting the "user" table.</p><div><pre><code class="language-none">local_infile OFF</code></pre></div><p>Loading local files is now prevented by default. If you have a workflow that requires such operations, you need to enable it.</p><div><pre><code class="language-none">log_bin ON
log_slave_updates ON</code></pre></div><p>We've seen from an inspection of the local directory that binary logging is enabled by default. But also very important is that <code>log_slave_update</code> is enabled. This is important to have slaves ready to replace a master, but will severely affect performance in those scenarios where some slaves were supposed to run without that feature.</p><div><pre><code class="language-none">master_info_repository TABLE
relay_log_info_repository TABLE</code></pre></div><p>Also impacting performance is the setting for replication repositories, which are now on TABLE by default. This is something that should have happened already in MySQL 5.6 and was long overdue.</p><p>Surprisingly, something that DOES NOT get enabled by default is Global Transaction Identifiers (GTID). This is also a legacy from decisions taken in MySQL 5.6. Due to the GTID implementation, enabling them by default is not possible when upgrading from a previous version. With new data in a fresh installation, it is safe to enable GTID from the start.</p><br />
<h3 id="toc_4">Users</h3><br />
<p>There are two new users when the server is created:</p><div><pre><code class="language-none">mysql.infoschema
mysql.session </code></pre></div><p>Theoretically, <code>mysql.session</code> also exists in 5.7, but it was introduced long after GA, so it still qualifies as a novelty.</p><p>Then, when the server starts, you get a grand total of 4 users (<code>root</code> and <code>mysql.sys</code> are inherited from MySQL 5.7.)</p><br />
<h3 id="toc_5">Mixed oddities</h3><br />
<p>When MySQL initializes, i.e. when the server starts for the first time and creates the database, you will notice some slowness, compared to previous versions. This is in part due to the data dictionary, which needs to create and fill 30 tables, but it is not a big deal in terms of performance. In some systems, though, the slowness is so acute that you start worrying about the server being stuck.</p><p>I noticed this problem in my Intel NUC running with SSD storage. In this box, the initialization time took a serious hit:</p><table border="1" cellpadding="1" cellspacing="0"><thead>
<tr> <th>Version</th> <th style="text-align: right">time</th> </tr>
</thead> <tbody>
<tr> <td>5.0.96</td> <td style="text-align: right">1.231s</td> </tr>
<tr> <td>5.1.72</td> <td style="text-align: right">1.346s</td> </tr>
<tr> <td>5.5.52</td> <td style="text-align: right">2.441s</td> </tr>
<tr> <td>5.6.39</td> <td style="text-align: right">5.540s</td> </tr>
<tr> <td>5.7.21</td> <td style="text-align: right">6.080s</td> </tr>
<tr> <td>8.0.3</td> <td style="text-align: right">7.826s</td> </tr>
<tr> <td>8.0.4</td> <td style="text-align: right"><font color="red"><b>38.547s</b></font></td> </tr>
</tbody> </table><p>There is no mistype. The initialization for 8.0.4 lasts 6 times more than 5.7. <br />
This doesn't happen everywhere. On a Mac laptop running on SSD the same operation takes almost 9 seconds, while 5.7 deploys in less than 5. It is still a substantial difference, one that has totally disrupted my regular operations in the NUC. I investigated the matter, and I found the reason. In 8.0, we have a new (hidden) table in the data dictionary, called <code>st_spatial_reference_systems</code>. Up to MySQL 8.0.3, this table was filled using a single transaction containing roughly 5,000 <code>REPLACE INTO</code> statements. It is a lot of data, but it happens quickly. For comparison, in MySQL 8.0.3 the initialization is only 2 seconds slower than 5.7. <br />
The reason for the slowness in 8.0.4 is that there was a new command added to the syntax: <a href="https://dev.mysql.com/doc/refman/8.0/en/create-spatial-reference-system.html">CREATE SPATIAL REFERENCE SYSTEM</a>, which is now used 5,000 times to fill the table that was previously filled with a single transaction. I don't know why someone in the MySQL team thought that changing this operation that is hidden from users was a good idea. The data is contained in the server itself and it goes into a data dictionary table, also not visible to users. I am sure I can find at least two methods to load the data faster. I was told that this <em>glitch</em> will be fixed in the next release. I'm waiting.</p><p>Speaking of initialization, the <code>mysql_install_db</code> script has been removed for good in 8.0. If you are still using it instead of the recommended <code>mysqld --initialize</code>, you should adapt asap.</p><p>This list is far from being exhaustive. I recommend reading <a href="https://dev.mysql.com/doc/refman/8.0/en/mysql-nutshell.html">What's new in MySQL 8</a> before upgrading. <br />
If you are impatient, <a href="http://datacharmer.blogspot.com/2018/02/meet-dbdeployer-new-sandbox-maker.html">dbdeployer</a> can help you test MySQL 8 quickly and safely.</p><br />
<br />
Giuseppe Maxiahttp://www.blogger.com/profile/15801583338057324813noreply@blogger.com2tag:blogger.com,1999:blog-16959946.post-26059316688776279762018-02-28T10:10:00.000+01:002018-02-28T10:10:04.775+01:00The confusing strategy for MySQL shell<h3>Where the hell is it?</h3><p>The <a href="https://dev.mysql.com/doc/refman/8.0/en/mysql-shell.html">MySQL shell</a> is a potentially useful tool that has been intentionally made difficult to use properly.</p><p>It was introduced, with much fanfare, with the <a href="https://dev.mysql.com/doc/refman/5.7/en/document-store.html">MySQL Document Store</a>, as <i>THE</i> tool to bridge the SQL and no-SQL worlds. The release was less than satisfactory, though: MySQL 5.7.12 introduced a new feature (the X-protocol plugin) bundled with the server. The maturity of the plugin was unclear, as it popped out of the unknown into a GA release, without any public testing. It was allegedly GA quality, although the quantity of bug reports that were filed soon after the release proved otherwise. The maturity of the shell was known as "development preview", and so we had a supposedly GA feature that could only be used with an alpha quality tool.</p><p>The situation with the MySQL shell got worse in a few months. A new product was brewing (<a href="https://dev.mysql.com/doc/refman/5.7/en/group-replication.html">MySQL Group Replication</a>) and went rapidly from something released in the <a href="https://labs.mysql.com/">Labs</a> without docs to being part of the regular server distribution, and it was evolving into a more complex and ambitious project (the <a href="https://dev.mysql.com/doc/refman/5.7/en/mysql-innodb-cluster-userguide.html">InnoDB Cluster</a>) which used the MySQL shell as its main tool.</p><p>Since the announcement of InnoDB Cluster, using the MySQL shell has been a nightmare. You saw examples in blog posts and presentations, and when you tried them at home, they did not work. There were different releases of MySQL shell <b>with the same version number</b> but different capabilities, depending on whether they were released through the main downloads site or through the labs.</p><p>When I asked why the shell wasn't distributed with the server, like the other tools, I was told that a non-GA product could not be released with a GA server. Considering that the Document Store is still walking around with a <a href="https://dev.mysql.com/doc/refman/5.7/en/document-store-legalnotice.html">Pre-Production status legal notice</a>, this was an odd excuse.</p><p>Still, I kept waiting, trying to figure out how to pair a given version of MySQL shell with a given version of the server. Unlike the server, there are no release notes for the shell, so every release was a surprising experience.</p><p>Eventually, the MySQL shell reached the GA state, with which merit I can't tell. Given the obstacles in the path to its usage, I doubt it has had any serious testing from the community. Despite the state being GA, it keeps being released separately, leaving the puzzled users with the ungrateful task of determining with which server version that shell could be used safely.</p><p>With the upcoming release of MySQL 8.0, a new version of MySQL shell appeared, with a colorful prompt and new features that the GA shell doesn't have. The public perception of the tool keeps getting more confused. In the presentations given by the MySQL team we see the new shell doing wonders, while the GA shell keeps its monochromatic features. Shall I use the 8.0.x shell with a 5.7 server or should I stick with the 1.0 version?</p><p>In MySQL 8.0, the situation is still divided. Both products (the server and the shell) are, as of today, not GA yet. It would make sense to finally end the craziness and put the two things together, so that users don't have to hunt around for the right shell version. But the two products are still released separately.</p><br />
<h3>How can I do stuff with MySQL shell?</h3><p>So far, we have only seen the availability of the shell. What about the functionality?</p><p>I have heard that Oracle wants to convert the shell into the only tool to deal with MySQL. I can't prove it, as Oracle doesn't release its development plans to the public, but I can see the emphasis on the shell in talks and articles authored by MySQL team engineers. If this is the plan, I think it needs a lot more work.</p><p>If you try to use MySQL shell the same way as the regular "mysql" client, you get in trouble soon.</p><pre><code>mysqlsh --user root --password=msandbox --port=5721 --host 127.0.0.1
mysqlx: [Warning] Using a password on the command line interface can be insecure.
Creating a Session to 'root@127.0.0.1:5721'
Your MySQL connection id is 38
Server version: 5.7.21 MySQL Community Server (GPL)
No default schema selected; type \use <schema> to set one.
MySQL Shell 1.0.11
Copyright (c) 2016, 2017, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type '\help' or '\?' for help; '\quit' to exit.
Currently in JavaScript mode. Use \sql to switch to SQL mode and execute queries.
</code></pre><p>I see two problems here:</p><ul><li>The warning about the password on the command line is legitimate. The trouble is that there is no alternative. mysqlsh <b>does not support</b> --defaults-file, and there is no way of giving a password other than directly at invocation. There is an option "--passwords-from-stdin" which does not seem to work, and even if it did, I can't see the advantage of using the password from a pipe.</li>
<li>The default mode is Javascript. I can see that this makes operations simpler when you want to perform setup tasks for InnoDB Cluster, but certainly doesn't help me to use this tool as the primary drive for database management. There is a "--sql" option that does what I expect, but if this is not the default, I can't see this replacement being very successful.</li>
<li>Due to the previous items, using the tool in batch mode (with -e "SQL commands") is impossible, as every invocation will start with the freaking password warning.</li>
</ul><p>I'm afraid that it's too late to take action for MySQL 8.0. The MySQL team is probably packaging the GA release while I write these notes. But I offer some suggestions nonetheless.</p><br />
<h3>Wish list</h3><br />
<ol><li>Package MySQL shell with the server. Past experience shows that the MySQL team keeps adding features into a GA release, thus exposing users to the risk of getting the wrong tool for the job. Having the shell and the server in the same tarball will help users pick the right version for the task. This is similar to what happens with <i>mysqldump</i>: using the tool from 5.5 with a 5.7+ server will not work properly. There is no reason for mysqlsh to be treated differently. </li>
<li>Make sure that all the features of the mysql client work seamlessly in mysqlsh. Perhaps run the test suite replacing <i>mysql</i> with <i>mysqlsh</i> and pick up from there. </li>
<li>Make the MySQL shell compatible with other tools. Specifically, it should support option files (--defaults-file, --defaults-extra-file, --defaults-group-suffix, --no-defaults) </li>
</ol><p>In short, if the plan is to replace <i>mysql</i> with <i>mysqlsh</i>, put the thing in the open, and please make sure it can do what users can reasonably expect.</p>Giuseppe Maxiahttp://www.blogger.com/profile/15801583338057324813noreply@blogger.com12tag:blogger.com,1999:blog-16959946.post-76001031550579638722018-02-21T05:32:00.000+01:002018-02-28T15:11:32.333+01:00Meet dbdeployer: the new sandbox maker<br />
<h3 id="toc_1">How it happened</h3><br />
<p>A few years ago I started thinking about refactoring <a href="https://github.com/datacharmer/mysql-sandbox">MySQL-Sandbox</a>. I got lots of ideas and a name for the project (<a href="https://github.com/datacharmer/dbdeployer">dbdeployer</a>) but went no further. The initial idea (this was 2013!) was to rewrite the project in Ruby: I had been using Ruby at work and it looked like a decent replacement for Perl. My main problem was the difficulty of installation in an uncontrolled environment. If you have control over your environment (it's your laptop or you are in charge of the server configuration via Puppet or similar) then the task is easy. But if you ever need to deploy somewhere with little or no notice, it becomes a problem: there are servers where Perl is not installed, and is common that the server also have a policy forbidding all scripting languages from being deployed. Soon I found out that Ruby has the same problem as Perl. In the meantime, my work also required heavy involvement with Python, and I started thinking that maybe it would be a better choice than Ruby.<br />
My adventures with deployment continued. In some places, I would find old versions of Perl, Ruby, Python, and no way of replacing them easily. I also realized that, if I bit the bullet and wrote my tools in C or C++, my distribution problems would not end, as I had to deal with library dependencies and conflict with existing ones.<br />
At the end of 2017 I finally did what I had postponed for so long: I took a serious look at <a href="https://golang.org">Go</a>, and I decided that it was the best candidate for solving the distribution problem. I had a few adjustment problems, as the Go philosophy is different from my previously used languages, but the advantages were so immediate that I was hooked. Here's what I found compelling:</p><ul><li>Shift in responsibility: with all the other languages I have used, the user is responsible for providing the working environment, such as installing libraries, the language itself, solve conflicts, and so on, until the program can work. With Go, the responsibility is on the developers only: they are supposed to know how to collect the necessary packages and produce a sound executable. Users only need to <a href="https://github.com/datacharmer/dbdeployer/releases">download the executable</a> and run it.</li>
<li>Ease of deployment. A Go executable doesn't have dependencies. Binaries can be compiled for several platforms from a single origin (I can build Linux executables in my Mac and vice versa) and they just work.</li>
<li>Ease of development. Go is a strongly typed language, and has a different approach at code structure than Perl or Python. But this doesn't slow down my coding: it forces me to write better code, resulting in something that is at the same time more robust and easy to extend.</li>
<li>Wealth of packages. Go has an amazingly active community, and there is <a href="https://github.com/avelino/awesome-go">an enormous amount of packages</a> ready for anything. </li>
</ul><br />
<h3 id="toc_2">What is dbdeployer?</h3><br />
<p><b>UPDATE 28-Feb-2018</b>: The commands "single", "replication", and "multiple" are now subcommand of "deploy". Also, "templates" is now a subcommand of "defaults".</p><p>The first goal of <a href="https://github.com/datacharmer/dbdeployer">dbdeployer</a> is to replace MySQL-Sandbox completely. As such, it has all the main features of MySQL Sandbox, and many more (See the full list of features at the end of this text.)</p><p>You can deploy a single sandbox, or multiple unrelated sandboxes, or several servers in replication. That you could do also with MySQL-Sandbox. The first difference is in the command structure:</p><div><pre><code class="language-none">$ dbdeployer
dbdeployer makes MySQL server installation an easy task.
Runs single, multiple, and replicated sandboxes.
Usage:
dbdeployer [command]
Available Commands:
admin administrative tasks
delete delete an installed sandbox
global Runs a given command in every sandbox
help Help about any command
multiple create multiple sandbox
replication create replication sandbox
sandboxes List installed sandboxes
single deploys a single sandbox
templates Admin operations on templates
unpack unpack a tarball into the binary directory
usage Shows usage of installed sandboxes
versions List available versions
Flags:
--base-port int Overrides default base-port (for multiple sandboxes)
--bind-address string defines the database bind-address (default "127.0.0.1")
--config string configuration file (default "$HOME/.dbdeployer/config.json")
--custom-mysqld string Uses an alternative mysqld (must be in the same directory as regular mysqld)
-p, --db-password string database password (default "msandbox")
-u, --db-user string database user (default "msandbox")
--expose-dd-tables In MySQL 8.0+ shows data dictionary tables
--force If a destination sandbox already exists, it will be overwritten
--gtid enables GTID
-h, --help help for dbdeployer
-i, --init-options strings mysqld options to run during initialization
--keep-auth-plugin in 8.0.4+, does not change the auth plugin
--keep-server-uuid Does not change the server UUID
--my-cnf-file string Alternative source file for my.sandbox.cnf
-c, --my-cnf-options strings mysqld options to add to my.sandbox.cnf
--port int Overrides default port
--post-grants-sql strings SQL queries to run after loading grants
--post-grants-sql-file string SQL file to run after loading grants
--pre-grants-sql strings SQL queries to run before loading grants
--pre-grants-sql-file string SQL file to run before loading grants
--remote-access string defines the database access (default "127.%")
--rpl-password string replication password (default "rsandbox")
--rpl-user string replication user (default "rsandbox")
--sandbox-binary string Binary repository (default "$HOME/opt/mysql")
--sandbox-directory string Changes the default sandbox directory
--sandbox-home string Sandbox deployment direcory (default "$HOME/sandboxes")
--skip-load-grants Does not load the grants
--use-template strings [template_name:file_name] Replace existing template with one from file
--version version for dbdeployer
Use "dbdeployer [command] --help" for more information about a command.
</code></pre></div><br />
<p>MySQL-Sandbox was created in 2006, and its structure changed as needed, without a real plan. dbdeployer, instead, was designed to have a hierarchical command structure, similar to git or docker, to give users a better feeling. As a result, it has a leaner set of commands, a non-awkward way of using options, and offers a better control of the operations out of the box.</p><p>For example, here's how we would start to run sandboxes:</p><div><pre><code class="language-none">$ dbdeployer --unpack-version=8.0.4 unpack mysql-8.0.4-rc-linux-glibc2.12-x86_64.tar.gz
Unpacking tarball mysql-8.0.4-rc-linux-glibc2.12-x86_64.tar.gz to $HOME/opt/mysql/8.0.4
.........100.........200.........292</code></pre></div><p>The first (mandatory) operation is to expand binaries from a tarball. By default, the files will be expanded to <strong>$HOME/opt/mysql</strong>. Once this is done, we can create sandboxes at will, with simple commands:</p><div><pre><code class="language-none">$ dbdeployer single 8.0.4
Database installed in $HOME/sandboxes/msb_8_0_4
run 'dbdeployer usage single' for basic instructions'
. sandbox server started
$ dbdeployer replication 8.0.4
[...]
Replication directory installed in /$HOME/sandboxes/rsandbox_8_0_4
run 'dbdeployer usage multiple' for basic instructions'
$ dbdeployer multiple 8.0.4
[...]
Multiple directory installed in $HOME/sandboxes/multi_msb_8_0_4
run 'dbdeployer usage multiple' for basic instructions'
$ dbdeployer sandboxes
msb_8_0_4 : single 8.0.4 [8004]
multi_msb_8_0_4 : multiple 8.0.4 [24406 24407 24408]
rsandbox_8_0_4 : master-slave 8.0.4 [19405 19406 19407]</code></pre></div><br />
<p>Three differences between dbdeployer and MySQL-Sandbox: </p><ul><li>There is only one executable, with different commands;</li>
<li>After each deployment, there is a suggestion on how to get help about the sandbox usage.</li>
<li>There is a command that displays which sandboxes were installed, the kind of deployment, and the ports in use. This will be useful when the ports increase, as in group replication.</li>
</ul><br />
<p>Here's another take, after deploying group replication:</p><div><pre><code class="language-none">$ dbdeployer sandboxes
group_msb_8_0_4 : group-multi-primary 8.0.4 [20405 20530 20406 20531 20407 20532]
group_sp_msb_8_0_4 : group-single-primary 8.0.4 [21405 21530 21406 21531 21407 21532]
msb_8_0_4 : single 8.0.4 [8004]
multi_msb_8_0_4 : multiple 8.0.4 [24406 24407 24408]
rsandbox_8_0_4 : master-slave 8.0.4 [19405 19406 19407]</code></pre></div><br />
<p>A few more differences from MySQL-Sandbox are the "global" and "delete" commands. <br />
The "global" command can broadcast a command to all the sandboxes. You can start, stop, restart all sandboxes at once, or run a query everywhere.</p><div><pre><code class="language-none">$ dbdeployer global use "select @@server_id, @@port, @@server_uuid"
# Running "use_all" on group_msb_8_0_4
# server: 1
@@server_id @@port @@server_uuid
100 20405 00020405-1111-1111-1111-111111111111
# server: 2
@@server_id @@port @@server_uuid
200 20406 00020406-2222-2222-2222-222222222222
# server: 3
@@server_id @@port @@server_uuid
300 20407 00020407-3333-3333-3333-333333333333
# Running "use_all" on group_sp_msb_8_0_4
# server: 1
@@server_id @@port @@server_uuid
100 21405 00021405-1111-1111-1111-111111111111
# server: 2
@@server_id @@port @@server_uuid
200 21406 00021406-2222-2222-2222-222222222222
# server: 3
@@server_id @@port @@server_uuid
300 21407 00021407-3333-3333-3333-333333333333
# Running "use" on msb_8_0_4
@@server_id @@port @@server_uuid
1 8004 00008004-0000-0000-0000-000000008004
[...]</code></pre></div><br />
<p>You can run the commands manually. <code>dbdeployer usage</code> will show which commands are available for every sandbox.</p><div><pre><code class="language-none">$ dbdeployer usage single
USING A SANDBOX
Change directory to the newly created one (default: $SANDBOX_HOME/msb_VERSION
for single sandboxes)
[ $SANDBOX_HOME = $HOME/sandboxes unless modified with flag --sandbox-home ]
The sandbox directory of the instance you just created contains some handy
scripts to manage your server easily and in isolation.
"./start", "./status", "./restart", and "./stop" do what their name suggests.
start and restart accept parameters that are eventually passed to the server.
e.g.:
./start --server-id=1001
./restart --event-scheduler=disabled
"./use" calls the command line client with the appropriate parameters,
Example:
./use -BN -e "select @@server_id"
./use -u root
"./clear" stops the server and removes everything from the data directory,
letting you ready to start from scratch. (Warning! It's irreversible!)</code></pre></div><br />
<p>When you don't need the sandboxes anymore, you can dismiss them with a single command:</p><div><pre><code class="language-none">$ dbdeployer delete ALL
Deleting the following sandboxes
$HOME/sandboxes/group_msb_8_0_4
$HOME/sandboxes/group_sp_msb_8_0_4
$HOME/sandboxes/msb_8_0_4
$HOME/sandboxes/multi_msb_8_0_4
$HOME/sandboxes/rsandbox_8_0_4
Do you confirm? y/[N]</code></pre></div><p>There is an option to skip the confirmation, which is useful for scripting unattended tests.</p><br />
<h3 id="toc_3">Customization</h3><br />
<p>One of the biggest problems with MySQL-Sandbox was that most of the functioning is hard-coded, and the scripts needed to run the sandboxes are generated in different places, so that extending or modifying features became more and more difficult. When I designed dbdeployer, I gave myself the goal of making the tool easy to change, and the code easy to understand and extend.</p><p>For this reason, I organized everything related to code generation (the scripts that initialize and run the sandboxes) in a collection of templates and default variables that are publicly visible and modifiable.</p><div><pre><code class="language-none">$ dbdeployer templates -h
The commands in this section show the templates used
to create and manipulate sandboxes.
Usage:
dbdeployer templates [command]
Aliases:
templates, template, tmpl, templ
Available Commands:
describe Describe a given template
export Exports all templates to a directory
import imports all templates from a directory
list list available templates
reset Removes all template files
show Show a given template</code></pre></div><br />
<p>You can list the templates on the screen.</p><div><pre><code class="language-none">$ dbdeployer templates list single
[single] replication_options : Replication options for my.cnf
[single] load_grants_template : Loads the grants defined for the sandbox
[single] grants_template57 : Grants for sandboxes from 5.7+
[single] grants_template5x : Grants for sandboxes up to 5.6
[single] my_template : Prefix script to run every my* command line tool
[single] show_binlog_template : Shows a binlog for a single sandbox
[single] use_template : Invokes the MySQL client with the appropriate options
[single] clear_template : Remove all data from a single sandbox
[single] restart_template : Restarts the database (with optional mysqld arguments)
[single] start_template : starts the database in a single sandbox (with optional mysqld arguments)
[single] stop_template : Stops a database in a single sandbox
[single] send_kill_template : Sends a kill signal to the database
[single] show_relaylog_template : Show the relaylog for a single sandbox
[single] Copyright : Copyright for every sandbox script
[single] expose_dd_tables : Commands needed to enable data dictionary table usage
[single] init_db_template : Initialization template for the database
[single] grants_template8x : Grants for sandboxes from 8.0+
[single] add_option_template : Adds options to the my.sandbox.cnf file and restarts
[single] test_sb_template : Tests basic sandbox functionality
[single] sb_include_template : TBD
[single] gtid_options : GTID options for my.cnf
[single] my_cnf_template : Default options file for a sandbox
[single] status_template : Shows the status of a single sandbox</code></pre></div><br />
<p>Then it's possible to examine template contents:</p><div><pre><code class="language-none">$ dbdeployer templates describe --with-contents init_db_template
# Collection : single
# Name : init_db_template
# Description : Initialization template for the database
# Notes : This should normally run only once
# Length : 656
##START init_db_template
#!/bin/bash
{{.Copyright}}
# Generated by dbdeployer {{.AppVersion}} using {{.TemplateName}} on {{.DateTime}}
BASEDIR={{.Basedir}}
export LD_LIBRARY_PATH=$BASEDIR/lib:$BASEDIR/lib/mysql:$LD_LIBRARY_PATH
export DYLD_LIBRARY_PATH=$BASEDIR/lib:$BASEDIR/lib/mysql:$DYLD_LIBRARY_PATH
SBDIR={{.SandboxDir}}
DATADIR=$SBDIR/data
cd $SBDIR
if [ -d $DATADIR/mysql ]
then
echo "Initialization already done."
echo "This script should run only once."
exit 0
fi
{{.InitScript}} \
{{.InitDefaults}} \
--user={{.OsUser}} \
--basedir=$BASEDIR \
--datadir=$DATADIR \
--tmpdir={{.Tmpdir}} {{.ExtraInitFlags}}
##END init_db_template</code></pre></div><p>The one above is the template that generates the initialization script. In MySQL-Sandbox, this was handled in the code, and it was difficult to figure out what went wrong when the initialization failed. The Go language has an excellent support for code generation using templates, and with just a fraction of its features I implemented a few dozen scripts which I am able to modify with ease. Here's what the deployed script looks like</p><pre><code>#!/bin/bash
# DBDeployer - The MySQL Sandbox
# Copyright (C) 2006-2018 Giuseppe Maxia
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Generated by dbdeployer 0.1.24 using init_db_template on Tue Feb 20 14:45:29 CET 2018
BASEDIR=$HOME/opt/mysql/8.0.4
export LD_LIBRARY_PATH=$BASEDIR/lib:$BASEDIR/lib/mysql:$LD_LIBRARY_PATH
export DYLD_LIBRARY_PATH=$BASEDIR/lib:$BASEDIR/lib/mysql:$DYLD_LIBRARY_PATH
SBDIR=$HOME/sandboxes/msb_8_0_4
DATADIR=$SBDIR/data
cd $SBDIR
if [ -d $DATADIR/mysql ]
then
echo "Initialization already done."
echo "This script should run only once."
exit 0
fi
$HOME/opt/mysql/8.0.4/bin/mysqld \
--no-defaults \
--user=$USER \
--basedir=$BASEDIR \
--datadir=$DATADIR \
--tmpdir=$HOME/sandboxes/msb_8_0_4/tmp \
--initialize-insecure --default_authentication_plugin=mysql_native_password
</code></pre><br />
<p>Let's see the quick-and-dirty usage. If you want to change a template and use it just once, do the following:</p><ol><li><code>$ dbdeployer templates show init_db_template</code></li>
<li>Save it to a file init_db.txt and edit it. Be careful, though: removing or altering essential labels may block the sandbox initialization.</li>
<li>Use the template file in the next command: </li>
</ol><p><code>$ dbdeployer single 8.0.4 --use-template=init_db_template:init_db.txt</code> </p><p>For more permanent results, when you'd like to change a template, or several ones, permanently, you can use the export/import commands</p><br />
<ol><li>List the templates related to replication (<code>dbdeployer templates list replication</code>) </li>
<li>Export the templates to the directory "mydir" <code>$ dbdeployer templates export replication mydir</code></li>
<li>edit the templates you want to change inside "mydir/replication"</li>
<li>Import the templates <code>dbdeployer templates import replication mydir</code></li>
</ol><p>The templates will end inside <code>$HOME/.dbdeployer/templates_$DBDEPLOYER_VERSION</code> and dbdeployer will load then instead of using the ones stored internally. The next time that one of those templates will be needed, it will be collected from the file. If you run <code>dbdeployer templates list</code> or <code>describe</code>, the ones saved to file will be marked with <code>{F}</code>.<br />
To go back to the built-in behavior, simply run <code>dbdeployer templates reset</code></p><p>In addition to templates, dbdeployer uses a set of values when creating sandboxes. Like templates, this set is used from internal store, but it can be exported to a configuration file. </p><div><pre><code class="language-none">$ dbdeployer admin show
# Internal values:
{
"version": "0.1.24",
"sandbox-home": "$HOME/sandboxes",
"sandbox-binary": "$HOME/opt/mysql",
"master-slave-base-port": 11000,
"group-replication-base-port": 12000,
"group-replication-sp-base-port": 13000,
"multiple-base-port": 16000,
"group-port-delta": 125,
"sandbox-prefix": "msb_",
"master-slave-prefix": "rsandbox_",
"group-prefix": "group_msb_",
"group-sp-prefix": "group_sp_msb_",
"multiple-prefix": "multi_msb_"
}</code></pre></div><br />
<p>The values named <code>*-base-port</code> are used to calculate the port for each node in a multiple deployment. The calculation goes:</p><div><pre><code class="language-none">sandbox_port + base_port + (revision_number * 100)</code></pre></div><p>So, for example, when deploying replication for 5.7.21, the sandbox port would be 5721, and the final base port will be calculated as follows:</p><div><pre><code class="language-none">5721 + 11000 + 21 * 100 = 18821</code></pre></div><p>This number will be incremented for each node in the cluster, so that the master will get 18822, and the first slave 18823. </p><p>Using the commands <code>dbdeployer admin export</code> and <code>import</code> you can customize the default values in a way similar to what we saw for the templates.</p><br />
<h3 id="toc_4">Thanks</h3><br />
<p>I'd like to thank:</p><ul><li><a href="http://ronaldbradford.com/">Ronald Bradford</a> and <a href="http://www.proxysql.com/">René Cannaò</a>, for priceless advice on the usability when the tool was in its early stage of development;</li>
<li><a href="http://code.openark.org">Shlomi Noach</a>, for <a href="http://code.openark.org/blog/mysql/using-dbdeployer-in-ci-tests">adopting dbdeployer</a> even before it was feature complete;</li>
<li><a href="http://lefred.be">Frédéric "lefred" Descamps</a>, for allowing me to present dbdeployer at the pre-FOSDEM MySQL event;</li>
<li>The Go community, for the exciting environment offered to newcomers.</li>
</ul><br />
<h3 id="toc_5">A note about unpacking MySQL tarball</h3><p>When using MySQL tarballs, we may have some problems due to the enormous size that the tarballs have reached. Look at this:</p><pre><code>690M 5.5.52
1.2G 5.6.39
2.5G 5.7.21
3.6G 8.0.0
1.3G 8.0.1
1.5G 8.0.2
1.9G 8.0.3
1.9G 8.0.4
</code></pre><p>This becomes a serious problem when you want to unpack the tarball inside a low-resource virtual machine or a Docker container. I have asked the MySQL team to provide reduced tarballs, possibly in a fixed location, so that sandboxes creation could be fully automated. I was told that something will be done soon. In the meantime, <a href="https://github.com/datacharmer/mysql-docker-minimal">I provide such reduced tarballs</a>, which have a more reasonable size:</p><pre><code> 49M 5.5.52
61M 5.6.39
346M 5.7.21
447M 8.0.0
462M 8.0.1
254M 8.0.2
270M 8.0.3
244M 8.0.4
</code></pre><p>Using these reduced tarballs, which are conveniently packed in a docker container (<b>datacharmer/mysql-sb-full</b> contains all major MySQL versions), I have automated dbdeployer tests with minimal storage involvement, and that improves the test speed as well.</p><h3 id="toc_6">Detailed list of features</h3><br />
<table border="1" cellpadding="5" cellspacing="0" style="border-collapse:collapse;" ><thead>
<tr> <th>Feature</th> <th style="text-align: left">MySQL-Sandbox</th> <th style="text-align: left">dbdeployer</th> <th style="text-align: left">dbdeployer planned</th> </tr>
</thead> <tbody>
<tr> <td>Single sandbox deployment</td> <td style="text-align: left">yes</td> <td style="text-align: left">yes</td> <td style="text-align: left"></td> </tr>
<tr> <td>unpack command</td> <td style="text-align: left">sort of <sup id="fnref1"><a href="#fn1" rel="footnote">1</a></sup></td> <td style="text-align: left">yes</td> <td style="text-align: left"></td> </tr>
<tr> <td>multiple sandboxes</td> <td style="text-align: left">yes</td> <td style="text-align: left">yes</td> <td style="text-align: left"></td> </tr>
<tr> <td>master-slave replication</td> <td style="text-align: left">yes</td> <td style="text-align: left">yes</td> <td style="text-align: left"></td> </tr>
<tr> <td>"force" flag</td> <td style="text-align: left">yes</td> <td style="text-align: left">yes</td> <td style="text-align: left"></td> </tr>
<tr> <td>pre-post grants SQL action</td> <td style="text-align: left">yes</td> <td style="text-align: left">yes</td> <td style="text-align: left"></td> </tr>
<tr> <td>initialization options</td> <td style="text-align: left">yes</td> <td style="text-align: left">yes</td> <td style="text-align: left"></td> </tr>
<tr> <td>my.cnf options</td> <td style="text-align: left">yes</td> <td style="text-align: left">yes</td> <td style="text-align: left"></td> </tr>
<tr> <td>custom my.cnf</td> <td style="text-align: left">yes</td> <td style="text-align: left">yes</td> <td style="text-align: left"></td> </tr>
<tr> <td>friendly UUID generation</td> <td style="text-align: left">yes</td> <td style="text-align: left">yes</td> <td style="text-align: left"></td> </tr>
<tr> <td>global commands</td> <td style="text-align: left">yes</td> <td style="text-align: left">yes</td> <td style="text-align: left"></td> </tr>
<tr> <td>test replication flow</td> <td style="text-align: left">yes</td> <td style="text-align: left">yes</td> <td style="text-align: left"></td> </tr>
<tr> <td>delete command</td> <td style="text-align: left">yes <sup id="fnref2"><a href="#fn2" rel="footnote">2</a></sup></td> <td style="text-align: left">yes</td> <td style="text-align: left"></td> </tr>
<tr> <td>group replication SP</td> <td style="text-align: left">no</td> <td style="text-align: left">yes</td> <td style="text-align: left"></td> </tr>
<tr> <td>group replication MP</td> <td style="text-align: left">no</td> <td style="text-align: left">yes</td> <td style="text-align: left"></td> </tr>
<tr> <td>prevent port collision</td> <td style="text-align: left">no</td> <td style="text-align: left">yes <sup id="fnref3"><a href="#fn3" rel="footnote">3</a></sup></td> <td style="text-align: left"></td> </tr>
<tr> <td>visible initialization</td> <td style="text-align: left">no</td> <td style="text-align: left">yes <sup id="fnref4"><a href="#fn4" rel="footnote">4</a></sup></td> <td style="text-align: left"></td> </tr>
<tr> <td>visible script templates</td> <td style="text-align: left">no</td> <td style="text-align: left">yes <sup id="fnref5"><a href="#fn5" rel="footnote">5</a></sup></td> <td style="text-align: left"></td> </tr>
<tr> <td>replaceable templates</td> <td style="text-align: left">no</td> <td style="text-align: left">yes <sup id="fnref6"><a href="#fn6" rel="footnote">6</a></sup></td> <td style="text-align: left"></td> </tr>
<tr> <td>configurable defaults</td> <td style="text-align: left">no</td> <td style="text-align: left">yes <sup id="fnref7"><a href="#fn7" rel="footnote">7</a></sup></td> <td style="text-align: left"></td> </tr>
<tr> <td>list of source binaries</td> <td style="text-align: left">no</td> <td style="text-align: left">yes <sup id="fnref8"><a href="#fn8" rel="footnote">8</a></sup></td> <td style="text-align: left"></td> </tr>
<tr> <td>list of installed sandboxes</td> <td style="text-align: left">no</td> <td style="text-align: left">yes <sup id="fnref9"><a href="#fn9" rel="footnote">9</a></sup></td> <td style="text-align: left"></td> </tr>
<tr> <td>test script per sandbox</td> <td style="text-align: left">no</td> <td style="text-align: left">yes <sup id="fnref10"><a href="#fn10" rel="footnote">10</a></sup></td> <td style="text-align: left"></td> </tr>
<tr> <td>integrated usage help</td> <td style="text-align: left">no</td> <td style="text-align: left">yes <sup id="fnref11"><a href="#fn11" rel="footnote">11</a></sup></td> <td style="text-align: left"></td> </tr>
<tr> <td>custom abbreviations</td> <td style="text-align: left">no</td> <td style="text-align: left">yes <sup id="fnref12"><a href="#fn12" rel="footnote">12</a></sup></td> <td style="text-align: left"></td> </tr>
<tr> <td>version flag</td> <td style="text-align: left">no</td> <td style="text-align: left">yes <sup id="fnref13"><a href="#fn13" rel="footnote">13</a></sup></td> <td style="text-align: left"></td> </tr>
<tr> <td>fan-in</td> <td style="text-align: left">no</td> <td style="text-align: left">no</td> <td style="text-align: left">yes <sup id="fnref14"><a href="#fn14" rel="footnote">14</a></sup></td> </tr>
<tr> <td>all-masters</td> <td style="text-align: left">no</td> <td style="text-align: left">no</td> <td style="text-align: left">yes <sup id="fnref15"><a href="#fn15" rel="footnote">15</a></sup></td> </tr>
<tr> <td>Galera/PXC/NDB</td> <td style="text-align: left">no</td> <td style="text-align: left">no</td> <td style="text-align: left">yes <sup id="fnref18"><a href="#fn18" rel="footnote">18</a></sup></td> </tr>
<tr> <td>finding free ports</td> <td style="text-align: left">yes</td> <td style="text-align: left">no</td> <td style="text-align: left">yes</td> </tr>
<tr> <td>pre-post grants shell action</td> <td style="text-align: left">yes</td> <td style="text-align: left">no</td> <td style="text-align: left">maybe</td> </tr>
<tr> <td>getting remote tarballs</td> <td style="text-align: left">yes</td> <td style="text-align: left">no</td> <td style="text-align: left">yes</td> </tr>
<tr> <td>circular replication</td> <td style="text-align: left">yes</td> <td style="text-align: left">no</td> <td style="text-align: left">no <sup id="fnref16"><a href="#fn16" rel="footnote">16</a></sup></td> </tr>
<tr> <td>master-master (circular)</td> <td style="text-align: left">yes</td> <td style="text-align: left">no</td> <td style="text-align: left">no</td> </tr>
<tr> <td>Windows support</td> <td style="text-align: left">no</td> <td style="text-align: left">no</td> <td style="text-align: left">no <sup id="fnref17"><a href="#fn17" rel="footnote">17</a></sup></td> </tr>
</tbody> </table><br />
<div class="footnotes"><hr><ol><li id="fn1">It's achieved using --export_binaries and then abandoning the operation. <a href="#fnref1" rev="footnote">↩</a><br />
</li>
<li id="fn2">Uses the sbtool command <a href="#fnref2" rev="footnote">↩</a><br />
</li>
<li id="fn3">dbdeployer sandboxes store their ports in a description JSON file, which allows the tool to get a list of used ports and act before a conflict happens. <a href="#fnref3" rev="footnote">↩</a><br />
</li>
<li id="fn4">The initialization happens with a script that is generated and stored in the sandbox itself. Users can inspect the <em>init_db</em> script and see what was executed. <a href="#fnref4" rev="footnote">↩</a><br />
</li>
<li id="fn5">All sandbox scripts are generated using templates, which can be examined and eventually changed and re-imported. <a href="#fnref5" rev="footnote">↩</a><br />
</li>
<li id="fn6">See also note 5. Using the flag --use-template you can replace an existing template on-the-fly. Group of templates can be exported and imported after editing. <a href="#fnref6" rev="footnote">↩</a><br />
</li>
<li id="fn7">Defaults can be exported to file, and eventually re-imported after editing. <a href="#fnref7" rev="footnote">↩</a><br />
</li>
<li id="fn8">This is little more than using an O.S. file listing, with the added awareness of the source directory. <a href="#fnref8" rev="footnote">↩</a><br />
</li>
<li id="fn9">Using the description files, this command lists the sandboxes with their topology and used ports. <a href="#fnref9" rev="footnote">↩</a><br />
</li>
<li id="fn10">It's a basic test that checks whether the sandbox is running and is using the expected port. <a href="#fnref10" rev="footnote">↩</a><br />
</li>
<li id="fn11">The "usage" command will show basic commands for single and multiple sandboxes. <a href="#fnref11" rev="footnote">↩</a><br />
</li>
<li id="fn12">The abbreviations file allows user to define custom shortcuts for frequently used commands. <a href="#fnref12" rev="footnote">↩</a><br />
</li>
<li id="fn13">Strangely enough, this simple feature was never implemented for MySQL-Sandbox, while it was one of the first additions to dbdeployer. <a href="#fnref13" rev="footnote">↩</a><br />
</li>
<li id="fn14">Will use the multi source technology introduced in MySQL 5.7. <a href="#fnref14" rev="footnote">↩</a><br />
</li>
<li id="fn15">Same as n. 13. <a href="#fnref15" rev="footnote">↩</a><br />
</li>
<li id="fn16">Circular replication should not be used anymore. There are enough good alternatives (multi-source, group replication) to avoid this old technology. <a href="#fnref16" rev="footnote">↩</a><br />
</li>
<li id="fn17">I don't do Windows, but you can fork the project if you do. <a href="#fnref17" rev="footnote">↩</a><br />
</li>
<li id="fn18">For Galera/PXC and MySQL Cluster I have ideas, but I may need help to implement. <a href="#fnref18" rev="footnote">↩</a><br />
</li>
</ol></div>Giuseppe Maxiahttp://www.blogger.com/profile/15801583338057324813noreply@blogger.com2tag:blogger.com,1999:blog-16959946.post-64727858996695957142017-09-24T15:15:00.000+02:002017-09-24T15:15:16.957+02:00Revisiting roles in MySQL 8.0<p>In my previous <a href="https://datacharmer.blogspot.com/2017/03/mysql-80-roles.html">article about roles</a> I said that one of the problems with role usage is that roles need to be activated before they kick in. Let's recap briefly what the problem is: </p><div><pre><code class="language-none">## new session, as user `root`
mysql [localhost] {root} ((none)) > create role viewer;
Query OK, 0 rows affected (0.01 sec)
mysql [localhost] {root} ((none)) > grant select on *.* to viewer;
Query OK, 0 rows affected (0.01 sec)
mysql [localhost] {root} ((none)) > create user see_it_all identified by 'msandbox';
Query OK, 0 rows affected (0.01 sec)
mysql [localhost] {root} ((none)) > grant viewer to see_it_all;
Query OK, 0 rows affected (0.01 sec)
## NEW session, as user `see_it_all`
mysql [localhost] {see_it_all} ((none)) > use test
ERROR 1044 (42000): Access denied for user 'see_it_all'@'%' to database 'test'
mysql [localhost] {see_it_all} ((none)) > show grants\G
*************************** 1. row ***************************
Grants for see_it_all@%: GRANT USAGE ON *.* TO `see_it_all`@`%`
*************************** 2. row ***************************
Grants for see_it_all@%: GRANT `viewer`@`%` TO `see_it_all`@`%`
2 rows in set (0.00 sec)
mysql [localhost] {see_it_all} (test) > select current_role();
+----------------+
| current_role() |
+----------------+
| NONE |
+----------------+
1 row in set (0.00 sec)</code></pre></div><p>We can create a simple role that gives read-only access to most database objects, and assign it to a user. However, when the new user tries accessing one database, it is rejected. The problem is that the role must be <strong>activated</strong>, either permanently, or for the current session. </p><div><pre><code class="language-none">mysql [localhost] {see_it_all} ((none)) > set role viewer;
Query OK, 0 rows affected (0.00 sec)
mysql [localhost] {see_it_all} (test) > select current_role();
+----------------+
| current_role() |
+----------------+
| `viewer`@`%` |
+----------------+
1 row in set (0.00 sec)
mysql [localhost] {see_it_all} ((none)) > use test
Database changed
mysql [localhost] {see_it_all} (test) > show grants\G
*************************** 1. row ***************************
Grants for see_it_all@%: GRANT SELECT ON *.* TO `see_it_all`@`%`
*************************** 2. row ***************************
Grants for see_it_all@%: GRANT `viewer`@`%` TO `see_it_all`@`%`
2 rows in set (0.00 sec)</code></pre></div><br />
<p>The main issue here is that the role is not active immediately. If we grant a given privilege to a user, the user will be able to operate under that privilege straight away. If we grant a role, instead, the user can't use it immediately. Roles need to be <strong>activated</strong>, either by the giver or by the receiver. </p><br />
<h2 id="toc_0">Auto activating roles</h2><br />
<p>In <a href="https://dev.mysql.com/doc/relnotes/mysql/8.0/en/news-8-0-2.html">MySQL 8.0.2</a> there are two new features related to roles, and one of them addresses the main problem we have just seen. When we use <a href="https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_activate_all_roles_on_login"><code>activate_all_roles_on_login</code></a>, all roles become active when the user starts a session, regardless of any role activation that may pre-exist. Let's try. In the previous example, as root, we issue this command: </p><br />
<div><pre><code class="language-none">mysql [localhost] {root} ((none)) > set global activate_all_roles_on_login=ON;
Query OK, 0 rows affected (0.00 sec)</code></pre></div><p>Then, we connect as user <code>see_it_all</code> </p><div><pre><code class="language-none">mysql [localhost] {see_it_all} ((none)) > select current_role();
+----------------+
| current_role() |
+----------------+
| `viewer`@`%` |
+----------------+
1 row in set (0.00 sec)
mysql [localhost] {see_it_all} ((none)) > use test
Database changed</code></pre></div><br />
<p>The role is active. The current role can be overridden temporarily using <code>SET ROLE</code>: </p><div><pre><code class="language-none">mysql [localhost] {see_it_all} ((none)) > use test
Database changed
mysql [localhost] {see_it_all} (test) > set role none;
Query OK, 0 rows affected (0.00 sec)
mysql [localhost] {see_it_all} (test) > select current_role();
+----------------+
| current_role() |
+----------------+
| NONE |
+----------------+
1 row in set (0.00 sec)
mysql [localhost] {see_it_all} (test) > show tables;
ERROR 1044 (42000): Access denied for user 'see_it_all'@'%' to database 'test'</code></pre></div><p>This is a good option, which can further simplify DBAs work. There are, as usual, a few caveats: </p><ul><li>This option has effect only on login, i.e. when the user starts a new session. Users that are already logged in when the option is changed will not be affected until they re-connect.</li>
<li>Use of this option can have adverse effects when using combinations of roles. If the DBA intent is to give users several roles that should be used separately, using <code>activate_all_roles_on_login</code> will make the paradigm more difficult to use. Let's see an example:</li>
</ul><br />
<div><pre><code class="language-sql">CREATE ROLE payroll_viewer ;
GRANT SELECT ON payroll.* TO payroll_viewer;
CREATE ROLE payroll_updater;
GRANT CREATE, INSERT, UPDATE, DELETE ON payroll.* TO payroll_updater;
CREATE ROLE personnel_viewer ;
GRANT SELECT ON personnel.* TO personnel_viewer;
CREATE ROLE personnel_updater;
GRANT CREATE, INSERT, UPDATE, DELETE ON personnel.* TO personnel_updater;
CREATE ROLE payroll;
GRANT payroll_updater, payroll_viewer, personnel_viewer to payroll;
CREATE ROLE personnel;
GRANT personnel_updater, personnel_viewer to personnel;
CREATE USER pers_user identified by 'msandbox';
CREATE USER pers_manager identified by 'msandbox';
CREATE USER pay_user identified by 'msandbox';
GRANT personnel to pers_user;
GRANT personnel, payroll_viewer to pers_manager;
GRANT payroll to pay_user;
SET DEFAULT ROLE personnel TO pers_user;
SET DEFAULT ROLE personnel TO pers_manager;
SET DEFAULT ROLE payroll TO pay_user;</code></pre></div><br />
<p>In the above situation, we want the user <code>pers_manager</code> to see the personnel records by default, but she needs to manually activate payroll_viewer to see the payroll.<br />
If we set <code>activate_all_roles_on_login</code>, <code>pers_manager</code> would be able to see payroll info without further action. </p><br />
<h2 id="toc_1">Mandatory roles</h2><br />
<p>Another option introduced in 8.0.2 is <a href="https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_mandatory_roles"><code>mandatory_roles</code></a>. This variable can be set with a list of roles. When set, the roles in the list will be added to the privileges of all users, including future ones.<br />
<br />
Here's an example of how this feature could be useful. We want a schema containing data that should be accessible to all users, regardless of their privileges. </p><div><pre><code class="language-sql">CREATE SCHEMA IF NOT EXISTS company;
DROP TABLE IF EXISTS company.news;
CREATE TABLE company.news(
id int not null auto_increment primary key,
news_type ENUM('INFO', 'WARNING', 'ALERT'),
contents MEDIUMTEXT);
DROP ROLE IF EXISTS news_reader;
CREATE ROLE news_reader;
GRANT SELECT ON company.* TO news_reader;
SET PERSIST mandatory_roles = news_reader;</code></pre></div><br />
<p>In this example, every user that starts a session after <code>mandatory_roles</code> was set will be able to access the "company" schema and read the news from there.<br />
<br />
There are at least two side effects of this feature: </p><ul><li>When a role is included in the list of mandatory roles, it can't be dropped.</li></ul><div><pre><code class="language-none">mysql [localhost] {root} (mysql) > drop role news_reader;
ERROR 4527 (HY000): The role `news_reader`@`%` is a mandatory role and can't be revoked or dropped.
The restriction can be lifted by excluding the role identifier from the global variable mandatory_roles.</code></pre></div><ul><li>users who have already a broad access that include the privileges in the mandatory role will nonetheless have the global role show up in the user list of grants. For example, here is how 'root'@'localhost' grants look like:</li>
</ul><div><pre><code class="language-none">mysql [localhost] {root} ((none)) > show grants \G
*************************** 1. row ***************************
Grants for root@localhost: GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, RELOAD,
SHUTDOWN, PROCESS, FILE, REFERENCES, INDEX, ALTER, SHOW DATABASES, SUPER,
CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, REPLICATION SLAVE, REPLICATION CLIENT,
CREATE VIEW, SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, CREATE USER, EVENT, TRIGGER,
CREATE TABLESPACE, CREATE ROLE, DROP ROLE ON *.* TO `root`@`localhost` WITH GRANT
OPTION
*************************** 2. row ***************************
Grants for root@localhost: GRANT BACKUP_ADMIN,BINLOG_ADMIN,CONNECTION_ADMIN,
ENCRYPTION_KEY_ADMIN,GROUP_REPLICATION_ADMIN,PERSIST_RO_VARIABLES_ADMIN,
REPLICATION_SLAVE_ADMIN,RESOURCE_GROUP_ADMIN,RESOURCE_GROUP_USER,ROLE_ADMIN,
SET_USER_ID,SYSTEM_VARIABLES_ADMIN,XA_RECOVER_ADMIN ON *.* TO `root`@`localhost`
WITH GRANT OPTION
*************************** 3. row ***************************
Grants for root@localhost: GRANT <b>SELECT ON `company`.*</b> TO `root`@`localhost`
*************************** 4. row ***************************
Grants for root@localhost: GRANT PROXY ON ''@'' TO 'root'@'localhost' WITH GRANT OPTION
*************************** 5. row ***************************
Grants for root@localhost: GRANT <b>`news_reader`@`%`</b> TO `root`@`localhost`
5 rows in set (0.00 sec)</code></pre></div><br />
<h2 id="toc_2">More gotchas</h2><p>There are several commands for setting roles. One of them uses <a href="https://dev.mysql.com/doc/refman/8.0/en/alter-user.html"><code>ALTER USER</code></a>, while the rest uses a <strong>SET</strong> command. </p><ul><li><p>First gotcha: <a href="https://dev.mysql.com/doc/refman/8.0/en/set-role.html"><code>SET ROLE</code></a> and <a href="https://dev.mysql.com/doc/refman/8.0/en/set-default-role.html"><code>SET DEFAULT ROLE</code></a> don't need an <u>equals</u> sign (=). The syntax is similar to <a href="ttps://dev.mysql.com/doc/refman/8.0/en/set-character-set.html"><code>SET CHARACTER SET</code></a>, not to <a href="https://dev.mysql.com/doc/refman/8.0/en/set-variable.html"><code>SET</code> variable</a>. This is a bit confusing, because another security related command (<a href="ttps://dev.mysql.com/doc/refman/8.0/en/set-password.html"><code>SET PASSWORD</code></a>) requires the '=' in the assignment. </p></li>
<li><p>Now, for the really entertaining part, here's a list of commands that can give any DBA an headache.</p></li>
</ul><br />
<table border="1"><thead>
<tr> <th>Command</th> <th>meaning</th> </tr>
</thead> <tbody>
<tr> <td><strong><code>SET ROLE</code></strong> role_name</td> <td>Activates the role role_name for the current session.</td> </tr>
<tr> <td><strong><code>SET DEFAULT ROLE</code></strong> role_name</td> <td>Sets the role role_name as default permanently.</td> </tr>
<tr> <td><strong><code>SET ROLE DEFAULT</code></strong></td> <td>Activates the default role for the current session.</td> </tr>
</tbody> </table><br />
<h2 id="toc_3">State of bugs</h2><br />
<p>Community members have reported several bugs about roles. While I am happy of the MySQL team response concerning the usability of roles (the automatic activation came after I gave feedback) I am less thrilled by seeing that none of the <a href="https://bugs.mysql.com/search.php?search_for=role&status%5B%5D=All&severity=&limit=30&order_by=&cmd=display&phpver=8.0&os=0&os_details=&bug_age=0&tags=&similar=&target=&last_updated=0&defect_class=all&workaround_viability=all&impact=all&fix_risk=all&fix_effort=all&triageneeded=">public bugs reported on this matter</a> have been addressed.<a href="https://bugs.mysql.com/bug.php?id=85561">Bug#85561</a> is particularly vexing. I reported that users can be assigned non-existing roles as default. I was answered with a sophism about the inner being of a default role, and the bug report was closed with a "Won't fix" state. I disagree with this characterisation. The behaviour that I reported is a bug because it allows users to write a wrong statement without a warning or an error. I hope the team will reconsider and take action to improve the usability of roles.</p>Giuseppe Maxiahttp://www.blogger.com/profile/15801583338057324813noreply@blogger.com10tag:blogger.com,1999:blog-16959946.post-30970732714700658542017-05-28T10:51:00.002+02:002017-05-28T10:51:51.557+02:00How to break MySQL InnoDB cluster<p>A few weeks ago I started experimenting with <a href="https://datacharmer.blogspot.com.es/2017/05/getting-to-know-mysql-innodb-cluster.html">MySQL InnoDB cluster</a>. As part of the testing, I tried to kill a node to see what happens to the cluster.</p><p>The good news is that the cluster is resilient. When the primary node goes missing, the cluster replaces it immediately, and operations continue. This is one of the features of an High Availability system, but this feature alone does not define the usefulness or the robustness of the system. In one of my previous jobs, I worked at testing a <a href="http://continuent.com/">commercial HA system</a> and I've learned a few things about what makes a reliable system.</p><p>Armed with this knowledge, I did some more experiments with InnoDB Cluster. The attempt from my previous article had no other expectation than seeing operations continue with ease (primary node replacement.) In this article, I examine a few more features of an HA system:</p><ul><li>Making sure that a failed primary node does not try to force itself back into the cluster;</li>
<li>Properly welcoming a failed node into the cluster;</li>
<li>Handling a <a href="https://en.wikipedia.org/wiki/Split-brain_(computing)">Split Brain</a> cluster.</li>
</ul><p>To explore the above features (or lack of) we are going to simulate some mundane occurrences. We start with the same cluster seen in the previous article, using <a href="https://github.com/mattlord/Docker-InnoDB-Cluster">Docker InnoDB Cluster</a>. The initial state is </p><pre><code>{
"clusterName": "testcluster",
"defaultReplicaSet": {
"name": "default",
"primary": "mysqlgr1:3306",
"status": "OK",
"statusText": "Cluster is ONLINE and can tolerate up to ONE failure.",
"topology": {
"mysqlgr1:3306": {
"address": "mysqlgr1:3306",
"mode": "R/W",
"readReplicas": {},
"role": "HA",
"status": "ONLINE"
},
"mysqlgr2:3306": {
"address": "mysqlgr2:3306",
"mode": "R/O",
"readReplicas": {},
"role": "HA",
"status": "ONLINE"
},
"mysqlgr3:3306": {
"address": "mysqlgr3:3306",
"mode": "R/O",
"readReplicas": {},
"role": "HA",
"status": "ONLINE"
}
}
}
}
</code></pre><p>The first experiment is to restart a non-primary node</p><pre><code>$ docker restart mysqlgr2
</code></pre><p>and see what happens to the cluster</p><pre><code>$ ./tests/check_cluster.sh | grep 'primary\|address\|status'
"primary": "mysqlgr1:3306",
"status": "OK_NO_TOLERANCE",
"statusText": "Cluster is NOT tolerant to any failures. 1 member is not active",
"address": "mysqlgr1:3306",
"status": "ONLINE"
"address": "mysqlgr2:3306",
"status": "(MISSING)"
"address": "mysqlgr3:3306",
"status": "ONLINE"
</code></pre><p>The cluster detects that one member is missing. But after a few seconds, it goes back to normality:</p><pre><code>$ ./tests/check_cluster.sh | grep 'primary\|address\|status'
"primary": "mysqlgr1:3306",
"status": "OK",
"statusText": "Cluster is ONLINE and can tolerate up to ONE failure.",
"address": "mysqlgr1:3306",
"status": "ONLINE"
"address": "mysqlgr2:3306",
"status": "ONLINE"
"address": "mysqlgr3:3306",
"status": "ONLINE"
</code></pre><p>This looks good. Now, let's do the same to the primary node</p><pre><code>$ docker restart mysqlgr1
$ ./tests/check_cluster.sh 2| grep 'primary\|address\|status'
"primary": "mysqlgr2:3306",
"status": "OK_NO_TOLERANCE",
"statusText": "Cluster is NOT tolerant to any failures. 1 member is not active",
"address": "mysqlgr1:3306",
"status": "(MISSING)"
"address": "mysqlgr2:3306",
"status": "ONLINE"
"address": "mysqlgr3:3306",
"status": "ONLINE"
</code></pre><p>As before, the cluster detects that a node is missing, and excludes it from the cluster. Since it was the primary node, another one becomes primary.</p><p>However, this time the node does not come back in the cluster. Checking the cluster status again after several minutes, node 1 is still reported missing. This is <strong>not a bug</strong>. This is a feature of well behaved HA systems: <strong>a primary node that has been already replaced should not come back to the cluster automatically</strong>. </p><p>Also this experiment was good. Now, for the interesting part, let's see the <strong>Split-Brain</strong> situation. </p><p><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhYAtOmXkmt2IiAWoHEhNeeS7oaJ_9B6vi_jG2E8Ig1Dt-asTYsT4cNIkPMBuomC_OANzv2Nvc7Dl0C3xQV14LqTo6Znudp9QYWEiusosB3BbuXlmFVINTYsRu67jZ-o3dRRtuT/?imgmax=1600" alt="Np brain 987746 000000" title="np_brain_987746_000000.png" border="0" width="200" /></p><p>At this moment, there are two parts of the cluster, and each one sees it in a different way. The view from the current primary node is the one reported above and what we would expect: node 1 is not available. But if we ask the cluster status to node 1, we get a different situation:</p><pre><code>$ ./tests/check_cluster.sh 1 | grep 'primary\|address\|status'
"primary": "mysqlgr1:3306",
"status": "OK_NO_TOLERANCE",
"statusText": "Cluster is NOT tolerant to any failures. 2 members are not active",
"address": "mysqlgr1:3306",
"status": "ONLINE"
"address": "mysqlgr2:3306",
"status": "(MISSING)"
"address": "mysqlgr3:3306",
"status": "(MISSING)"
</code></pre><p>Node 1 thinks it's the primary, and two nodes are missing. Node 2 and three think that node 1 is missing.</p><p>In a sane system, the logical way to operate is to admit the failed node back into the cluster, after checking that it is safe to do so. In the InnoDB cluster management there is a <a href="https://dev.mysql.com/doc/dev/mysqlsh-api-javascript/classmysqlsh_1_1dba_1_1_cluster.html">rejoinInstance method</a> that allows us to get an instance back:</p><pre><code>$ docker exec -it mysqlgr2 mysqlsh --uri root@mysqlgr2:3306 -p$(cat secretpassword.txt)
mysql-js> cluster = dba.getCluster()
<Cluster:testcluster>
mysql-js> cluster.rejoinInstance('mysqlgr1:3306')
Rejoining the instance to the InnoDB cluster. Depending on the original
problem that made the instance unavailable, the rejoin operation might not be
successful and further manual steps will be needed to fix the underlying
problem.
Please monitor the output of the rejoin operation and take necessary action if
the instance cannot rejoin.
Please provide the password for 'root@mysqlgr1:3306':
Rejoining instance to the cluster ...
The instance 'root@mysqlgr1:3306' was successfully rejoined on the cluster.
The instance 'mysqlgr1:3306' was successfully added to the MySQL Cluster.
</code></pre><p>Sounds good, eh? Apparently, we have node 1 back in the fold. Let's check:</p><pre><code>$ ./tests/check_cluster.sh 2| grep 'primary\|address\|status'
"primary": "mysqlgr2:3306",
"status": "OK_NO_TOLERANCE",
"statusText": "Cluster is NOT tolerant to any failures. 1 member is not active",
"address": "mysqlgr1:3306",
"status": "(MISSING)"
"address": "mysqlgr2:3306",
"status": "ONLINE"
"address": "mysqlgr3:3306",
"status": "ONLINE"
</code></pre><p>Nope. Node 1 is still missing. And if we try to rescan the cluster, we see that the rejoin call was not effective:</p><pre><code>mysql-js> cluster.rescan()
Rescanning the cluster...
Result of the rescanning operation:
{
"defaultReplicaSet": {
"name": "default",
"newlyDiscoveredInstances": [],
"unavailableInstances": [
{
"host": "mysqlgr1:3306",
"label": "mysqlgr1:3306",
"member_id": "6bd04911-4374-11e7-b780-0242ac170002"
}
]
}
}
The instance 'mysqlgr1:3306' is no longer part of the HA setup. It is either offline or left the HA group.
You can try to add it to the cluster again with the cluster.rejoinInstance('mysqlgr1:3306') command or you can remove it from the cluster configuration.
Would you like to remove it from the cluster metadata? [Y|n]: n
</code></pre><p>It's curious (and frustrating) that we get a recommendation to run the very same function that we've attempted a minute ago.</p><p>But, just as a devilish thought, let's try the same experiment from the invalid cluster.</p><pre><code>$ docker exec -it mysqlgr1 mysqlsh --uri root@mysqlgr1:3306 -p$(cat secretpassword.txt)
mysql-js> cluster = dba.getCluster()
<Cluster:testcluster>
mysql-js> cluster.rejoinInstance('mysqlgr2:3306')
Rejoining the instance to the InnoDB cluster. Depending on the original
problem that made the instance unavailable, the rejoin operation might not be
successful and further manual steps will be needed to fix the underlying
problem.
Please monitor the output of the rejoin operation and take necessary action if
the instance cannot rejoin.
Please provide the password for 'root@mysqlgr2:3306':
Rejoining instance to the cluster ...
The instance 'root@mysqlgr2:3306' was successfully rejoined on the cluster.
The instance 'mysqlgr2:3306' was successfully added to the MySQL Cluster.
mysql-js> cluster.status()
{
"clusterName": "testcluster",
"defaultReplicaSet": {
"name": "default",
"primary": "mysqlgr1:3306",
"status": "OK_NO_TOLERANCE",
"statusText": "Cluster is NOT tolerant to any failures. 1 member is not active",
"topology": {
"mysqlgr1:3306": {
"address": "mysqlgr1:3306",
"mode": "R/W",
"readReplicas": {},
"role": "HA",
"status": "ONLINE"
},
"mysqlgr2:3306": {
"address": "mysqlgr2:3306",
"mode": "R/O",
"readReplicas": {},
"role": "HA",
"status": "ONLINE"
},
"mysqlgr3:3306": {
"address": "mysqlgr3:3306",
"mode": "R/O",
"readReplicas": {},
"role": "HA",
"status": "(MISSING)"
}
}
}
}
</code></pre><p>Now <strong>this</strong> was definitely <strong>not supposed to happen</strong>. The former failed node has invited a healthy node into its minority cluster and the operation succeeded!</p><p>The horrible part? This illegal operation succeeded into reconciling the views from node 1 and node2. Now also node 2 thinks that node1 is again the primary node, and node 3 (which was minding its own business and never had any accidents) is considered missing:</p><pre><code>$ ./tests/check_cluster.sh 2| grep 'primary\|address\|status'
"primary": "mysqlgr1:3306",
"status": "OK_NO_TOLERANCE",
"statusText": "Cluster is NOT tolerant to any failures. 1 member is not active",
"address": "mysqlgr1:3306",
"status": "ONLINE"
"address": "mysqlgr2:3306",
"status": "ONLINE"
"address": "mysqlgr3:3306",
"status": "(MISSING)"
</code></pre><p>And node 3 all of a sudden finds itself in the role of failed node, while it had had nothing to do about the previous operations:</p><pre><code>$ ./tests/check_cluster.sh 3| grep 'primary\|address\|status'
"primary": "mysqlgr3:3306",
"status": "OK_NO_TOLERANCE",
"statusText": "Cluster is NOT tolerant to any failures. 2 members are not active",
"address": "mysqlgr1:3306",
"status": "(MISSING)"
"address": "mysqlgr2:3306",
"status": "(MISSING)"
"address": "mysqlgr3:3306",
"status": "ONLINE"
</code></pre><p>In short, while we were attempting to fix a split brain, we ended up with a different split brain, and an unexpected node promotion. This is clearly a bug, and I hope the MySQL team can make the system more robust. </p>Giuseppe Maxiahttp://www.blogger.com/profile/15801583338057324813noreply@blogger.com4tag:blogger.com,1999:blog-16959946.post-13321394675202969602017-05-08T00:09:00.000+02:002017-05-08T00:09:38.471+02:00Getting to know MySQL InnoDB cluster, the new kid in the block<p><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg4ZL5tVe5vs7dB-_vTrBbK_5d-BRmduLXhFosuuNCi-8hZ9UrujI1db9nlbAZamOucMdGe8NUHIy0Cs4VYaU65c6ps-Nc92aX3EuCKbgIVWOP4Jtl9F3iuPFBLDSRACgue7dFF/?imgmax=1600" alt="Innodb cluster3" title="innodb_cluster3.png" border="0" width="400" /></p><p><a href="https://dev.mysql.com/doc/refman/5.7/en/mysql-innodb-cluster-userguide.html">InnoDB Cluster</a> was <a href="http://mysqlserverteam.com/mysql-innodb-cluster-ga/">released as GA</a> a few weeks ago. I remember the initial announcement of the product at OOW 2016, promising a seamless solution for replication and high availability with great ease of use. I was a bit disappointed to see that, at GA release time, the InnoDB Cluster is a patchwork of three separate products (Group Replication, MySQL Router, MySQL Shell) which the users have to collect and install separately.</p><p>Given this situation, I was very pleased when <a href="https://twitter.com/mattalord">Matthew Lord</a> published <a href="https://github.com/mattlord/Docker-InnoDB-Cluster">Docker-InnoDB-Cluster</a>, an image for Docker that contains everything you need to get the system up and running. The associated scripts make the experience even easier: not only we don't have to hunt for components, but the cluster deployment procedure is completely automated.</p><h2>Installation</h2><p>The process is painless. After cloning the <a href="https://github.com/mattlord/Docker-InnoDB-Cluster">repository</a> the start script takes care of everything. It will create a network, deploy three database nodes, and fire up the router.</p><pre><code>$ ./start_three_node_cluster.sh
Creating dedicated grnet network...
# network grnet already exists
NETWORK ID NAME DRIVER SCOPE
8fa365076198 grnet bridge local
Bootstrapping the cluster...
12fb4bd975c2fb2e7152ed64e12d2d212bbc9f1d3b39d715ea0c73eeb37fed45
Container mysqlgr1 is up at Sun May 7 22:02:38 CEST 2017
Starting mysqlgr1 container...
Starting mysqlgr1 container...
MySQL init process done. Ready for start up.
Getting GROUP_NAME...
Adding second node...
a2b504ea1920d35b1555f65de24cd364fc1bc7a6ac87ca4eb32f4c02f5afce7c
Container mysqlgr2 is up at Sun May 7 22:02:48 CEST 2017
Starting mysqlgr2 container...
Starting mysqlgr2 container...
MySQL init process done. Ready for start up.
Adding third node...
393d46b9a1795531d99f68645087393a54b2463ef88b9b3c4cbe735c1527fe57
Container mysqlgr3 is up at Sun May 7 22:02:58 CEST 2017
Starting mysqlgr3 container...
Starting mysqlgr3 container...
MySQL init process done. Ready for start up.
Sleeping 10 seconds to give the cluster time to sync up
Adding a router...
830c3125bad70b09b057cee370ee490bcb88b1d4a1bfec347cda847942f3b56e
Container mysqlrouter1 is up at Sun May 7 22:03:17 CEST 2017
Done!
Connecting to the InnoDB cluster...
</code></pre><p>Most of the configuration (which has been simplified thanks to the usage of MySQL shell to add nodes) is handled inside the container initialization script. Just a few details are needed in the cluster deployment script to get the result.</p><p>The deployment script will also invoke the mysql shell in one of the nodes to show the status of the cluster:</p><pre><code>Creating a Session to 'root@mysqlgr1:3306'
Classic Session successfully established. No default schema selected.
{
"clusterName": "testcluster",
"defaultReplicaSet": {
"name": "default",
"primary": "mysqlgr1:3306",
"status": "OK",
"statusText": "Cluster is ONLINE and can tolerate up to ONE failure.",
"topology": {
"mysqlgr1:3306": {
"address": "mysqlgr1:3306",
"mode": "R/W",
"readReplicas": {},
"role": "HA",
"status": "ONLINE"
},
"mysqlgr2:3306": {
"address": "mysqlgr2:3306",
"mode": "R/O",
"readReplicas": {},
"role": "HA",
"status": "ONLINE"
},
"mysqlgr3:3306": {
"address": "mysqlgr3:3306",
"mode": "R/O",
"readReplicas": {},
"role": "HA",
"status": "ONLINE"
}
}
}
}
</code></pre><p>The above status is the result of <code>dba.getCluster().status()</code>, which is a convenient way of collecting a bunch of information about the cluster and then present them in a compact JSON structure. If you enable the general log prior to running this command, you would see something like this:</p><pre><code>select count(*) from performance_schema.replication_group_members where MEMBER_ID = @@server_uuid AND MEMBER_STATE IS NOT NULL AND MEMBER_STATE != 'OFFLINE';
select count(*) from mysql_innodb_cluster_metadata.instances where mysql_server_uuid = @@server_uuid;
SELECT @@server_uuid, VARIABLE_VALUE FROM performance_schema.global_status WHERE VARIABLE_NAME = 'group_replication_primary_member';
SELECT MEMBER_STATE FROM performance_schema.replication_group_members WHERE MEMBER_ID = '0030396b-3300-11e7-a8b6-0242ac170002';
SELECT CAST(SUM(IF(member_state = 'UNREACHABLE', 1, 0)) AS SIGNED) AS UNREACHABLE, COUNT(*) AS TOTAL FROM performance_schema.replication_group_members;
select count(*) from performance_schema.replication_group_members where MEMBER_ID = @@server_uuid AND MEMBER_STATE IS NOT NULL AND MEMBER_STATE != 'OFFLINE';
select count(*) from mysql_innodb_cluster_metadata.instances where mysql_server_uuid = @@server_uuid;
SELECT @@server_uuid, VARIABLE_VALUE FROM performance_schema.global_status WHERE VARIABLE_NAME = 'group_replication_primary_member';
SELECT MEMBER_STATE FROM performance_schema.replication_group_members WHERE MEMBER_ID = '0030396b-3300-11e7-a8b6-0242ac170002';
SELECT CAST(SUM(IF(member_state = 'UNREACHABLE', 1, 0)) AS SIGNED) AS UNREACHABLE, COUNT(*) AS TOTAL FROM performance_schema.replication_group_members;
SELECT cluster_id, cluster_name, default_replicaset, description, options, attributes FROM mysql_innodb_cluster_metadata.clusters WHERE attributes->'$.default' = true;
show databases like 'mysql_innodb_cluster_metadata';
SELECT replicaset_name, topology_type FROM mysql_innodb_cluster_metadata.replicasets WHERE replicaset_id = 7;
select count(*) from performance_schema.replication_group_members where MEMBER_ID = @@server_uuid AND MEMBER_STATE IS NOT NULL AND MEMBER_STATE != 'OFFLINE';
select count(*) from mysql_innodb_cluster_metadata.instances where mysql_server_uuid = @@server_uuid;
SELECT @@server_uuid, VARIABLE_VALUE FROM performance_schema.global_status WHERE VARIABLE_NAME = 'group_replication_primary_member';
SELECT MEMBER_STATE FROM performance_schema.replication_group_members WHERE MEMBER_ID = '0030396b-3300-11e7-a8b6-0242ac170002';
SELECT CAST(SUM(IF(member_state = 'UNREACHABLE', 1, 0)) AS SIGNED) AS UNREACHABLE, COUNT(*) AS TOTAL FROM performance_schema.replication_group_members;
SELECT @@group_replication_single_primary_mode;
SHOW STATUS LIKE 'group_replication_primary_member';
SELECT mysql_server_uuid, instance_name, role, MEMBER_STATE, JSON_UNQUOTE(JSON_EXTRACT(addresses, "$.mysqlClassic")) as host FROM mysql_innodb_cluster_metadata.instances LEFT JOIN performance_schema.replication_group_members ON `mysql_server_uuid`=`MEMBER_ID` WHERE replicaset_id = 7;
</code></pre><p>In short, these commands check that the cluster is resilient, summarized in the <strong>statusText</strong> field, which says that we can lose up to one node and the cluster will keep working.</p><h2>High Availability</h2><p>What we have after deployment is a system that is highly available:</p><ul><li>Group replication with one primary node;</li>
<li>Access to the cluster through the router, which provides one port for Read/write and one for Read-Only;</li>
<li>Automatic failover. When the primary node fails, another one is promoted on the spot, without any manual labor.</li>
</ul><p>Let's start a test. We can check whether the data inserted from the R/W port is then retrieved by other nodes using the R/O port.</p><pre><code>$ docker exec -it mysqlrouter1 /opt/ic/tests/test_router.sh
Server ID of current master
--------------
SELECT @@global.server_id
--------------
+--------------------+
| @@global.server_id |
+--------------------+
| 100 |
+--------------------+
Create content using router
--------------
create schema if not exists test
--------------
--------------
create table t1(id int not null primary key, name varchar(50))
--------------
--------------
insert into t1 values (1, "aaa")
--------------
</code></pre><p>The first part of the test will show the server ID of the primary node, by using the router R/W port (6446.) Then it will create a table and insert one record.</p><pre><code>Server ID of a RO node
--------------
SELECT @@global.server_id
--------------
+--------------------+
| @@global.server_id |
+--------------------+
| 200 |
+--------------------+
retrieving contents using router
--------------
SELECT * from test.t1
--------------
+----+------+
| id | name |
+----+------+
| 1 | aaa |
+----+------+
</code></pre><p>Using the read-only port (6447), we get a different node, and we retrieve the data created in the primary node.</p><p>Now we can test the high availability. Since we are using Docker, instead of simply kill the MySQL service, we can simulate an anvil falling on the server, by wiping away the container:</p><pre><code>$ docker rm -f -v mysqlgr1
mysqlgr1
</code></pre><p>The primary node is gone for good. Let's see what the cluster status says now:</p><pre><code>$ ./tests/check_cluster.sh 2
Creating a Session to 'root@mysqlgr2:3306'
Classic Session successfully established. No default schema selected.
{
"clusterName": "testcluster",
"defaultReplicaSet": {
"name": "default",
"primary": "mysqlgr2:3306",
"status": "OK_NO_TOLERANCE",
"statusText": "Cluster is NOT tolerant to any failures. 1 member is not active",
"topology": {
"mysqlgr1:3306": {
"address": "mysqlgr1:3306",
"mode": "R/O",
"readReplicas": {},
"role": "HA",
"status": "(MISSING)"
},
"mysqlgr2:3306": {
"address": "mysqlgr2:3306",
"mode": "R/W",
"readReplicas": {},
"role": "HA",
"status": "ONLINE"
},
"mysqlgr3:3306": {
"address": "mysqlgr3:3306",
"mode": "R/O",
"readReplicas": {},
"role": "HA",
"status": "ONLINE"
}
}
}
}
</code></pre><p>There are a few differences compared to the initial report:</p><ul><li>The primary is now node 2 (mysqlgr2);</li>
<li>Node 1 is marked as MISSING;</li>
<li>The cluster has lost its resilience. Unless we add another node, no further failures will be handled automatically.</li>
</ul><p>We can run the router test again, and it will work just as well, with the differences reported below:</p><pre><code>Server ID of current master
--------------
SELECT @@global.server_id
--------------
+--------------------+
| @@global.server_id |
+--------------------+
| 200 |
+--------------------+
Create content using router
--------------
create schema if not exists test
--------------
--------------
drop table if exists t1
--------------
--------------
create table t1(id int not null primary key, name varchar(50))
--------------
--------------
insert into t1 values (1, "aaa")
--------------
Server ID of a RO node
--------------
SELECT @@global.server_id
--------------
+--------------------+
| @@global.server_id |
+--------------------+
| 300 |
+--------------------+
</code></pre><p>We see that the primary has now ID 200, and the R/O node is 300 (the only other node that has survived.)</p><h2>Summarizing</h2><ul><li><p>The good</p><ul><li>I can see that some of the ease of use promised in San Francisco is already available. We can create a cluster with little effort. </li>
<li>The recovery from the master failure is transparent.</li>
<li>The cluster status gives clear information about the system.</li>
</ul></li>
<li><p>The bad</p><ul><li>MySQL shell is difficult to use. The command line help is insufficient: some options require trial and error to work correctly. It also does not use an options file like other MySQL clients.</li>
<li>Adding a node after the primary has become unavailable is harder than it should be, and the manual does not contemplate this case. It only mentions a server that can be restarted.</li>
<li>Restarting the router after the primary died is impossible with the current configuration.</li>
<li>The metadata for replication is now in three different schemas: mysql, performance_schema, and mysql_innodb_cluster_metadata. I understand the reasons, but I believe that a simplification would be possible.</li>
</ul></li>
<li><p>The bottom line: quite good to start a cluster, but not enough to deal effectively with simple HA cases. Possibly released too early.</p></li>
</ul>Giuseppe Maxiahttp://www.blogger.com/profile/15801583338057324813noreply@blogger.com2tag:blogger.com,1999:blog-16959946.post-74698828556414740842017-04-30T09:35:00.001+02:002017-11-15T18:54:00.062+01:00Revisiting the hidden MySQL 8.0 data dictionary tables <p>A few months ago I wrote about <a href="https://datacharmer.blogspot.com/2016/09/showing-hidden-tables-in-mysql-8-data.html">showing the hidden tables in MySQL 8 data dictionary</a>. What I presented there was a naïve solution, which I am now able to improve with a few (bold) moves. In the solution given in my previous article, I was able to guess the table names somehow, but they were still hidden from the catalog. I did not think clearly then. I should have used the data dictionary itself to see the tables. Here, in addition to getting the real list of tables, I offer a feature to unhide them permanently.</p><p><a href="https://github.com/datacharmer/mysql-sandbox/releases/tag/3.2.08">MySQL-Sandbox 3.2.08</a> has now the ability of un-hide the data dictionary tables, and keep them available for inspection. This feature came to my mind after a chat with the MySQL team during PerconaLive 2017. They stressed the reason for hiding the tables, which is they want the freedom to change the tables if needed, without being constrained by what is published. They also say that if there is something missing in the information_schema views we could ask for an enhancement. I thought immediately that asking for I_S views enhancements would be easier if we could see the original tables. In the interest of science, then, I added an option <code>--expose_dd_tables</code> to MySQL-Sandbox, which will start the server with the changes needed to see and use the data dictionary tables.</p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhBPIkbYWWBWs7GOLjF_a9mJlmwC2lD1Jzu5k4GWloTs2O6TLwwviXb2gv9xWGPJL8C_3DycTID_IM-HpDGzDOLtazijZg5g85nrlLFXsetFGPna_-xL8YH1KwPxF3k5cHS1xXj/s1600/Screenshot+2017-04-30+09.28.26.png" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhBPIkbYWWBWs7GOLjF_a9mJlmwC2lD1Jzu5k4GWloTs2O6TLwwviXb2gv9xWGPJL8C_3DycTID_IM-HpDGzDOLtazijZg5g85nrlLFXsetFGPna_-xL8YH1KwPxF3k5cHS1xXj/s320/Screenshot+2017-04-30+09.28.26.png" width="320" height="313" /></a><br />
<pre><code>$ make_sandbox 8.0.1 -- --expose_dd_tables
[...]
$ ~/sandboxes/msb_8_0_1/use mysql
mysql [localhost] {msandbox} (mysql) > select tables.name from mysql.tables inner join sys.dd_hidden_tables using (id);
+------------------------------+
| name |
+------------------------------+
| version |
| collations |
| tablespaces |
| tablespace_files |
| catalogs |
| schemata |
| st_spatial_reference_systems |
| tables |
| view_table_usage |
| view_routine_usage |
| columns |
| indexes |
| index_column_usage |
| column_type_elements |
| foreign_keys |
| foreign_key_column_usage |
| table_partitions |
| table_partition_values |
| index_partitions |
| table_stats |
| index_stats |
| events |
| routines |
| parameters |
| parameter_type_elements |
| triggers |
| character_sets |
+------------------------------+
27 rows in set (0.00 sec)
</code></pre><p>This is, without a doubt, the complete list of hidden tables.</p><p>As you can infer from the query above, MySQL-Sandbox adds a table to the sys schema with the list of hidden tables.<br />
If you want to hide the tables again, you can run this:</p><pre><code>mysql [localhost] {msandbox} (mysql) > update mysql.tables set hidden=1
where id in (select id from sys.dd_hidden_tables);
Query OK, 27 rows affected (0.04 sec)
Rows matched: 27 Changed: 27 Warnings: 0
</code></pre><p>But of course we need the tables visible, so now we can peruse the data dictionary tables at will, and find out if there is valuable information missing from information_schema views.</p><p>How does this work? </p><p>I used the same hack defined in <a href="https://datacharmer.blogspot.com/2016/09/showing-hidden-tables-in-mysql-8-data.html">my previous post</a>, combined with a new feature of MySQL 8.0 (<a href="http://mysqlserverteam.com/mysql-8-0-persisting-configuration-variables/">SET PERSIST</a>) that allows me to keep the special option enabled across restarts. Unlike the solution in my previous post, though, users only need to use the starting option (--expose_dd_tables) without remembering obscure debug sequences. </p><p>Happy hacking!</p>Giuseppe Maxiahttp://www.blogger.com/profile/15801583338057324813noreply@blogger.com0tag:blogger.com,1999:blog-16959946.post-49411108528942365252017-03-25T05:48:00.001+01:002017-09-23T02:00:46.736+02:00MySQL 8.0 roles<p>One of the most interesting features introduced in <a href="https://dev.mysql.com/doc/refman/8.0/en/mysql-nutshell.html">MySQL 8.0</a> is <a href="https://dev.mysql.com/doc/refman/8.0/en/roles.html">roles</a> or the ability of defining a set of privileges as a named role and then granting that set to one or more users. The main benefits are more clarity of privileges and ease of administration. Using roles we can assign the same set of privileges to several users, and eventually modify or revoke all privileges at once.</p><p><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEimo6pKcDuG9-lfn1oWvjyV1lcwbgSzmXhUMDW9M-RqHCRnlk2BElDq1jQjWic6i_1Woy-U8AVCmuy5P_vOwjcA-fylskmx2MefFe8aJ4EcTGiYzGbi0JKpl_LiPWhXBIFKVphQ/?imgmax=1600" alt="Roles nutshell 2" title="roles_nutshell_2.png" border="0" width="591" height="39" /></p><h2>Roles in a nutshell</h2><p>Looking at the <a href="https://dev.mysql.com/doc/refman/8.0/en/roles.html">manual</a>, we see that using roles is a matter of several steps.</p><p>(1) <strong>Create a role.</strong> The statement is similar to <code>CREATE USER</code> though the effects are slightly different (we will see it in more detail later on.) </p><pre><code>mysql [localhost] {root} ((none)) > CREATE ROLE r_lotr_dev;
Query OK, 0 rows affected (0.02 sec)
</code></pre><p>(2) <strong>Grant privileges to the role.</strong> Again, this looks like granting privileges to a user.</p><pre><code>mysql [localhost] {root} ((none)) > GRANT ALL on lotr.* TO r_lotr_dev;
Query OK, 0 rows affected (0.01 sec)
</code></pre><p>(3) <strong>Create a user.</strong> This is the same that we've been doing until version 5.7.</p><pre><code>mysql [localhost] {root} (mysql) > create user aragorn identified by 'lotrpwd';
Query OK, 0 rows affected (0.01 sec)
</code></pre><p>Notice that the role is in the <code>mysql.user</code> table, and looks a lot like a user. </p><pre><code>mysql [localhost] {root} ((none)) > select host, user, authentication_string from mysql.user where user not like '%sandbox%;
+-----------+-------------+-------------------------------------------+
| host | user | authentication_string |
+-----------+-------------+-------------------------------------------+
| % | r_lotr_dev | |
| % | aragorn | *3A376D0203958F6EB9E6166DC048EC04F84C00B9 |
| localhost | mysql.sys | *THISISNOTAVALIDPASSWORDTHATCANBEUSEDHERE |
| localhost | root | *6C387FC3893DBA1E3BA155E74754DA6682D04747 |
+-----------+-------------+-------------------------------------------+
</code></pre><p>(4) <strong>Grant the role to the user.</strong> Instead of granting single privileges, we grant the role. Note that when we use this syntax we can't specify the <code>ON xxx</code> clause, because it is already implicit in the role definition.</p><pre><code>mysql [localhost] {root} (mysql) > grant r_lotr_dev to aragorn;
Query OK, 0 rows affected (0.03 sec)
</code></pre><p>The relationship between user and role is recorded in a new table in the <code>mysql</code> database:</p><pre><code>mysql [localhost] {root} (mysql) > select * from mysql.role_edges;
+-----------+------------+---------+---------+-------------------+
| FROM_HOST | FROM_USER | TO_HOST | TO_USER | WITH_ADMIN_OPTION |
+-----------+------------+---------+---------+-------------------+
| % | r_lotr_dev | % | aragorn | N |
+-----------+------------+---------+---------+-------------------+
</code></pre><p>(5) Finally we <strong>set the default role.</strong> Until this point, the role is assigned to the user, but <em>not active</em>. We either set the default role permanently (as done below) or we let the user activate the role.</p><pre><code>mysql [localhost] {root} (mysql) > alter user aragorn default role r_lotr_dba;
Query OK, 0 rows affected (0.01 sec)
</code></pre><p>If a default role has been set, it is recorded in another new table, <code>mysql.default_roles</code>.</p><pre><code>mysql [localhost] {root} (mysql) > select * from mysql.default_roles;
+------+---------+-------------------+-------------------+
| HOST | USER | DEFAULT_ROLE_HOST | DEFAULT_ROLE_USER |
+------+---------+-------------------+-------------------+
| % | aragorn | % | r_lotr_dba |
+------+---------+-------------------+-------------------+
</code></pre><h2>Common gotchas</h2><p>If we follow all the steps described above, using roles would not feel any different than using old style grants. But it is easy to go astray if we skip something. Let's see an example. First, we create an user and grant the same role as the one given to user <code>aragorn</code>:</p><pre><code>mysql [localhost] {root} (mysql) > create user legolas identified by 'lotrpwd';
Query OK, 0 rows affected (0.03 sec)
mysql [localhost] {root} (mysql) > grant r_lotr_dev to legolas;
Query OK, 0 rows affected (0.01 sec)
</code></pre><p>Then we connect using user <code>legolas</code>:</p><pre><code>mysql [localhost] {legolas} ((none)) > use lotr;
ERROR 1044 (42000): Access denied for user 'legolas'@'%' to database 'lotr'
mysql [localhost] {legolas} ((none)) > show grants;
+-----------------------------------------+
| Grants for legolas@% |
+-----------------------------------------+
| GRANT USAGE ON *.* TO `legolas`@`%` |
| GRANT `r_lotr_dev`@`%` TO `legolas`@`%` |
+-----------------------------------------+
2 rows in set (0.00 sec)
mysql [localhost] {legolas} ((none)) > show grants for legolas using r_lotr_dev;
+---------------------------------------------------+
| Grants for legolas@% |
+---------------------------------------------------+
| GRANT USAGE ON *.* TO `legolas`@`%` |
| GRANT ALL PRIVILEGES ON `lotr`.* TO `legolas`@`%` |
| GRANT `r_lotr_dev`@`%` TO `legolas`@`%` |
+---------------------------------------------------+
3 rows in set (0.00 sec)
</code></pre><p>The role is assigned to the user, but it is not active, as no default role was defined. To use a role, the user must activate one:</p><pre><code>mysql [localhost] {legolas} ((none)) > select current_role();
+----------------+
| current_role() |
+----------------+
| NONE |
+----------------+
1 row in set (0.01 sec)
</code></pre><p>There is no default role for <code>legolas</code>. We need to assign one.</p><pre><code>mysql [localhost] {legolas} ((none)) > set role r_lotr_dev;
Query OK, 0 rows affected (0.00 sec)
mysql [localhost] {legolas} ((none)) > select current_role();
+------------------+
| current_role() |
+------------------+
| `r_lotr_dev`@`%` |
+------------------+
1 row in set (0.00 sec)
</code></pre><p>Now the role is active, and all its privileges kick in:</p><pre><code>mysql [localhost] {legolas} ((none)) > use lotr
Database changed
mysql [localhost] {legolas} (lotr) > show tables;
Empty set (0.00 sec)
mysql [localhost] {legolas} (lotr) > create table t1 (i int not null primary key);
Query OK, 0 rows affected (0.15 sec)
</code></pre><p>Note that the role activation is volatile. If the user reconnects, the role activation goes away:</p><pre><code>mysql [localhost] {legolas} ((none)) > connect;
Connection id: 33
Current database: *** NONE ***
mysql [localhost] {legolas} ((none)) > select current_role();
+----------------+
| current_role() |
+----------------+
| NONE |
+----------------+
1 row in set (0.01 sec)
</code></pre><p>For a permanent assignment, the user can use the <code>SET DEFAULT ROLE</code> statement:</p><pre><code>mysql [localhost] {legolas} ((none)) > set default role r_lotr_dev to legolas;
Query OK, 0 rows affected (0.01 sec)
</code></pre><p>The above statement corresponds to <code>ALTER USER ... DEFAULT ROLE ...</code>. Every user can set its own role with this statement, without needing additional privileges.</p><pre><code>mysql [localhost] {legolas} ((none)) > alter user legolas default role r_lotr_dev;
Query OK, 0 rows affected (0.01 sec)
</code></pre><p>Now if the user reconnects, the role persists:</p><pre><code>mysql [localhost] {legolas} ((none)) > connect
Connection id: 34
Current database: *** NONE ***
mysql [localhost] {legolas} ((none)) > select current_role();
+------------------+
| current_role() |
+------------------+
| `r_lotr_dev`@`%` |
+------------------+
1 row in set (0.01 sec)
</code></pre><p>However, if an user sets its own default role using <code>ALTER USER</code>, the change will be available only in the next session, or after calling <code>SET ROLE</code>.</p><p>Let's try:</p><pre><code>mysql [localhost] {legolas} ((none)) > set default role none to legolas;
Query OK, 0 rows affected (0.02 sec)
mysql [localhost] {legolas} ((none)) > select current_role();
+------------------+
| current_role() |
+------------------+
| `r_lotr_dev`@`%` |
+------------------+
1 row in set (0.00 sec)
</code></pre><p>The role stays what it was before. This is similar to what happens when using <code>SET GLOBAL var_name</code> vs <code>SET SESSION var_name</code>. In the first case the effect persists, but it is not activated immediately, while a session set will be immediately usable, but will not persist after a new connection.</p><pre><code>mysql [localhost] {legolas} ((none)) > connect
Connection id: 35
Current database: *** NONE ***
select current_role();
+----------------+
| current_role() |
+----------------+
| NONE |
+----------------+
1 row in set (0.00 sec)
</code></pre><p>It's worth mentioning that <code>SET DEFAULT ROLE</code> implies an hidden update of a <code>mysql</code> table (<code>default_roles</code>), similar to what happens with <code>SET PASSWORD</code>. In both cases, a user without explicit access to the <code>mysql</code> database will be unable to check the effects of the operation.</p><h2>Advanced role management.</h2><p>Dealing with one or two roles is no big deal. Using the statements seen above, we can easily see what privileges were granted to a role or an user. When we have many roles and many users, the overview become more difficult to achieve.</p><p>Before we see the complex scenario, let's have a deeper look at <strong>what constitutes a role</strong>. </p><pre><code>mysql [localhost] {root} (mysql) > create role role1;
Query OK, 0 rows affected (0.05 sec)
mysql [localhost] {root} (mysql) > create user user1 identified by 'somepass';
Query OK, 0 rows affected (0.01 sec)
mysql [localhost] {root} (mysql) > select host, user, authentication_string , password_expired , account_locked from user where user in ('role1', 'user1');
+------+-------+-------------------------------------------+------------------+----------------+
| host | user | authentication_string | password_expired | account_locked |
+------+-------+-------------------------------------------+------------------+----------------+
| % | role1 | | Y | Y |
| % | user1 | *13883BDDBE566ECECC0501CDE9B293303116521A | N | N |
+------+-------+-------------------------------------------+------------------+----------------+
2 rows in set (0.00 sec)
</code></pre><p>The main difference between user and role is that a role is created with <code>password_expired</code> and <code>account_locked</code>. Apart from that, an user could be used as a role and vice versa.</p><pre><code>mysql [localhost] {root} (mysql) > alter user role1 identified by 'msandbox';
Query OK, 0 rows affected (0.01 sec)
mysql [localhost] {root} ((none)) > alter user role1 account unlock;
Query OK, 0 rows affected (0.02 sec)
</code></pre><p>Now <code>role1</code> can access the database as any other user.</p><pre><code>mysql [localhost] {root} ((none)) > grant root@'localhost' to user1;
Query OK, 0 rows affected (0.03 sec)
</code></pre><p>And user1 inherits all privileges from root, but it can access the server from any host.</p><p>Now let's see some complex usage of roles. In a typical organisation, we would define several roles to use the <code>lotr</code> database:</p><pre><code>CREATE ROLE r_lotr_observer;
CREATE ROLE r_lotr_tester;
CREATE ROLE r_lotr_dev;
CREATE ROLE r_lotr_dba;
GRANT SELECT on lotr.* TO r_lotr_observer;
GRANT SELECT, INSERT, UPDATE, DELETE on lotr.* TO r_lotr_tester;
GRANT ALL on lotr.* TO r_lotr_dev;
GRANT ALL on *.* TO r_lotr_dba;
</code></pre><p>And then assign those roles to several users:</p><pre><code>CREATE USER bilbo IDENTIFIED BY 'msandbox' PASSWORD EXPIRE;
CREATE USER frodo IDENTIFIED BY 'msandbox' PASSWORD EXPIRE;
CREATE USER sam IDENTIFIED BY 'msandbox' PASSWORD EXPIRE;
CREATE USER pippin IDENTIFIED BY 'msandbox' PASSWORD EXPIRE;
CREATE USER merry IDENTIFIED BY 'msandbox' PASSWORD EXPIRE;
CREATE USER boromir IDENTIFIED BY 'msandbox' PASSWORD EXPIRE;
CREATE USER gimli IDENTIFIED BY 'msandbox' PASSWORD EXPIRE;
CREATE USER aragorn IDENTIFIED BY 'msandbox' PASSWORD EXPIRE;
CREATE USER legolas IDENTIFIED BY 'msandbox' PASSWORD EXPIRE;
CREATE USER gollum IDENTIFIED BY 'msandbox' PASSWORD EXPIRE;
CREATE USER galadriel IDENTIFIED BY 'msandbox' PASSWORD EXPIRE;
CREATE USER gandalf IDENTIFIED BY 'msandbox';
GRANT r_lotr_observer TO pippin, merry, boromir, gollum;
SET DEFAULT ROLE r_lotr_observer to pippin, merry, boromir, gollum;
GRANT r_lotr_tester TO sam, bilbo, gimli;
SET DEFAULT ROLE r_lotr_tester to sam, bilbo, gimli;
GRANT r_lotr_dev to frodo, aragorn, legolas;
SET DEFAULT ROLE r_lotr_dev to frodo, aragorn, legolas;
GRANT r_lotr_dba TO gandalf, galadriel;
SET DEFAULT ROLE r_lotr_dba to gandalf, galadriel;
</code></pre><p>Now we have 12 users with 4 different roles. Looking at the user table, we don't get a good overview:</p><pre><code>mysql [localhost] {root} (mysql) > select host, user, authentication_string from mysql.user where user not like '%sandbox%';
+-----------+-----------------+-------------------------------------------+
| host | user | authentication_string |
+-----------+-----------------+-------------------------------------------+
| % | aragorn | *6C387FC3893DBA1E3BA155E74754DA6682D04747 |
| % | bilbo | *6C387FC3893DBA1E3BA155E74754DA6682D04747 |
| % | boromir | *6C387FC3893DBA1E3BA155E74754DA6682D04747 |
| % | frodo | *6C387FC3893DBA1E3BA155E74754DA6682D04747 |
| % | galadriel | *6C387FC3893DBA1E3BA155E74754DA6682D04747 |
| % | gandalf | *6C387FC3893DBA1E3BA155E74754DA6682D04747 |
| % | gimli | *6C387FC3893DBA1E3BA155E74754DA6682D04747 |
| % | gollum | *6C387FC3893DBA1E3BA155E74754DA6682D04747 |
| % | legolas | *6C387FC3893DBA1E3BA155E74754DA6682D04747 |
| % | merry | *6C387FC3893DBA1E3BA155E74754DA6682D04747 |
| % | pippin | *6C387FC3893DBA1E3BA155E74754DA6682D04747 |
| % | r_lotr_dba | |
| % | r_lotr_dev | |
| % | r_lotr_observer | |
| % | r_lotr_tester | |
| % | sam | *6C387FC3893DBA1E3BA155E74754DA6682D04747 |
| localhost | mysql.sys | *THISISNOTAVALIDPASSWORDTHATCANBEUSEDHERE |
| localhost | root | *6C387FC3893DBA1E3BA155E74754DA6682D04747 |
+-----------+-----------------+-------------------------------------------+
</code></pre><p>And even the <code>roles_edges</code> table does show a clear picture:</p><pre><code>mysql [localhost] {root} (mysql) > select * from role_edges;
+-----------+-----------------+---------+-----------+-------------------+
| FROM_HOST | FROM_USER | TO_HOST | TO_USER | WITH_ADMIN_OPTION |
+-----------+-----------------+---------+-----------+-------------------+
| % | r_lotr_dba | % | galadriel | N |
| % | r_lotr_dba | % | gandalf | N |
| % | r_lotr_dev | % | aragorn | N |
| % | r_lotr_dev | % | frodo | N |
| % | r_lotr_dev | % | legolas | N |
| % | r_lotr_observer | % | boromir | N |
| % | r_lotr_observer | % | gollum | N |
| % | r_lotr_observer | % | merry | N |
| % | r_lotr_observer | % | pippin | N |
| % | r_lotr_tester | % | bilbo | N |
| % | r_lotr_tester | % | gimli | N |
| % | r_lotr_tester | % | sam | N |
+-----------+-----------------+---------+-----------+-------------------+
</code></pre><p>Here's a better use of that table's data. Which users are using the dev role?</p><pre><code>select to_user as users from role_edges where from_user = 'r_lotr_dev';
+---------+
| users |
+---------+
| aragorn |
| frodo |
| legolas |
+---------+
3 rows in set (0.00 sec)
</code></pre><p>And the testers?</p><pre><code>select to_user as users from role_edges where from_user = 'r_lotr_tester';
+-------+
| users |
+-------+
| bilbo |
| gimli |
| sam |
+-------+
3 rows in set (0.00 sec)
</code></pre><p>Or, even better, we could see all the roles at once:</p><pre><code>select from_user as role, count(*) as how_many_users, group_concat(to_user) as users from role_edges group by role;
+-----------------+----------------+-----------------------------+
| role | how_many_users | users |
+-----------------+----------------+-----------------------------+
| r_lotr_dba | 2 | galadriel,gandalf |
| r_lotr_dev | 3 | aragorn,frodo,legolas |
| r_lotr_observer | 4 | boromir,gollum,merry,pippin |
| r_lotr_tester | 3 | bilbo,gimli,sam |
+-----------------+----------------+-----------------------------+
4 rows in set (0.01 sec)
</code></pre><p>Similarly, we could list the default role for several users:</p><pre><code>select default_role_user as default_role, group_concat(user) as users from default_roles group by default_role;
+-----------------+-----------------------------+
| default_role | users |
+-----------------+-----------------------------+
| r_lotr_dba | galadriel,gandalf |
| r_lotr_dev | aragorn,frodo,legolas |
| r_lotr_observer | boromir,gollum,merry,pippin |
| r_lotr_tester | bilbo,gimli,sam |
+-----------------+-----------------------------+
</code></pre><p>The latest two queries should be good candidates for <code>information_schema</code> views.</p><p>Another candidate for an <code>information_schema</code> view is the list of roles, for which there is no satisfactory workaround now. The best we could think of is a list of users with <code>password_expired</code> and <code>account_locked</code>:</p><pre><code>select host, user from mysql.user where password_expired='y' and account_locked='y';
+------+-----------------+
| host | user |
+------+-----------------+
| % | r_lotr_dba |
| % | r_lotr_dev |
| % | r_lotr_observer |
| % | r_lotr_tester |
+------+-----------------+
4 rows in set (0.00 sec)
</code></pre><p>Until a feature to differentiate users and roles is developed, it is advisable to use a name format that helps identify roles without help from the server. In this article I am using <code>r_</code> as a prefix, which makes role listing easier.</p><pre><code>mysql [localhost] {root} (mysql) > select host, user from mysql.user where user like 'r_%';
+------+-----------------+
| host | user |
+------+-----------------+
| % | r_lotr_dba |
| % | r_lotr_dev |
| % | r_lotr_observer |
| % | r_lotr_tester |
+------+-----------------+
4 rows in set (0.00 sec)
</code></pre><h2>Known bugs</h2><ul><li><a href="https://bugs.mysql.com/bug.php?id=85562">bug#85562</a> Dropping a role does not remove privileges from active users.</li>
<li><a href="https://bugs.mysql.com/bug.php?id=85561">bug#85561</a> Users can be assigned non-existing roles as default.</li>
<li><a href="https://bugs.mysql.com/bug.php?id=85559">bug#85559</a> Dropping a role does not remove the associated default roles.</li>
<li><a href="https://bugs.mysql.com/bug.php?id=84244">bug#84244</a> Distinguish roles and plain users in mysql.user.</li>
<li><a href="https://bugs.mysql.com/bug.php?id=82987">bug#82987</a> SHOW CREATE USER doesn't show default role.</li>
</ul><h2>Summing up</h2><p>Roles are a great addition to the MySQL feature set. The usability could be improved with some views in <code>information_schema</code> (users<em>for</em>role, default<em>role</em>for<em>user) and functions in <code>sys</code> schema (default</em>role<em>for</em>user, is_role.)</p><p>Perhaps some commands to activate or deactivate roles could make administration easier. The current bugs don't affect the basic functionality but decrease usability. I think that another run of bug fixing and user feedback would significantly improve this feature.</p><p>More about roles in <a href="https://www.percona.com/live/17/sessions/quick-tour-mysql-80-roles">my talk at PerconaLive 2017.</a> </p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiA0Kjo7WxwD2Rzwoz7v8clzQCKhop48yj-BPXSCAKb598DVq0nZ7qOePQxtOPOtc8nICaF27NG8ap3nI4MBuGwKH2P6cn7DDVB8PvCucymLFjxov6f5Qc_lwbpwO5JV8E1jpsP/s1600/GiuseppeMaxia-3.jpg" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiA0Kjo7WxwD2Rzwoz7v8clzQCKhop48yj-BPXSCAKb598DVq0nZ7qOePQxtOPOtc8nICaF27NG8ap3nI4MBuGwKH2P6cn7DDVB8PvCucymLFjxov6f5Qc_lwbpwO5JV8E1jpsP/s1600/GiuseppeMaxia-3.jpg" /></a><br />
Giuseppe Maxiahttp://www.blogger.com/profile/15801583338057324813noreply@blogger.com2tag:blogger.com,1999:blog-16959946.post-59049913274425307332017-01-22T15:31:00.000+01:002017-11-19T21:41:47.328+01:00MySQL Group Replication vs. Multi Source<p>In my previous post, we saw the usage of MySQL Group Replication (MGR) <a href="https://datacharmer.blogspot.com/2017/01/mysql-group-replication-installation.html">in single-primary mode</a>. We know that Oracle does not recommends using MGR in multi-primary mode, but there is so much in the documentation and in presentations about MGR behavior in multi-primary, that I feel I should really give it a try, and especially compare this technology with the already existing multiple master solution introduced in 5.7: <a href="https://dev.mysql.com/doc/refman/5.7/en/replication-multi-source.html">multi-source replication</a>.</p><h2>Installation</h2><p>To this extent, I will set up two clusters using <a href="http://mysqlsandbox.net">MySQL-Sandbox</a>. The instructions for MGR in <a href="https://dev.mysql.com/doc/refman/5.7/en/group-replication-getting-started-deploying-instances.html">the manual</a> use three nodes in the same host without using MySQL Sandbox. Here we can see that using MySQL-Sandbox simplifies operations considerably (the scripts are available in <a href="https://github.com/datacharmer/mysql-replication-samples/tree/master/multi-master-comparison">GitHub</a>):</p><h3>Group replication</h3><pre><code># ----------------------------------------------------------------------------
#!/bin/bash
# mm_gr.sh : installs MySQL Group Replication
MYSQL_VERSION=$1
[ -z "$MYSQL_VERSION" ] && MYSQL_VERSION=5.7.17
make_multiple_sandbox --gtid --group_directory=GR $MYSQL_VERSION
if [ "$?" != "0" ] ; then exit 1 ; fi
multi_sb=$HOME/sandboxes/GR
baseport=$($multi_sb/n1 -BN -e 'select @@port')
baseport=$(($baseport+99))
port1=$(($baseport+1))
port2=$(($baseport+2))
port3=$(($baseport+3))
for N in 1 2 3
do
myport=$(($baseport+N))
options=(
binlog_checksum=NONE
log_slave_updates=ON
plugin-load=group_replication.so
group_replication=FORCE_PLUS_PERMANENT
group_replication_start_on_boot=OFF
group_replication_bootstrap_group=OFF
transaction_write_set_extraction=XXHASH64
report-host=127.0.0.1
loose-group_replication_group_name="aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee"
loose-group_replication_local_address="127.0.0.1:$myport"
loose-group_replication_group_seeds="127.0.0.1:$port1,127.0.0.1:$port2,127.0.0.1:$port3"
loose-group-replication-single-primary-mode=off
)
$multi_sb/node$N/add_option ${options[*]}
user_cmd='reset master;'
user_cmd="$user_cmd CHANGE MASTER TO MASTER_USER='rsandbox', MASTER_PASSWORD='rsandbox' FOR CHANNEL 'group_replication_recovery';"
$multi_sb/node$N/use -v -u root -e "$user_cmd"
done
START_CMD="SET GLOBAL group_replication_bootstrap_group=ON;"
START_CMD="$START_CMD START GROUP_REPLICATION;"
START_CMD="$START_CMD SET GLOBAL group_replication_bootstrap_group=OFF;"
$multi_sb/n1 -v -e "$START_CMD"
sleep 1
$multi_sb/n2 -v -e 'START GROUP_REPLICATION;'
sleep 1
$multi_sb/n3 -v -e 'START GROUP_REPLICATION;'
sleep 1
$multi_sb/use_all 'select * from performance_schema.replication_group_members'
# ----------------------------------------------------------------------------
</code></pre><p>Using this script, we get a cluster with MGR up and running. Here's a trimmed-out sample of its output:</p><pre><code>$ ./mm_gr.sh
installing node 1
installing node 2
installing node 3
group directory installed in $HOME/sandboxes/GR
# option 'binlog_checksum=NONE' added to configuration file
# option 'log_slave_updates=ON' added to configuration file
# option 'plugin-load=group_replication.so' added to configuration file
# option 'group_replication=FORCE_PLUS_PERMANENT' added to configuration file
# option 'group_replication_start_on_boot=OFF' added to configuration file
# option 'group_replication_bootstrap_group=OFF' added to configuration file
# option 'transaction_write_set_extraction=XXHASH64' added to configuration file
# option 'loose-group_replication_group_name=aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee' added to configuration file
# option 'loose-group_replication_local_address=127.0.0.1:14518' added to configuration file
# option 'loose-group_replication_group_seeds=127.0.0.1:14518,127.0.0.1:14519,127.0.0.1:14520' added to configuration file
# option 'loose-group-replication-single-primary-mode=off' added to configuration file
.. sandbox server started
reset master
CHANGE MASTER TO MASTER_USER='rsandbox', MASTER_PASSWORD='rsandbox' FOR CHANNEL 'group_replication_recovery'
# [ ...]
.. sandbox server started
reset master
CHANGE MASTER TO MASTER_USER='rsandbox', MASTER_PASSWORD='rsandbox' FOR CHANNEL 'group_replication_recovery'
# [...]
.. sandbox server started
reset master
CHANGE MASTER TO MASTER_USER='rsandbox', MASTER_PASSWORD='rsandbox' FOR CHANNEL 'group_replication_recovery'
SET GLOBAL group_replication_bootstrap_group=ON
START GROUP_REPLICATION
SET GLOBAL group_replication_bootstrap_group=OFF
--------------
--------------
START GROUP_REPLICATION
--------------
START GROUP_REPLICATION
--------------
</code></pre><h3>Multi-source replication</h3><p>We have a similar (but much shorter) script to run multi-source replication in sandboxes.</p><pre><code># ----------------------------------------------------------------------------
#!/bin/bash
# mm_ms.sh : installs MySQL multi-source replication
MYSQL_VERSION=$1
[ -z "$MYSQL_VERSION" ] && MYSQL_VERSION=5.7.16
make_multiple_sandbox --gtid --group_directory=MS $MYSQL_VERSION
if [ "$?" != "0" ] ; then exit 1 ; fi
multi_sb=$HOME/sandboxes/MS
$multi_sb/use_all 'reset master'
for N in 1 2 3
do
user_cmd=''
for node in 1 2 3
do
if [ "$node" != "$N" ]
then
master_port=$($multi_sb/n$node -BN -e 'select @@port')
user_cmd="$user_cmd CHANGE MASTER TO MASTER_USER='rsandbox', "
user_cmd="$user_cmd MASTER_PASSWORD='rsandbox', master_host='127.0.0.1', "
user_cmd="$user_cmd master_port=$master_port FOR CHANNEL 'node$node';"
user_cmd="$user_cmd START SLAVE FOR CHANNEL 'node$node';"
fi
done
$multi_sb/node$N/use -v -u root -e "$user_cmd"
done
# ----------------------------------------------------------------------------
Sample run:
$ ./mm_ms.sh
installing node 1
installing node 2
installing node 3
group directory installed in $HOME/sandboxes/MS
# server: 1:
# server: 2:
# server: 3:
--------------
CHANGE MASTER TO MASTER_USER='rsandbox', MASTER_PASSWORD='rsandbox', master_host='127.0.0.1', master_port=14318 FOR CHANNEL 'node2'
START SLAVE FOR CHANNEL 'node2'
CHANGE MASTER TO MASTER_USER='rsandbox', MASTER_PASSWORD='rsandbox', master_host='127.0.0.1', master_port=14319 FOR CHANNEL 'node3'
START SLAVE FOR CHANNEL 'node3'
--------------
CHANGE MASTER TO MASTER_USER='rsandbox', MASTER_PASSWORD='rsandbox', master_host='127.0.0.1', master_port=14317 FOR CHANNEL 'node1'
START SLAVE FOR CHANNEL 'node1'
CHANGE MASTER TO MASTER_USER='rsandbox', MASTER_PASSWORD='rsandbox', master_host='127.0.0.1', master_port=14319 FOR CHANNEL 'node3'
START SLAVE FOR CHANNEL 'node3'
--------------
CHANGE MASTER TO MASTER_USER='rsandbox', MASTER_PASSWORD='rsandbox', master_host='127.0.0.1', master_port=14317 FOR CHANNEL 'node1'
START SLAVE FOR CHANNEL 'node1'
CHANGE MASTER TO MASTER_USER='rsandbox', MASTER_PASSWORD='rsandbox', master_host='127.0.0.1', master_port=14318 FOR CHANNEL 'node2'
START SLAVE FOR CHANNEL 'node2'
--------------
</code></pre><h3>Simple test data</h3><p>Finally, we have a script that will create one table for each node and insert one record.<br />
<pre><code>
# ----------------------------------------------------------------------------
#!/bin/bash
multi_sb=$1
if [ -z "$multi_sb" ]
then
echo multiple sandbox path needed
exit 1
fi
if [ ! -d $multi_sb ]
then
echo directory $multi_sb not found
exit 1
fi
if [ ! -d "$multi_sb/node3" ]
then
echo directory $multi_sb/node3 not found
exit 1
fi
cd $multi_sb
for N in 1 2 3 ; do
./n$N -e "create schema if not exists test"
./n$N -e "drop table if exists test.t$N"
./n$N -e "create table test.t$N(id int not null primary key, sid int)"
./n$N -e "insert into test.t$N values ($N, @@server_id)"
done
./use_all 'select * from test.t1 union select * from test.t2 union select * from test.t3'
# ----------------------------------------------------------------------------
</code></pre><p>We run the script in both clusters, and at the end we'll have the <em>test</em> database with three tables, each one created and filled by a different node.</p><h2>Checking replication status</h2><h3>The old topology: multi-source</h3><p>Let's start with the the old technology, so we can easily compare it with the new one. </p><pre><code>node1 [localhost] {msandbox} (performance_schema) > select * from replication_connection_status\G
*************************** 1. row ***************************
CHANNEL_NAME: node2
GROUP_NAME:
SOURCE_UUID: 00014318-2222-2222-2222-222222222222 # ----
THREAD_ID: 32
SERVICE_STATE: ON
COUNT_RECEIVED_HEARTBEATS: 244
LAST_HEARTBEAT_TIMESTAMP: 2017-01-22 13:31:54
RECEIVED_TRANSACTION_SET: 00014318-2222-2222-2222-222222222222:1-4
LAST_ERROR_NUMBER: 0
LAST_ERROR_MESSAGE:
LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00
*************************** 2. row ***************************
CHANNEL_NAME: node3
GROUP_NAME:
SOURCE_UUID: 00014319-3333-3333-3333-333333333333 # ----
THREAD_ID: 34
SERVICE_STATE: ON
COUNT_RECEIVED_HEARTBEATS: 244
LAST_HEARTBEAT_TIMESTAMP: 2017-01-22 13:31:55
RECEIVED_TRANSACTION_SET: 00014319-3333-3333-3333-333333333333:1-4
LAST_ERROR_NUMBER: 0
LAST_ERROR_MESSAGE:
LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00
2 rows in set (0.00 sec)
</code></pre><p>Notice that we are benefitting from a feature of MySQL-Sandbox that creates a more readable version of the server UUID. This way we can easily identify the nodes. Here we see that each transaction set has a clearly defined origin. We can see similar information in the replication tables from the mysql database:</p><pre><code>node1 [localhost] {msandbox} (mysql) > select * from slave_master_info\G
*************************** 1. row ***************************
Number_of_lines: 25
Master_log_name: mysql-bin.000001
Master_log_pos: 154
Host: 127.0.0.1 # ----
User_name: rsandbox
User_password: rsandbox
Port: 14318 # ----
Connect_retry: 60
Enabled_ssl: 0
Ssl_verify_server_cert: 0
Heartbeat: 30
Bind:
Ignored_server_ids: 0
Uuid: 00014318-2222-2222-2222-222222222222 # ----
Retry_count: 86400
Ssl_crlpath:
Enabled_auto_position: 0
Channel_name: node2
Tls_version:
*************************** 2. row ***************************
Number_of_lines: 25
Master_log_name: mysql-bin.000001
Master_log_pos: 154
Host: 127.0.0.1 # ----
User_name: rsandbox
User_password: rsandbox
Port: 14319 # ----
Connect_retry: 60
Enabled_ssl: 0
Ssl_verify_server_cert: 0
Heartbeat: 30
Bind:
Ignored_server_ids: 0
Uuid: 00014319-3333-3333-3333-333333333333 # ----
Retry_count: 86400
Ssl_crlpath:
Enabled_auto_position: 0
Channel_name: node3
Tls_version:
2 rows in set (0.00 sec)
</code></pre><p>Additionally, we have SHOW SLAVE STATUS, which, although not the ideal monitoring tool, is still the only place where we can see at once both the received and executed transactions, and the corresponding binary log and relay log records.</p><p>Here's an abridged version:</p><pre><code>node1 [localhost] {msandbox} (performance_schema) > SHOW SLAVE STATUS\G
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 127.0.0.1
Master_User: rsandbox
Master_Port: 14318
Connect_Retry: 60
Master_Log_File: mysql-bin.000001
Read_Master_Log_Pos: 965
Relay_Log_File: mysql-relay-node2.000002
Relay_Log_Pos: 1178
Relay_Master_Log_File: mysql-bin.000001
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Exec_Master_Log_Pos: 965
Relay_Log_Space: 1387
Master_Server_Id: 102
Master_UUID: 00014318-2222-2222-2222-222222222222
Master_Info_File: mysql.slave_master_info
Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
Master_Retry_Count: 86400
Retrieved_Gtid_Set: 00014318-2222-2222-2222-222222222222:1-4
Executed_Gtid_Set: 00014317-1111-1111-1111-111111111111:1-4,
00014318-2222-2222-2222-222222222222:1-4,
00014319-3333-3333-3333-333333333333:1-4
Channel_Name: node2
*************************** 2. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 127.0.0.1
Master_User: rsandbox
Master_Port: 14319
Connect_Retry: 60
Master_Log_File: mysql-bin.000001
Read_Master_Log_Pos: 965
Relay_Log_File: mysql-relay-node3.000002
Relay_Log_Pos: 1178
Relay_Master_Log_File: mysql-bin.000001
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Exec_Master_Log_Pos: 965
Relay_Log_Space: 1387
Until_Condition: None
Master_Server_Id: 103
Master_UUID: 00014319-3333-3333-3333-333333333333
Master_Info_File: mysql.slave_master_info
Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
Master_Retry_Count: 86400
Retrieved_Gtid_Set: 00014319-3333-3333-3333-333333333333:1-4
Executed_Gtid_Set: 00014317-1111-1111-1111-111111111111:1-4,
00014318-2222-2222-2222-222222222222:1-4,
00014319-3333-3333-3333-333333333333:1-4
Channel_Name: node3
2 rows in set (0.00 sec)
</code></pre><p>Finally, we'll have a look at the data itself:</p><pre><code>node1 [localhost] {msandbox} (mysql) > show binlog events;
+------------------+-----+----------------+-----------+-------------+-------------------------------------------------------------------+
| Log_name | Pos | Event_type | Server_id | End_log_pos | Info |
+------------------+-----+----------------+-----------+-------------+-------------------------------------------------------------------+
| mysql-bin.000001 | 4 | Format_desc | 101 | 123 | Server ver: 5.7.16-log, Binlog ver: 4 |
| mysql-bin.000001 | 123 | Previous_gtids | 101 | 154 | |
| mysql-bin.000001 | 154 | Gtid | 101 | 219 | SET @@SESSION.GTID_NEXT= '00014317-1111-1111-1111-111111111111:1' |
| mysql-bin.000001 | 219 | Query | 101 | 325 | create schema if not exists test |
| mysql-bin.000001 | 325 | Gtid | 101 | 390 | SET @@SESSION.GTID_NEXT= '00014317-1111-1111-1111-111111111111:2' |
| mysql-bin.000001 | 390 | Query | 101 | 518 | DROP TABLE IF EXISTS `test`.`t1` /* generated by server */ |
| mysql-bin.000001 | 518 | Gtid | 101 | 583 | SET @@SESSION.GTID_NEXT= '00014317-1111-1111-1111-111111111111:3' |
| mysql-bin.000001 | 583 | Query | 101 | 711 | create table test.t1(id int not null primary key, sid int) |
| mysql-bin.000001 | 711 | Gtid | 101 | 776 | SET @@SESSION.GTID_NEXT= '00014317-1111-1111-1111-111111111111:4' |
| mysql-bin.000001 | 776 | Query | 101 | 844 | BEGIN |
| mysql-bin.000001 | 844 | Table_map | 101 | 890 | table_id: 108 (test.t1) |
| mysql-bin.000001 | 890 | Write_rows | 101 | 934 | table_id: 108 flags: STMT_END_F |
| mysql-bin.000001 | 934 | Xid | 101 | 965 | COMMIT /* xid=72 */ |
+------------------+-----+----------------+-----------+-------------+-------------------------------------------------------------------+
13 rows in set (0.00 sec)
</code></pre><p>The binary log contains only the data produced in this node. </p><h3>The new topology: MGR</h3><p>Turning to the new software, let's first check whether replication is working. An important note here: <strong>SHOW SLAVE STATUS is not available in MGR</strong>. That's not entirely true. The channel architecture used for multi-master has been hijacked to convey information about group problems. If something goes wrong during the setup, you will find the information in the group<em>replication</em>recovery channel.</p><pre><code>node1 [localhost] {msandbox} (performance_schema) > SHOW SLAVE STATUS for channel 'group_replication_recovery';
Empty set (0.00 sec)
</code></pre><p>When things are fine, the tables in performance_schema report a satisfactory status:</p><pre><code>node1 [localhost] {msandbox} (performance_schema) > select * from replication_group_members;
+---------------------------+--------------------------------------+-------------+-------------+--------------+
| CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE |
+---------------------------+--------------------------------------+-------------+-------------+--------------+
| group_replication_applier | 00014418-1111-1111-1111-111111111111 | gmini | 14418 | ONLINE |
| group_replication_applier | 00014419-2222-2222-2222-222222222222 | gmini | 14419 | ONLINE |
| group_replication_applier | 00014420-3333-3333-3333-333333333333 | gmini | 14420 | ONLINE |
+---------------------------+--------------------------------------+-------------+-------------+--------------+
</code></pre><p>The above command tells us that all nodes are online.</p><p>Next, we ask what are the stats of the current member.</p><pre><code>node1 [localhost] {msandbox} (performance_schema) > select * from replication_group_member_stats\G
*************************** 1. row ***************************
CHANNEL_NAME: group_replication_applier
VIEW_ID: 14850806532423012:3
MEMBER_ID: 00014418-1111-1111-1111-111111111111
COUNT_TRANSACTIONS_IN_QUEUE: 0
COUNT_TRANSACTIONS_CHECKED: 12
COUNT_CONFLICTS_DETECTED: 0
COUNT_TRANSACTIONS_ROWS_VALIDATING: 0
TRANSACTIONS_COMMITTED_ALL_MEMBERS: aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee:1-7:1000003-1000006:2000003-2000006
LAST_CONFLICT_FREE_TRANSACTION: aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee:2000006
1 row in set (0.00 sec)
</code></pre><p>The same operation from a different member will give a very similar result.</p><pre><code>node2 [localhost] {msandbox} (performance_schema) > select * from replication_group_member_stats\G
*************************** 1. row ***************************
CHANNEL_NAME: group_replication_applier
VIEW_ID: 14850806532423012:3
MEMBER_ID: 00014419-2222-2222-2222-222222222222
COUNT_TRANSACTIONS_IN_QUEUE: 0
COUNT_TRANSACTIONS_CHECKED: 12
COUNT_CONFLICTS_DETECTED: 0
COUNT_TRANSACTIONS_ROWS_VALIDATING: 0
TRANSACTIONS_COMMITTED_ALL_MEMBERS: aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee:1-7:1000003-1000006:2000003-2000006
LAST_CONFLICT_FREE_TRANSACTION: aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee:2000006
1 row in set (0.00 sec)
</code></pre><p>Then, we check the more classical replication status:</p><pre><code>node1 [localhost] {msandbox} (performance_schema) > select * from replication_connection_status\G
*************************** 1. row ***************************
CHANNEL_NAME: group_replication_applier
GROUP_NAME: aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee
SOURCE_UUID: aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee # ----
THREAD_ID: NULL
SERVICE_STATE: ON
COUNT_RECEIVED_HEARTBEATS: 0
LAST_HEARTBEAT_TIMESTAMP: 0000-00-00 00:00:00
RECEIVED_TRANSACTION_SET: aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee:1-7:1000003-1000006:2000003-2000006
LAST_ERROR_NUMBER: 0
LAST_ERROR_MESSAGE:
LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00
1 row in set (0.00 sec)
</code></pre><p>There are a few things that strike the observer immediately:</p><ul><li>As we saw in the single-primary topology, all transactions bear the UUID of the group, not of the server that generated them. While in single-primary mode this could be considered an asset, as it simplifies a failover procedure, in multi-primary mode I consider it to be a loss. We lose the knowledge of the transaction provenience. As you can see, the <strong>SOURCE_UUID</strong> field shows the group ID instead of the node.</li>
<li>The GTID numbers look odd. There is a set that stars at 1, another set that starts at 1 million, and a third one that starts at 2 million. What's going on? The answer is in the value of <a href="https://dev.mysql.com/doc/refman/5.7/en/group-replication-options.html#sysvar_group_replication_gtid_assignment_block_size">group_replication_gtid_assignment_block_size</a>, which determines the block of values for each node. When the values in the block are exhausted, the node allocates another block. Someone could naively think that we could use this block to identify which node the data comes from, but this would be ultimately wrong for two reasons: <ul><li>The blocks are assigned on a first-come-first-served basis. If we start operations in node 2, its transactions will bear the lowest numbers. </li>
<li>When the blocks are exhausted, the node starts a new block, meaning that with a busy cluster we will have hard time identifying which nodes uses which block.</li>
</ul><p></li>
</ul></p><p>If someone thought that we could get some more information from the replication tables in mysql, they are in for a disappointment:</p><pre><code>node2 [localhost] {msandbox} (mysql) > select * from slave_master_info\G
*************************** 1. row ***************************
Number_of_lines: 25
Master_log_name:
Master_log_pos: 4
Host: <NULL> # ----
User_name:
User_password:
Port: 0 # ----
Connect_retry: 60
Enabled_ssl: 0
Ssl_verify_server_cert: 0
Heartbeat: 30
Bind:
Ignored_server_ids: 0
Uuid: # ----
Retry_count: 86400
Enabled_auto_position: 1
Channel_name: group_replication_applier
Tls_version:
*************************** 2. row ***************************
Number_of_lines: 25
Master_log_name:
Master_log_pos: 4
Host: <NULL>
User_name: rsandbox
User_password: rsandbox
Port: 0
Connect_retry: 60
Enabled_ssl: 0
Ssl_verify_server_cert: 0
Heartbeat: 30
Bind:
Ignored_server_ids: 0
Uuid:
Retry_count: 1
Enabled_auto_position: 1
Channel_name: group_replication_recovery
Tls_version:
2 rows in set (0.00 sec)
</code></pre><p>The table shows group operations rather than individual hosts connections. There is no origin information here.</p><p>Looking at the events, we will notice immediately some more differences.</p><pre><code>node2 [localhost] {msandbox} (mysql) > show binlog events;
+------------------+------+----------------+-----------+-------------+-------------------------------------------------------------------------+
| Log_name | Pos | Event_type | Server_id | End_log_pos | Info |
+------------------+------+----------------+-----------+-------------+-------------------------------------------------------------------------+
| mysql-bin.000001 | 4 | Format_desc | 102 | 123 | Server ver: 5.7.17-log, Binlog ver: 4 |
| mysql-bin.000001 | 123 | Previous_gtids | 102 | 150 | |
| mysql-bin.000001 | 150 | Gtid | 101 | 211 | SET @@SESSION.GTID_NEXT= 'aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee:1' |
| mysql-bin.000001 | 211 | Query | 101 | 270 | BEGIN |
| mysql-bin.000001 | 270 | View_change | 101 | 369 | view_id=14850806532423012:1 |
| mysql-bin.000001 | 369 | Query | 101 | 434 | COMMIT |
| mysql-bin.000001 | 434 | Gtid | 101 | 495 | SET @@SESSION.GTID_NEXT= 'aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee:2' |
| mysql-bin.000001 | 495 | Query | 101 | 554 | BEGIN |
| mysql-bin.000001 | 554 | View_change | 101 | 693 | view_id=14850806532423012:2 |
| mysql-bin.000001 | 693 | Query | 101 | 758 | COMMIT |
| mysql-bin.000001 | 758 | Gtid | 102 | 819 | SET @@SESSION.GTID_NEXT= 'aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee:3' |
| mysql-bin.000001 | 819 | Query | 102 | 878 | BEGIN |
| mysql-bin.000001 | 878 | View_change | 102 | 1017 | view_id=14850806532423012:3 |
| mysql-bin.000001 | 1017 | Query | 102 | 1082 | COMMIT |
| mysql-bin.000001 | 1082 | Gtid | 101 | 1143 | SET @@SESSION.GTID_NEXT= 'aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee:4' |
| mysql-bin.000001 | 1143 | Query | 101 | 1250 | create schema if not exists test |
| mysql-bin.000001 | 1250 | Gtid | 101 | 1311 | SET @@SESSION.GTID_NEXT= 'aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee:5' |
| mysql-bin.000001 | 1311 | Query | 101 | 1440 | DROP TABLE IF EXISTS `test`.`t1` /* generated by server */ |
| mysql-bin.000001 | 1440 | Gtid | 101 | 1501 | SET @@SESSION.GTID_NEXT= 'aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee:6' |
| mysql-bin.000001 | 1501 | Query | 101 | 1630 | create table test.t1(id int not null primary key, sid int) |
| mysql-bin.000001 | 1630 | Gtid | 101 | 1691 | SET @@SESSION.GTID_NEXT= 'aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee:7' |
| mysql-bin.000001 | 1691 | Query | 101 | 1755 | BEGIN |
| mysql-bin.000001 | 1755 | Table_map | 101 | 1797 | table_id: 219 (test.t1) |
| mysql-bin.000001 | 1797 | Write_rows | 101 | 1837 | table_id: 219 flags: STMT_END_F |
| mysql-bin.000001 | 1837 | Xid | 101 | 1864 | COMMIT /* xid=51 */ |
| mysql-bin.000001 | 1864 | Gtid | 102 | 1925 | SET @@SESSION.GTID_NEXT= 'aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee:1000003' |
| mysql-bin.000001 | 1925 | Query | 102 | 2032 | create schema if not exists test |
| mysql-bin.000001 | 2032 | Gtid | 102 | 2093 | SET @@SESSION.GTID_NEXT= 'aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee:1000004' |
| mysql-bin.000001 | 2093 | Query | 102 | 2222 | DROP TABLE IF EXISTS `test`.`t2` /* generated by server */ |
| mysql-bin.000001 | 2222 | Gtid | 102 | 2283 | SET @@SESSION.GTID_NEXT= 'aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee:1000005' |
| mysql-bin.000001 | 2283 | Query | 102 | 2412 | create table test.t2(id int not null primary key, sid int) |
| mysql-bin.000001 | 2412 | Gtid | 102 | 2473 | SET @@SESSION.GTID_NEXT= 'aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee:1000006' |
| mysql-bin.000001 | 2473 | Query | 102 | 2542 | BEGIN |
| mysql-bin.000001 | 2542 | Table_map | 102 | 2584 | table_id: 220 (test.t2) |
| mysql-bin.000001 | 2584 | Write_rows | 102 | 2624 | table_id: 220 flags: STMT_END_F |
| mysql-bin.000001 | 2624 | Xid | 102 | 2651 | COMMIT /* xid=62 */ |
| mysql-bin.000001 | 2651 | Gtid | 103 | 2712 | SET @@SESSION.GTID_NEXT= 'aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee:2000003' |
| mysql-bin.000001 | 2712 | Query | 103 | 2819 | create schema if not exists test |
| mysql-bin.000001 | 2819 | Gtid | 103 | 2880 | SET @@SESSION.GTID_NEXT= 'aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee:2000004' |
| mysql-bin.000001 | 2880 | Query | 103 | 3009 | DROP TABLE IF EXISTS `test`.`t3` /* generated by server */ |
| mysql-bin.000001 | 3009 | Gtid | 103 | 3070 | SET @@SESSION.GTID_NEXT= 'aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee:2000005' |
| mysql-bin.000001 | 3070 | Query | 103 | 3199 | create table test.t3(id int not null primary key, sid int) |
| mysql-bin.000001 | 3199 | Gtid | 103 | 3260 | SET @@SESSION.GTID_NEXT= 'aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee:2000006' |
| mysql-bin.000001 | 3260 | Query | 103 | 3324 | BEGIN |
| mysql-bin.000001 | 3324 | Table_map | 103 | 3366 | table_id: 221 (test.t3) |
| mysql-bin.000001 | 3366 | Write_rows | 103 | 3406 | table_id: 221 flags: STMT_END_F |
| mysql-bin.000001 | 3406 | Xid | 103 | 3433 | COMMIT /* xid=68 */ |
+------------------+------+----------------+-----------+-------------+-------------------------------------------------------------------------+
47 rows in set (0.00 sec)
</code></pre><p>Two important points:</p><ul><li>All transaction IDs are assigned to the group, not to the node. The only way to see where the data is coming from is to look at the binary log itself and check the good old server-id. One wonders why we have come all this way with the ugly UUIDs in the global transaction identifier only to maim their usefulness by removing one of the most important feature, which is tracking the data origin.</li>
</ul><p>For example:</p><pre><code># at 434
#170122 11:24:11 server id 101 end_log_pos 495 GTID last_committed=1 sequence_number=2
SET @@SESSION.GTID_NEXT= 'aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee:2'/*!*/;
# at 495
#170122 11:24:11 server id 101 end_log_pos 554 Query thread_id=7 exec_time=6 error_code=0
SET TIMESTAMP=1485080651/*!*/;
BEGIN
/*!*/;
</code></pre><ul><li>Because <em>log-slave-updates</em> is mandatory, the binary log in every node will have all the transactions of every other node. This can have disagreeable side effects when dealing with large data. Here is an example when we load <a href="https://github.com/datacharmer/test_db">the sample employee database</a> from node #1:</li>
</ul><p>With Group Replication, the load takes 2 minutes and 16 seconds, and the binary logs have the same size in every node.</p><pre><code>[GR]$ ls -lh node?/data/*bin*
-rw-r----- 1 gmax staff 8.2K Jan 22 10:22 node1/data/mysql-bin.000001
-rw-r----- 1 gmax staff 63M Jan 22 10:24 node1/data/mysql-bin.000002
-rw-r----- 1 gmax staff 38B Jan 22 10:22 node1/data/mysql-bin.index
-rw-r----- 1 gmax staff 63M Jan 22 10:24 node2/data/mysql-bin.000001
-rw-r----- 1 gmax staff 19B Jan 22 10:12 node2/data/mysql-bin.index
-rw-r----- 1 gmax staff 63M Jan 22 10:24 node3/data/mysql-bin.000001
-rw-r----- 1 gmax staff 19B Jan 22 10:12 node3/data/mysql-bin.index
</code></pre><p>The same operation in multi-source replication takes 1 minute and 30 seconds. The binary logs are kept only in the origin.</p><pre><code>[MS]$ ls -lh node?/data/*bin*
-rw-r----- 1 gmax staff 4.9K Jan 22 10:26 node1/data/mysql-bin.000001
-rw-r----- 1 gmax staff 63M Jan 22 10:27 node1/data/mysql-bin.000002
-rw-r----- 1 gmax staff 38B Jan 22 10:26 node1/data/mysql-bin.index
-rw-r----- 1 gmax staff 1.4K Jan 22 10:14 node2/data/mysql-bin.000001
-rw-r----- 1 gmax staff 19B Jan 22 10:14 node2/data/mysql-bin.index
-rw-r----- 1 gmax staff 1.4K Jan 22 10:14 node3/data/mysql-bin.000001
-rw-r----- 1 gmax staff 19B Jan 22 10:14 node3/data/mysql-bin.index
</code></pre><h2>Conflict resolution</h2><p>One of the strong points of MGR is conflict resolution. </p><p>We can try a conflicting operations in two nodes, inserting the same data at the same time:</p><pre><code>use test;
set autocommit=0;
insert into t2 values (3, @@server_id);
commit;
</code></pre><p>In multi source, we get a replication error, on both nodes. It's an ugly result, but it tells the user immediately that something went wrong in a given node, and doesn't let the error propagate to other nodes.</p><p>In MGR, the situation varies. This is a possible outcome:</p><pre><code>node1 [localhost] {msandbox} (test) > set autocommit=0; | node2 [localhost] {msandbox} (test) > set autocommit=0;
Query OK, 0 rows affected (0.00 sec) | Query OK, 0 rows affected (0.00 sec)
|
node1 [localhost] {msandbox} (test) > insert into t2 values (3, @@server_id); | node2 [localhost] {msandbox} (test) > insert into t2 values (3, @@server_id);
Query OK, 1 row affected (0.00 sec) | Query OK, 1 row affected (0.00 sec)
|
node1 [localhost] {msandbox} (test) > select * from t2; | node2 [localhost] {msandbox} (test) > select * from t2;
+----+------+ | +----+------+
| id | sid | | | id | sid |
+----+------+ | +----+------+
| 2 | 102 | | | 2 | 102 |
| 3 | 101 | | | 3 | 102 |
+----+------+ | +----+------+
2 rows in set (0.00 sec) | 2 rows in set (0.00 sec)
|
node1 [localhost] {msandbox} (test) > commit; | node2 [localhost] {msandbox} (test) > commit;
Query OK, 0 rows affected (0.01 sec) | ERROR 3101 (HY000): Plugin instructed the server to rollback the current transaction.
| node2 [localhost] {msandbox} (test) > select * from t2;
node1 [localhost] {msandbox} (test) > select * from t2; | +----+------+
+----+------+ | | id | sid |
| id | sid | | +----+------+
+----+------+ | | 2 | 102 |
| 2 | 102 | | | 3 | 101 |
| 3 | 101 | | +----+------+
+----+------+ | 2 rows in set (0.00 sec)
2 rows in set (0.00 sec) |
</code></pre><p>Here node # 2 got the transaction a fraction of second later, and its transaction was rolled back. Thus the transaction that was ultimately kept in the database was the one from node1 (server-id 101.) However, this behavior is not predictable. If we try the same operation again, we get a different outcome:</p><pre><code>node1 [localhost] {msandbox} (test) > insert into t2 values (4, @@server_id); | node2 [localhost] {msandbox} (test) > insert into t2 values (4, @@server_id);
Query OK, 1 row affected (0.00 sec) | Query OK, 1 row affected (0.00 sec)
|
node1 [localhost] {msandbox} (test) > select * from t2; | node2 [localhost] {msandbox} (test) > select * from t2;
+----+------+ | +----+------+
| id | sid | | | id | sid |
+----+------+ | +----+------+
| 2 | 102 | | | 2 | 102 |
| 3 | 101 | | | 3 | 101 |
| 4 | 101 | | | 4 | 102 |
+----+------+ | +----+------+
3 rows in set (0.00 sec) | 3 rows in set (0.00 sec)
|
node1 [localhost] {msandbox} (test) > commit; | node2 [localhost] {msandbox} (test) > commit;
Query OK, 0 rows affected (0.01 sec) |
ERROR 3101 (HY000): Plugin instructed the server to rollback |
the current transaction. |
node1 [localhost] {msandbox} (test) > select * from t2; | node2 [localhost] {msandbox} (test) > select * from t2;
+----+------+ | +----+------+
| id | sid | | | id | sid |
+----+------+ | +----+------+
| 2 | 102 | | | 2 | 102 |
| 3 | 101 | | | 3 | 101 |
| 4 | 102 | | | 4 | 102 |
+----+------+ | +----+------+
4 rows in set (0.00 sec) | 3 rows in set (0.00 sec)
</code></pre><p>In the second attempt, the transaction was rolled back by node 1, and the surviving one is the one that was inserted from node 2. This means that conflict resolution works, but it may not be what the user wants, as the resolved conflict if aleatory.</p><h2>Summing up</h2><p>On the plus side, MGR keeps what it promises. We can set up a cluster of peer nodes and replicate data between nodes with some advantages compared to older multi-source topologies. </p><p>On the minus side, the documentation could be vastly improved, especially for multi-primary setup. Moreover, users need to be aware of the <a href="https://dev.mysql.com/doc/refman/5.7/en/group-replication-limitations.html">limitations</a>, such as serializable isolation level and foreign keys with constraints not being supported.</p><p>Most important from my standpoint is the reduction of monitoring information for this technology, namely the loss of information about the data origin.</p>Giuseppe Maxiahttp://www.blogger.com/profile/15801583338057324813noreply@blogger.com0tag:blogger.com,1999:blog-16959946.post-66979930709311354932017-01-15T23:34:00.000+01:002017-01-15T23:34:02.114+01:00MySQL group replication: installation with Docker<h2>Overview</h2><p><a href="https://dev.mysql.com/doc/refman/5.7/en/group-replication.html">MySQL Group Replication</a> was released as GA with MySQL 5.7.17. It is essentially a plugin that, when enabled, allows users to set replication with this new way.</p><p>There has been some confusion about the stability and usability of this release. Until recently, MySQL Group Replication (MGR) was only available in the <a href="https://labs.mysql.com/">Labs</a>, which traditionally denotes a preview or an <em>use-at-your-own-risk</em> feature. Several months ago we saw the release of Group Replication as a Docker image, which allowed users to deploy a peer-to-peer cluster (every node is a master.) However, about one month after such release, word came from Oracle discouraging this setup, and inviting users to use Group Replicator in <a href="https://dev.mysql.com/doc/refman/5.7/en/group-replication-single-primary-mode.html">Single Primary mode</a> which is functionally equivalent to traditional replication, with just some synchronous component more. There hasn't been an update of MGR for Docker since.</p><p>BTW, some more confusion came from the use of "synchronous replication" to refer to Group Replication operations. In reality, what in many presentations was called <em>synchronous replication</em> is only a synchronous transfer of binary logs data. The replication itself, i.e. the operation that makes a node able to retrieve the data inserted in the master, is completed asynchronously. Therefore, if you looked at MGR as a way of using multiple masters without conflicts, this is not the solution.</p><p>What we have is a way of replicating from a node that is the Primary in the group, with some features designed to facilitate high availability solutions. And all eyes are on the next product, which is based on MGR, named <a href="http://mysqlserverteam.com/mysql-innodb-cluster-a-hands-on-tutorial/">MySQL InnoDB Cluster</a> which is MGR + an hormone pumped <a href="https://dev.mysql.com/doc/refman/5.7/en/mysql-shell.html">MySQL Shell</a> (released with the same version number 1.0.5 in two different packages,) and <a href="https://dev.mysql.com/doc/mysql-router/en/">MySQL-Router</a>.</p><p>MGR has several <a href="https://dev.mysql.com/doc/refman/5.7/en/group-replication-limitations.html">limitations</a>, mostly related to multi-primary mode.</p><p>Another thing that users should know is that the <a href="http://mysqlhighavailability.com/performance-evaluation-mysql-5-7-group-replication/">performance</a> of MGR is inferior to that of asynchronous replication, even in Single-Primary mode. As an example, loading the <a href="https://github.com/datacharmer/test_db">test employees database</a> takes 92 seconds in MGR, against 49 seconds in asynchronous replication (same O.S., same MySQL version, same server setup.)</p><h2>Installing MySQL Group Replication</h2><p>One of the biggest issue with MGR has been the quality of its documentation, which for a while was just lack of documentation altogether. <a href="https://dev.mysql.com/doc/refman/5.7/en/group-replication-getting-started-deploying-instances.html">What we have now</a> has a set of instructions that refers to installing group replication in three nodes on the same host. You know, sandboxes, although without the benefit of using <a href="http://mysqlsandbox.net">a tool</a> to simplify operations. It's just three servers on the same host, and you drive with stick shift.</p><p>What we'll see in this post is how to set group replication using three servers in Docker. The advantage of using this approach is that the servers look and feel like real ones. Since the instructions assume that you are only playing with sandboxes (an odd assumption for a GA product) we lack the instructions for a real world setup. The closest thing to a useful manual is the <a href="https://www.percona.com/live/plam16/sessions/mysql-group-replication-nutshell-hands-tutorial">tutorial</a> given by Frédéric Descamps and Kenny Gryp at PerconaLive Amsterdam in October. The instructions, however, are muddled up by the fact that they were using the still unreliable InnoDB Cluster instead of a bare bones Group Replicator. What follows is my own expansion of the sandboxed rules as applied to distinct servers.</p><h3>The environment:</h3><p>I am using Docker 1.12.6 on Linux, and the image for <a href="https://store.docker.com/community/images/mysql/mysql-server">mysql/mysql-server:5.7.17</a>. I deploy three containers, with a customized my.cnf containing the bare minimum options to run Group Replication. Here's the template for the configuration files:</p><pre><code>$ cat my-template.cnf
[mysqld]
user=mysql
server_id=_SERVER_ID_
gtid_mode=ON
enforce_gtid_consistency=ON
master_info_repository=TABLE
relay_log_info_repository=TABLE
binlog_checksum=NONE
log_slave_updates=ON
log_bin=mysql-bin
relay-log=relay
binlog_format=ROW
log-error=mysqld.err
transaction_write_set_extraction=XXHASH64
loose-group_replication_group_name="aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee"
loose-group_replication_start_on_boot=off
loose-group_replication_local_address= "172.19.0._IP_END_:6606"
loose-group_replication_group_seeds= "172.19.0.2:6606,172.19.0.3:6606,172.19.0.4:6606"
loose-group_replication_ip_whitelist="172.19.0.2,172.19.0.3,172.19.0.4,127.0.0.1"
loose-group_replication_bootstrap_group= off
</code></pre><p>Here I take a shortcut. Recent versions of Docker assign a predictable IP address to new containers. To make sure I get the right IPs, I use a private network to deploy the containers. In a perfect world, I should use the container names for this purpose, but the manual lacks the instructions to set up the cluster progressively. For now, this method requires full knowledge about the IPs of the nodes, and I play along with what I have.</p><p>This is the deployment script:</p><pre><code>#!/bin/bash
exists_net=$(docker network ls | grep -w group1 )
if [ -z "$exists_net" ]
then
docker network create group1
fi
docker network ls
for node in 1 2 3
do
export SERVERID=$node
export IPEND=$(($SERVERID+1))
perl -pe 's/_SERVER_ID_/$ENV{SERVERID}/;s/_IP_END_/$ENV{IPEND}/' my-template.cnf > my${node}.cnf
datadir=ddnode${node}
if [ ! -d $datadir ]
then
mkdir $datadir
fi
unset SERVERID
docker run -d --name=node$node --net=group1 --hostname=node$node \
-v $PWD/my${node}.cnf:/etc/my.cnf \
-v $PWD/data:/data \
-v $PWD/$datadir:/var/lib/mysql \
-e MYSQL_ROOT_PASSWORD=secret \
mysql/mysql-server:5.7.17
ip=$(docker inspect --format '{{ .NetworkSettings.Networks.group1.IPAddress}}' node${node})
echo "${node} $ip"
done
</code></pre><p>This script deploys three nodes, called node1, node2, and node3. For each one, the template is modified to use a different server ID. They use an external data directory created on the current directory (see <a href="http://datacharmer.blogspot.com/2015/11/mysql-docker-operations-part-2.html">Customizing MYSQL in Docker</a> for more details on this technique.) Moreover, each node can access the folder /data, which contains this set of SQL commands:</p><pre><code>reset master;
SET SQL_LOG_BIN=0;
CREATE USER rpl_user@'%';
GRANT REPLICATION SLAVE ON *.* TO rpl_user@'%' IDENTIFIED BY 'rpl_pass';
SET SQL_LOG_BIN=1;
CHANGE MASTER TO MASTER_USER='rpl_user', MASTER_PASSWORD='rpl_pass' FOR CHANNEL 'group_replication_recovery';
INSTALL PLUGIN group_replication SONAME 'group_replication.so';
</code></pre><h3>Operations</h3><p>After deploying the containers using the above script, I wait a few seconds to give time to the servers to be ready. I can peek at the error logs, which are in the directories ddnode1, ddnode2, and ddnode3, as defined in the installation command. Then I run the SQL code:</p><p><code><pre>$ for N in 1 2 3; do docker exec -ti node$N bash -c 'mysql -psecret < /data/user.sql' ; done</pre></code></p><p>At this stage, the plugin is installed in all three nodes. I can start the cluster:</p><pre><code>$ docker exec -ti node1 mysql -psecret
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 5
Server version: 5.7.17-log MySQL Community Server (GPL)
Copyright (c) 2000, 2016, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> SET GLOBAL group_replication_bootstrap_group=ON;
Query OK, 0 rows affected (0.00 sec)
mysql> START GROUP_REPLICATION;
Query OK, 0 rows affected (1.14 sec)
mysql>SET GLOBAL group_replication_bootstrap_group=OFF;
Query OK, 0 rows affected (0.00 sec)
mysql> select * from performance_schema.replication_group_members;
+---------------------------+--------------------------------------+-------------+-------------+--------------+
| CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE |
+---------------------------+--------------------------------------+-------------+-------------+--------------+
| group_replication_applier | ecba1582-db68-11e6-a492-0242ac130002 | node1 | 3306 | ONLINE |
+---------------------------+--------------------------------------+-------------+-------------+--------------+
1 row in set (0.00 sec)
</code></pre><p>The above operations have started the replication with the <em>bootstrap</em>, an operation that must be executed only once, and that defines the primary node.</p><p>After setting the replication, I can enter some data, and then see what happens in the other nodes:</p><pre><code>mysql> create schema test;
Query OK, 1 row affected (0.01 sec)
mysql> use test
Database changed
mysql> create table t1 (id int not null primary key, msg varchar(20));
Query OK, 0 rows affected (0.06 sec)
mysql> insert into t1 values (1, 'hello from node1');
Query OK, 1 row affected (0.01 sec)
mysql> show binlog events;
+------------------+------+----------------+-----------+-------------+----------------------------------------------------------------------------+
| Log_name | Pos | Event_type | Server_id | End_log_pos | Info |
+------------------+------+----------------+-----------+-------------+----------------------------------------------------------------------------+
| mysql-bin.000001 | 4 | Format_desc | 1 | 123 | Server ver: 5.7.17-log, Binlog ver: 4 |
| mysql-bin.000001 | 123 | Previous_gtids | 1 | 150 | |
| mysql-bin.000001 | 150 | Gtid | 1 | 211 | SET @@SESSION.GTID_NEXT= 'aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee:1' |
| mysql-bin.000001 | 211 | Query | 1 | 270 | BEGIN |
| mysql-bin.000001 | 270 | View_change | 1 | 369 | view_id=14845163185775300:1 |
| mysql-bin.000001 | 369 | Query | 1 | 434 | COMMIT |
| mysql-bin.000001 | 434 | Gtid | 1 | 495 | SET @@SESSION.GTID_NEXT= 'aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee:2' |
| mysql-bin.000001 | 495 | Query | 1 | 554 | BEGIN |
| mysql-bin.000001 | 554 | View_change | 1 | 693 | view_id=14845163185775300:2 |
| mysql-bin.000001 | 693 | Query | 1 | 758 | COMMIT |
| mysql-bin.000001 | 758 | Gtid | 1 | 819 | SET @@SESSION.GTID_NEXT= 'aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee:3' |
| mysql-bin.000001 | 819 | Query | 1 | 912 | create schema test |
| mysql-bin.000001 | 912 | Gtid | 1 | 973 | SET @@SESSION.GTID_NEXT= 'aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee:4' |
| mysql-bin.000001 | 973 | Query | 1 | 1110 | use `test`; create table t1 (id int not null primary key, msg varchar(20)) |
| mysql-bin.000001 | 1110 | Gtid | 1 | 1171 | SET @@SESSION.GTID_NEXT= 'aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee:5' |
| mysql-bin.000001 | 1171 | Query | 1 | 1244 | BEGIN |
| mysql-bin.000001 | 1244 | Table_map | 1 | 1288 | table_id: 219 (test.t1) |
| mysql-bin.000001 | 1288 | Write_rows | 1 | 1341 | table_id: 219 flags: STMT_END_F |
| mysql-bin.000001 | 1341 | Xid | 1 | 1368 | COMMIT /* xid=144 */ |
+------------------+------+----------------+-----------+-------------+----------------------------------------------------------------------------+
19 rows in set (0.00 sec)
</code></pre><p>The binary log events show that we are replicating using the ID of the group, instead of the ID of the single server.</p><p>In the other two nodes I run the operation a bit differently:</p><pre><code>$ docker exec -ti node2 mysql -psecret
mysql> select * from performance_schema.global_variables where variable_name in ('read_only', 'super_read_only');
+-----------------+----------------+
| VARIABLE_NAME | VARIABLE_VALUE |
+-----------------+----------------+
| read_only | OFF |
| super_read_only | OFF |
+-----------------+----------------+
2 rows in set (0.01 sec)
mysql> START GROUP_REPLICATION;
Query OK, 0 rows affected (5.62 sec)
mysql> select * from performance_schema.global_variables where variable_name in ('read_only', 'super_read_only');
+-----------------+----------------+
| VARIABLE_NAME | VARIABLE_VALUE |
+-----------------+----------------+
| read_only | ON |
| super_read_only | ON |
+-----------------+----------------+
2 rows in set (0.01 sec)
mysql> select * from performance_schema.replication_group_members;
+---------------------------+--------------------------------------+-------------+-------------+--------------+
| CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE |
+---------------------------+--------------------------------------+-------------+-------------+--------------+
| group_replication_applier | ecba1582-db68-11e6-a492-0242ac130002 | node1 | 3306 | ONLINE |
| group_replication_applier | ecf2eae5-db68-11e6-a492-0242ac130003 | node2 | 3306 | ONLINE |
+---------------------------+--------------------------------------+-------------+-------------+--------------+
2 rows in set (0.01 sec)
</code></pre><p>Now the cluster has two nodes, and I've seen that the nodes are automatically defined as read-only. I can repeat the same operation in the third one.</p><pre><code>$ docker exec -ti node2 mysql -psecret
mysql> START GROUP_REPLICATION;
Query OK, 0 rows affected (2.35 sec)
mysql> select * from performance_schema.replication_group_members;
+---------------------------+--------------------------------------+-------------+-------------+--------------+
| CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE |
+---------------------------+--------------------------------------+-------------+-------------+--------------+
| group_replication_applier | ecba1582-db68-11e6-a492-0242ac130002 | node1 | 3306 | ONLINE |
| group_replication_applier | ecf2eae5-db68-11e6-a492-0242ac130003 | node2 | 3306 | ONLINE |
| group_replication_applier | ed259dfc-db68-11e6-a4a6-0242ac130004 | node3 | 3306 | ONLINE |
+---------------------------+--------------------------------------+-------------+-------------+--------------+
3 rows in set (0.00 sec)
</code></pre><p>What about the data? It's been replicated:</p><pre><code>mysql> show schemas;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| performance_schema |
| sys |
| test |
+--------------------+
5 rows in set (0.00 sec)
mysql> show tables from test;
+----------------+
| Tables_in_test |
+----------------+
| t1 |
+----------------+
1 row in set (0.01 sec)
</code></pre><h3>Monitoring</h3><p>In this flavor of replication there is no SHOW SLAVE STATUS. Everything I've got is in performance<em>schema tables and in mysql.slave</em>master<em>info and mysql.slave</em>relay<em>log</em>info, and sadly it is not a lot.</p><pre><code>mysql> select * from replication_group_member_stats\G
*************************** 1. row ***************************
CHANNEL_NAME: group_replication_applier
VIEW_ID: 14845163185775300:3
MEMBER_ID: ecba1582-db68-11e6-a492-0242ac130002
COUNT_TRANSACTIONS_IN_QUEUE: 0
COUNT_TRANSACTIONS_CHECKED: 3
COUNT_CONFLICTS_DETECTED: 0
COUNT_TRANSACTIONS_ROWS_VALIDATING: 0
TRANSACTIONS_COMMITTED_ALL_MEMBERS: aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee:1-6
LAST_CONFLICT_FREE_TRANSACTION: aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee:5
1 row in set (0.00 sec)
mysql> select * from replication_connection_status\G
*************************** 1. row ***************************
CHANNEL_NAME: group_replication_recovery
GROUP_NAME:
SOURCE_UUID:
THREAD_ID: NULL
SERVICE_STATE: OFF
COUNT_RECEIVED_HEARTBEATS: 0
LAST_HEARTBEAT_TIMESTAMP: 0000-00-00 00:00:00
RECEIVED_TRANSACTION_SET:
LAST_ERROR_NUMBER: 0
LAST_ERROR_MESSAGE:
LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00
*************************** 2. row ***************************
CHANNEL_NAME: group_replication_applier
GROUP_NAME: aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee
SOURCE_UUID: aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee
THREAD_ID: NULL
SERVICE_STATE: ON
COUNT_RECEIVED_HEARTBEATS: 0
LAST_HEARTBEAT_TIMESTAMP: 0000-00-00 00:00:00
RECEIVED_TRANSACTION_SET: aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee:1-6
LAST_ERROR_NUMBER: 0
LAST_ERROR_MESSAGE:
LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00
2 rows in set (0.00 sec)
</code></pre><p>Compared to regular replication, we lose the ID of the node where the data was originated. Instead, we get the ID of the group replication (which we set in the configuration file.) This is useful for a smoother operation of replacing the primary node (a.k.a. the master) with another node, but we have lost some valuable information that could have been added to the output rather than simply being replaced. Another valuable piece of information that is missing is the transactions that were executed (we only see RECEIVED_TRANSACTION_SET.) As in regular replication, we can get this information with "SHOW MASTER STATUS" or "SELECT @@global.gtid_executed", but as mentioned in <a href="https://datacharmer.blogspot.com/2016/09/improving-design-of-mysql-replication.html">improving the design of MySQL replication</a> there are several flaws in this paradigm. What we see in MGR is a reduction of replication monitoring data, while we would have expected some improvement, given the complexity of the operations for this new technology.</p><h2>Summing up</h2><p>MySQL Group Replication is an interesting technology. If we consider it in the framework of a component for high availability (which will be completed when the InnoDB Cluster is released) it might improve the workflow of many database users.</p><p>As it is now, however, it gives the feeling of being a rushed up piece of software that does not offer any noticeable advantage to users, especially considering that the documentation released with it is far below the standards of other MySQL products.</p>Giuseppe Maxiahttp://www.blogger.com/profile/15801583338057324813noreply@blogger.com0tag:blogger.com,1999:blog-16959946.post-43265082529317391422016-11-03T06:00:00.000+01:002016-11-03T06:00:11.213+01:00MySQL-Sandbox 3.2.03 with customized initialization<p><a href="https://github.com/datacharmer/mysql-sandbox">MySQL-Sandbox</a> installs the MySQL server in isolation, by rejecting existing option files using the option <code>--no-defaults</code>. This is usually a good thing, because you don't want the initialization to be influenced by options in your <code>/etc/my.cnf</code> or other options files in default positions. </p><p>However, such isolation is also a problem when you need to add options during the initialization. One example is <a href="http://dev.mysql.com/doc/refman/5.7/en/innodb-parameters.html#sysvar_innodb_page_size">innodb-page-size</a>, which can be set to many values, but only if the server was initialized accordingly. Thus, you can't set <code>innodb-page-size=64K</code> in your configuration file because the default value is different. It would fail, as InnoDB would conflict. </p><p><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgMQAUBoEBXdFu82NbnWAiOdh-G2w7W4OxvDmY-aVsLbZVg7x42TG0oGuswvJKwWcySMDhfWhxS8G0cFO5lohupKrN86an8iFxbYuu4FicM67OnxFU6JOCkXkCsnyjioS7QXLHX/?imgmax=1600" alt="Mysql init " title="mysql initialization" border="0" width="200" /></p><p><a href="https://github.com/datacharmer/mysql-sandbox">MySQL-Sandbox 3.2.03</a> introduces three options that allow flexibility during initialization.</p><ul><li><code>--init_option='some options'</code> will add 'some options' to the initialization command. </li>
<li>Another possibility is <code>--init_my_cnf</code> which will load the sandbox configuration file. This is simple, but sometimes it may case initialization issues, depending on what else is in the options file.</li>
<li>Finally, <code>--init_use_cnf</code> allows you to define a custom configuration file, which will be used during initialization.</li>
</ul><p>The following three examples will all produce the wanted result, i.e. install MySQL with a custom <code>innodb-page-size</code> of 64K.</p><pre><code>make_sandbox 5.7.16 -- -c innodb-page-size=64K --init_option='--innodb-page-size=64K'
make_sandbox 5.7.16 -- -c innodb-page-size=64K --init_my_cnf
cat /tmp/my.cnf
[mysqld]
innodb-page-size=64K
make_sandbox 5.7.16 -- -c innodb-page-size=64K --init_use_cnf=/tmp/my.cnf
</code></pre>Giuseppe Maxiahttp://www.blogger.com/profile/15801583338057324813noreply@blogger.com0