Monday, April 22, 2013

Installing and administering Tungsten Replicator - Part 2 : advanced

Switching roles

To get a taste of the power of Tungsten Replicator, we will show how to switch roles. This is a controlled operation (as opposed to fail-over), where we can decide when to switch and which nodes are involved.

In our topology, host1 is the master, and we have three slaves. We can either ask for a switch and let the script select the first available slave, or tell the script which slave should be promoted. The script will show us the steps needed to perform the operation.

IMPORTANT! Please note that this operation is not risk free. Tungsten replicator is a simple replication system, not a complete management tool like Continuent Tungsten. WIth the replicator, you must make sure that the applications have stopped writing to the master before starting the switch, and then you should address the application to the new master when the operation is done.

$ cookbook/switch host2
# Determining current roles
host1 master
host2 slave
host3 slave
host4 slave
# Will promote host2 to be the new master server
# Waiting for slaves to catch up and pausing replication
trepctl -host host2 wait -applied 5382
trepctl -host host2 offline
trepctl -host host3 wait -applied 5382
trepctl -host host3 offline
trepctl -host host4 wait -applied 5382
trepctl -host host4 offline
trepctl -host host1 offline
# Reconfiguring server roles and restarting replication
trepctl -host host2 setrole -role master
trepctl -host host2 online
trepctl -host host1 setrole -role slave -uri thl://host2:2112
trepctl -host host1 online
trepctl -host host3 setrole -role slave -uri thl://host2:2112
trepctl -host host3 online
trepctl -host host4 setrole -role slave -uri thl://host2:2112
trepctl -host host4 online
--------------------------------------------------------------------------------------
Topology: 'MASTER_SLAVE'
--------------------------------------------------------------------------------------
# node host1
cookbook  [slave]   seqno:      5,384  - latency:   2.530 - ONLINE

# node host2
cookbook  [master]  seqno:      5,384  - latency:   2.446 - ONLINE

# node host3
cookbook  [slave]   seqno:      5,384  - latency:   2.595 - ONLINE

# node host4
cookbook  [slave]   seqno:      5,384  - latency:   2.537 - ONLINE

As you can see from the listing above, The script displays the steps for the switch, using trepctl as a centralized tool.

Under load

After the simple installation in Part 1, we saw that we can test the flow of replication using 'cookbook/test_cluster'. That's a very simple set of operations that merely checks if replication is working. If we want to perform more serious tests, we should apply a demanding load to the replication system.

If you don't have applications that can exercise the servers to your liking, you should be pleased to know that Tungsten Replicator ships with a built-in application for data loading and benchmarking. Inside the expanded tarball, there is a directory named bristlecone, containing the software for such testing tools. There is a detailed set of instructions under './bristlecone/doc'. For the impatient, there is a cookbook recipe that starts a reasonable load with a single command:

$ cookbook/load_data start
# Determining current roles
# Evaluator started with pid 28370
# Evaluator details are available at /home/tungsten/installs/cookbook/tungsten/load/host1/evaluator.job
# Evaluator output can be monitored at /home/tungsten/installs/cookbook/tungsten/load/host1/evaluator.log

$ cat /home/tungsten/installs/cookbook/tungsten/load/host1/evaluator.job
Task started at   : Sun Apr  7 18:20:00 2013
Task started from : /home/tungsten/tinstall/current
Executable        : /home/tungsten/installs/cookbook/tungsten/bristlecone/bin/evaluator.sh
Process ID        : 28370
Using             : /home/tungsten/installs/cookbook/tungsten/load/host1/evaluator.xml
Process id        : /home/tungsten/installs/cookbook/tungsten/load/host1/evaluator.pid
Log               : /home/tungsten/installs/cookbook/tungsten/load/host1/evaluator.log
Database          : host1
Table prefix      : tbl
Host              : host1
Port              : 3306
User              : tungsten
Test duration     : 3600

$  tail /home/tungsten/installs/cookbook/tungsten/load/host1/evaluator.log
18:22:18,672 INFO  1365351738672 10/10 5035.0 ops/sec 0 ms/op 28380 rows/select 41 updates 54 deletes 166 inserts
18:22:20,693 INFO  1365351740693 10/10 4890.0 ops/sec 0 ms/op 26746 rows/select 57 updates 37 deletes 144 inserts
18:22:22,697 INFO  1365351742697 10/10 4986.0 ops/sec 0 ms/op 28183 rows/select 59 updates 46 deletes 162 inserts
18:22:24,716 INFO  1365351744716 10/10 5208.0 ops/sec 0 ms/op 29067 rows/select 51 updates 51 deletes 171 inserts
18:22:26,736 INFO  1365351746736 10/10 4856.0 ops/sec 0 ms/op 27695 rows/select 46 updates 68 deletes 141 inserts
18:22:28,739 INFO  1365351748739 10/10 5022.0 ops/sec 0 ms/op 28269 rows/select 51 updates 58 deletes 145 inserts
18:22:30,758 INFO  1365351750758 10/10 4893.0 ops/sec 0 ms/op 28484 rows/select 47 updates 50 deletes 165 inserts
18:22:32,777 INFO  1365351752777 10/10 4501.0 ops/sec 0 ms/op 26481 rows/select 42 updates 52 deletes 130 inserts
18:22:34,781 INFO  1365351754781 10/10 5057.0 ops/sec 0 ms/op 30450 rows/select 58 updates 53 deletes 157 inserts
18:22:36,801 INFO  1365351756801 10/10 5087.0 ops/sec 0 ms/op 30845 rows/select 55 updates 56 deletes 156 inserts

What happens here?

The evaluator process is started using a file named 'evaluator.xml,' which is generated dynamically. The cookbook recipe detects which is the current master in the replication system and directs the operations there (in our case, it's 'host1'). The same task takes note of the process ID, which will be used to stop the evaluator when done, and the output is sent to a file, where you can look at it if needed.

Looking at evaluator.log, you can see that there are quite a lot of operations going on. Most of them are read queries, as the application was designed to solicit a database server as much as possible. Nonetheless, there are quite a lot of update operations, as a call to 'show_cluster' can confirm.

$ cookbook/show_cluster
--------------------------------------------------------------------------------------
Topology: 'MASTER_SLAVE'
--------------------------------------------------------------------------------------
# node host1
cookbook  [master]  seqno:     30,292  - latency:   0.566 - ONLINE

# node host2
cookbook  [slave]   seqno:     30,277  - latency:   0.531 - ONLINE

# node host3
cookbook  [slave]   seqno:     30,269  - latency:   0.511 - ONLINE

# node host4
cookbook  [slave]   seqno:     30,287  - latency:   0.550 - ONLINE

The load will continue for one hour (unless you defined a different duration). SHould you want to stop it before that period, you can run:

$ cookbook/load_data stop
# Determining current roles
# Stopping Evaluator at pid 28370

One important piece of information about this load application is that it looks for the masters in your cluster, and starts a load in every master. This is useful if you want to test a multi-master topology, as the ones we will see in another article.

If the default behavior of load_data is not what you expect, you can further customize the load by fine tuning the application launcher. First, you run 'load_data' with the print option:

$ cookbook/load_data print
# Determining current roles
$HOME/installs/cookbook/tungsten/bristlecone/bin/concurrent_evaluator.pl \
    --deletes=1 \
    --updates=1 \
    --inserts=3 \
    --test-duration=3600 \
    --host=host1 \
    --port=3306 \
    -u tungsten \
    -p secret  \
    --continuent-root=/home/tungsten/installs/cookbook \
    -d host1 \
    -s /home/tungsten/installs/cookbook/tungsten/load/host1  start

Then, you can copy and paste the resulting command, and eventually run the concurrent_evaluator script with your additions.

There are many options available. The manual is embedded in the application itself:

$ ./bristlecone/bin/concurrent_evaluator.pl --manual

An important option that we can use is --instances=N. This option will launch concurrently the evaluator N times, each time using a different schema. We will use this option to test parallel replication.

Backup

I am not going to stress here how important backups are. I assume (perhaps foolishly) that everyone reading this article knows why. Instead, I want to show how Tungsten Replicator supports backup and restore as integrated methods.

When you install Tungsten, you can add options to select a backup method and fine tune its behavior.

$ ./tools/tungsten-installer --help-master-slave -a |grep backup
--backup-directory            Permanent backup storage directory [$TUNGSTEN_HOME/backups]
                              This directory should be accessible by every replicator to ensure simple operation of backup and restore.
--backup-method               Database backup method (none|mysqldump|xtrabackup|script) [xtrabackup-full]
                              Tungsten integrates with a variety of backup mechanisms. We strongly recommend you configure one of these to help with provisioning servers. Please consult the Tungsten
                              Replicator Guide for more information on backup configuration.
--backup-dump-directory       Backup temporary dump directory [/tmp]
--backup-retention            Number of backups to retain [3]
--backup-script               What is the path to the backup script
--backup-command-prefix       Use sudo when running the backup script? [false]
--backup-online               Does the backup script support backing up a datasource while it is ONLINE [false]

First off, the default directory for backups is under your installation directory ($TUNGSTEN_HOME/backups). If you want to take backups through Tungsten, you must make sure that there is enough storage in that path to hold at least one backup. Tungsten will keep up to three backups in that directory, but you can define this action differently.

Second, the default backup method is 'mysqldump,' not because it is recommended, but because it is widely available. As you probably know, though, if your database is more than a few dozen GB, mysqldump is not an adequate method.

Tungsten Replicator provides support for xtrabackup. If xtrabackup is installed in your servers, you can define it as your default backup method. When you are installing a new cluster, you can do this:

$ export MORE_OPTIONS='-a --backupmethod=xtrabackup --backup-command-prefix=true'
$ cookbook/install_master_slave

If you have just installed and need to reconfigure, you can call 'configure_service' to accomplish the task:

$ cookbook/configure_service -U -a --backup-method=xtrabackup --backup-command-prefix=true cookbook

(Where 'cookbook' is the service name). VERY IMPORTANT: configure_service acts on a single host, and by default it acts on the current host, unless you say otherwise. For example:

$ cookbook/configure_service -U --host=host2 -a --backup-method=xtrabackup --backup-command-prefix=true cookbook

You will have to restart the replicator in node 'host2' for the changes to take effect.

$ ssh host2 "cd $TUNGSTEN_BASE/tungsten/ ; ./cookbook/replicator restart"

Using the backup is quite easy. You only need to call 'trepctl', indicate in which host you want to take a backup, and Tungsten will do the rest.

$ cookbook/trepctl -host host3 backup
Backup completed successfully; URI=storage://file-system/store-0000000001.properties

$ cookbook/trepctl -host host2 backup
Backup completed successfully; URI=storage://file-system/store-0000000001.properties

Apparently, we have two backups with the same contents, taken from two different nodes. However, since we have changed the backup method for host2, we will have a mysqldump small file for host3, and a rather larger xtrabackup file for host2. Again, the cookbook has a method that shows the backups that are available in all the nodes:

$ ./cookbook/backups
   backup-agent : (service: cookbook) mysqldump
     backup-dir : (service: cookbook) /home/tungsten/installs/cookbook/backups/cookbook
# [node: host1] 0 files found
# [node: host2] 3 files found
++ /home/tungsten/installs/cookbook/backups/cookbook
total 2.4G
-rw-r--r-- 1 tungsten tungsten   72 Apr  7 21:52 storage.index
-rw-r--r-- 1 tungsten tungsten 2.4G Apr  7 21:52 store-0000000001-full_xtrabackup_2013-04-07_21-50_59.tar
-rw-r--r-- 1 tungsten tungsten  323 Apr  7 21:52 store-0000000001.properties
drwxr-xr-x 2 tungsten tungsten 4.0K Apr  7 21:52 xtrabackup

# [node: host3] 3 files found
++ /home/tungsten/installs/cookbook/backups/cookbook
total 6.3M
-rw-r--r-- 1 tungsten tungsten   72 Apr  7 21:50 storage.index
-rw-r--r-- 1 tungsten tungsten 6.3M Apr  7 21:50 store-0000000001-mysqldump_2013-04-07_21-50_28.sql.gz
-rw-r--r-- 1 tungsten tungsten  315 Apr  7 21:50 store-0000000001.properties

# [node: host4] 0 files found

WARNING: This example was here only to show how to change the backup method. It is NOT recommended to have mixed methods for backups in different nodes. Unless you have a specific need, and understand the consequence of this choice, you should have the same backup method everywhere.

Restore

A backup is only good if you can use to restore your data. Using the same method shown to take a backup, you can restore your data. For this example, let's use mysqldump in all nodes (just because it's quicker), and show the operations for a backup and restore.

First, we take a backup in node 'host3', and then we will restore the data in 'host2'.

$ cookbook/trepctl -host host3 backup
Backup completed successfully; URI=storage://file-system/store-0000000001.properties

$ cookbook/backups
   backup-agent : (service: cookbook) mysqldump
     backup-dir : (service: cookbook) /home/tungsten/installs/cookbook/backups/cookbook
# [node: host1] 0 files found
# [node: host2] 0 files found
# [node: host3] 3 files found
++ /home/tungsten/installs/cookbook/backups/cookbook
total 6.2M
-rw-r--r-- 1 tungsten tungsten   72 Apr  7 22:05 storage.index
-rw-r--r-- 1 tungsten tungsten 6.1M Apr  7 22:05 store-0000000001-mysqldump_2013-04-07_22-05_43.sql.gz
-rw-r--r-- 1 tungsten tungsten  315 Apr  7 22:05 store-0000000001.properties

# [node: host4] 0 files found

Now, we have the backup files in host3, but we have an issue in host2, and we need to take a restore there. Assuming that the database server is unusable (this is usually the case when we must take a restore), we have the unpleasant situation where the backups are in one node, and we need to use in another. In a well organized environment, we would have a shared storage for the backup directory, and thus we could just move ahead and perform our restore. In this case, though, we have no such luxury. Then, we use yet another feature of the cookbook:

$ cookbook/copy_backup
syntax: copy_backup SERVICE SOURCE_NODE DESTINATION_NODE

$ cookbook/copy_backup cookbook host3 host2
# No message = success

$ cookbook/backups
   backup-agent : (service: cookbook) mysqldump
     backup-dir : (service: cookbook) /home/tungsten/installs/cookbook/backups/cookbook
# [node: host1] 0 files found
# [node: host2] 3 files found
++ /home/tungsten/installs/cookbook/backups/cookbook
total 6.2M
-rw-r--r-- 1 tungsten tungsten   72 Apr  7 22:05 storage.index
-rw-r--r-- 1 tungsten tungsten 6.1M Apr  7 22:05 store-0000000001-mysqldump_2013-04-07_22-05_43.sql.gz
-rw-r--r-- 1 tungsten tungsten  315 Apr  7 22:05 store-0000000001.properties

# [node: host3] 3 files found
++ /home/tungsten/installs/cookbook/backups/cookbook
total 6.2M
-rw-r--r-- 1 tungsten tungsten   72 Apr  7 22:05 storage.index
-rw-r--r-- 1 tungsten tungsten 6.1M Apr  7 22:05 store-0000000001-mysqldump_2013-04-07_22-05_43.sql.gz
-rw-r--r-- 1 tungsten tungsten  315 Apr  7 22:05 store-0000000001.properties

# [node: host4] 0 files found

The 'copy_backup' command has copied the files from one host to another, and now we are ready to perform a restore in host2.

$ cookbook/trepctl -host host2 restore
Operation failed: Restore operation failed: Operation irrelevant in current state

Hmm. Probably not the friendliest of error messages. What this scoundrel means is that it can't perform a restore when the replicator is online.

$ cookbook/trepctl -host host2 offline
$ cookbook/trepctl -host host2 restore
Restore completed successfully

$ cookbook/trepctl -host host2 services
Processing services command...
NAME              VALUE
----              -----
appliedLastSeqno: 17955
appliedLatency  : 0.407
role            : slave
serviceName     : cookbook
serviceType     : local
started         : true
state           : ONLINE
Finished services command...

The restore operation was successful. We could have used xtrabackup just as well. THe only difference is that the operation takes way longer.

Parallel replication

Slave lagging is a common occurrence in MySQL replication. Most of the time, the reason for this problem is that while the master updates data using many threads concurrently, the slave applies the replication stream using a single thread. In Tungsten there is a built-in feature that applies changes in parallel, when the updates are happening in separate schemas. For database servers that are sharded by database or for the ones the serve multi-tenancy application, this is an ideal case. It is likely that the action happens in several schemas at once, and thus Tungsten can parallelize the changes successfully. Notice, however, that if you are running operations using a single schema, parallel replication won't give you any relief. Also, the operations must be really independent from each other. If a schema has foreign keys that reference to another schema, or if a transaction mixes data from two or more schemas, Tungsten will stop parallelizing and resume working in single thread until the end of the unclean operation, resulting in an overall decrease of performance, instead of increase.

To activate parallel replication, you need to enable two options:

  • --channels=N where you indicate how many parallel threads you want to establish. You should indicate as many channels as the number of schemas where you are operating. Some benchmarks will help you find the limits. Defining too many schemas will eventually exhaust the system resources. If the number of schemas is larger than the channels, Tungsten will use the channels in a round-robin fashion.
  • --svc-parallelization-type=disk: This option will activate a fast queue-creation algorithm that acts directly to the THL files. Contrary to common perception, where one would believe that in-memory queues are faster, this method is very efficient and less likely to exhaust system resources.

If you want to install all the servers with parallel replication, you can do this:

$ export MORE_OPTIONS='-a --channels=5 --svc-parallelization-type=disk'
$ cookbook/install_master_slave

If you need parallel replication only on one particular slave service, you can enable parallel replication there, using 'configure_service', same as we have seen before for the backup-method.

In this example, we're going to use the second method

$ cookbook/configure_service -U -a --host=host4 --channels=5 --svc-parallelization-type=disk cookbook
WARN  >> host4 >> THL schema tungsten_cookbook already exists at tungsten@host4:3306 (WITH PASSWORD)
NOTE >> host4 >> Deployment finished

$ cookbook/replicator restart
Stopping Tungsten Replicator Service...
Stopped Tungsten Replicator Service.
Starting Tungsten Replicator Service...

Now parallel replication is enabled. But how do we make sure that the service has indeed been enhanced?

The quickest method is to check the Tungsten service schema. Every replicator service creates a database schema named 'tungsten_$SERVICE_NAME', where it stores the current replication status. For example, in our default system, where the only replication service is called 'cookbook', we will find a schema named 'tungsten_cookbook'. The table that we want to inspect is one named 'trep_commit_seqno', where we store the global transaction ID, the schema where the transaction was applied, the data origin, and the time stamps at extraction and apply time. What is relevant in this table is that there will be one record for each channel that we have enabled. Thus, in host2 and host3 there will be only one line, while in host4 we should find 5 lines.

There is one useful recipe to get this result at once:

$ cookbook/query_all_nodes 'select count(*) from tungsten_cookbook.trep_commit_seqno'
+----------+
| count(*) |
+----------+
|        1 |
+----------+
+----------+
| count(*) |
+----------+
|        1 |
+----------+
+----------+
| count(*) |
+----------+
|        1 |
+----------+
+----------+
| count(*) |
+----------+
|        5 |
+----------+

Right! So we have 5 channels. Before inspecting what is going on in these channels, let's apply some load. You may recall that our load_data script can show you a command that we can customize for our purpose.

$ cookbook/load_data print
/home/tungsten/installs/cookbook/tungsten/bristlecone/bin/concurrent_evaluator.pl \
    --deletes=1 \
    --updates=1 \
    --inserts=3 \
    --test-duration=3600 \
    --host=host1 \
    --port=3306 \
    -u tungsten \
    -p secret  \
    --continuent-root=/home/tungsten/installs/cookbook \
    -d host1 \
    -s /home/tungsten/installs/cookbook/tungsten/load/host1 start

We just copy-and-paste this command, adding --instances=5 at the end, and we get 5 messages indicating that an evaluator was started. Let's see:

$ cookbook/query_node host4 'show schemas'
+--------------------+
| Database           |
+--------------------+
| information_schema |
| host11             |
| host12             |
| host13             |
| host14             |
| host15             |
| mysql              |
| test               |
| tungsten_cookbook  |
+--------------------+

Since we indicated that the database was to be named 'host1' and we have asked for 5 instances, the evaluator has created host11. host12, and so on.

Now that there is some action, we can have a look at our replication. Rather than querying the database directly, asking for the contents of trep_commit_seqno, we use another cookbook recipe:

$ cookbook/tungsten_service all
# node: host1 - service: cookbook
+--------+-----------+-----------------+----------+---------------------+---------------------+
| seqno  | source_id | applied_latency | shard_id | update_timestamp    | extract_timestamp   |
+--------+-----------+-----------------+----------+---------------------+---------------------+
| 324246 | host1     |               1 | host12   | 2013-04-07 23:02:16 | 2013-04-07 23:02:15 |
+--------+-----------+-----------------+----------+---------------------+---------------------+
# node: host2 - service: cookbook
+--------+-----------+-----------------+----------+---------------------+---------------------+
| seqno  | source_id | applied_latency | shard_id | update_timestamp    | extract_timestamp   |
+--------+-----------+-----------------+----------+---------------------+---------------------+
| 324383 | host1     |               0 | host13   | 2013-04-07 23:02:16 | 2013-04-07 23:02:16 |
+--------+-----------+-----------------+----------+---------------------+---------------------+
# node: host3 - service: cookbook
+--------+-----------+-----------------+----------+---------------------+---------------------+
| seqno  | source_id | applied_latency | shard_id | update_timestamp    | extract_timestamp   |
+--------+-----------+-----------------+----------+---------------------+---------------------+
| 324549 | host1     |               0 | host13   | 2013-04-07 23:02:16 | 2013-04-07 23:02:16 |
+--------+-----------+-----------------+----------+---------------------+---------------------+
# node: host4 - service: cookbook
+--------+-----------+-----------------+----------+---------------------+---------------------+
| seqno  | source_id | applied_latency | shard_id | update_timestamp    | extract_timestamp   |
+--------+-----------+-----------------+----------+---------------------+---------------------+
| 324740 | host1     |               0 | host11   | 2013-04-07 23:02:16 | 2013-04-07 23:02:16 |
| 324736 | host1     |               0 | host12   | 2013-04-07 23:02:16 | 2013-04-07 23:02:16 |
| 324739 | host1     |               0 | host13   | 2013-04-07 23:02:16 | 2013-04-07 23:02:16 |
| 324737 | host1     |               0 | host14   | 2013-04-07 23:02:16 | 2013-04-07 23:02:16 |
| 324735 | host1     |               0 | host15   | 2013-04-07 23:02:16 | 2013-04-07 23:02:16 |
+--------+-----------+-----------------+----------+---------------------+---------------------+

Here you see that host1 has only one channel: it is the master, and it must serialize according to the binary log. Slaves host2 and host3 have only one channel, because we have enabled parallel replication only in host4. And finally we see that in host4 there are 5 channels, each showing a different shard_id (= database schema), with its own transaction ID being applied. This shows that replication is working.

Tungsten Replicator has, however, several tools that help monitoring parallel replication:

$ cookbook/trepctl -host host4 status -name stores
Processing status command (stores)...
NAME                      VALUE
----                      -----
activeSeqno             : 475395
doChecksum              : false
flushIntervalMillis     : 0
fsyncOnFlush            : false
logConnectionTimeout    : 28800
logDir                  : /home/tungsten/installs/cookbook/thl/cookbook
logFileRetainMillis     : 604800000
logFileSize             : 100000000
maximumStoredSeqNo      : 475449
minimumStoredSeqNo      : 0
name                    : thl
readOnly                : false
storeClass              : com.continuent.tungsten.replicator.thl.THL
timeoutMillis           : 2147483647
NAME                      VALUE
----                      -----
criticalPartition       : -1
discardCount            : 0
estimatedOfflineInterval: 0.0
eventCount              : 457459
headSeqno               : 475415
intervalGuard           : AtomicIntervalGuard (array is empty)
maxDelayInterval        : 60
maxOfflineInterval      : 5
maxSize                 : 10
name                    : parallel-queue
queues                  : 5
serializationCount      : 0
serialized              : false
stopRequested           : false
store.0                 : THLParallelReadTask task_id=0 thread_name=store-thl-0 hi_seqno=475415 lo_seqno=17957 read=457459 accepted=93357 discarded=364102 events=0
store.1                 : THLParallelReadTask task_id=1 thread_name=store-thl-1 hi_seqno=475415 lo_seqno=17957 read=457459 accepted=92567 discarded=364892 events=0
store.2                 : THLParallelReadTask task_id=2 thread_name=store-thl-2 hi_seqno=475415 lo_seqno=17957 read=457459 accepted=91197 discarded=366262 events=0
store.3                 : THLParallelReadTask task_id=3 thread_name=store-thl-3 hi_seqno=475415 lo_seqno=17957 read=457459 accepted=90492 discarded=366967 events=0
store.4                 : THLParallelReadTask task_id=4 thread_name=store-thl-4 hi_seqno=475415 lo_seqno=17957 read=457459 accepted=89846 discarded=367613 events=0
storeClass              : com.continuent.tungsten.replicator.thl.THLParallelQueue
syncInterval            : 10000
Finished status command (stores)...

This command shows the status of parallel replication in each channels. Notable information in this screen:

  • eventCount is the number of transaction being processed
  • serializationCount:0 means that all events have been parallelized, and there was no need to serialize any.
  • 'read' ... 'accepted' ... 'discarded' are the operation in the disk queue. Each channel parses all the events, and queues only the ones that belong in its shard.
$ cookbook/trepctl -host host3 status -name shards
Processing status command (shards)...
 ...
NAME                VALUE
----                -----
appliedLastEventId: mysql-bin.000006:0000000169567337;0
appliedLastSeqno  : 660707
appliedLatency    : 1.314
eventCount        : 130325
shardId           : host11
stage             : q-to-dbms
NAME                VALUE
----                -----
appliedLastEventId: mysql-bin.000006:0000000169566006;0
appliedLastSeqno  : 660702
appliedLatency    : 1.312
eventCount        : 129747
shardId           : host12
stage             : q-to-dbms
 ...

This command (only a portion reported here) displays the status of each shard, showing for each one which event, transaction ID and event count were recorded.

There should be much more to mention about the monitoring tools, but for now I want just to mention a last important point. When the replicator goes offline, parallel replication stops, and the replication operations are consolidated into a single thread. This makes sure that replication can later resume using a single thread, or it can be safely handed over to native MySQL replication. This behavior also makes sure that a slave can be safely promoted to master. A switch operation requires that the slave service be offline before being reconfigured to become a master. When the replicator goes offline, the N channels become 1.

$ ./cookbook/trepctl offline
$ cookbook/tungsten_service all
# node: host1 - service: cookbook
+--------+-----------+-----------------+----------+---------------------+---------------------+
| seqno  | source_id | applied_latency | shard_id | update_timestamp    | extract_timestamp   |
+--------+-----------+-----------------+----------+---------------------+---------------------+
| 769652 | host1     |               0 | host12   | 2013-04-07 23:18:07 | 2013-04-07 23:18:07 |
+--------+-----------+-----------------+----------+---------------------+---------------------+
# node: host2 - service: cookbook
+--------+-----------+-----------------+----------+---------------------+---------------------+
| seqno  | source_id | applied_latency | shard_id | update_timestamp    | extract_timestamp   |
+--------+-----------+-----------------+----------+---------------------+---------------------+
| 769699 | host1     |               0 | host13   | 2013-04-07 23:18:08 | 2013-04-07 23:18:08 |
+--------+-----------+-----------------+----------+---------------------+---------------------+
# node: host3 - service: cookbook
+--------+-----------+-----------------+----------+---------------------+---------------------+
| seqno  | source_id | applied_latency | shard_id | update_timestamp    | extract_timestamp   |
+--------+-----------+-----------------+----------+---------------------+---------------------+
| 769866 | host1     |               0 | host15   | 2013-04-07 23:18:08 | 2013-04-07 23:18:08 |
+--------+-----------+-----------------+----------+---------------------+---------------------+
# node: host4 - service: cookbook
+--------+-----------+-----------------+----------+---------------------+---------------------+
| seqno  | source_id | applied_latency | shard_id | update_timestamp    | extract_timestamp   |
+--------+-----------+-----------------+----------+---------------------+---------------------+
| 767064 | host1     |               0 | host15   | 2013-04-07 23:18:01 | 2013-04-07 23:18:01 |
+--------+-----------+-----------------+----------+---------------------+---------------------+

If we put it back online, we see again the channels expanding.


Further info:

No comments:

Vote on Planet MySQL