PHP, the universe and all the rest - MySQL

Happy Birthday Bug# 30994, here is a present for you ...

nospam@example.com (Hartmut Holzgraefe) — Thu, 13 Sep 2012 21:01:05 +0200

One of the MySQL bugs having birthday today is Bug #30994 which i filed five years ago.

So i decided to give it a little birthday present by finally writing a patch for it myself:

https://code.launchpad.net/~hartmut-php/maria/5.5-bug30994

MySQL Bug Anniversaries

nospam@example.com (Hartmut Holzgraefe) — Wed, 12 Sep 2012 20:14:28 +0200

Ten years ago today the first public bug was filed on bugs.mysql.com.

As we also stumbled across an unfixed bug today that was reported almost exactly two years ago i came up with the idea of writing a little script that would report all bugs filed on exactly the same day one or more years ago that are still in an "active" state (open, verified, etc.).

The list for today can be found at

http://php-groupies.de/mysql-bugs/2012-09-12.html

And as i was already on it i also created a RSS feed for this at

http://php-groupies.de/mysql-bugs/bugs.rss

Fun with Cluster and Locking

nospam@example.com (Hartmut Holzgraefe) — Fri, 27 Apr 2012 19:22:44 +0200

I've been dealing with MySQL Cluster in one way or another since around 2005 or so (back in the MySQL 4.1 days) but it is still full of "funny" surprises. This post is a collection of different locking related issue i ran into during the previous weeks that i had not been aware of up to now (or simply may have forgotten over time)

== Unique hash indexes lock exclusively ==

This is the one that regular users are most likely to run into: in general row logs in MySQL Cluster distinguish between reads and writes so that writers can block other writers, but not readers and readers from other transactions always see the last committed row value (Cluster currently only supports the READ COMMITTED isolation level). As soon as you have a secondary unique index in addition to a primary key things are different though. Internally a unique index that is not the primary key is implemented as a unique hash index in Cluster (and optionally also as an additional ordered T-Tree), and for unique hash entries there only seems to be exclusive locking so that writers will block readers across transactions.

See http://bugs.mysql.com/65086 for actual examples.

== Long running transactions can block starting nodes ==

Less likely to happen, but very annoying if it happens and you don't know about it:

A starting node needs to lock all rows for a very short period of time near the end of start phase 5. At this point it needs to wait for all current transaction to free the locks they are holding (by either COMMIT or ROLLBACK). This is usually not a problem as cluster transaction are not supposed to last very long anyway, but if there happens to be a long running transaction it can potentially block a node start forever. A stopping node will wait for pending transactions for 5 seconds and will then terminate these transactions the hard way, a starting node on the other hand will gracefully wait forever.

This is bad in two ways:

1) it is obviously bad availability wise as you may end up with a non-redundant configuration for extended periods of time and without any way to automatically identify and terminate the offending transaction(s)

2) currently the blocked starting node does not even tell what it is waiting for, it just silently sits and waits. So unless you know what is going on you are faced with a node that is simply stuck without doing anything (CPU, disc and network load next to zero)

The related bug report is http://bugs.mysql.com/65037 , proposed solutions are meaningful log messages in the short term and killing active transactions after a grace period in the long run.

== Weird error message on lock timeout ==

This last item is related to the non-standard INSERT ... ON DUPLICATE KEY UPDATE construct. What can happen here is that the INSERT part fails with a lock wait timeout as the key is write locked by another transaction. The expected error message in this case would simply be "Lock wait timeout exceed" but what you'll actually get is "Transaction already aborted". What seems to happen here is that the ON DUPLICATE KEY UPDATE part is triggered by any error in the INSERT phase and not only by duplicate key violation. So the INSERT part is tried tried, fails with "Lock wait timeout" (which implicitly rolls back the current transaction), next the UPDATE part is tried, fails with "Transaction already aborted" and overwrites the previous error ...

The related bug report is http://bugs.mysql.com/65130

Partition fun again, today with ARCHIVE

nospam@example.com (Hartmut Holzgraefe) — Fri, 06 Apr 2012 18:50:00 +0200

After the fun with InnoDB and MyISAM and massive partitioning it's time to move on to some other storage engines. So for the next round of fun i chose ARCHIVE:

  mysql> create table t1 (id int primary key auto_increment, val int) 
      engine=archive partition by hash(id) partitions 1023;
  Query OK, 0 rows affected (0.26 sec)

  mysql> insert into t1 values(NULL,1);
  ERROR 1194 (HY000): Table 't1' is marked as crashed 
      and should be repaired

  mysql> drop table t1; 
  ERROR 2013 (HY000): Lost connection to MySQL server 
      during query

PS: MariaDB does not seem to be affected ...

A different kind of static?

nospam@example.com (Hartmut Holzgraefe) — Mon, 12 Mar 2012 10:30:55 +0100

While actually trying to check for something completely different (more on that later maybe)
i found that mysql, mysqladmin and other client binaries in my MySQL 5.5 install dir did
not list libmysqlclient.so as a dependency in ldd output anymore. Looks as if since the
switch to Cmake it gets statically linked into all the bundled client libraries and so
increases binary sizes by a factor of about 10x now, and i can't see any Cmake build
option that would produce the old behavior either.

So my /usr/local/mysql-5.5.21/bin directory now uses 130MB instead of the 17MB that
the /usr/local/mysql-5.1.60/bin one used. Doesn't sound much by itself, but for
someone like me who tries to keep as many versions as possible installed in parallel
for testing purposes it sums up to several GB quickly, and it totally defies the
purpose of having a shared libmysqlclient.so in the first place ...

Fun with partitions and MyISAM, part #2

nospam@example.com (Hartmut Holzgraefe) — Wed, 07 Mar 2012 05:29:36 +0100

This is part of the reply i got on my bug report on the ALTER TABLE issue with MyISAM and a large number of partitions from my previous post:

This is a design limitation, MyISAM have one file handle open for the .MYI file shared by all instances of one table, and one file handle open for each table instance for the .MYD file. And the ha_partition generic partitioning engine always open all partitions when opening a table/handler.

So this should not only affect ALTER TABLE but all operations on MyISAM tables with large number of partitions should eventually run into this. Lets try and see what happens ...

Continue reading "Fun with partitions and MyISAM, part #2"

More fun with partitions, this time with MyISAM

nospam@example.com (Hartmut Holzgraefe) — Wed, 29 Feb 2012 14:22:59 +0100

mysql> CREATE TABLE t1 (id INT PRIMARY KEY ) 
             engine=myisam PARTITION BY HASH(id) PARTITIONS 1000;
Query OK, 0 rows affected (0.18 sec)

mysql> alter table t1 add column v int;
ERROR 1016 (HY000): Can't open file: './test/t1.frm' (errno: 24)

mysql> quit
Bye

$ perror 24
OS error code  24:  Too many open files

Looks as if MyISAM tries to open all the 1000 partition .MYD/.MYI files at once and runs out of file handles during this operation ...

Some fun with partitions and InnoDB ...

nospam@example.com (Hartmut Holzgraefe) — Wed, 29 Feb 2012 12:55:36 +0100

Creating or dropping a partitioned table on InnoDB can become a quite expensive operation, on my laptop i'm seeing the following times for a simple table with 100 or 1000 partitions (using 5.1.58 right now as i'm testing on stock Ubuntu 11.10):

  mysql> CREATE TABLE t1 (id INT PRIMARY KEY )
               engine=innodb PARTITION BY HASH(id) PARTITIONS 100;
  Query OK, 0 rows affected (5.21 sec)

  mysql> drop table t1;
  Query OK, 0 rows affected (5.11 sec)

  mysql> CREATE TABLE t1 (id INT PRIMARY KEY )
               engine=innodb PARTITION BY HASH(id) PARTITIONS 1000;
  Query OK, 0 rows affected (52.76 sec)

So the time to create a partitioned InnoDB table grows linearly with the number of partitions at a 'speed' of about 20 partitions per second, and during that time the hard disk LED is always on. The rate for creating regular InnoDB tables on this machine is about 10 tables per second by the way.

MyIsam on the other had can create around 17 tables per second, and creating a single partitioned MyISAM table only takes about a 10th of a second.

So what is going on with InnoDB here? First of all from the engines point of view each partition of a partitioned table is an actual table, all the partitioning magic happens on a layer above the storage engine one, and the storage engine only receives handler requests for the actual partition tables involved. On the storage engine layer the actual engines are not aware that these tables are part of a larger partitioned table at all (with the exception of ndbcluster tables, but MySQL Cluster is a different story in this respect anyway).

So when creating an InnoDB table with 1000 partitions the InnoDB storage engine actually receives 1000 individual requests to create a table. Now what seems to happen is that for each
created partition/table InnoDB flushes at least its internal data dictionary to disk. Given the constant disk activity and rather low internal table creation rate it may even do a full checkpoint for each table, i haven't checked the details on this yet. And when dropping a partitioned table the same thing happens in reverse.

So this may be something to keep in mind when planning to use large numbers of partitions on InnoDB ...

MySQL features i forgot about #1 : slave_compressed_protocol

nospam@example.com (Hartmut Holzgraefe) — Thu, 23 Feb 2012 06:10:40 +0100

It is probably about time to re-read the MySQL manual end-to-end as i more and more find myself discovering features i either completely forgot about or which i never was aware of in the first place ...

Todays guest is slave_compressed_protocol, an option that has probably been there ever since MySQL 3.23 at least (so that i can't claim that i seem to have missed the ChangeLog entry as i usually do

With slave_compressed_protocol enabled the communication between slave and master uses the MYSQL_OPT_COMPRESS option to compress the protocol stream if both sides support it (and it's very unlikely to find an installation that does *not* support it these days), so this can be a big savior if your master and slave are at different sites with only limited bandwidth between the two.

Unfortunately this is a global server option though, IMHO this should be part of the options provided by the CHANGE MASTER command, similar to all the SSL encryption related stuff that is part of CHANGE MASTER.

Looks as if it is about time for YAFR (Yet Another Feature Request) ...

Overriding bind=... in included MySQL option files

nospam@example.com (Hartmut Holzgraefe) — Tue, 04 Oct 2011 08:29:10 +0200

Debian and Ubuntu make use of the !includedir directive at the end of the packaged my.cnf to allow for extending and overriding their default configuration without having to modify the distribution config file.

This is especially nice in combination with configuration tools like puppet as it is usually much easier to add a file to the system than to modify an existing file.

I ran into a small gotcha with this though: the default my.cnf binds mysqld to the localhost TCP interface so that it listens IP 127.0.0.1 only. To make the mysql server reachable from other hosts the bind=127.0.0.1 setting needs to be overwritten.
How to change the binding to a different IP should be obvious, but how to disable an existing binding completely if you want your mysqld to listen on all available interfaces?

The most obvious would be to simply say

bind=

, this does not have any effect though. What needs to be done instead is

bind=0.0.0.0

to make mysqld listen on all interfaces again.

This is obviously a solution for IPv4 only, for allowing incoming connections from all v4 and v6 interfaces allowing bind= without any argument to reset any existing bindings would probably be more clever?

MySQL Conference talk slides

nospam@example.com (Hartmut Holzgraefe) — Fri, 15 Apr 2011 18:53:55 +0200

Slides for my two MySQL Conference talks are now available from the O'Reilly conference site:

Why go with O(n) if you can have O(n²)?

nospam@example.com (Hartmut Holzgraefe) — Thu, 03 Mar 2011 10:49:01 +0100

Found this while waiting for the CSV export of a search result from (*undisclosed project*) ...

The export took almost half an hour even though it was just extracting the snail mail addresses of about 40K contacts, so what caused it to only spit out about 20 rows per second? Looking at the PROCESSLIST output a pattern was easy to detect. What greeted me was a single running query which had only been running for a few seconds even though i was already waiting for export results for about 20 minutes:

SELECT ... LIMIT 32900, 100;

And then a few seconds later:

SELECT ... LIMIT 33000, 100;

So instead of retrieving the search results for export by firing a single query and processing the results in one go the same pagination logic seemed to be used as for displaying search results. About 400 queries instead of a single one, and each iteration taking more query time, with finally the last one taking about the same time as the only query in the all-at-once alone.

Somewhere, somehow, something terribly went wrong it seems ....

MySQL Plugin generator now supporting system variables

nospam@example.com (Hartmut Holzgraefe) — Mon, 16 Feb 2009 08:28:26 +0100

CodeGen_MySQL_Plugin had basic support for status variables (for SHOW STATUS) for quite a while, now it also has basic support for system variables (SHOW [GLOBAL|SESSION] VARIABLES, SET) in the 0.9.2 package version i just released as well

Continue reading "MySQL Plugin generator now supporting system variables"

MySQL UDF generator finally supporting windows

nospam@example.com (Hartmut Holzgraefe) — Wed, 11 Feb 2009 09:04:42 +0100

So i finally defeated my 'inner Schweinehund' and set up a working Windows development system again. Using the Visual Studio C++ 2008 Express edition i'm now able to compile UDF projects generated by CodeGen_MySQL_UDF aka. udf-gen, and the udf-gen script now works natively on Windows, too.

Continue reading "MySQL UDF generator finally supporting windows"

Speaking at the MySQL User Conference 2008

nospam@example.com (Hartmut Holzgraefe) — Thu, 06 Mar 2008 16:14:54 +0100

I'm talking at the MySQL User Conference once again this year presenting my Code Generators for MySQL Plugins and User Defined Functions in a formal session this time after my original "pizza & hacking" session on this topic on the 2006 UC (which ended up in hacking some other things on the way first ... photographs may follow ...).

The plugin and UDF code generators are based on the same foundation as my "pecl-gen" generator for PHP extensions so this might be of interest for hardcore PHP folks, too.

For more information see the Conference Schedule.