Lua and mysql-proxy: how to send a query to an email

March 11th, 2011 by Leandro Morgado

I recently got an inquiry on how to receive an email every time a query was executed via the MySQL proxy. This is very simple and you can achieve it by simply piping the query to the *nix mail command. Here is the sample code (in a file caled send_mail.lua):

1
2
3
4
5
6
7
function read_query(packet)
  if string.byte(packet) == proxy.COM_QUERY then
    print("Hello world! Seen the query: " .. string.sub(packet, 2))
    print("Sending query to email:")
    os.execute("echo " .. string.sub(packet, 2) .. "| mail -s'Lua test' your.email.here@example.com ")
  end
end

To execute this, install proxy as per the instructions in the manual. Then simply run the proxy and point it at your script:

shell> bin/mysql-proxy --proxy-lua-script=send_email.lua &

Finally, connect to the proxy and issue a query:

shell> mysql -h127.0.0.1 --port=4040
...
mysql> SELECT "Test 1";
Hello world! Seen the query: SELECT "Test 1"
Sending query to email:
+--------+
| Test 1 |
+--------+
| Test 1 |
+--------+
1 row in set (0.00 sec)

You’ll shorty receive email with the query above. Happy Lua scripting.

Configuring MySQL Enterprise Monitor to authenticate from LDAP

October 11th, 2010 by Leandro Morgado

In the last post, we saw how to create a test OpenLDAP server, populate it and secure it with SSL certificates. Now we are going to have a look at how to configure MySQL Enterprise Manager (MEM) to authenticate against LDAP. We will be examining a few different kinds of setup methods.

1. Using LDAP to fetch just the user password

The simplest form is to configure a user with MEM and set it to the LDAP type. The user’s role is setup in MEM during user creation time and is not fetched from LDAP. Below you can see the user definition page:

How to create a LDAP user in MEM - password only

How to create a LDAP user in MEM - password only

The username is user1 as specified in MEM, but where do we get the password from? We need to tell MEM how to find this user. Here is a sample user as defined by the LDIF file used in my previous post:

# user1, People, example.com
dn: uid=user1,ou=People,dc=example,dc=com
objectClass: person
objectClass: inetOrgPerson
objectClass: organizationalPerson
objectClass: top
cn: Aaren Atp
sn: Atp
description: This is the description for Aaren Atp.
employeeNumber: 1
givenName: Aaren
homePhone: +1 280 375 4325
initials: ALA
l: New Haven
mail: user.1@maildomain.net
mobile: +1 680 734 6300
ou: admin
pager: +1 850 883 8888
postalAddress: Aaren Atp$70110 Fourth Street$New Haven, OH  93694
postalCode: 936942
st: OH
street: 70110 Fourth Street
telephoneNumber: +1 390 103 6917
uid: user1
userPassword:: e1NTSEF9Z0tsZjU4cm50Wit4b045N0U4cWlldVJQK1RMOVAzTGw=

Now tell MEM to look for the uid by navigating to Settings > Global Settings > LDAP Authentication:

MEM settings for LDAP password lookup only

MEM settings for LDAP password lookup only

The password is specified in the userPassword attribute and uses standard LDAP supported formats. Technically what MEM does is an LDAP simple bind operation (Authentication Mode = Bind as User) with the given username/password and then attempts to search for the given user using the DN pattern.  If the binding and search succeeds, we assume the password is correct. With this in place, you can now login with user1 without having to specify the password in MEM.

2. Adding MEM Roles from LDAP

If we want to fetch the user’s role from LDAP in addition to the password, we need to enable the Map LDAP Roles to Application Roles check box in the LDAP settings page. We’ll see the two ways of doing this shortly.

The good thing about enabling role mapping is that manual user creation in the Web dashboard is no longer required. If the user does not already exist within MEM, it will be “copied” from LDAP to MEM when that user first logs in. The lookup order is Built-in users first, then LDAP users. If you want user authentication to be exclusively done against LDAP then make LDAP authoritative by ticking the check box:

MEM authoritative LDAP authentication

MEM authoritative LDAP authentication

WARNING: Enabling this prevents Built-in users from being authenticated. Make sure that you test with non authoritative authentication first and make sure role mapping is working. Also, ensure that you have a valid LDAP user who has admin privileges or risk locking yourself out.

Below is an example of how my test user table looked before logging in with user1:

User table before LDAP login

User table before LDAP login

And this is how it looks after a successful login:

User table after LDAP login

User table after LDAP login

Notice how the LDAP user1 entry was created automatically in the MEM user table with values from the LDAP directory.

2.2.1 MEM roles in a LDAP user’s attribute

The first way of achieving role mapping is to simply have an attribute in the user’s entry, whose attribute value will be the role. In our example, we use the ou attribute. The relevant attributes for our sample users are:

uid: user1
ou: admin
....
uid: user2
ou: dba

The relevant settings in MEM are:

MEM settings for LDAP role mapping with user attributes

MEM settings for LDAP role mapping with user attributes

Now when you login with user2 for example, its role will be fetched from LDAP. We can verify this in the User Preferences section as shown by the screenshot below:

A user that has authenticated via LDAP with role mapping

A user that has authenticated via LDAP with role mapping

If we try user1, then we will get an admin (manager) role.

Note that some versions have a bug that prevents this setup from working. Check if you are affected.

2.2.2 MEM roles specified by a LDAP group

There is another way to specify a user’s role. Instead of each user having an attribute representing the role, a roles group is created, which enumerates the users that are members of that role. In our example, this is represented in the LDIF file as:

dn: ou=groups,dc=example,dc=com
objectClass: organizationalUnit
ou: groups

dn: cn=admin,ou=groups,dc=example,dc=com
objectClass: groupOfUniqueNames
cn: admin
uniqueMember: uid=user1,ou=People,dc=example,dc=com

dn: cn=dba,ou=groups,dc=example,dc=com
objectClass: groupOfUniqueNames
cn: dba
uniqueMember: uid=user2,ou=People,dc=example,dc=com

So again, we need to tell MEM how to do role lookups from the LDAP directory.  This is how the LDAP settings in MEM would look like for our particular case:

MEM settings for LDAP role mapping in groups

MEM settings for LDAP role mapping in groups

Note, that due this bug, you might not get this particular setup to work.

2.3 Configuring LDAP lookups to use SSL and STARTTLS

Once plain text authentication is working, you can choose to secure communications between MEM and the LDAP server with either STARTTLS or SSL. Obviously the LDAP server must support it. Note that SSL (ldaps) is deprecated in the latest LDAP v3 and STARTTLS is prefered. MEM supports both types. To use STARTTLS, simply select it in the menu as follows:

MEM with STARTTLS LDAP authentication

MEM with STARTTLS LDAP authentication

There is no need to change the default port. Using SSL (ldaps) however, implies using the ldaps port, which by default is 636, and setting MEM to use SSL(ldaps) :

MEM with SSL (ldaps) LDAP authentication

MEM with SSL (ldaps) LDAP authentication

One thing to note is that the (Primary) Server Hostname should match the SSL certificate of the LDAP server.

If you are using self signed certificates that were not issued by a commercial SSL company, MEM will not have the root CA (Certificate Authority) in it’s repository. You should see a message in the $INSTALL_DIR/monitor/apache-tomcat/logs/catalina.out logfile:

javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed:
sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

To fix this, you will need to import the your root CA certificate that was used to generate the LDAP server’s certificate. This can be achieve with the Java Keytool utility as follows:

shell> cd  /opt/mysql/enterprise/monitor/java
shell> bin/keytool -import -trustcacerts -alias ldapssl \
       -file /etc/openldap/ssl/ca-cert.pem -keystore lib/security/cacerts

This needs to be run under MEM’s Java installation directory. Unless you have previously modified it, the default password is changeit . We saw how to create a CA certificate (ca-cert.pem) in the previous post. Finally, restart the MEM monitor service.

In case you have problems importing your root CA certificate, make sure it is correctly format and acceptable by the Java keytool utility:

keytool can import X.509 v1, v2, and v3 certificates, and PKCS#7 formatted certificate chains consisting of certificates of that type. The data to be imported must be provided either in binary encoding format, or in printable encoding format (also known as Base64 encoding) as defined by the Internet RFC 1421 standard. In the latter case, the encoding must be bounded at the beginning by a string that starts with ‘—–BEGIN’, and bounded at the end by a string that starts with ‘—–END’.”

http://download.oracle.com/docs/cd/E17409_01/javase/6/docs/technotes/tools/windows/keytool.html

To troubleshoot, the best place is to look at the catalina.out logfile. If you need extra debugging information, you can start MEM’s JVM as follows:

JAVA_OPTS="$JAVA_OPTS -Djavax.net.debug=ALL"

In the current version (2.2.1.1721) this should be added at around line 180 of $INSTALL_DIR/monitor/apache-tomcat/bin/catalina.sh. Note that catalina.sh will get overwritten when you next upgrade so if you want JAVA_OPTS changes to be persistent, then use setenv.sh (or setenv.bat for Windows.). A Tomcat restart is needed. Note that a lot of information is logged and you’re log files will grow rather quickly. Please remember to turn this off in regular production mode.

Hopefully this will help when configuring MEM to authenticate it’s users against a LDAP directory.

Setting up OpenLDAP for MySQL Enterprise Monitor

October 11th, 2010 by Leandro Morgado

The latest 2.2 release of MySQL Enterprise Monitor (MEM) has the ability to authenticate against LDAP. I decided to test this setup and for that, I had to create and populate an OpenLDAP server, including STARTTLS/SSL certificates. This guide was done on CentOS 5.5 but it shouldn’t be much different in other Linux/Unix distributions. First, start off by installing the packages with:

root@shell> yum install openldap openldap-clients openldap-servers

Then head to /etc/openldap where you can set you domain and the DN for the LDAP manager user. I’ve inserted some useful comments into the slapd.conf file. Lines without comments have not been changed from the default slapd.conf file.

shell> grep -v "^#" /etc/openldap/slapd.conf | grep -v "^$"
include		/etc/openldap/schema/core.schema
include		/etc/openldap/schema/cosine.schema
include		/etc/openldap/schema/inetorgperson.schema
include		/etc/openldap/schema/nis.schema
allow bind_v2
pidfile		/var/run/openldap/slapd.pid
argsfile	/var/run/openldap/slapd.args
#The lines below are for SSL and STARTTLS.
#I'll show you how to generate certs later on
TLSCipherSuite HIGH:MEDIUM:-SSLv2
TLSCACertificateFile /etc/openldap/ssl/ca-cert.pem
TLSCertificateFile /etc/openldap/ssl/server-cert.pem
TLSCertificateKeyFile /etc/openldap/ssl/server-key.pem
#This allows ldapsearch command to connect without a client cert
TLSVerifyClient never
database	bdb
# this is your domain. I used example.com for my tests.
suffix		"dc=example,dc=com"
# this is the "username" of the LDAP admin for this domain
rootdn		"cn=Manager,dc=example,dc=com"
# this is the encripted password. To generate a SSHA password use slappasswd.
rootpw {SSHA}8diJsdIYFRr/wt7vqk3SGj6b/ZZZ21eno
directory	/var/lib/ldap
index objectClass                       eq,pres
index ou,cn,mail,surname,givenname      eq,pres,sub
index uidNumber,gidNumber,loginShell    eq,pres
index uid,memberUid                     eq,pres,sub
index nisMapName,nisMapEntry            eq,pres,sub

If for some reason, you need to compile your own version of OpenLDAP, see this guide:
http://www.openldap.org/doc/admin24/quickstart.html

The next step is to generate our certificates. First we generate the Certificate Authority and the LDAP Server certificates. One important thing is to set the CN attribute to your server’s hostname in both certificates. You can run the hostname command in the shell to find that out. This is the same hostname you will be using in the MEM setup or to test with the ldapsearch command line utility.

# Create CA and Server Certs
shell> openssl genrsa 2048 > ca-key.pem
shell> openssl req -new -x509 -nodes -days 1000 -key ca-key.pem > ca-cert.pem
shell> openssl req -newkey rsa:2048 -days 1000 -nodes -keyout server-key.pem > server-req.pem
shell> openssl x509 -req -in server-req.pem -days 1000 -CA ca-cert.pem -CAkey ca-key.pem -set_serial 01 > server-cert.pem

These files should be located under /etc/openldap/ssl and the server key needs to be chmoded:

shell> chmod 600 /etc/openldap/server-key.pem

Then we create the certificate for our client utilities, and let openldap know about it:

# Create client certificate
shell> openssl req -newkey rsa:2048 -days 1000 -nodes -keyout client-key.pem > client-req.pem
shell> openssl x509 -req -in client-req.pem -days 1000 -CA ca-cert.pem -CAkey ca-key.pem -set_serial 01 > client-cert.pem
shell> cat /etc/openldap/ldap.conf
#
# LDAP Defaults
#
# See ldap.conf(5) for details
# This file should be world readable but not world writable.
#BASE	dc=example, dc=com
#URI	ldap://ldap.example.com ldap://ldap-master.example.com:666
#SIZELIMIT	12
#TIMELIMIT	15
#DEREF		never
#TLS_CACERTDIR /etc/openldap/cacerts
TLS_CACERT /etc/openldap/ssl/ca-cert.pem

Here are how the permissions on these files look like for me:

shell> ls -la /etc/openldap/ssl/
total 40K
drwxr-xr-x 2 root root 4.0K Jun 23 13:53 .
drwxr-xr-x 5 root root 4.0K Jun 25 20:59 ..
-rw-r--r-- 1 root root 1.5K Jun 23 13:51 ca-cert.pem
-rw-r--r-- 1 root root 1.7K Jun 23 13:51 ca-key.pem
-rw-r--r-- 1 root root 1.2K Jun 23 13:53 client-cert.pem
-rw-r--r-- 1 root root 1.7K Jun 23 13:53 client-key.pem
-rw-r--r-- 1 root root 1.1K Jun 23 13:53 client-req.pem
-rw-r--r-- 1 root root 1.2K Jun 23 13:52 server-cert.pem
-rw------- 1 ldap root 1.7K Jun 23 13:52 server-key.pem
-rw-r--r-- 1 root root 1.1K Jun 23 13:52 server-req.pem

Next enable SSL for OpenLDAP and start up the server:

shell> grep -v "^#" /etc/sysconfig/ldap | grep -v "^$"
ULIMIT_SETTINGS=
STOP_DELAY=3s
SLAPD_LDAP=yes
SLAPD_LDAPS=yes
SLAPD_LDAPI=no
shell> /etc/init.d/ldap start
Checking configuration files for slapd:  bdb_db_open: Warning - No DB_CONFIG file found in directory /var/lib/ldap: (2)
Expect poor performance for suffix dc=example,dc=com.
config file testing succeeded                                                           [  OK  ]
Starting slapd:                                            [  OK  ]
shell> ps aux | grep slapd
ldap     25224  0.0  2.3 392036 191288 ?       Ssl  21:01   0:00 /usr/sbin/slapd -h ldap:/// ldaps:/// -u ldap

Don’t worry about the warning. Let’s try and query the LDAP directory now:

shell> ldapsearch -x -h localhost -b 'dc=example,dc=com'
# extended LDIF
#
# LDAPv3
# base  with scope subtree
# filter: (objectclass=*)
# requesting: ALL
#
# search result
search: 2
result: 32 No such object
# numResponses: 1

As we can see, there are no entries yet. So let’s populate the directory with this LDIF file. You will be prompted for your password. It’s the one used to setup /etc/openldap/slapd.conf.

shell> ldapadd -x -D "cn=Manager,dc=example,dc=com" -W -f mem-ldap.ldif.txt
Enter LDAP Password:
adding new entry "dc=example,dc=com"
adding new entry "ou=People,dc=example,dc=com"
adding new entry "uid=user1,ou=People,dc=example,dc=com"
adding new entry "uid=user2,ou=People,dc=example,dc=com"
adding new entry "ou=groups,dc=example,dc=com"
adding new entry "cn=admin,ou=groups,dc=example,dc=com"
adding new entry "cn=dba,ou=groups,dc=example,dc=com"

And now we can see the results with:

shell> ldapsearch -x -H ldap:///localhost -b 'dc=example,dc=com'
# extended LDIF
#
# LDAPv3
# base <dc=example,dc=com> with scope subtree
# filter: (objectclass=*)
# requesting: ALL
#
# example.com
dn: dc=example,dc=com
objectClass: domain
objectClass: top
dc: example
# People, example.com
dn: ou=People,dc=example,dc=com
objectClass: organizationalUnit
objectClass: top
ou: People
# user1, People, example.com
dn: uid=user1,ou=People,dc=example,dc=com
objectClass: person
objectClass: inetOrgPerson
objectClass: organizationalPerson
objectClass: top
cn: Aaren Atp
sn: Atp
description: This is the description for Aaren Atp.
employeeNumber: 1
givenName: Aaren
homePhone: +1 280 375 4325
initials: ALA
l: New Haven
mail: user.1@maildomain.net
mobile: +1 680 734 6300
ou: admin
pager: +1 850 883 8888
postalAddress: Aaren Atp$70110 Fourth Street$New Haven, OH  93694
postalCode: 936942
st: OH
street: 70110 Fourth Street
telephoneNumber: +1 390 103 6917
uid: user1
userPassword:: e1NTSEF9Z0tsZjU4cm50Wit4b045N0U4cWlldVJQK1RMOVAzTGw=
# user2, People, example.com
dn: uid=user2,ou=People,dc=example,dc=com
objectClass: person
objectClass: inetOrgPerson
objectClass: organizationalPerson
objectClass: top
cn: Aaren Atp
sn: Atp
description: This is the description for Aaren Atp.
employeeNumber: 2
givenName: Aaren
homePhone: +1 280 375 4325
initials: ALA
l: New Haven
mail: user2@maildomain.net
mobile: +1 680 734 6300
ou: dba
pager: +1 850 883 8888
postalAddress: Aaren Atp$70110 Fourth Street$New Haven, OH  93694
postalCode: 936941
st: OH
street: 70110 Fourth Street
telephoneNumber: +1 390 103 6917
uid: user2
userPassword:: e1NTSEF9Z0tsZjU4cm50Wit4b045N0U4cWlldVJQK1RMOVAzTGw=
# groups, example.com
dn: ou=groups,dc=example,dc=com
objectClass: organizationalUnit
ou: groups
# admin, groups, example.com
dn: cn=admin,ou=groups,dc=example,dc=com
objectClass: groupOfUniqueNames
cn: admin
uniqueMember: uid=user1,ou=People,dc=example,dc=com
# dba, groups, example.com
dn: cn=dba,ou=groups,dc=example,dc=com
objectClass: groupOfUniqueNames
cn: dba
uniqueMember: uid=user2,ou=People,dc=example,dc=com
# search result
search: 2
result: 0 Success
# numResponses: 8
# numEntries: 7

You should also test encrypted connections, first with STARTTLS, by adding the -ZZ option:

shell> ldapsearch -x -h localhost -ZZ -b 'dc=example,dc=com'

You can also do it with SSL (ldaps), but this has been deprecated in LDAPv3:

shell> ldapsearch -x -H ldaps:///localhost -b 'dc=example,dc=com'

If you need to look at the OpenLDAP log files, you should enable them in syslog.conf:

# Log LDAP stuff
local4.*                        /var/log/ldap.log

By default, OpenLDAP writes to the LOG_LOCAL 4 facility. You should also set the loglevel to the amount of detail needed. Either check man slapd.conf or the documentation for the available levels.

At the moment, we have a running openldap server which is populated with two test users, ready to be used by MEM. In the next post, I will discuss how to setup MEM to authenticate against our LDAP server. If you need to manipulate your LDAP directory and need a graphical tool, I found Apache Directory Studio to be very nice.

Connecting JBDC to MySQL Enterprise Monitor’s Query Analyzer

October 11th, 2010 by Leandro Morgado

With the release of MySQL Enterprise Monitor (MEM) 2.2, there is now the ability to monitor queries using the Query Analyzer (QUAN) without needing the agent proxy to be running. You can use a .NET or JDBC connector plugin to directly gather the query statistics. In the example below, we will use the MySQL Enterprise Plugin for Connector/J.

First, make sure both the Connector/J, the Connector/J plugin and the Apache Commons Logging jars are in the $CLASSPATH. At the time of writing, these are the files needed:

mysql-connector-java-5.1.12-bin.jar
c-java-mysql-enterprise-plugin-1.0.0.42.jar
required/commons-logging-1.1.1.jar

Then, add the plugin to the connection string so that it changes from something like this:

conn =	DriverManager.getConnection("jdbc:mysql://localhost:3306/test?"+"user=root&password=PASSWORD");

to something like this:

conn = DriverManager.getConnection("jdbc:mysql://localhost:3306/test?"
       +"user=root&password=PASSWORD"
       +"&statementInterceptors=com.mysql.etools.jdbc.StatementPerformanceCounters"
       +"&serviceManagerUrl=http://memserverhost.com:18080/"
       +"&serviceManagerUser=agent&serviceManagerPassword=AGENTPASSWORD");

Here is the full example code:

import java.sql.*;
 
// Notice, do not import com.mysql.jdbc.*
// or you will have problems!
 
public class LoadDriver {
    public static void main(String[] args) {
 
        try {
            // The newInstance() call is a work around for some
            // broken Java implementations
 
           	Class.forName("com.mysql.jdbc.Driver").newInstance();
 
	        Connection conn = null;
		conn =	DriverManager.getConnection("jdbc:mysql://127.0.0.1:33306/test?"+"user=root&password=PASSWORD"+"&statementInterceptors=com.mysql.etools.jdbc.StatementPerformanceCounters&serviceManagerUrl=http://memserverhostname.com:18080/&serviceManagerUser=agent&serviceManagerPassword=AGENTPASSWORD");
 
		//Get a Statement object
	      	Statement stmt = conn.createStatement();
    		ResultSet rs = stmt.executeQuery("SELECT SLEEP(10)");
		System.out.println("Closing connection");
		rs.close();
		conn.close();
        } catch (Exception ex) {
            // handle the error
	    ex.printStackTrace();
        }
    }
}

Now, when you Java application runs queries, they will be analyzed by QUAN. In my example code, there is a sleep call for 10 seconds. You can clearly see that this query was caught by QUAN:

Slow queries fed to QUAN using the Connector/J plugin

This now eliminates the need for the monitoring agent to be running it’s own instance of mysql-proxy for QUAN. Java and .NET applications can simply use the interceptor, saving on resources.

Of course, prior to setting this up, you should always check the online documentation.

MySQL Locks (and a bit of the Query Cache)

September 10th, 2008 by Leandro Morgado

MySQL uses locks for concurrency control. Whenever a client/thread acquires a lock, it will have exclusive access to that table or row (depending on the granularity of the lock). Other clients however, will be prevented from writing and possibly reading to/from the locked resource. The two main existing locks are:

READ LOCK – A read lock will allow the other clients to read from the locked resource but not write to it.
WRITE LOCK – a write lock will prevent the other clients from reading or writing to the locked resource.

Also, different storage engines have different lock granularity. For example, MyISAM will lock the whole table while InnoDB will only lock the rows it needs.

Let’s try this out using a MyISAM table. We open two concurrent sessions and in the first lock the City table from the world database (get it here).

Session 1:

mysql> LOCK TABLE City READ;
Query OK, 0 rows affected (0.19 sec)

Session 2:

mysql> SELECT * FROM City LIMIT 1;
+----+------+-------------+----------+------------+
| ID | Name | CountryCode | District | Population |
+----+------+-------------+----------+------------+
|  1 |      | AFG         | Kabol    |    1780000 |
+----+------+-------------+----------+------------+
1 row in set (1.09 sec)
 
mysql> DELETE FROM City WHERE ID=1;

Here we can see that in Session 2, we are able to read from the locked table, but the query just hangs when trying to write. This happens because Session 1 has acquired a read lock on the table. As soon as Session 1 releases the lock, Session 2 is able to write to the table.

Session 1:

mysql> UNLOCK TABLES;
Query OK, 0 rows affected (0.05 sec)

Session 2:

mysql> DELETE FROM City WHERE ID=1;
Query OK, 1 row affected (1 min 39.07 sec)

We can see that the lock was in place for over a minute before Session 1 released it and Session 2 was able to grab it. Now let’s see what happens with a WRITE LOCK.

Session 1:

mysql> LOCK TABLE City WRITE;
Query OK, 0 rows affected (0.05 sec)

Session 2:

mysql> DELETE FROM City WHERE ID=2;
Query aborted by Ctrl+C
ERROR 1317 (70100): Query execution was interrupted
mysql> SELECT * FROM City LIMIT 1;

In the example above I tried to write to City, saw that it hung waiting for the lock and then aborted the query. I then tried to do a read. Due to the write lock, Session 2 can’t do a read or a write until Session 1 releases the lock:

Session 1:

mysql> UNLOCK TABLES;
Query OK, 0 rows affected (0.00 sec)

Session 2:

+----+----------+-------------+----------+------------+
| ID | Name     | CountryCode | District | Population |
+----+----------+-------------+----------+------------+
|  2 | Qandahar | AFG         | Qandahar |     237500 |
+----+----------+-------------+----------+------------+
1 row in set (25.64 sec)

One thing that can confuse you is the following situation. Say you read something from a table before issuing any locks:

Session 2:

mysql> SELECT * FROM City LIMIT 1;
+----+----------+-------------+----------+------------+
| ID | Name     | CountryCode | District | Population |
+----+----------+-------------+----------+------------+
|  2 | Qandahar | AFG         | Qandahar |     237500 |
+----+----------+-------------+----------+------------+
1 row in set (0.00 sec)

Then you write lock the table (preventing reads and writes from other sessions) and try the read again:

Session 1:

mysql> LOCK TABLE City WRITE;
Query OK, 0 rows affected (0.00 sec)

Session 2:

mysql> SELECT * FROM City LIMIT 1;
+----+----------+-------------+----------+------------+
| ID | Name     | CountryCode | District | Population |
+----+----------+-------------+----------+------------+
|  2 | Qandahar | AFG         | Qandahar |     237500 |
+----+----------+-------------+----------+------------+
1 row in set (0.00 sec)

Wow! Session 2 was able to read the City table even though Session 1 has write locked it! What the hell is happening? The truth is that MySQL’s query cache is tricking you. The query cache gets checked beforehand and if the query is cached, it won’t even need to look at the table or care about locks. One way to test this is by issuing a different read query that’s not in cache:

Session 2:

mysql> SELECT * FROM City LIMIT 2;

It hangs as expected, waiting for the write lock to be released. Remember that MySQL does a hash on the SQL query and if it’s different (even in case), it won’t use the query cache. We can also try the same query after emptying the query cache:

Session 2:

mysql> RESET QUERY CACHE;
Query OK, 0 rows affected (0.00 sec)
 
mysql> SELECT * FROM City LIMIT 1;

Once again, this leaves us waiting on the lock.

Concurrency is complicated stuff and there is a lot of ground to cover. There are other ways to do locks (eg: GET_LOCK(), SELECT … FOR UPDATE) with different purposes and applications. You can find out a little bit more in the excellent MySQL documentation (1 2 3)

The query cache is very handy and allows you to boost performance greatly. As with any cache, it can be confusing, so you should remember that it will serve out cached results even when you don’t expect them (it can turn it off though). You can find more info here.

One thing you can do to get prevent your queries being served from the query cache is to issue them like this:

mysql> SELECT SQL_NO_CACHE * FROM City LIMIT 2;

I’ll close off with some notes about locking:

  1. WRITE locks have higher priority than READ locks
  2. When you issue a second LOCK TABLE, it will implicitly unlock all previously held locks.
  3. To lock mutiple tables, you need to lock them simultaneously. Eg:LOCK TABLE City WRITE, Country READ;

Happy lock picking!

MySQL Passwordless Login

August 12th, 2008 by Leandro Morgado

MySQL allows user specific options to be placed in a user options file under your home directory. This is handy if you use automated scripts or want to avoid typing in your MySQL user’s password everytime you login in. To do this, create the ~/.my.cnf with the options below:

[client]
user=mysqlusername
password=mysqlpassword

Then make sure no one else on the system can read this file. It’s essential for security as it contains your password in clear text!

$ chmod 600 ~/.my.cnf
$ ls -l ~/.my.cnf
-rw------- 1 alice alice 36 Aug 12 17:49 /home/alice/.my.cnf

Now you can just type mysql to login. Note that this will work for other MySQL clients like mysqldump. And you can always override the .my.cnf settings in the command line. For example:

$ mysql -ubob -pbobspassword

You can add other parameters to this file. For a full reference check out:
http://dev.mysql.com/doc/refman/5.1/en/option-files.html