Arjen's Journal

Move to Open Query blog

2009-04-21T23:06:18Z

I'm shifting away from LiveJournal. It lacks ability to search and otherwise peruse archived blog posts. And of course it's only me, while Open Query has more people.

From now on the posts will be at http://openquery.com/blog/ and this is aggregated to Planet MySQL as a group blog. You may have already seen Walter posting from his seat at the MySQL Conf. All posts and comments from my LJ blog have been migrated to WordPress, thanks to magic performed by young Akash Mehta. Unfortunately the comment threading can't be exported.

The full export means that my personal posts are now also present at Open Query, although I may move those elsewhere later. The existing blog entries on LJ will stay for a while at least, although I do have to pay for the LiveJournal subdomain to keep the URLs alive.

A little sidenote on the export... it actually to considerable effort and script hacking and possibly even some screen scraping. Consider this is my own data... Remember, LJ used to have a perfect export, which actually enticed one to stay put. Funny, isn't it. But that was in the Danga days, and we're two owners down the line from that (SixApart and then the spinoff-sale to the Russians). Lock-in makes people want to leave more.

Oracle agrees to buy Sun

2009-04-20T13:10:00Z

See Oracle Agrees to Acquire Sun Microsystems (NY Times, 20 April 2009). Wellwell, that's quite interesting, isn't it... as to what it means for MySQL, ZFS, OpenSolaris, OpenOffice.org, VirtualBox, Java, and numerous other tidbits, that remains to be seen.

It must be a buzzy kind of day at the MySQL Users Conference 2009, with this news coming in just before the opening... and it's only been a year since Sun bought MySQL and Jonathan Schwartz did a key note at the conference. I'm sure we'll hear a lot more from a lot of people, not necessarily content but definitely conjecture and pre-emptive opinions ;-)

Open Query at MySQL Users Conference 2009

2009-04-20T05:02:55Z

I'm not personally there this year, but Walter Heck will be. In case you haven't met Walter yet, photo enclosed ;-)

He's a techie, like you, and he'd love to meet you and hear how you're using MySQL and surrounding technologies and what things might make your life easier in terms of application architecture, development, deployment, maintenance, and so on.

This may or may not fit with the services that Open Query provides, but the key point is to listen, not sell. If there's a good match, of course that's fine too!

As a sidenote, just to pre-empt the inevitable question of "Arjen why are you not here?": flying over to the US has always been fine, but coming back is a pile of grief every time taking multiple weeks to recover. I've never had that issue with for instance trips to Europe. That's not just grief in terms of personal discomfort or lost work time, but it means my daughter really loses out on her papa for about 3 weeks.

Before I made the decision to note make the trip this year, I did enter various items into the session submission process months back - they didn't get accepted, but I'm not unique in that respect. Many well known speakers from previous years did not get a slot this year. No worries. There's plenty of things going on in the surroundings, including Sheeri's game day and community conference and Percona's performance sessions, as well as all the excellent encounters in the corridors and in the evenings.

Having been absent in 2007 as well, I would like to add a little bit of outside perspective on the event... while for this week everything appears to be focused on the conference (at least when you look at Planet MySQL), most MySQL users around the world will of course not be at the conf, and have the same needs like the rest of the year. I believe it's really important to tell about insights and developments all year round, as well as being active in the various interaction media (such as forums, user groups and events).

Anyway, have a great time at the conference, and do have a chat with Walter!

InnoDB lock timeout before query execution

2009-04-17T23:34:58Z

I found this yesterday while tracking down a locking issue for a client. They had a connection time out on a lock, but before it times out, SHOW PROCESSLIST had the status 'statistics' so it wasn't actually executing the query yet. So, what was it doing and why did it time out there?

The answer actually was remarkably simple, but I did have to take a peek in the MySQL server source code (horay for Open Source!) The server sets the thd_proc_info to 'statistics' when calling the join optimiser, that's the part of the optimiser that works out the best join order.

A lesser known feature of the join optimiser is that if it works out that only one row can match (lookup on primary or unique key field), it'll try to retrieve the row. If there's no match, there's an "impossible WHERE clause" and essentially the entire query is optimised away and 0 rows returned.

If there is a match, all references to columns in that table can be replaced with the values that were just retrieved, turning them into constants. After all, there's only one row matching! This can optimise away tables, removing the need to execute some or all joins. You don't actually see this directly if you do EXPLAIN EXTENDED and then SHOW WARNINGS, because that output is from the parse tree which does not know about the join structure. But what you will see there is that the columns were replaced with constants, so from that you can deduce what's going on.

For a query
EXPLAIN EXTENDED SELECT name from Country where code='AUS'
you would see access type const in the explain (which is the indicator for this optimisation), and SHOW WARNINGS brings up
select 'Australia' AS `name` from `world`.`country` where 1
so you see the WHERE clause is gone completely and the name column in the SELECT has been fed the value Australia so it's all constants.

Back to the original issue... if another connection holds an exclusive (write) lock on that particular row, the join optimiser will be waiting for the lock to be released and that's how you can actually get a lock timeout at this stage. It's a perfectly normal thing to happen.

What a Google data center really looks like

2009-04-03T00:16:35Z

Google has fessed up: apparently it looks like a bunch of shipping containers, since 2005. And it does not use the usual rack stuff or centralised backup power. See Google uncloaks once-secret server for a decent overview, plus pictures.

I think there's quite a lof of good information in there, with important lessons. Just a few:

It's not fancy-brand stuff, they're simple Gigabyte mainboards
An open frame rather than enclosures.
12V battery on the frame, rather than a centralised UPS. More energy efficient, and no single point of failure.
Heatsinks, not lots of fans (mechanical stuff fails). There'll be airflow-creating foo for the whole container, but since it's a single enclosed space there's neat separation between hot/cold already. Efficient!
Two regular SATA harddisks

So, it's mostly commodity stuff in a custom frame and and other minor tidbits. There's 1160 servers per container, and by the looks of it it should work out pretty cheap to buy, build and operate.

The energy efficiency has more aspects... if you use less power, you're also generating less heat that then needs to be got rid of. Cool machines are more reliable and live longer, and even fractions of a degree can make a significant difference.

You can see it all from a monetary perspective, but it also makes sense in terms of maintenance effort, waste/disposal and other environmental impact aspects. Of course these containers still eat a lot of electricity, but there's a lot in there and it will be way "greener" than most data centers.

The problem with April Fools in the MySQL/web space...

2009-04-01T23:52:33Z

...is that truth is stranger than fiction. Reality does not appear any more plausible than plain nonsense.

We were discussing this yesterday on #ourdelta (Freenode IRC) in the context of How MySQL really executes a query by Baron. Antony Curtis noted that if he'd write a truthful post on that topic, people would think it was made-up regardless of the day of the year!

Another proof of the premise: Baron has now put a giant banner above/below his post, explaining that it was a joke. Apparently that's necessary?

I tend to come up with neat ideas for April Fools throughout the year, neglect to write them down, and come the day I have a blank. But, given the above, there's another option: you just write a truthful story, still leaving people wondering whether it's for real. I reckon the main issue is probably with the rest of the year, where lots of people still write nonsense, and the truth remains weird as usual. At least on April 1st you know to question. I hope.

Predictive caching in a MySQL-backed infrastructure

2009-04-01T02:07:49Z

Sounds a bit far fetched (pun intended ;-), but we're doing it. This is not inside of the MySQL server, but rather the overall application design. Let me run you through the logic...

Some key aspects to scaling are: not doing unnecessary queries, and caching what you can. Just a quick baseline. The fastest query is the one you don't do, or the one you've already done before - the latter being caching.

A simple yet brilliant example of this is the Youtube trick where a script reads the relay log, converting updates into appropriate selects and running them so that the InnoDB cache will have the blocks in memory when the slave SQL thread executes the actual update. Maatkit now has a tool for this, so it's publically available. It's not quite predictive, but it's a neat trick anyway that sometimes comes in handy. Search engines use similar tricks.

Extending on this, with certain applications you actually tell what is likely to happen next, sometimes for a particular user and often for many users. Individual user behaviour may sometimes appear random, but as a group it can be highly predictable. The analysis needs to be done properly though, otherwise averaging will make certain interesting behavioural patterns disappear.

Anyway, if you can identify these patterns you can take appropriate measures, such as do some queries so they get cached, and/or schedule other relevant actions (so it's more than just caching, but it's a reasonably suitable name anyway). This allows the app to deal with higher peak load, as well as improving response time for individual user.

I might do a talk or article on the predictive caching concept some time, as I appreciate that the short description may appear a bit abstract or obscure. But I assure you it's entirely practical and real.

It's one example of how Open Query helps its clients scale well, by design. We focus on preventing emergencies, which includes not just scenarios where stuff fails (and does a safe failover), but also the "oh dear we suddenly have so many more users than a minute ago" type of happening, which should actually be an occasion to enjoy, not stress about.

Relax! A Failure is NOT an Emergency - a talk in Melbourne

2009-03-27T01:31:01Z

While I'm in Melbourne in a few weeks for training I'm once again visiting my friends at Linux Users of Victoria (LUV). It's been a while since my schedule coincided with their meeting schedule!

I've been invited to do one of the talks (Sandrine Balbo doing the other), and my topic is "Relax! A Failure is NOT an Emergency." which is although somewhat MySQL-related not actually MySQL-specific.

This will be on Tuesday April 7th. You can find more detailed description and location/time info in the LUV announcement.

Arjen also on twitter & identi.ca

2009-03-19T11:46:09Z

If you'd like to follow shorter scribbles of what I get up to, like my work at Open Query, OurDelta and of course the BlueHackers initiative, I've got myself organised at http://twitter.com/arjenlentz or http://identi.ca/arjenlentz (you can pick one, they have the same feed).

Open Query is now a DotCom

2009-03-16T11:02:58Z

Yes, Open Query is now a DotCom. In the sense of the domain name, that is (openquery.com). It just took us quite a while to "catch" the domain off a hog that was somehow really attached to it!

Other than that, we are

not moving to Silicon Valley; the world is a big place with lots of beautiful places to live and work from. Different OQ people are in different such places.
not seeking or accepting any venture capital; self-funded organic growth works well for us.
not aiming to get bought, or float...

Just in case anybody was wondering! ;-)

Email to either the .com or .com.au will work, and the website simply redirects so it's zero-fuss for everybody. If you experience any hassles, do let me know.

What a school can look like, work better, and cheaper

2009-03-13T11:58:22Z

Schools Designed for Learning: The Denver School of Science and Technology, presented by the American Architectural Foundation, KnowledgeWorks Foundation, and Target, this video features the innovative Denver School of Science and Technology.

Building it was cheaper (per sq.ft) than a "regular" school, and the results are so much better. Fantastic. Take note everybody, and feel very embarassed if you dare to build any more "same old" school blocks!

On Oracle (and MySQL), Enterprise, Suitability and Sense

2009-03-13T11:02:11Z

50 things to know before migrating Oracle to MySQL by Baron Schwartz is an interesting read, it points out clearly that MySQL is not Oracle. However, Oracle is not the benchmark by which all others are to be judged. So what do we compare with, or actually, why do we compare at all?

Hmm, so we take three steps back, and get a much better view... Marten Mickos (MySQL CEO from 2001 until the Sun acquisition in 2008) said it all along "MySQL does not compete with Oracle". I don't think people actually appreciated what he was saying, or even believed that he meant precisely what he said. They might have thought "oh that's just positioning and protesting too much to make the opposite point". But he wasn't, it was the clear plain truth and it still is today (and so it should remain, I think).

In my conference talks and community relations work at the time I described the same thing differently. The only way to compete with Oracle is to become Oracle, and why would you want to do that? Oracle is already very good at being Oracle, because they are Oracle. Seen on the business level, trying to compete head-on against an incumbent tends to not be a winning proposition, even if you're cheaper. It's not really smart business at all.

You don't have to be Oracle to be a valid choice for many deployments, or "horses for courses" as the Australians love to say. MySQL does many things differently, which -in part by chance and in part by being in the right spot at the right time- has allowed it to become one of the core components of the modern web world. And it's doing very well there (and beyond), as we all know. And I'm judging this is on sites and business relying on it and doing well, not anyone in particular making money off MySQL as that in my opinion is not relevant for judging success (that said, there's plenty of profitable and sustainable business in the MySQL space).

MySQL doesn't have to be suited to every possible need, and its not suiting certain needs is not a failure. A knife and a screwdriver are both useful types of tools, but they are not interchangeable in terms of purpose. You can cut some things with a screwdriver, and you can try to turn a screw with a knife, and sometimes such uses even make sense - but it can also get rather awkward, break the tool, cut your hand, and so on. It's not necessarily pretty.

MySQL has a different market from Oracle, and that is why they are not in direct competition. It's not about low end and high end on a single scale, it's quite different in many aspects. You may try to simplify it if you want, but you won't capture the essence of it all. There is a bit of market overlap, and there are also historical deployments of Oracle where it might not have been the most suitable choice (using a knife where a screwdriver might have worked better). So, in some fairly rare cases (seen over the huge numbers of both Oracle and MySQL deployments) some people have a choice where both are suitable, and in some cases migration makes sense if the wrong tool was getting used in the first place. But again you don't want to generalise either.

Presenting to government about migrating from Oracle to MySQL should really be seen in the context of the above two corner cases, and not as a general pitch of "MySQL can replace Oracle". 'cos that'd be ridiculous, right? I know that some sales people would love to close even more such big gigs, but you shouldn't sell something into an inappropriate space. Related communications have muddied the waters somewhat, and I really think that's unfortunate. It has set expectations and a focus in "the market" that is neither appropriate nor sensible, and in the 5.0 and 5.1 releases of MySQL we've also seen how that "reaching for enterprise" (whatever your definition of it is) has not been all that great.

By the way, the term "Enterprise" is as polluted as the term "Cluster". They are both dreadful terms now to use for defining anything. You wouldn't convey a clear concept at all, your picture of it is not the same as the person you're talking with: you won't be on the same wavelength!

The traditional Enterprise database market (defining it broadly as large corporations, government, defence) is a very small and limited space; lots of money there, but it's still limited. There's a much larger space out there, with very high demands in terms of rapid growth, larger numbers of concurrent users than any enterprise organisation has internally, and so on. "Enterprise features" and "Enterprise architecture" are not actually all that suited for those different needs, and in some cases they are hideously bad fits. This is why I bring up issues like SAN deployments. SANs are very good, but in the MySQL space they are generally (not always) the wrong choice. But someone thinks on a linear scale, reaches for the top, and goes for an Enterprise solution... must be good and they won't get fired for it. Horay? Not.

Summary:
MySQL is not Oracle. MySQL is not Enterprise. MySQL is MySQL, it's very very good at it, and the "usable market space" for its functionality is absolutely huge. If you're wondering whether MySQL is good for your needs, just ask. Ask the tough questions relevant to you. And then judge it on its suitability for the particular purpose, not by some arbitrary scale or feature set of another product, or even generic corporate policy. Those comparisons are "easy", but not relevant for your needs and they could cause you to pick the wrong tool for the job.

At Open Query we do get asked about these things, and we explicitly recommend other products over MySQL when we feel it's most appropriate. We have an operational rule I've brought with me from long ago: "I'd rather have a happy non-customer". For instance, for serious GIS (geospatial) applications, PostGIS (PostgresSQL's GIS extensions) are far superior not just in the open source arena but also compared to the proprietary space. Horses for courses.

Can I have your horror-stories, please? (SANs and VMs)

2009-03-13T01:15:22Z

Please make it descriptive, graphic, and if anything burnt or exploded I'd love to have pictures.
Include an approximate timeline of when things happened and when it was all working again (if ever).
Thanks!

This somewhat relates to the earlier post A SAN is a single point-of-failure, too. Somehow people get into scenarios where highly virtualised environments with SANs get things like replication and everything, but it all runs on the same hardware and SAN backend. So if this admittedly very nice hardware fails (and it will!), the degree of "we're stuffed" is particularly high. The reliance in terms of business processes is possibly a key factor there, rather than purely technical issues.

Anyway, if you have good stories of (distributed?) SAN and VM infra failure, please step up and tell all. It'll help prevent similar issues for others. Thanks. I'm not anti-SAN or anti-VM, but both need to be used when appropriate and in the proper context.

ramtest86 hangs but it's not the RAM's fault!

2009-03-12T05:16:35Z

I was using the ramtest86 from a Ubuntu Intrepid startup disk (32 and 64 bit CDs). It was a machine with RAM problems (IBM eServer x346), so I wasn't too surprised that ramtest86 hung during testing. Still, kinda annoying.
I had another similar machine (I have them here for some MySQL related testing) that was known to be probably good (no things are certain ;-) and interestingly it hung there also.

Using ramtest is just my "default" way of finding bad RAM, as it works on any box. But these IBM systems have a diagnostics mode which is essentially a flashrom-based PC-DOS with some basic drivers and a test tool. Cool!
So I tried that, and it worked fine on the "good" box. And found problems in a specific DIMM on the "bad" box.
Excellent. Progress!

Oh and it's not realy that "cool" really since these boxes currently reside in my office, and apart form the noise from the hotswappable fans and disks, they are very good at heat generation. Let me remind you that it's currently summer in Australia, Brisbane is in the subtropics anyway, and my office does not have aircon. If it were winter, it could perhaps act as a decent heater! ;-)

Back to the story... it's now going through the other ram banks in the "bad" box, and then I can rip out the faulty ones. Nice. But the question is, why does ramtest86 hang? These are dual Xeon 3 GHz boxes, in case you're not familiar with the x346. Anyone else familiar with this tool just hanging on certain boxes, or perhaps architectures?

High Performance MySQL (2nd edition) book wins Jolt Productivity Award!

2009-03-12T04:00:27Z

TechWeb Announces Winners of the 19th Annual Jolt Product Excellence & Productivity Awards - The 'Oscars' of Software Development Industry Honor Innovation and Technological Achievement.

The winners of the 19th Jolt Product Excellence Awards and the recipients of the Jolt Productivity Awards are:
[...]
2. Technical Books
[...]
Productivity Winners:
High Performance MySQL: Optimization, Backups, Replication, and More
by Baron Schwartz, Peter Zaitsev, Vadim Tkachenko, Jeremy Zawodny,
Arjen Lentz and Derek J. Balling (O'Reilly Media)
[...]

Migrating MySQL MyISAM apps to the InnoDB storage engine

2009-03-12T01:47:45Z

Today, most of us professionally involved with MySQL deployments regard InnoDB as the default engine to use, although the server "default default" is still MyISAM and many (even new) deployments still end up on MyISAM. It used to be the case that InnoDB had advantages in terms of write-concurrency, crash recovery and consistency/durability as well as in some cases speed, however it uses more disk space and to perform adequately it requires at least more RAM. For deployments that didn't care for write-concurrency, MyISAM often remained the favourite. But most servers now have enough basic horsepower and sufficient RAM, and the disk-usage issue has also been addressed in the 5.1 InnoDB plugin with compressed pages. Disk use is often not the biggest issue though.

So anyway, if you don't have FULLTEXT indexes (use Sphinx!), can't you simple run ALTER TABLE ... ENGINE=InnoDB on all your tables (or use the mysql_convert_table_format script) and live on happily? Not quite...
First of all you'll want to tune the InnoDB subsystem in terms of buffer pool, log file size, and some other aspects.
But also, your application can encounter additional error situations when using InnoDB, which it must handle. In my previous post (Error handling for MySQL applications) I already indicated that many apps just don't handle errors properly at all. Many errors are not fatal, and many are transient. That is, if you do the same command again, it'll generally succeed.

InnoDB being transactional and using row-level locking, it's possible and entirely normal to encounter deadlocks. They are distinctly non-fatal, but do require appropriate handling. For a multi-statement transaction (i.e. more than one query, wrapped in a BEGIN or START TRANSACTION and COMMIT construct), when you get one of the lock timeout errors or a deadlock error, you can simply ROLLBACK and then re-run the whole transaction construct. This should not cause an endless loop, but re-running at least once is sensible.

If you're not (yet) using InnoDB but just MyISAM, you're not in the clear though: regular lock timeouts can also occur with MyISAM, although problems are, in my experience, less common.

I'm always intrigued about the "why" of things, so in this case... why aren't MySQL application developers aware of these things? And I think I might be able to answer that question. Anyone with a background in using transactional database systems (Oracle, etc) has been taught to expert and deal with these things, so it'll be an intrinsic part of how they write code. Contrarily, MySQL's non-transactional origins and the fact that many MySQL application developers do not have a background in database engineering or other database systems, explains most of it.

But it's also MyISAM being deadlock free by design, that has allowed MySQL app developers to be particularly lazy about these things, AND get away with it! MyISAM always locks specified tables in a particular order whereas InnoDB, with its row-level locking, can't control the flow of locking needs, and thus cannot guarantee to be deadlock free. It's not a flaw (or magic) on either side, it's just consequences and possibilities due to the capabilities of the engines and their resulting internal architecture (also covered by one of the Open Query training day-modules).

So, learning this and without needing to pass blame on MySQL, any particular engine, or developers, we can now get on with things and develop (and fix up!) applications to have appropriate logic, whichever engine might be used underneath. But most importantly, please remember that just converting an app from MyISAM to InnoDB without checking up on the error handling is not safe.

Error handling for MySQL applications

2009-03-12T01:12:52Z

When connecting to MySQL, or executing a query, proper error handling is required. Many take this very seriously, and do a construct like mysql_connect() or die() or the equivalent with mysql_query(). For web apps this generally makes error codes end up on the user page, you can easily see this by doing a Google search for some of the common error texts. Slightly improved apps are nicer to the user and log the error separately.

But both approaches fail, fundamentally, as they don't take into account that not all errors are the same and, most importantly, many error are not fatal but require other forms of action. So let's look at that, look at what causes the errors so you truly understand that it's not fatal stuff, and how you can handle them.

When you get a "not successful" response back from a MySQL API function, you first need find out what the error is with mysql_errno() (and mysql_error() for a corresponding text string, if you want to log it).

So when in the below I write about an API function returning an error, I don't mean that the function itself literally returns the error code... it'll return false (or whatever the appropriate return code for non-success is for that function) and you need to call mysql_error() to retrieve the actual error code. Oki? Let's continue!

If mysql_connect() to the local Unix socket comes up with OS (operating system, the <1000 codes come from the OS not MySQL server as such) error code 11 (MySQL error text "Resource temporarily unavailable") this is actually defined as EAGAIN in your operating system. That's quite descriptive actually and can be read as "failed now, but do try again".
Typically you can get this error when the maximum # of connections has been reached. A possible cause can be that apps keep a connection open even if they don't need to use it for an extended period, or an app server doing persistent connections even though the infrastructure does not effectively manage them (Apache/PHP is the prime example there). MySQL has a very fast connection handshake so speed-wise persistent connections are not necessary.
There are some things you can tune on the MySQL server end and some other things to look at on the application server end... I posted a note about the PHP issue and solution 4 years back.
If the error code is 1040 (Too many connections) the situation is similar but you were trying to open a TCP/IP connection to the server. The handling is similar though. You will want to retry, but not keep on looping forever of course ;-)
By the way, you can't just increase max_connections in my.cnf as some buffers are allocated per-connection and thus particularly on 32-bit installations you could get into memory trouble.
Error 1046 I'm tossing in here just to make the point about non-fatal-errors... it means "No database selected" and indicates you simply forgot to do mysql_select_db() before running a query that doesn't explicitly specify the database for each table it uses.
Error 2013 (Lost connection to MySQL server during query) is actually a generated inside your client library, and if your environment is using the C client library then simply re-issuing the query will make the library reconnect as well. You can also reconnect explicitly and then re-run the query. Why does it happen?
It could be a network error that caused the TCP/IP connection to drop, or the wait_timeout was exceeded on the server; the latter can actually be useful in keeping the # of open connections down, but the app will then need to handle errorcode 2013 correctly!!
Error 1213 (Deadlock found when trying to get lock; try restarting transaction) can only occur when using InnoDB, since MyISAM is by its design deadlock-free. You just need to re-run the query. If it's a multi-statement transaction, the entire transaction should be re-run. (I'll clarify this in a separate blog post immediately after this one!)
Error 1614 (Transaction branch was rolled back: deadlock was detected) is XA (distributed transaction, two-phase commit) related, handling essentially the same as for 1213.
Error 1205 (Lock wait timeout exceeded; try restarting transaction) can occur with both InnoDB and MyISAM, and again simply reissuing the statement or transaction is the way to go (also see note below).

Next up a related post about deadlocks and lock wait timeouts in both MyISAM and InnoDB, and more specifically about some things to look out for when converting from MyISAM to InnoDB (Migrating MySQL MyISAM apps to the InnoDB storage engine). Also, to be very clear, when I write about "re-run" or "re-issue" the query, I don't mean "create an endless loop". Normal coding sense applies.

The error codes mentioned in this post are merely very common ones, there are more! For ref, see Errors, Error Codes, and Common Problems in MySQL's online reference manual.

Safari 4 browser beta.... first impressions

2009-03-06T03:31:50Z

On my MacBook, I use both Safari and Firefox anyway, so this is not a "browser switch" story... merely that Safari 4 beta is now available.

It feels fast, and has some pretty interesting features... does take some getting used to, particularly having the tabs all the way at the top now. And the size of the tabs scales with the # present, which I find to be a nuisance; I prefer them to be a fixed size, this also keep the closing button [X] in a consistent position if you close multiple tabs. I'll work with it for some days.

Do you have a fork stuck in your head?

2009-03-06T00:20:36Z

This is actually a MySQL-related post, but I like odd analogies - stuff that makes you laugh tends to be remembered better!

So to get back to the fork... it's stuck in your head and you have a serious headache.
You go to the doctor. Would you like the doctor to
A) prescribe something for the headache, or
B) identify that the fork might be causing the headache, and suggest that it be removed?
There may be other factors causing the headache, but leaving the fork there is probably a bad idea anyway, right?

When someone asks me questions like "what's the maximum number of columns MySQL can have in a table", a little alarmbell goes off in my head (better than a fork, eh!), and I try to explore the "why" - much to some people's annoyance... they just want "the answer". But I do this because it's one of those anti-patterns, indicative of other issues. There are indeed valid reasons to require lots of columns, but not many.

I have standards, and I apply them on Freenode #mysql IRC as well as with Open Query clients. I don't want to just help you "fix" your headache while leaving a fork in your bleeding head.
Instead, I'd like to make sure there's no fork, first. Once you're aware of the issues in the anti-pattern, you can make an assessment (with or without assistance) whether the problem you were trying to address is actually the real problem, or whether you want to deal with any underlying issues.
Am I an annoying doctor? ;-)

mysql_install_db, mysqld --bootstrap, binary log, cPanel

2009-03-05T04:55:15Z

Warning... what follows is a murky mess.
It's filed as MySQL bug#43398 (verified!) but it's triggered by cPanel doing evil.

Scenario....

Start a mysql server, as normal
Then run mysql_install_db (as root, like you would when you first install MySQL)
See a new binlog file get created, with ownership/group root!

Of course you generally wouldn't run mysql_install_db while a server is running, but there's nothing to prevent you (or something else) from doing so!
--bootstrap just shouldn't initialise binlog, then there wouldn't be a issue.

cPanel runs mysql_install_db in its automatic upgrade scripts (dangerous already, automatically upgrading MySQL Server on a system!), it's run every night on cPanel systems even if no upgrade is done, and it behaves exactly as described above. It then chowns the binlog files to mysql:mysql which is of course a hack and not a fix, also there's still a brief moment where the new logfile has the wrong permissions, and mysqld may still encounter various race conditions by the mysql --bootstrap just adding a binlog file independently (as described in the bug report).

Then some people filed a bug with cPanel about binlog files ending up as root when datadir is changed from the default, so cPanel added this:

# temporary fix for non-standard mysql directory.

echo " "
echo -n "POSTUPCP: running temporary MySQL permissions fix: "
for dir in data import tmp logs ; do

if [ -d "/home/mysql/${dir}" ]; then
  chown -R mysql.mysql "/home/mysql/${dir}" &>/dev/null
fi
done

echo "OK"
echo " "
# end temp fix

That's right, it's an additional hack to set the permissions right, but meanwhile the server is still running and doing its thing. The potential for race conditions is just ridiculous.

Key take-away from this story... look, I'm not opposed to workarounds, but it's very important to understand what cause a particular symptom has. Then, a workaround may be "ok" within a specific context. But if the root cause is not understood (or not found at all) then applying workarounds like this is beyond dangerous.
In this case it's in a commercial product that's deployed in thousands of managed server deployments. That's just freaky.

grsecurity

2009-03-05T03:25:14Z

Hint... don't install stuff like this on your production servers unless you have advanced Linux skills... http://www.grsecurity.net/
It's custom kernel modifications. So any time you upgrade kernel, you need an update of this stuff as well. And never mind the potential bugs that in themselves can cause security and reliability issues.
You don't "just" want to install any such package on a production server (MySQL or otherwise), unless you really really know what you're doing. It's just not a userspace kind of thing.

Apparently it's something that some cPanel deployments install to further secure their system, but that comes to the fundamental question of whether you want to be running cPanel in serious production environments.... beef for another post, and I'll get to that shortly.

OpenSolaris on an Eee PC?

2009-03-04T07:47:29Z

No, I didn't do it, honestly! ;-)
I just got an email from Sun (Open Query being a Partner and all) to participate in a survey, and this is the incentive prize. It's kinda funny, but also a bit indulgent. I really don't think that's where OpenSolaris shines, so why even go there... just to prove that it'll run in a tiny environment? Who cares? I mean, you'd usually run it on a nice 64bit multi-core machine with plenty of RAM and storage, right? Just like any other new server...

I don't get it... perhaps someone else will offer a more plausible suggestion, but so far I'm guessing indulgence ;-)

Challenge: build a queue/IPC in standard MySQL without "polling"

2009-03-01T13:21:20Z

Following scenario... you want to pass on some basic info to another task, through MySQL, and the other task can just wait for it - naturally you don't want to poll (keep issuing a query to see if there's anything new) as that's hideously inefficient even if you use the query cache.

Rules: standard MySQL (no special storage engines); and stay within MySQL, no other tools.

I have some thoughts on this, for instance using the behaviour of InnoDB with isolation levels and locking... can be engineered to either acquire a lock (after waiting for someone else to release) or timeout, where the timeout can be set so you don't have to poll all the time... but curious if anyone might have more nifty ideas!

Open Query - job opportunities for MySQL and related skillsets

2009-02-27T02:01:52Z

Economic downturn? Perhaps, our service offering and pricing is such that we've actually only seen an increase in demand, which is of course great! So, we're looking for additional skilled and enthusiastic co-workers.

Open Query is not structured like your typical company, and this is quite on purpose. Our funding, business structure and approach, human resources, decisions on offering and pricing, are all "a bit different" to jive with our thoughts on building a company that provides a range of excellent services, while also really looking at what customers actually need, and providing a healthy (literally) working environment for our people.

One of our key aims is towards no-stress, both for ourselves as well as for our clients. You may figure stress and emergencies are an inevitable "fact of life", but we feel that that would be a wrong starting point. When instead we put our minds to creating no-stress solutions, we will at least end up with something low-stress rather than "same old".

For our workers, we offer ongoing contracts for a guaranteed minimum of hours a month and an agreed maximum. So you can have more free time, or do other things. We pay decent rates so you may find that part-time is quite sufficient for your life needs! We fully expect "an honest day's work" but won't load you until you're run off your feet - and this contract structure guarantees that.

Your principal place of work would be your home. That's not for everyone, but works well for many and is absolutely ideal for some - we have experience with this and can help you with ideas on how to arrange things.

Most work is done remotely (you'll need a reliable ADSL connection, although I sometimes work from a cafe with a UMTS connection for my laptop - that's fine for some work, but not all), and for those tasks we really don't mind where you live. We have some strategic needs in terms of availability in certain ranges of timezones.

If you are (also) interested in a trainer position, there are additional requirements in terms of experience and skill, as well as location. You would never be doing just training with us though, otherwise you'd a) lose touch with the real world and b) find plane seat 23B and hotel room 843 to be your primary residence, which we think neither of those things would be good.

Did I get your attention, and you're not put off but rather intrigued and attracted by Open Query's operational "quirkyness"? Then I'd love to talk with you! For more details and contact info, see http://openquery.com/company/jobs

On Value and Cost - part 2 (failed loyalty scheme)

2009-02-19T23:50:50Z

Some offerings are just a complete fail. I'll give you an example from my real world. Supermarket chains around here give out a voucher for a petrol discount when you spend more than $30 in one go. That's kinda smart, since petrol pricing is something people care about so it can drive changes in customer behaviour (to a point).
Then one of the big chains has introduced a kind of loyalty card last year, whereby the discount gets linked to the card rather than getting printed on the receipt. Saves hassle with paper, but the real objective (like any loyalty card) is to better track customer behaviour by allowing the supermarket to link your purchases over time.

The key fail is that you still only get the discount if you spend $30 in one go. Of course this is aimed to entice people to do fewer big shops rather than more smaller ones. But apart from the paper hassle, we've merely identified two benefits to the vendor, and nothing much for the customer. And it doesn't even work for the supermarket, because now people won't bother showing their card if they're spending less than $30, and thus their behaviour doesn't get tracked for those purchases. That's the thing you'd want to specifically track, if you'd like to get people to change their purchasing behaviour. And the discounts aren't sufficient to drive a change in behaviour, they still do ad-hoc shops. So it just doesn't work.

What could help? Keeping the loyalty card concept, the supermarket could record points for every purchase, with a bonus number (or percentage) for bigger purchases. At a certain level of points, the customer gets their fuel discount. It's all automated through the cash registers anyway, so all that is easy enough to implement.

But I haven't told you about the other fail, on the customer side. With the cards, people now can't easily tell whether there's a discount on their card, unless they login to some website to check (and remember yet another login and password). They used to at least have their discount voucher, which you handed over when you pay for the petrol.

If you don't show your card now you still get your voucher (although no special offers that only apply if you show the card but again are only valid if the purchase is greater than $30), and if that were scrapped I'm sure that people would still use the card when doing bigger purchases just to not lose out on the discounts. That doesn't mean it's a good system though. It sucks on all sides as it pretty much fails all its objectives.