Traditionally, message queues have been implemented in enterprise systems as large-scale “message queue middle tier” deployments. Most of these are provided by the big boys (read big investment) such as IBM, Tibco, etc. and usually require a dedicated team of engineers to integrate (though this has changed lately). However, there weren’t really any independent or open source solutions ready for production use until the emergence of RabbitMQ – an Open Source Enterprise Messaging Broker.

RabbitMQ is built on the foundations of Erlang and implements AMQP – an emerging standard of high performance messaging. It is based on a proven platform, is interoperable and offers low latency and good throughput. I’ve been hearing good things lately about it, specifically about the ease-of-setup (packaged EC2 server configs), performance and scalability, numerous protocol bindings and integration frameworks, etc.

This Google Tech Talk video (embedded below) explains the goals of RabbitMQ, their internal architecture (and decisions behind their design) and their work so far:

Here are the slides:

• Part 1: http://www.rabbitmq.com/resources/google-tech-talk-final/alexis-google-rabbitmq-talk.pdf
• Part 2: http://www.rabbitmq.com/resources/google-tech-talk-final/google.html

What seems most interesting is that RabbitMQ seems to be the first real step towards open sourcing the messaging middleware, which traditionally has suffered from vendor lock-in. Since it’s a $1 billion industry, I don’t quite expect the vendors to take this lying down. So either they will compete by lowering client costs, provide better support, or offer more managed integrations, or open source products will win the critical mass game by getting most small business/startups to adopt rapidly (similar to MySQL). In either case, it should be interesting to watch what RabbitMQ has started.

A natural extension message queues is where many processes consume a message from the queue (i.e. Publish/Subscribe pattern). PubSub is great for broadcasting messages to a variety of consuming processes. E.g. In the e-commerce order placement example above, once the credit card processing is complete, the message could be broadcasted to a variety of processes down the pipeline (order fulfillment, inventory update, shipping tracking, etc.). Fortunately, LShift has implemented a PubSub layer on top of RabbitMQ.

With the growing use of cloud computing services like Amazon Web Services, web developers are going to have start thinking more enterprisy (if that’s a word?). If you’re a web developer, what are your thoughts? If you’re a web developer who is already thinking in this direction (good for you!), how has your design/architecture process changed from the early days of web development?

July 6, 2009 Update: Thanks to @andrewgleave who linked to this presentation which goes more in-depth about AMQP, PubSub and RabbitMQ.

Richard Jones (from Last.fm) has compiled an useful comparison of the various distributed key-values stores out there including MemcacheDB, Cassandra and CouchDB. He seems most interested in Scalaris and Project Voldemort.

redis seems to be a very interesting key-value store. It behaves like memcached but supports strings, lists and sets as values. It also includes atomic operations like increments, push/pop, etc. They even wrote a Twitter clone using redis + PHP.

Bret Taylor from Friendfeed goes in-depth about how they built a ’schema-less’ MYSQL schema which attempts to solve the problem of extensibility and maintainability in most large RDBMS solutions. It doesn’t hurt improving performance as a result.

As our database has grown, we have tried to iteratively deal with the scaling issues that come with rapid growth. We did the typical things, like using read slaves and memcache to increase read throughput and sharding our database to improve write throughput. However, as we grew, scaling our existing features to accomodate more traffic turned out to be much less of an issue than adding new features.

Leonard Lin also has some notes on distributed key-value stores. He compares some current tools out there, but finding none viable, creates his own. He has some words of wisdom:

• The distributed stores out there is currently pretty half-baked at best right now. Your comfort-level running in prod may vary, but for most sane people, I doubt you’d want to.
• If you’re dealing w/ a reasonable number of items (<50M), Tokyo Tyrant is crazy fast. If you’re looking for a known, MySQL is probably an acceptable solution.
• Don’t believe the hype. There’s a lot of talk, but I didn’t find any public project that came close to the (implied?) promise of tossing nodes in and having it figure things out.
• Based on the maturity of projects out there, you could write your own in less than a day. It’ll perform as well and at least when it breaks, you’ll be more fond of it. Alternatively, you could go on the conference circuit and talk about how awesome your half-baked distributed keystore is.

The popular consensus seems to be that even though numerous solutions exist out there, none seem to be widely adopted and production-proven yet.

July 1st, 2009 Update: Think Vitamin has a good article on reasons for switching to a non-relational database and also compares the tradeoffs between the different data storage options.

July 21st, 2009 Update: Evan Weaver (of Twitter) has a good intro article to getting up and running with Cassandra.