headphono.us

Incase you’re not familiar with the advantages of message queues in web applications, they allow you to (among other things) offload tasks and processes that may be initiated by users interacting with your application but the results of which need not return back to the user synchronously. E.g. a user submits an e-commerce order, the application captures the order and returns a confirmation to the user. As an offline (to the user) process, the order is processed, fulfilled from existing inventory and marked to be shipped out. Each of these steps would consume a message (from a queue) and after their task is complete, if necessary, add a message to the queue for the next process to consume. The use cases for message queues aren’t limited to user initiated events, and have general applications in most systems.

Traditionally, message queues have been implemented in enterprise systems as large-scale “message queue middle tier” deployments. Most of these are provided by the big boys (read big investment) such as IBM, Tibco, etc. and usually require a dedicated team of engineers to integrate (though this has changed lately). However, there weren’t really any independent or open source solutions ready for production use until the emergence of RabbitMQ – an Open Source Enterprise Messaging Broker.

RabbitMQ is built on the foundations of Erlang and implements AMQP – an emerging standard of high performance messaging. It is based on a proven platform, is interoperable and offers low latency and good throughput. I’ve been hearing good things lately about it, specifically about the ease-of-setup (packaged EC2 server configs), performance and scalability, numerous protocol bindings and integration frameworks, etc.

This Google Tech Talk video (embedded below) explains the goals of RabbitMQ, their internal architecture (and decisions behind their design) and their work so far:

Here are the slides:

• Part 1: http://www.rabbitmq.com/resources/google-tech-talk-final/alexis-google-rabbitmq-talk.pdf
• Part 2: http://www.rabbitmq.com/resources/google-tech-talk-final/google.html

What seems most interesting is that RabbitMQ seems to be the first real step towards open sourcing the messaging middleware, which traditionally has suffered from vendor lock-in. Since it’s a $1 billion industry, I don’t quite expect the vendors to take this lying down. So either they will compete by lowering client costs, provide better support, or offer more managed integrations, or open source products will win the critical mass game by getting most small business/startups to adopt rapidly (similar to MySQL). In either case, it should be interesting to watch what RabbitMQ has started.

A natural extension message queues is where many processes consume a message from the queue (i.e. Publish/Subscribe pattern). PubSub is great for broadcasting messages to a variety of consuming processes. E.g. In the e-commerce order placement example above, once the credit card processing is complete, the message could be broadcasted to a variety of processes down the pipeline (order fulfillment, inventory update, shipping tracking, etc.). Fortunately, LShift has implemented a PubSub layer on top of RabbitMQ.

With the growing use of cloud computing services like Amazon Web Services, web developers are going to have start thinking more enterprisy (if that’s a word?). If you’re a web developer, what are your thoughts? If you’re a web developer who is already thinking in this direction (good for you!), how has your design/architecture process changed from the early days of web development?

July 6, 2009 Update: Thanks to @andrewgleave who linked to this presentation which goes more in-depth about AMQP, PubSub and RabbitMQ.