DropboxMQ: Home

1.2 - Minor fix-ups, jboss support and some future ESB fun Friday, Aug 12, 2011

Just as stated below on the previous news, the 1.2 release is all about jboss support and some future ESB work. The ESB work is tucked away for a future release, but should anyone feel curious, that what open source is all about. Take a look!

What is DropboxMQ?

How well does your JMS recover?

Here is the scenario: You have a JMS application that reads a message, does some work, stores something in a database, and finally sends a reply message. Say the database crashed about two hours ago and the application has burned through hundreds of messages, erroring out on each message. So you fix the database and everything is running fine again. But how do you recover all of those lost messages?

The answer depends largely on which JMS provider you use. With DropboxMQ, you simply cd to the appropriate dropbox's error directory and

`mv * ../target/`

Done!

Welcome to DropboxMQ! A 'dropbox' is a common way to share information electronically by 'dropping' files into a shared folder in one process (the message producer) then reading the files from the same shared folder in another process (the message consumer). DropboxMQ formalizes this communication and wraps it in the Java Message Service (JMS) API.

Most if not all file systems, shared and local, move files from directory to directory in an 'atomic' operation; the move either entirely succeeds or entirely fails but partial success or failure is not possible. DropboxMQ uses this 'atomicity' to resolve contention between multiple consumers. In a dropbox directory, each message is contained in a separate file. Multiple message consumers can be attempting to consume a message at the same moment in time but because each consumer first attempts to move the file, only one consumer will succeed in consuming the message.

Dropboxes are cross-platform and language independent. At the company where I work, we have applications that interact with dropboxes running on Solaris, IRIX and Linux written in Java, C and a variety of shell languages. In fact, directly from the command line, messages can be sent to or removed from a destination without any adverse effects on other applications.

Dropboxes are as scalable and fault tolerant as the file system on which they are stored. My company uses a NetApp with a fully-redundant, active-active controller configuration. We are passing hundreds of thousands of messages per day between dozens of servers while using less than 5% of the NetApp's bandwidth.

Previous News:

1.1 - Tying up loose ends Saturday, Nov 20, 2010

Only a handful of minor fixes in this release. The main fix was to resolve contention when running on NFS in a clustered or any multiple consumer and multiple server environment. Consumed messages will now have the client id prepended to the message filename to avoid NFS reporting more than one successful consumer. It seems to be working in our production environment, so give it a try and let us know how it goes.

Up next: Some minor fixes for jboss (but most likely applicable to all app servers except OAS). We have also been looking at some fun ESB functionality.

1.0 - Fully JMS Compliant Sunday, Oct 25, 2009

The functionality is 100% there and this release has been in production at my day job for several months now. Sorry for holding it back but life has its demands. Not a lot changed between 0.9 and 1.0. Looking at README.txt nothing really stands out.

0.9 - 99.9 % Complete Monday, Dec 1, 2008

Only a very few (all minor) issues need to be addressed before we get to 1.0. Here are the release notes for 0.9:

JCA and XA complete
Selector based routing (was defer mechanism based on JMSType in 0.8)
Read locks (both global and per destination) to stop consumers
Write locks to slow or stop producers
Message selector support for LIKE, message selector implementation complete
Consumer can delete expired messages with deleteExpiredMessages configuration
Consumer can bypass expired checking with consumeExpiredMessages configuration

0.8 - The devil is in the details Sunday, Sep 2, 2008

Looking at previous posts, XA is still at 90%, JCA is still nearly complete and message selectors still don't support LIKE, but much progress has still been made. Hopefully we can avoid having the last 10% taking 90% of the time. What is in this release is a much more correct JCA and a very solid core.

There have been a couple of interim builds that were not widely published. 0.7 is in production currently at my day job and this 0.8 release is slated to go in mid-October.

0.5 - Interim build Sunday, Jan 13, 2008

This is an interim build to tag a release that will be going into production at my day job. There has been considerable work since 0.4 but unfortunately none of it is 100%. The most egregious bug is in Session.rollback() for which I already have a fix but like I said I need to tag this as is. XA is 90% complete but doesn't recover transactions when there is a system outage. JCA is nearly complete but doesn't work with XA. Message selectors don't currently support the LIKE identifier.

I obviously haven't been twiddling my thumbs but it hard to believe that it's been over a 1 1/2 years since the last release. 0.6 should be just a few days past 0.5. 1.0 will happen when all of the issues mentioned above are 100% which feels very close.

0.4 - Today's Topic: Pub-Sub is now available Monday, Jul 17, 2006

Until now dropboxes were typically used in a point-to-point messaging domain. DropboxMQ 0.4 changes that while maintaining the same basic architecture of previous versions. We have also added some administrative tools and freshened up the documentation.

I'm thinking the next release will contain support for distributed transactions which sounds much more interesting than message selectors.

Enjoy and let me know what you think.

0.3 - Docs, Message types and a `QueueBrowser` Saturday, March 4, 2006

The primary goal of this release was to tie up loose ends and greatly reduce the number of filtered-out JMSCTS tests. The basic architecture remains unchanged but we now support all JMS message types and the QueueBrowser interface. And the documentation should actually make this thing usable now.

Onward to pub-sub!

0.2 - All JMSCTS Point-to-Point Tests Pass Saturday, January 14, 2006

DropboxMQ is now in full compliance with all JMSCTS point-to-point tests. The exact list of filtered out tests can be seen here. Now the prioritized list of new features is as follows:

QueueBrowser implementation
Support for all message types (only Message and TextMessage are currently supported)
Pub-sub messaging
Message selectors
XA transactions

0.1 - Functional JMS from a 36 K jar! Sunday, October 23, 2005

DropboxMQ is far from complete, but it is already functional with several JMS applications. This release only supports point-to-point message model (pub/sub will be supported in a later release).