Last year we started to work on a project that was pretty high profile within the business. The dates had been adjusted several times once we had more information, which added more and more levels of stress. In parallel, we have been re-architecting our platform so there were opportunities to make changes in how we process business transactions. Based on these factors we had to make several design decisions.
We decided that we wanted to start making our business transactions more asynchronous. One of the easy ways to do this is with message queueing. Leading up to this, for another initiative, I had been doing some investigation into the various queueing tools that are available in both open source and commercial and my conclusion was that Kafka was the choice that we should use.
Given the tight schedule we decided that we will go with it for now, foregoing any prototyping. I had ulterior motives for choosing Kafka. We currently shuttle all of our click data to Adobe SiteCatalyst, which many of us feel we should keep that data internally. Kafka was designed at LinkedIn to handle a ridiculous amount of data that they were producing. I believed that by choosing Kafka as our message queue we would be setup to bring our data in-house.
When started to integrate the queue into our processes, the lack of prototyping that we skipped floated to the surface. I took on the task to create the infrastructure so that the client application can leverage it. I learned that there were a lot of things that weren’t as straightforward as I thought.
Some of the challenges were the available tooling to interact with the queue. I tried a few of the libraries that were out there, but for one reason or another I didn’t think that they would meet our needs. I decided to use the rest proxy from Confluent. This part of the integration was promising, everyone knows rest! There was one problem…this rest proxy is stateful.
I don’t want to detail all of the challenges that we encountered and continue to in this post, maybe another day. Skipping ahead, we have turned off the queue for now until we can build integrations tests to make sure that Kafka or any queue will fit our needs. It may turn out that we need to choose a different technology like RabbitMQ or ZeroMQ, but I will eat that crow when/if the time comes. I still believe that Kafka is a great tool and can meet our needs short and long term. For now, I need to iron out all of the wrinkles so we can get it back into production. I wonder what crow tastes like if you brine it first?