Kafka Client for .NET

They say that every journey start with a single step, well even those can get screwed up.  I am going to attempt to write a comprehensive client for Kafka written in my native tongue, c# and I want to document the progress.

And now for the misstep… I have been working on this client for several days now and I have been making a great deal of progress.  I felt that this evening was a good enough time to add my work to Github.  Turns out that when Xamarin says that the folder is not empty, and do I want it to be deleted, they mean it.  I didn’t notice that the folder I had in the box was at the root of my projects.  Luckily this a fairly new laptop so I only had a single project.

The documentation that I am using as a reference guide is located here https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-Requests.

Some of my first struggles were due to my lack of familiarity with the backus-naur syntax.  Another nuance that I glanced over where the default serialization that Kafka expects.  Obvious now, but i had missed some of the required around the inclusion of variable width sized fields such as strings.  In Kafka, they specify that for arrays it expects the field to be preceded by an 32-bit byte count.  For instance, an array of 32 fields you would add the 32 to the stream of data and then the content of the array.  The same pattern was true for string fields, except they need to be preceded by a 16-bit count.

Another big thing that luckily I picked up on early was that this was built on big endian so I had to reverse all of the bytes.  I am going to start the journey again tomorrow on the Metadata API.  Here is the git repo if you want to follow along.




Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s