Well Oiled Machine

As a leader, I am always looking at situations to find the lessons that are hiding.  The other day I watched a team of engineers working together and it was inspiring.  Here are some of my observations and commentary.

Over the last several weeks, the team has been working on refactoring some of the code that is deep within the bowels of our infrastructure. There has been no shortage of challenges in making these changes.

The code was refactored with backwards compatibility in mind, but that doesn’t always work out so nicely.  It was decided to only retrofit the systems that were in their domain since they are closer and have more intimate knowledge.  They started out strong, but the challenges started to pile up.

There were issues with namespaces, assembly references and code that was removed that needed to be swapped out.  Once the new assemblies were referenced the scope of the issues started to be clear.  One interesting issue that cropped up was a ghost assembly that was somehow being inserted back into the project.  The problem is that was the one library that is getting ripped out.  I blame NuGet (easy target).

We are big fans of swarming our work, but work like this refactoring didn’t seem to lend it self to a swarm.  For those of you who are not familiar with swarming, it is where the entire team works together on a single work item until it is complete.  We have found that in many situations this works well, but that is a topic for another day.

Back to the inspiration.

I walked out of my office and I notice a huddle around one of the engineers cube.  I was curious so I hung out listening and watching, and I was very impressed.  This team was swarming together to help one of the engineers with their refactoring task.

What I saw was an even playing field.  There were no pretenses of superiority and no egos.  Just four engineers working together to solve a problem.  It was a beautiful thing to see!

All to often I see the game of “it was his code” or “that is her PBI”.  That doesn’t sound very teamish (feel free to use that word) to me.  You maybe thinking that all engineers should work together to deliver business value, right?  Unfortunately that is the exception and not the rule in my experience.

It is not often that I get to see that level of team work within a team.  These engineers are some of the most professional teammates that I have had the pleasure to work with.  The interesting piece is that half of the team members are junior, but they operate as a well oiled machine.  I was inspired by their cohesion around the problem, it was awesome!  I was so inspired, I took a photo (pixelated for security 🙂 ).





Hero Complex

I want to spend a minute expounding on a previous post where I talked about having (past tense) a hero complex.  I guess the logical place to start is with a definition (albeit, mine) of hero complex.

It is similar in context to the arsonist firefighter lighting fires and then serving as a first responder.  Now to be fair, the arsonist firefighter has a much more negative connotation.  The hero complex is where you need to be the one who solves the hard problems and saves the day.  There are a few problems with this.

At face value, the hero is the one that everyone can depend on to solve the problems and put out fires.  This sounds like a great thing to have someone so dependable, but…  The team dynamic, when there is a hero, can turn ugly.

If there is always someone there to do for you what you can do yourself, you are then robbed of the ability to be accountable.  It also robs you of the glory of saving the day on your own.

I previously held the opinion that this is what a servant leader is.  Shielding the team from the stuff that was less fun aka site down.  That was something that I thought made a good leader.

In the Marine Corps, we were taught that you shouldn’t ask someone to do something that you were unwilling to do yourself.  I think that is a good starting point, but it will not win the race on its own.

When I reflect on those days, I realize that I had an issue because the people were not meeting my expectation.  This was not because they weren’t capable, but rather because I expected that they would be me.  Do things the way I would do it.  Looking back, I can say that this was pretty dumb.

The fact is they were all very talented engineers who could have solved any one of the many problems in an equal number of ways, and maybe one of them was my way.  The important piece is that they solved the problem and not how it was solved.

Now I can take another look at the servant leader and it’s role.  How can a leader best serve the team that he is responsible for?  That begs another question to be asked.  What is the team responsible for?  Your job as a leader is to make sure that they can do their job!  Get them the tools, training, extra help, etc. that is needed to get the job done.

It is a wise leader who can spot and draw out what the team needs to help them achieve their goals.  It is a wise leader who can inspire the team to improve and solve impossible problems.  Sorry, but it is time for a Simon Sinek quote.

It is the leader who should help the team develop why they do what they do instead of what they do.  It is the why they do what they do that will inspire them to come to work, login and do great things.

I started this post talking about the hero complex and how I was once afflicted with that disease.  I can say that I am hero free.  My focus now is to help my team develop our “Why” so that we can inspire each other.  Cheers…

Out Of Touch – Just Rambling

How often do you call your friends or family?  What do you talk about?  I don’t  call my friends very frequently, which I think makes me a bad friend or at least my wife thinks so.

She constantly tells me that I need to call so and so, but do I have to?  I know that the answer is yes, but why don’t I?

I suffer from significant anxiety and talking on the phone is difficult. and the moments that leads up to the call spirals out of control for me.  I know that once we get talking everything will be fine, but it is that moment just before I pick up the phone and dial that creates the internal pressure.

I get the same way when people call me.  I don’t like the awkward pauses or how to end the call.  During the call my brain  tells me that I need to think about the next thing to talk about, or am I talking to much about me.What do I do to break the uncomfortable silence we run out of things to chat about.

I think that this is why I am OK to recede into the dark corners of text messaging.  There are no repetitious sequences of “OK Bye”, “OK Bye, Talk to you Later”, “OK Later” that must be stopped by one of the parties.

One of the things I think hurts me is that I compartmentalize the last time I spoke with a friend or family. I hold on to that memory and we stay connected in my mind.  When we finally connect, it is like I we had interacted more recently than they know.

I have two friends that are very dear to me, Chris and Sara.  I haven’t talked to Sara in probably five years, but for me it was just yesterday that we were on the phone talking about my wife being pregnant with my son.  Chris on the other hand calls me about once a year and I think we are OK with it.

I regret that I don’t keep in touch the way I think a normal person would, but it is not that simple for me.  I am the same way with my brother and sister, which is sad!  The only person that I talk to on a semi-frequent basis is my dad.  I think it is easy to talk to him because he loves me the way that I am (hope so, he has been there for 38 years :)).

I do think that I am not the best friend or brother that I can be, but it is time that I suck it up and get over it.  I will make some much overdue phone calls this week!  I hope that you do not take contact with the important people in your lives for granted.  They don’t live in your head.  Living in the memories is not healthy.  Human contact is needed.  Cheers…


A Thousand Words & Lessons Learned

I have read articles that have stated that to become a better writer – you need to write.  Its not just the act of writing, there are expectations to achieve the benefit.  A thousand words a day, every day is purported to make you a better writer.

I think that it is very possible that writing a thousand words a day will work, but as an average person (aka not a Kardashian) I run out of words pretty quick. I want for this blog to be a platform to share my knowledge as well as a medium to practice writing.


I have spent so much time reading personal development and communication to try to improve other rough edges that I have little time left to practice.  That is just an excuse.

I feel disappointed that there are days when I feel like I have done little more than prepare my next set of excuses.  I guess that is part of the ebb and flow of life.

I have been through trials and tribulations in the past, and each time I have come out better for it.  I am in a similar state lately, so I am working on ME to get ready for my next set of challenges.  I am fortunate to have been given the rope to fail (too heavy a word for the situation) which had the unexpected side affect of giving me a front row view to a valuable lesson.

I have created several opportunities in the last 24 months for myself to reflect and grow.

~Lessons Learned~

Let’s start with Focus.  I know that management is where I am most happy, but I made some mistakes when I had a team of ten.  I used my autonomy (not really, but hang on) to work on personal projects.  That sounds really unethical, but I mean projects that I wanted to work on that would ultimately serve the team.

At the time, I rationalized in many of ways that I was doing what was needed and I was serving the team but that wasn’t true.  I was taking something very important from the team, accountability.  It wasn’t until recently that this lesson became visible.  I was listening to the book “Notes To A Software Team Leader” by Roy Osherove that I saw that I had taken from the team that which I needed from them the most.

I used to feel proud of myself because of the value I created, since I often needed to step in and help with the firefight.  At the same time, I was thinking that they should be able to solve these issues on their own so why can’t they.  I realize now that I was the reason that they could not fix it.

They didn’t need to because “Heath would always be there to fix it”.  What a mistake that was!  I work with an amazing team of engineers and the fact is that they are the best ones positioned to solve these problem.  I wasn’t the one that was writing the code, so why did I think I was the best one to fix it?

That was a major aha moment for me. When I read that passage in the book it gave me a substantial pause of reflection.  I was very disappointed knowing that I took that away from them.

That was a very big one for me, but there is another that I think was the most disappointing.  I learned that mentorship will not come to you – you must actively seek it out.  That is something that I expected to take place.  Someone to help guide me in directions that I would not have otherwise seen and provide affirmation in the ones that I did.

I saw a youtube from John Somnez where someone had asked about how to find a mentor and that is where the lightening bulb moment happened.  Mentors are not always seeking apprentices.  Another thing that I came to realize is that you may not realize that you are being mentored.

I learn something everyday from at least one team member everyday.  I consider this to be a form of mentoring by committee.  I have also received personal mentoring from people that have reported to me.

What I am eluding to is that I no longer believe that mentoring happens in our field like a journeyman or apprentice.  You need to take the lessons as they come and try to learn from them.  You have to ask for clarity when things are unclear or get counseled when you need guidance.

At the start of this post I mentioned how you need to write a thousand words every day in order to improve your writing skills, but at this point I only have 796…

Personal Brand

How do you market yourself?  What are the characteristics that you will promote to compel someone to take a chance on you?  These are questions that I have been asking myself lately.  The one thing that I have not done well is frame the answers relative to what I want to achieve.

What do I want?

This is a question that I have been asked quite a few times lately.  My vision is to be in a position to help guide an organization to deliver the highest value possible.  I want to be able to share visions of customer satisfaction delivered through technology.

Simon Sinek’s Ted talk “How Greater Leaders Inspire Action” is a video that I have watched several times.  He speaks of how the importance is greater in the Why you deliver awesome when compared to what that awesome is.

NOTE:  If you like his Ted talk, you will love his book!

In business, many companies have mission or vision statements that speak of building the best product or dominating a particular market but those are goals.  That doesn’t mention why you want to meet that goal.  Simon talks about common “why”s   such as “to make  money.”  He states that these are by products of doing business as usual.  The more interesting questions is why do you build that product or sell that product in the first place.

I am trying to apply that same rationale to my career.  Why do I want to be a leader of an organization?  I have questioned my leadership abilities lately and in reflection I realize that I was not focused on leading.  My focus was on other things, things that are more reminiscent of an individual contributor.  I can look in the rear view and see what mistakes I made and how I will correct them in the future.  But that still doesn’t help me answer the question!

I think the question that I am missing is what position will give me the ability to lead and guide a team.  Recently a distinction has been shown to me with the comparison to an Architect path and that of a Director/VP.  It was sold to me that the Director route is a tactical execution path, where the Architect is more strategic.  This has been debunked!

Even though that notion imploded, it did highlight exactly what it was that I was after.  I want to be a strategic leader focusing on how technology can best serve the business.  So now knowing what I want and why I want it, how can I market myself to achieve that goal?

This is a tricky question for a technical leadership position.  On the one hand, you need to show that you have the leadership traits and on the other hand you need to have the technical chops.  I have seen leaders come and go and the one killer that happened to all was the lack of technical respect from the team.  That is a hard sell, but a critical one.

So again, what would be the compelling traits to move up the ladder?  I don’t have an answer for this question.  I don’t know which ones are more important that the others.  Technical chops, emotional IQ, empathy, all these are important but does the order matter?  I have been doing a lot of reading (listening :)) lately to different respected leadership books and there are some common threads that I am noticing, so I hope the answer will come eventually.

For now, I am just plugging away.  Writing everyday to improve my written communication skills.  I am also spending more time on reading technical articles and other research to help hone my technical skills.  I WILL reach my goal, it is just a matter of when!





Kafka, Sadly Its Time To Part Ways

I had big dreams for the perfect union between my company and Kafka.  I could see jagigabytes (technical term for a huge number) upon jagigabytes of data passing through our network from the massive clickstream data that we would produce.  The power of having our data in-house and not relying on the paid services to store and cull our data was huge.

That was the dream; and now for the reality :(.  We have tried to bend the will of Kafka to meet our use case but Kafka didn’t break.  I wanted badly for the pub/sub application to be able to work at our small scale.  When I say our scale, I mean somewhere south of 1000 messages per day for business transactions purposes.

My thinking was that if we could get it to work at our scale, then we would have learned a great deal to help us with my grander vision.  I can say that I achieved the goal of learning, but not much more.

The first issue that we had was the messages were not returned from the queue during a single fetch request.  I saw that during development, but I didn’t pay enough attention to what I was seeing.  That turned out to be a fatal flaw.

We were losing messages

When we configured our jobs to read from various topics, we configured them to poll at specific intervals.   When we spaced them out to an hour or greater, we were closing the window between the retention policy and the opportunities to read data.  For example, if we have a retention policy of 16 hours and a poll interval of one hour, then we have 16 chances to read data.  If during those 16 individual read attempts, data was not returned it was lost.

What happened is that we were missing critical event data and we couldn’t figure out why.  It took some time before I figured out that you have to ask for the data until it is returned.  That was issue number one.65831061

We were losing messages

Now that we were able to get the data back, all of a sudden all the data was gone.  This was really baffling!  I thought we had solved our problems with receiving the data, but to the outside it looked as if we were having the same issue again.  I couldn’t figure out why after 16 hours our queue was empty regardless of how recent the last message was published.

I did all the reading that someone should have to do in a lifetime (except for you, please continue reading) and I couldn’t solve it.  So I turned to the Kafka mailing list for help.  It turns out that Kafka will delete the log file with the message that is outside of the retention policy.  This was exactly what we were seeing.

We could send a steady stream of data and like clockwork, all of it would be gone once the flush began.  It turns out that the initial log file is a gigabyte in size.  Remember, my volumes are very low and we wouldn’t fill that up in a year.  That could be solved by setting the log file size really low, we set it to 1024B.

We were losing messages

That brings us to our third and last issue.  The straw that broke the camel’s back.  Nail in the coffin.  Ok, I will stop.  Now we are receiving data reliably and our logs files are right sized, what else could be going on?

With their rest client, there are two methods of committing back an offset when operating in a group.  You can auto-commit where you set your cursor to the last entry that was returned or you can wait and commit that cursor position once you are done with the data.  To be fair, we had some issues in our code that was causing the processing to halt and stop processing messages.  These were messages that were already committed, but were not processed.

Without the ability to grab a single message at a time we were stuck.  We had hoped that Confluent 3.0 (Kafka 0.10) was going to save the day with the max.poll.records, but they didn’t roll that into the rest client.  Disappointed, we realized that we had really hit a wall.

We sucked it up and decided to turn our backs to Kafka for now.  We were diligent to create abstractions that will allow us to change with reasonable ease.  We will be taking a day to research and design what the new solution will be.  I think that this was a good lesson on picking a solution that matches the current use case.  Even though I really wanted to set us up to use Kafka for my grander vision, it just wasn’t the right choice.

I haven’t turned my back on Kafka completely, I still think it is awesome and will have a home with us in the future.  Sadly, for now I can’t fit your size so I will have to leave you on the rack.  Goodbye.



Efficacy Of The Daily Scrum

For the last 8 years, I have seen many interpretations of how a scrum team is to operate.   All have been modified in some way, some good and some bad.  One component that I have not seen changed is the format of the daily scrum. What I would like to talk about is the efficacy of the daily scrum as it is defined in the scrum.org handbook.

All the way back to 2008, the daily stand-up has always answering the three following questions:

  1. What did I do yesterday that helped the Development Team meet the Sprint Goal?
  2. What will I do today to help the Development Team meet the Sprint Goal?
  3. Do I see any impediment that prevents me or the Development Team from meeting the Sprint Goal?

Each morning, we would go around the room (or voice chat) and talk about these three questions.  I noticed that there was very low engagement and most people were just reporting status to the scrum master.  I have even seen a fourth question asked, “What have I learned since yesterday” and it woke people up.  I think that it is a great question since it can help the team and I always liked to share anyway.

Recently I was on a particular scrum team working on a project that had a lot of dependencies.  As each person spoke up, I started to realize that I was not getting any value out of what they did yesterday.  This began to needle at me, since this was a ceremony that was required to happen daily.

I started to only answer the second question, “What will I do today to help the Development Team meet the Sprint Goal?”.  I felt that this was really what the team cared about.  What is interesting about the work that I am about to do that will help to drive us forward.

It didn’t take too long before it caught on and now the meetings are shorter.  Not only are the meetings shorter, but we are acting more collaboratively.  I don’t think that all of the credit for the increase collaboration can be attributed to the modified daily scrum, because the team was already pretty kick-ass to begin with!

I know that scrum is a guide and you have to inspect / adapt to make it work for you, but the daily stand-up has, in my experience, remained unchanged over the last 8 years.  If I were asked about the efficacy of the traditional daily stand-up, I would have to say it is low.

Changing the format of this meeting has helped our team be more successful.  One of the next posts that I am going to write is inspired by Allen Holub.  I want to write about the efficacy of scrum compared to a more bare bones agile process and kanban.  Until then, cheers.