Detective Optimizers

Supporting Applications

Having worked in a centralised application support team I have had the opportunity to study the personality traits that make a good fit for support and which don’t. When I say support I am talking about supporting the operation of an application in production, whether it is in a centralised team or in a dev ops team that supports what it builds.

I believe that the concept of a centralised support team is flawed for multiple reasons but that is another topic. However if you’re in a company that has centralised support teams then one of the disadvantages you may see is high turnover of team members in these teams. I have the strong opinion that you can’t maintain a highly motivated team unless they have interesting and challenging work. It has to be more than just recycling pools, moving queue messages off a dying server and one liner code changes. When you have tens or hundreds of tasks a day and you are having your productivity measured then you will get burned out, bored or just hunger for more.

That is why I advocate that centralised support teams should also be involved in performance optimization projects, code refactoring projects for buggy areas of applications, improving application robustness where weaknesses have caused incidents in the past and so on. Turn that team that sees the uglier side of the code base, day in and day out, into a centre of excellence for performance, refactoring and automated testing. Then you might get people asking to join the team.

If you’re a dev ops team then you should be doing all of this already.

Having said that, not all programmers are naturally suited to this work.

  • Most developers prefer green-field projects. They want to work on new functionality and not be constrained to work within the bounds of an existing application, with an existing architecture and possibly older technologies. Some developers don’t have confidence reading other people´s code and probably unconsciously avoid work that is heavy in reading code.

  • Most developers don’t think bug fixing is FUN. It is a necessary part of the job but they don’t get excited when they discover a bug. There are some developers who actively don’t like bug fixing. This is not a great attitude but sadly more common than it should be. This developers are the ones that get burned out quicker and often end up moving to a different career.

Much of this is about the personality of the programmer and where they are in their career and their life. These attitudes change over time and are not set in stone. However as a support team leader it is critical to identify the developers not suited to support, hopefully before you hire them.

Personality Traits

So what personality makes a great support/technical improvement developer? Personality-wise there are some key traits that I see in the most successful developers in my support team.

We are Detectives

“Code is a crime scene”. When things go wrong we love collating the evidence, reading code – other people’s code, testing hypotheses and talking to the development teams. Our code reading skills are excellent and we can almost sense our way to the problem just through reading the code. We find it very satisfying to diagnose a rare multi-threading bug by following the thread of execution in our minds through the source files.

We are OptimiZers

We love finding inefficient areas of an application and carrying out performance projects. Turning unstructured and messy code into clean elegant code.

We like Meta and Tools Projects

Log analysis, real-time monitoring, intelligent alerts, mapping application relationships and dependencies and tools to help automate common support tasks.

We like resolving hard to fix problems

When errors confound us we like to get stuck in, study, collaborate until we find the cause. The harder the bug the more satisfying it is when we resolve it.

I call these people Detective Optimizers.

Cultivating vs Hiring Detective Optimizers

Detective Optimizers are much less common than your greenfield developer which presents a problem when it comes to hiring. So investment in cultivating the detective optimizer spirit is essential in order to maintain the right people at the right level of technical craftsmanship.

Sometimes people need to be reminded to have fun, work can be stressful at times and we can get bogged down by the work, but the right word with the right tone can break someone out of it. The first thing you need to knock out of your developers is that bugs are bad. What needs to be communicated is that sure, BUGS ARE BAD, but they are also opportunities for us to practice our problem solving skills. Problem solving and investigation skills are highly valuable and having the opportunity to develop them is a POSITIVE thing. The key is to show how much you yourself relish the opportunity to investigate the issue. I try to make my positivity be infectious. When people see how much I relish investigating an error it can help them change the way they think about bug fixing. Teach them how to think like a detective and they will be better at it and enjoy it more. Investigation is a process and it can be learned.

While cultivating this spirit it is easy to fall into the trap of mocking the people responsible for the bad code. Some of the code that you see makes you wonder what the developer was thinking. You see some really crazy things and it is easy to say “What an idiot” but the reality is that everyone does dumb things. Show me a human who doesn´t screw up! However, seeing buggy code day in and day out can create this air of mocking where support developers speak more and more negatively about the other programmers. This is dangerous and if left unchecked it can ruin your team. It enhances the “us” and “them” mentality which needs to be avoided at all costs. Partnership, collaboration and respect are the base of any teamwork. So seriously, don´t engage in it and don´t let your team mates either. The next screw-up might be your teams then your self-righteous attitude will come and bite you in the ass.

Hiring Detective Optimizers

How easy it is to find these guys and girls really depends on your hiring techniques. One good way is to give an interviewee a programming test with a series of small challenges. They must complete 2 of the 5 tasks. Make one for Optimizers, one for Detectives and the other three mini development tasks. See which ones the interviewee solves for you. If they avoid the two targeted ones like the plague then that is probably a good indication.

Optimizers love to optimize. If they see, out of the five tasks available is a performance improvement or a code refactoring task they will go for it.

Detectives love finding bugs. If they see, out of the five tasks available is a “diagnose and fix the bug” task they will go for it.

Nothing in interviews is 100% but it is better than just testing their technical knowledge and skills.

Support teams are special beasts and you need to get tactical to create the high performance team that you want. Look for Detective Optimizers, cultivate the spirit and don’t forget to have fun!