Agile Flows Like A Waterfall

“It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way – in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only.”

— “A Tale of Two Cities,” Charles Dickens

A quote from “A Tale of Two Cities” might seem like an odd choice for this post, but surprisingly it isn’t: it’s a tale of the struggles that software engineers face when developing commercial software in a corporate environment and the challenges that businesses have with trying to deliver software products within the constraints that businesses have.

For those not in the tech and software industry a brief history lesson is in order. Back in the day writing software was a simple affair. Engineers hacked away at their keyboards, adding bit by bit to their code (iterating) until they were done. The only plan was the idea in your mind. Things were finished whenever they were finished. Not going to lie, it was kind of awesome. As software became more mainstream and businesses realized that there was serious money in developing software commercially, they started hiring software engineers. As team sizes and the feature lists of their software products grew. Simple iteration was insufficient. Businesses needed a way to manage projects (we’ll go into why this matters later) and they turned to the one model that everyone knew: Waterfall.

In Waterfall, as the name implies, phases of a project cascade over time. One phase follows another phase. It was a highly successful methodology for project management in other industries such as construction and manufacturing so it stands to reason that people would think that this would work well for building software. After all, writing a piece of software is much simpler than building a concrete factory, right? Well…

It kinda worked. Software was released, but it took forever. New version took six, eight, twelve, or even eighteen-plus months. You see, the water methodology is incredibly rigid. You first gather all the requirements of what problem your customers have, and this is what you are going to solve. Next, you go through a design phase where you go through all of the requirements and create the blueprints of your software system. Next you begin to code. Once you’re done with that, you get this over to your QA team to validate. They go through it and inevitably discover bugs, because software has a lot of complexity that folks forget about. But what about if the market shifted and you needed to change requirements? Never fear! File a Change Order and determine what needs to change or be added, the impact to the current system, and then when you’re going to schedule building that change. Easy and fast, right? There was another fun side effect with this entire thing, engineering teams were miserable. It took the fun out of programming.

We then had a revolution. People started coming up with new methodologies to write software and took a firm stance on changing the way we write software. These brave pioneers gave us things like extreme programming, SCRUM, The Agile Manifesto, etc. Hallelujah! We were saved!

Kind of. Businesses understood they needed a better way to write software, and they also understood that work culture can be a powerful tool in attracting new talent. The end result? Everyone jumped on Agile. One of the most popular agile frameworks is SCRUM so whenever businesses adopted “Agile” they inevitably adopted SCRUM.

Here is where it gets bumpy.

Although SCRUM aims to be agile, it was abused by businesses. They adopted the portions that suited them. Ignored others. They added their own secret blend of herbs and spices. It was (and still is) kind of terribad. The Agile landscape was so fragmented that one of my standard questions when I was interviewing at a company was, “What flavor of Agile do you practice here?/What does Agile look like here?” People would laugh and explain it. I can say with confidence that although everyone claimed they did Agile everyone ran a different process. Additionally, ask any engineer their thoughts on Agile and I’m willing to be that 9/10 times you will trigger them Winter Soldier style. You see, businesses adopted Agile, but to them that meant having the label and also a heavily structured process:

• Daily scrums/standups
• Backlog refinement sessions
• Sprint planning
• PI/Quarterly planning
• Minor exorcisms
• Retrospectives
• Sprint Demos

Engineers complain that they had too many meetings (fair). They also complain that their work output is being tracked. That their work is monitored and reported on. I hear questions like, “Why do I have to give estimates of how long this is going to take?” “This will be done when it’s done.” “This is all for TPS reports anyway.”

So, this is going to be spicy.

Us Engineers… we don’t get it. We are operating in a bubble and don’t understand that we get a monthly paycheck because we work at a business. That business’ success hinges on its ability to remain competitive. What does that mean? Developing better products faster than their competitors. It means using marketing and having marketing campaigns. And what does this have to do with Agile and the above meetings and capturing metrics? It means that a business needs predictability and consistency in its outputs. If the company is going to be able to plan out a marketing campaign, they need to have a high level of confidence that a particular product or feature is going to be done by a certain date. Sorry to us, but our, “it will be done when it will be done” is not good enough, and honestly, it’s childish.

This doesn’t mean that work should be a slog, and that we should move slowly. There are ways to deliver software predictably. There are ways to measure numbers without turning things into 1984. How?

Shift your engineering culture to create product (and therefore customer) focused engineers. Teach your engineers what matters to a business and why, especially cost and return on investment. I once worked at a company where the cost of things was hidden in every way possible from engineers. In fact, I got in trouble for bringing up the topic and asking about costs in front of my team. Why? Because clearly that undermines engineers’ creativity. I can categorically say that when an engineer understands the engine that drives a business and how their work helps drive it and also what “value” means to a business that they are going to do amazing things.

Beyond that, I also recommend doing away with in-person standups and to figure out an asynchronous process for conveying that information. Standups tend to be usurped by the project managers and they turn the meeting into a grueling status update. Reserve time for the team to discuss blockers or other issues if there is a need for it around the time of the standup. Use Kanban! No, not just for production and incident support. Use it for product and feature development. Measure your lead times, cycle times, and throughput! For the love of all that is good and holy, apply Work-In-Progress (WIP) limits to all your columns that aren’t the backlog. Make sure your engineers are finishing their tasks and helping others finish their tasks before picking up new work! Celebrate accomplishments frequently. Demo what you’ve done. Have retros. Empower your teams to take ownership of items coming out from retros and effect change. Don’t say you practice kaizen; live and embody kaizen. It’s easy to fall into a rut and demoralize the team.

We can do better, folks. It’s possible. We have the technology!

1