Transforming Chaos into Clarity: How Optimising Systems Fuels Engineering Excellence

Chaos costs money. In my experience working with various organisations, I’ve seen firsthand how they struggle to manage the chaos that often surrounds the delivery of usable, working products to their customers. The result? Substandard work, a barrage of bugs hitting production, and a frustrating cycle of rework because things simply don’t meet the minimum standards we expect. It’s a familiar tale, and one that I’ve encountered time and again.

Understanding the Shift

When I’m building products, I often find myself in a position where I need to solve a problem. You identify a challenge, envision a solution, and set off in that direction. However, as time passes, the landscape can shift dramatically. This shift might be due to emerging market opportunities or the need to scale operations.

The solution that once seemed perfect can quickly become inadequate. I’ve witnessed this in various sectors, particularly in industries like airlines and car rentals, where technical debt accumulates over time. Choices made years ago can lead to a tangled web of outdated systems that no longer serve the organisation’s needs.

The Dangers of Technical Debt

Take, for instance, a customer I worked with who built their own source control system three decades ago. At that time, the options available were limited, and they had no choice but to create a bespoke solution. Fast forward to today, and it’s clear that this approach no longer makes sense. With robust source control systems like Git readily available, the need to cling to outdated technology becomes a liability.

Investing time and money to transition from legacy systems to modern solutions is crucial. This is where technical leadership and engineering excellence come into play. We must focus on ensuring that our systems and processes are as effective as possible, making our teams’ jobs smoother and more efficient.

The Value of Optimisation

Satya Nadella at Microsoft exemplifies this approach. He prioritises optimising systems so that the best engineers can focus on delivering features rather than wrestling with outdated processes. This investment in system optimisation is not merely a cost; it’s a value centre that supports our ability to innovate and deliver new features.

A prime example of this is the Azure DevOps team. Back in 2012, they were delivering around 25 features to production each year with a workforce of 650 people. Through a commitment to technical excellence and leadership, they transformed their output to over 600 features annually. This remarkable increase was achieved by addressing both technical debt and the accumulation of what I like to call “technical cruft”, the sediment that builds up in systems over time.

The Impact of Testing

One of the pivotal changes they made was shifting from long-running system tests to unit tests. This transition took four years of dedicated effort, but the results were staggering. They reduced the time it took to verify changes from 48 hours to just 3.5 minutes. Imagine the productivity boost if you could instantly know whether a change was successful rather than waiting two days.

This rapid feedback loop allows teams to make smaller, more frequent changes, leading to a better product that is scalable and less reliant on quick fixes. Instead of patching over problems with Band-Aids, teams can focus on building robust solutions that stand the test of time.

Conclusion: The Cost of Chaos

Ultimately, the cost of chaos stems from poor technical leadership and a culture of engineering mediocrity. By empowering your best engineers to focus on optimising systems, you enable every team member to add value more easily. This shift not only enhances productivity but also fosters a culture of excellence that can propel your organisation forward.

In my journey, I’ve learned that addressing chaos is not just about managing the present; it’s about preparing for the future. By investing in the right systems and processes today, we can ensure that we’re not just surviving the chaos but thriving in it.