The Crucible of Learning: Transforming Organizational Effectiveness Through DevOps

When organizations engage with us for DevOps consulting, it’s rarely by chance. Typically, they’ve identified a problem, a problem that’s grown too big to ignore, a problem that requires expertise beyond their current capabilities. As a DevOps consultant, I’ve seen this scenario play out many times. One of the most significant engagements I’ve experienced involved a large organization in the oil and gas industry. This case illustrates the complexity and challenges organizations face when they try to implement DevOps, and the transformational impact that a well-executed DevOps strategy can have.

The Root Cause: Technical Debt in a Large-Scale System

Identifying the Problem

The company I worked with had a product that sold for $50,000 per license, a desktop product with significant complexity. They had 90 teams spread across 13 locations in nine different countries, all working on this product. Each team had its own long-term branch, leading to divergent development paths that could last as long as a year. The organization attempted to mitigate these divergences by applying force, a common but flawed approach.

They spun up a dedicated DevOps team, centralizing DevOps engineers in one area. However, this team faced a monumental challenge: they couldn’t directly influence what the 90 teams were doing but were responsible for ensuring continuous delivery and daily builds of the product.

The Complexity of the System

To achieve their goal, the DevOps team built an infrastructure that facilitated about 11,000 build executions per day, 1.2 million a year, just to keep the product functioning daily. The system had to manage code from 90 different teams using different source control systems, including Git, Team Foundation Version Control, Subversion, and even custom in-house systems. Each platform had its own branching and merging capabilities, which the DevOps system had to integrate and unify.

The complexity of the product was staggering. Developers needed workstations with 128GB of RAM and 24-core processors just to build the product. It was an obscene setup, but necessary due to the product’s massive and complicated architecture.

The Lesson Learned: The Impact of Technical Debt

Assessing the Situation

We conducted a DevOps assessment for this organization, identifying key areas of concern:

• Diverse Source Control Systems: Teams were using different systems, creating unnecessary complexity.

• Distributed Teams: With 90 teams in 13 locations, coordination was a nightmare.

• Multiple Funding Routes: The product had 13 different funding sources, limiting control over the entire system.

The product had been in development for 25 years, during which the company acquired competitors and integrated their technologies, often without addressing technical debt. Instead of migrating new acquisitions to their systems, they kept legacy systems intact, adding layers of complexity.

The Road to Simplification

Over four years, we worked to align the organization’s development practices. We consolidated everything into a single source control system, which allowed us to perform a unified build that produced one version of the product. We also reduced the number of branches and moved towards mainline development.

This process wasn’t easy. It required patience, collaboration across different locations, and a deep understanding of both the product and the organizational structure. The teams involved were not within the same reporting structure, making it impossible to simply dictate changes. Instead, we had to influence, persuade, and gradually bring everyone on board.

The Value of Refactoring

A significant part of this transformation was helping the organization understand the importance of refactoring. Over time, they had accumulated technical debt by integrating new systems without proper refactoring. This debt had to be paid back to simplify the product and make future development more manageable.

The organization had been integrating rather than refactoring, patching together systems rather than taking the time to rebuild them properly. This approach made the system increasingly unwieldy, obscuring the forest for the trees. The key lesson here is the importance of regular refactoring and simplifying as you go.

The Outcome: Simplification, Efficiency, and Happiness

What Organizations Can Expect

The outcomes of a successful DevOps transformation are profound:

• Reduced Cost and Time: Streamlining the development process reduces both the cost and time required to deliver new features.

• Improved Team Happiness: Developers spend less time grappling with unnecessary complexity and more time solving meaningful business problems.

• Increased Capability: With a focus on refactoring and clean code, teams become more effective and can deliver more value over time.

A great example of this is the Azure DevOps team at Microsoft. Before they embraced DevOps, they delivered about 25 new features to production each year, with 600 people working on the product. After investing in reducing build times, from 72 hours to just 3.5 minutes, they scaled up to nearly 300 features per year with the same number of people.

The Importance of Skill and Ownership

One critical point to understand is that DevOps is not something you can simply “install” in an organization. It requires skill, dedication, and a willingness to embrace change. If a consultancy promises to do all the work for you, your people won’t learn anything. They won’t go through the necessary pain of cleaning up their technical debt and will likely repeat the same mistakes.

Think of it like cleaning your teenager’s room. If you do it for them, they won’t learn the importance of keeping their space tidy. But if they do it themselves, especially if they have to clean up a particularly nasty mess, they’re more likely to avoid making the same mess in the future. The same principle applies to DevOps and engineering practices.

A Final Thought

Bringing DevOps into an organization is about upskilling and taking ownership of the problems you’ve created over time. It’s about dealing with your own “crap” so that you create less of it in the future. This approach enables teams to be slicker, more effective, and ultimately, more successful in delivering value to the business.

Conclusion

The journey of DevOps transformation is not easy, but it is immensely rewarding. By focusing on simplification, refactoring, and ownership, organizations can achieve significant improvements in efficiency, cost, and team satisfaction. The story of the oil and gas company is a powerful reminder that while the road may be long and complex, the destination is well worth the effort. Remember, the key to successful DevOps is not just in the tools or the processes but in the mindset and skills of the people involved. Embrace the challenge, learn from the experience, and watch as your organization transforms into a lean, agile, and highly effective machine. 🚀