Rethinking Continuous Delivery: Why Best Practices Don't Exist in Complex Environments

TL;DR

There are no universal best practices for continuous delivery in complex environments; instead, teams should adopt flexible, situation-specific approaches. Key strategies include audience-based delivery for rapid feedback, testing in production to validate real-world performance, and a commitment to quickly finding and fixing issues. Development managers should focus on continuous improvement and adaptability rather than rigid processes.

23 January 2025
Written by Martin Hinshelwood
3 minute read
Subscribe

I often find myself in discussions about the best practices for enabling continuous delivery within teams. It’s a question that comes up frequently, and I want to address it head-on: there are no best practices in complex environments. Best practices are a concept that applies to simple tasks in straightforward situations where a procedure can be followed consistently to achieve the same results. However, the world we operate in is anything but simple.

Instead, I prefer to say that there are only adequate practices tailored to the specific situation at hand, and these practices can, and often do, change. This is a fundamental truth we must embrace. While we may not have a one-size-fits-all solution, there are several practices that many organisations have successfully leveraged to enhance their continuous delivery efforts. Let’s explore some of these practices and how they can support cross-functional collaboration without compromising quality.

Audience-Based Delivery

One of the most powerful practices I advocate for is the implementation of an audience-based delivery strategy. Traditionally, the delivery pipeline followed a linear path: development, testing, staging, and finally production. This model, while familiar, is fraught with inefficiencies and costs. It often leads to a scenario where quality is tested in rather than built in, which is the most expensive way to ensure quality.

By shifting to an audience-based delivery model, we can deploy small changes quickly to a limited set of users. This allows us to validate our product in real-world scenarios, which is invaluable. The Windows team at Microsoft exemplifies this approach. They deploy updates to internal users nightly, allowing developers to test their code in production almost immediately. This rapid feedback loop is crucial for continuous improvement.

Testing in Production

The concept of testing in production is another critical aspect of this audience-based model. There is no perfect simulation of a production environment; the only way to truly validate our work is to deploy it. By allowing a small group of users to access new features, we can monitor how the product performs and make adjustments as necessary. This practice not only enhances our understanding of user interactions but also helps us identify and rectify issues before a wider rollout.

The Philosophy of Find It and Fix It

When we talk about continuous delivery, we must also adopt a philosophy of “find it and fix it.” If something slips through our automated checks and makes it into production, we need to investigate how that happened and adjust our processes accordingly. This isn’t just about fixing bugs; it’s about evolving our architecture and practices to prevent similar issues in the future.

For instance, the Azure DevOps team faced challenges when a non-essential service caused significant disruptions. They learned that it’s better to allow the system to continue functioning, even if it means displaying a less-than-ideal user experience, rather than taking everything offline. This approach not only maintains user productivity but also provides valuable insights into system resilience.

Embracing Change

Ultimately, the key takeaway here is that we must continuously seek to improve our products and our processes. This relentless pursuit of betterment is not merely a best practice; it’s a philosophy that should underpin everything we do. By embracing change and being willing to adapt, we can create a more robust and responsive delivery pipeline that meets the needs of our users.

In conclusion, while the term “best practices” may be misleading in the context of complex environments, there are certainly effective practices and philosophies we can adopt. By focusing on audience-based delivery, testing in production, and a commitment to continuous improvement, we can enhance our ability to deliver quality software efficiently. Let’s keep the conversation going and share our experiences as we navigate this ever-evolving landscape together.

Subscribe

What to read next

Video Product Development Engineering Excellence

Embrace Simplicity: How to Transform Complexity into Continuous Delivery Success

Explains how simplifying complex software and committing to change enables continuous delivery, highlighting the need for cultural shift, …

Watch video
Video DevOps Product Development

The Common Challenges of Adopting DevOps Practices

Explores key challenges in adopting DevOps, including managing evolving risks and closing feedback loops, with strategies for rapid …

Watch video
Video DevOps Engineering Excellence

Transforming Agility: How Azure DevOps Went from Two-Year Releases to 880,000 Deployments

Explores how Azure DevOps shifted from slow, two-year releases to rapid, continuous delivery, highlighting the benefits of fast feedback, …

Watch video
Video Engineering Excellence

Mastering Agility: Balancing Engineering Excellence and Effective Processes in a Rapidly Changing Business Landscape

Explores how to balance engineering excellence and effective Agile processes, highlighting the need for technical skills, continuous …

Watch video
Video Engineering Excellence DevOps

Unlocking Rapid Delivery: Overcoming Common DevOps Challenges for Agile Success

Explores common DevOps challenges to rapid delivery, highlighting automation, feedback loops, mindset shifts, and compliance for achieving …

Watch video
Article Engineering Excellence DevOps

Stop Hiding Behind Complexity and Start Delivering Continuously

Continuous delivery is achievable for any software, regardless of complexity. Success depends on investment in automation, quality, and …

Read article