Building an Agile Culture
Agile and Continuous Delivery — the Spotify Engineering Culture
The fall of the waterfall project management methodology
Many years ago I witnessed a massive “waterfall” IT project to replace an entire business application. The project was 3 years overdue and millions over budget, and when it was finally delivered, both internal and external customers rejected the solution. They had not been part of the engineering effort, maybe requirements had changed in the 5 years it took to build it, and more importantly, they had not seen the product at all until it was all done.
Numerous organizations shared the same experience and as a result, started adopting the agile methodology. The principle is simple, show progress incrementally as features are progressively coded and added to the solution over time in releases. The design is fluid and does not have to be defined completely at the beginning of the effort. And, more importantly, the stakeholders are able to use the software early and are able to provide feedback and influence the shaping and engineering of the solution before it is complete. Agile is a project management methodology, but it is more importantly a philosophy and a culture.
Agile also gave way to the DevOps culture, the age-old contention between developers, who want to push software to production before it is completely baked, and operations (Ops) teams who want to preserve a stable production environment with as few changes as possible. More than a release frequency issue, the goal of DevOps is also to end the enduring problem of “I built it, you run it”, which was causing Ops teams to be responsible for deploying and supporting software solutions they had little to no knowledge of. Not only does DevOps enable the Dev teams to build and deploy solutions quickly, but it also places the support and on-call responsibility into their hands as well, which in turn tends to improve the quality and stability of the software product. Continuous Integration and Continuous Deployment (CI/CD) pipelines have pushed agile to the extreme, each code commit can now in essence be automatically built, deployed, tested, and eventually promoted through the environments all the way to production. Many large IT companies are now accustomed to several hundred production deployments per day, a far cry from the old days.
Maintaining a “separation of duty” is also important for many organizations, a single disgruntled employee (or a clumsy junior developer) should never be allowed to write bad code and deploy it all the way to the production environment. As a result, in many cases, Dev teams are not able to deploy solutions to production, Site Reliability Engineering (SRE) teams serve as the “custodians of production” and also offer relief to the Dev teams by taking on on-call duty if proper troubleshooting documentation is provided.
Along with the agile and DevOps culture, the concept of microservices also gained more popularity, as described by the twelve-factor methodology. The idea isn’t new, Services Oriented Architecture (SOA) also promoted many of the same concepts, but JSON and the advancements of modern programming languages made its application much easier. Instead of building enormous monolithic applications, each distinct function can be broken into its own microservice, which can be reused by other applications. Think of it as functions made available via an API call. In order for this concept to work, microservices have to be decoupled and independent from each other. The concept of microservices, a distinct business function within the overall solution, also reinforced the notion of product ownership, as opposed to product management. DevOps teams are responsible for that overall business function from start to finish, they own that solution, and as such, are granted the freedom and independence to make choices such as the tools to use and sometimes programming languages and frameworks to use.
Agile and Continuous Delivery — the Spotify Engineering Culture
These videos are a bit dated now, but still beautifully executed and still very much relevant for many companies, both large and small.
Spotify’s organizational structure became popular after Henrik Kniberg published a video detailing the workings of the company’s internal structure, which is divided into squads, tribes, and guilds.
Lessons from Spotify: Building an Agile Business. The Spotify cultural norms, described in their engineering culture videos (Part 1 & Part 2), keep team alignment strong.
Agile Product Ownership in a Nutshell
Another great video also by Henrik Kniberg.
References:
