DevOps, an emerging practice that merges the disciplines and best practices of IT operations and application development, is designed to balance growth-oriented responsiveness with risk management and service-level predictability. In the DevOps world, nothing gets tossed over the wall. If you built it, you own it.
Contents |
Virtualization Enables IT Operations Responsiveness
IT organizations have come a long way from the era in which large teams needed months to requisition and provision new servers for new applications. Virtualization of servers, networks, and storage, together with systems automation and management tools, now make it possible for a smaller team to provision and decommission virtual servers in minutes.
Agile Enables Application Development Responsiveness
Application development organizations have gone through a similar transformation. Yesterday’s organization might have delivered annual or even biennial software releases, requiring a large staff and months of testing. Once tested, the application was installed, and the development team was done, as they moved on to the next 12-24 month development project.
Today’s development organizations use an agile and lean development approach, with a small, focused team delivering a new software release every couple of weeks. Developers can now quickly write a little code on a work station or cloud service, then send it to IT operations who can spin up a new virtual server and drop it into production. These new infrastructure and application-development teams are more exciting, more flexible, and more responsive to the business units. And they have learned to iterate like crazy. Both applications and infrastructure are on a near-continuous release cycle.
The Limitations of Walled Organizations
When walls remain between development and IT operations, however, this flexibility and responsiveness comes at a cost. Many of the applications fail to meet performance, scale, availability, data protection, and data security requirements. And too often the teams devolve into finger pointing when problems arise.
A Capital Management DevOps Success Story
On March 6, 2012, J. Wolfgang Goerlich joined the Wikibon community to describe how he lead his organization, a Midwestern capital management company, through a DevOps transformation. DevOps has been criticized by some as "too much 'dev' and not enough 'ops'," which makes the story at this company particularly interesting.
Preparing Operations for DevOps
Originally brought in to manage digital security, Goerlich moved to managing infrastructure, where he developed a reputation for efficiency. Under his direction, the company aggressively embraced server virtualization, beginning, as most organizations do, with servers for application development and test. This enabled the company to reduce the server count by almost 70% and to be more responsive for new server requests.
In addition, the company adopted best practices for service management, such as those proscribed in ITIL and ITIL alternatives. Finally, the company standardized on a common set of management and monitoring tools across the infrastructure, believing that the staff-efficiency benefits gained from a common approach and look-and-feel far outweighed any incremental benefits that might have been achieved from best-of-breed, but stove-piped, solutions. Specifically, the virtualization and management tools that this company uses include:
Inspired by Wolfgang's reputation for always being on time with new infrastructure deployments, combined with the efficiency gains achieved in IT operations, senior management gave him the added responsibility for application development and support.
Core Metrics for Development
At the time Wolfgang assumed responsibility for application development, the company had 317 custom and 57 shrink-wrap applications, which his newly-inherited team supported. His employer was looking for substantial improvements in both efficiency and responsiveness. One of the first actions he took was to develop an annual customer satisfaction survey and a focused set of key metrics designed to continuously assess progress. These metrics included:
- Implemented changes,
- Backed-out changes,
- Help desk tickets related to changes,
- Help desk tickets not related to changes,
- Time to repair.
Adopting Standard Platforms for Application Development
Much like his approach to infrastructure, Wolfgang began driving standards for application developers. In doing so, he began eliminating best-of-breed selections in favor of development environments that enabled skill transfer and skill sharing. His company is increasingly a Microsoft shop, leveraging the integration between system management and application development that Microsoft provides. Through standardization, Wolfgang has reduced the number of supported applications to 272 and is looking to consolidate further.
Beyond Standards: You Built It, You Own It
The walls between IT operations and application development have now been largely removed. Developers sit with IT operations and together address any application quality, availability, or performance issues that arise in production. Service-level and quality problems are not a development problem or an operations problem, but a DevOps problem. Since everyone has time on the help desk, developers now have a keener understanding of operations issues and best practices.
Similarly, IT operations professionals are gaining experience in application development. When working side-by-side with operations, developers can now more elegantly incorporate application installation, maintenance, monitoring and management best practices into their code.
Achieving Hyper-Productivity: Keep Them Interested and Educated
Even with the consolidation and automation gains achieved in both IT operations and application development, the task of maintaining 272 applications on 43 servers with a total team of 10 across development, operations, and support, is enormous. This level of hyper productivity comes from developing superstars.
To maintain a highly-motivated, high-performance team, Wolfgang has focused the team exclusively on applications that provide significant business value. Rather than using third-parties to develop applications in new and emerging areas, where outside skills might be more readily available, he outsources less interesting and lower value applications, and saves the “interesting” work for his internal team.
To address the potential internal skills gap, Wolfgang allocates 20% of staff time to ongoing training, where developers learn about operations, operations professionals learn about development, and both become highly-schooled and highly-skilled in the latest tools, methods, and applications. Some organizations fear that this level of investment in education leads to higher staff turnover, as employees might leverage newly-acquired skills for higher-paying jobs. Wolfgang's experience has been quite different, with employees who a decade ago were working in the then-hot area of Web development now gaining opportunities to work on applications and infrastructure that are today’s hottest trends.
With skill hopping and a continuous investment in education, employees know that they won’t get stuck supporting decade-old applications destined to be outsourced. This has resulted in reduced staff turnover and retention of superstars. Wolfgang describes his superstars as artisans who know infrastructure, know security, and take end-to-end responsibility for their applications.
According to Wolfgang, "Organizations are already paying for training. They just are not formally recognizing it as such and tracking it as such. Most project plans have padding to allow for the folks to figure out what they are doing. Most organizations have had extended outages while the team struggled to learn on the fly." For Wolfgang, "It is about making training explicit and measurable. Done right, this avoids the outage or the project delays that occur when people are 'training' in a customer-facing scenario."
Action Item: The migration to DevOps has been driven, in part, by application development teams embracing Agile software development methods and by IT operations organizations embracing virtualization and automation. Both have enabled an increased rate of change that can disrupt traditional IT organizations that are operating in the conventional silos of application development and IT operations. These organizations should evaluate a DevOps model and the potential significant benefits in efficiency, quality, and agility. Measurement of the current rate of change and the current quality of code released into production provide a good baseline to both identify the need for and document the benefit of a DevOps approach. If the current culture is one of assigning blame, it will need to move rapidly to one of accepting end-to-end responsibility. In order to fully embrace DevOps, walls must be broken down while budgets and reporting structures must be consolidated. Maximum benefit will come to those organizations and individuals who shift from a mentality of a fixed job role to a dynamic progression of skills. This will require an explicit investment in education, rather than ad hoc, on the fly, emergency training.