Storage Peer Incite: Notes from Wikibon’s March 6, 2012 Research Meeting
Recorded audio from the Peer Incite:
DevOps is a major reorganization of IT pioneered in the Cloud, where Web-based organizations in direct competition for consumer and business dollars need great agility in the services they provide and code that works right the first time. A social networking company that doesn't keep up with the competition or that puts up beta level code that crashes its site or just fails to work properly for customers is not likely to survive long.
But as with other revolutionary ideas pioneered in the cloud -- data center virtualization, big data -- DevOps is moving into the IT shows of traditional organizations. There DevOps is a radical reorganization that breaks down traditional silo walls between large development groups and equally large operations organizations and in many cases help desks. The advantage is that it makes the same individuals responsible for an application from initial development through implementation and support. IT eliminates the "it works fine on my laptop" attitude of developers who now must fix the problems with their code in the company's production environment and often must also deal directly with calls of exasperated end-users to the help desk.
On March 6 the Wikibon community participated in a discussion of this trend led by J Wolfgang Goerlich, who revolutionized the IT shop at the large, midwestern financial organization where he works using DevOps. In the process he: Reduced the number of physical servers from 140 to 43, Reduced the IT FTEs including full-time employees and consultants from 28 to 9. Wikibon and its business partner SiliconAngle believe that DevOps is an important trend for IT, to the extent that SiliconAngle has opened a new site, DevOps Angle, devoted to covering news and developments in DevOp. There Wikibon members can find an extensive report on the Peer Incite. Bert Latamore, Editor
Achieving Hyper Productivity Through DevOps
DevOps, an emerging practice that merges the disciplines and best practices of IT operations and application development, is designed to balance growth-oriented responsiveness with risk management and service-level predictability. In the DevOps world, nothing gets tossed over the wall. If you built it, you own it.
Virtualization Enables IT Operations Responsiveness
IT organizations have come a long way from the era in which large teams needed months to requisition and provision new servers for new applications. Virtualization of servers, networks, and storage, together with systems automation and management tools, now make it possible for a smaller team to provision and decommission virtual servers in minutes.
Agile Enables Application Development Responsiveness
Application development organizations have gone through a similar transformation. Yesterday’s organization might have delivered annual or even biennial software releases, requiring a large staff and months of testing. Once tested, the application was installed, and the development team was done, as they moved on to the next 12-24 month development project.
Today’s development organizations use an agile and lean development approach, with a small, focused team delivering a new software release every couple of weeks. Developers can now quickly write a little code on a work station or cloud service, then send it to IT operations who can spin up a new virtual server and drop it into production. These new infrastructure and application-development teams are more exciting, more flexible, and more responsive to the business units. And they have learned to iterate like crazy. Both applications and infrastructure are on a near-continuous release cycle.
The Limitations of Walled Organizations
When walls remain between development and IT operations, however, this flexibility and responsiveness comes at a cost. Many of the applications fail to meet performance, scale, availability, data protection, and data security requirements. And too often the teams devolve into finger pointing when problems arise.
A Capital Management DevOps Success Story
On March 6, 2012, J. Wolfgang Goerlich joined the Wikibon community to describe how he lead his organization, a Midwestern capital management company, through a DevOps transformation. DevOps has been criticized by some as "too much 'dev' and not enough 'ops'," which makes the story at this company particularly interesting.
Preparing Operations for DevOps
Originally brought in to manage digital security, Goerlich moved to managing infrastructure, where he developed a reputation for efficiency. Under his direction, the company aggressively embraced server virtualization, beginning, as most organizations do, with servers for application development and test. This enabled the company to reduce the server count by almost 70% and to be more responsive for new server requests.
In addition, the company adopted best practices for service management, such as those proscribed in ITIL and ITIL alternatives. Finally, the company standardized on a common set of management and monitoring tools across the infrastructure, believing that the staff-efficiency benefits gained from a common approach and look-and-feel far outweighed any incremental benefits that might have been achieved from best-of-breed, but stove-piped, solutions. Specifically, the virtualization and management tools that this company uses include:
Inspired by Wolfgang's reputation for always being on time with new infrastructure deployments, combined with the efficiency gains achieved in IT operations, senior management gave him the added responsibility for application development and support.
Core Metrics for Development
At the time Wolfgang assumed responsibility for application development, the company had 317 custom and 57 shrink-wrap applications, which his newly-inherited team supported. His employer was looking for substantial improvements in both efficiency and responsiveness. One of the first actions he took was to develop an annual customer satisfaction survey and a focused set of key metrics designed to continuously assess progress. These metrics included:
- Implemented changes,
- Backed-out changes,
- Help desk tickets related to changes,
- Help desk tickets not related to changes,
- Time to repair.
Adopting Standard Platforms for Application Development
Much like his approach to infrastructure, Wolfgang began driving standards for application developers. In doing so, he began eliminating best-of-breed selections in favor of development environments that enabled skill transfer and skill sharing. His company is increasingly a Microsoft shop, leveraging the integration between system management and application development that Microsoft provides. Through standardization, Wolfgang has reduced the number of supported applications to 272 and is looking to consolidate further.
Beyond Standards: You Built It, You Own It
The walls between IT operations and application development have now been largely removed. Developers sit with IT operations and together address any application quality, availability, or performance issues that arise in production. Service-level and quality problems are not a development problem or an operations problem, but a DevOps problem. Since everyone has time on the help desk, developers now have a keener understanding of operations issues and best practices.
Similarly, IT operations professionals are gaining experience in application development. When working side-by-side with operations, developers can now more elegantly incorporate application installation, maintenance, monitoring and management best practices into their code.
Achieving Hyper-Productivity: Keep Them Interested and Educated
Even with the consolidation and automation gains achieved in both IT operations and application development, the task of maintaining 272 applications on 43 servers with a total team of 10 across development, operations, and support, is enormous. This level of hyper productivity comes from developing superstars.
To maintain a highly-motivated, high-performance team, Wolfgang has focused the team exclusively on applications that provide significant business value. Rather than using third-parties to develop applications in new and emerging areas, where outside skills might be more readily available, he outsources less interesting and lower value applications, and saves the “interesting” work for his internal team.
To address the potential internal skills gap, Wolfgang allocates 20% of staff time to ongoing training, where developers learn about operations, operations professionals learn about development, and both become highly-schooled and highly-skilled in the latest tools, methods, and applications. Some organizations fear that this level of investment in education leads to higher staff turnover, as employees might leverage newly-acquired skills for higher-paying jobs. Wolfgang's experience has been quite different, with employees who a decade ago were working in the then-hot area of Web development now gaining opportunities to work on applications and infrastructure that are today’s hottest trends.
With skill hopping and a continuous investment in education, employees know that they won’t get stuck supporting decade-old applications destined to be outsourced. This has resulted in reduced staff turnover and retention of superstars. Wolfgang describes his superstars as artisans who know infrastructure, know security, and take end-to-end responsibility for their applications.
According to Wolfgang, "Organizations are already paying for training. They just are not formally recognizing it as such and tracking it as such. Most project plans have padding to allow for the folks to figure out what they are doing. Most organizations have had extended outages while the team struggled to learn on the fly." For Wolfgang, "It is about making training explicit and measurable. Done right, this avoids the outage or the project delays that occur when people are 'training' in a customer-facing scenario."
Action Item: The migration to DevOps has been driven, in part, by application development teams embracing Agile software development methods and by IT operations organizations embracing virtualization and automation. Both have enabled an increased rate of change that can disrupt traditional IT organizations that are operating in the conventional silos of application development and IT operations. These organizations should evaluate a DevOps model and the potential significant benefits in efficiency, quality, and agility. Measurement of the current rate of change and the current quality of code released into production provide a good baseline to both identify the need for and document the benefit of a DevOps approach. If the current culture is one of assigning blame, it will need to move rapidly to one of accepting end-to-end responsibility. In order to fully embrace DevOps, walls must be broken down while budgets and reporting structures must be consolidated. Maximum benefit will come to those organizations and individuals who shift from a mentality of a fixed job role to a dynamic progression of skills. This will require an explicit investment in education, rather than ad hoc, on the fly, emergency training.
CIOs: Time to Think DevOps
If you're a CIO who cares about IT operations and wants things to run smoothly, it's time to look into DevOps. At the simplest level, DevOps brings together the development and operations mindsets and organizes them around highly motivated teams, trained and well versed in both disciplines. The benefits are substantial and seen in the form of lower operational costs (cut by as much as 2/3rds), faster deployment and much greater flexibility to respond to business needs.
This was the premise put forth at the March 6, 2012 Wikibon Peer Incite by J. Wolfgang Goerlich, an IT practitioner in the financial services sector.
DevOps is a concept and IT management methodology that is designed to address the disconnects between operations and development. The problem is that operations people are focused on making sure IT infrastructure doesn't go down. Indeed most system failures come from human error, not faulty machines-- leading to a "don't touch my storage network" mentality. Development people on the other hand are motivated and paid to push code, change with business needs and develop new function that helps differentiate their organizations and provide competitive advantage. To a development professional, change is good. To an operations person, change is scary bad.
The schism between development and operations becomes obvious when a new code release is ready to go into production. Development will "toss" the code over the fence to operations folks, who then are responsible for getting it deployed. In doing so they must configure the code to match the operational environment, which is designed never to go down and is almost certainly different than the development infrastructure. So even though the code release is "finished", operations people must change it to really be ready for prime time. In doing so, they invariably uncover other bugs or change the code, which breaks something else. The operations people then have to call in the developers, who often get blamed for shipping faulty code. Of course the developers are frustrated, because it worked fine on their laptops. This process is duplicative and adds significant time to the deployment, increasing costs and decreasing business value.
DevOps is an organizational approach to break down the silos that have built up between development and operational teams. Key aspects of DevOps are:
- IT staff are cross trained in both development and operations,
- Teams are organized under the same structure, breaking down the barriers between development and operations,
- Teams are measured and incentivized in a much more similar, if not identical manner,
- The lifecycle of development and operations is unified,
- Tool sets between development and operations are much more common if not identical.
A key outcome of DevOps is to automate as much as possible. Think "programmable infrastructure."
What we heard on the Peer Incite call with Goerlich is that DevOps has completely transformed his IT organization by:
- Accelerating application rationalization,
- Speeding the consolidation of infrastructure,
- Reducing FTE counts by 2/3rds,
- Improving quality and speed of deployment,
- Increasing morale.
The key challenges to successfully implementing DevOps are organizational and process, as these must change, and change is typically difficult. But the results of DevOps appear promising, and in these days of "do more faster with less", the DevOps mindset is a trend that is your friend.
Action item: DevOps is becoming a proven approach to creating hyper-productivity and forming highly motivated IT teams. It is not only the next step beyond agile development; it is the future of IT operations management for IT-as-a-Service and cloud scale environments. DevOps successes at global giants like Google and Amazon have begun to trickle down to other organizations, and CIOs must investigate this new approach as a means of streamlining operations and delivering greater value to the business.
Integration and Consolidation Imperatives change IT Organization
Just because you can, doesn't mean you should. The business case for purchasing integrated technologies, whether it be server blade infrastructure with built-in storage from HP or a VCE Vblock complete solution, is usually overwhelming. There is nothing that can't be done in house. It may even be installed for a lower initial cost internally. The cost savings of integration and consolidation come over time from higher availability and lower costs of infrastructure management and maintenance.
Good integration is easy to recognize when it is there; it just works for the job or workload it was designed for. The functionality should reflect the business requirements of the business, neither too little or too much. As much care should be taken about what is left out as to what is included.
Leading IT organizations are focusing on streamlining IT by consolidating the development and operational functions. During the Wikibon Peer Incite on March 6, 2012, Wolfgang Goerlich, Information Systems and Security Manager at a multi-billion dollar investment management company, gave crisp advice about how the human capital within an integrated development/operations (DevOps) group should be deployed. The tasks that should be performed should meet three criteria. They should:
- Deliver business value,
- Excite staff,
- Utilize high levels of skill in staff.
If tasks don't meet these three criteria, they should either not be initiated or outsourced.
His suggested metrics for measuring success were also simple, and reflect the imperative to make changes that improve IT value to the business, from improved availability, function, lower cost or improved agility. Wolfgang's recommendations are:
- Number of changes (value of implementations or updates, with an emphasis on number),
- Percentage of changes backed out (quality of integration).
Action item: Integration, consolidation and virtualization of infrastructure brings with it an imperative to streamline the operational and development functions and to be highly selective of the projects undertaken. Stress should be on not adding additional products or procedures but on either improving or replacing those already in place. The removal of organizational barriers by cross-training all staff has been shown to lead to a more agile and higher quality delivery of IT services to the business.
DevOps - One Team, One System
The benefits of a streamlined DevOps model are many: Better efficiency, higher productivity, and reduced costs. But integrating previously siloed application development and IT operations teams into a single agile, cohesive unit is not without its organizational challenges.
DevOps team members must adopt new mindsets. For application developers, this means taking responsibility for the maintenance and performance of applications after they’ve gone into production. On the flip side, IT operations staff must expand the scope of its role to include supporting the application development process.
To do this, DevOps team leaders should dedicate a significant portion of each team member’s time – say 20% - to training across disciplines. Operations pros should spend some time coding, for example, while developers should get their hands dirty working with the hardware.
Support silos must also be broken down. At one Midwestern capital management company, for example, application support and operations support were previously directed to separate help desks. As part of its new DevOps model, support is consolidated into a single help desk, with just one phone line and one email address for help tickets, manned by both developers and operations pros.
DevOps teams are lean, meaning most organizations that adopt the DevOps model will ultimately end up reducing staff. This company went from more than 25 staff members between its siloed application development and IT operations staffs to a nine-member DevOps team, a 65% reduction. DevOps team leaders should focus on retaining those team members who embrace fast-paced development and release cycles, thrive in highly collaborative environments, and are eager to develop and master new skills.
Budgets must also be integrated, a process that can take a year or more.
Action item: To address the organizational challenges associated with the DevOps model, Wolfgang Goerlich, who heads this company's DevOps team, developed a helpful Venn diagram to guide his decision-making process.
The first circle includes those activities that drive business value – “what’s saving me money or helping my business make money.” The second circle includes those activities that motivate DevOps team members – “what gets my people fired up, what gets them excited, what makes them challenged at work and really puts them in the zone.” The third circle includes those activities that team members are most skillful and knowledgeable about.
Goerlich focuses much if his effort on growing the third ring through training his staff to develop new skills, including cross-training between the development and operations sides of the team. This has the residual effect of growing the other two rings, resulting in a lean DevOps team prepared to meet the current needs of the business and agile enough to adapt to changing conditions with minimal disruptions.
Vendors Need to Keep in Step with Customers Moving to DevOps
Similar to server virtualization, the business value of DevOps is that it allows businesses to streamline operations, which delivers a significant reduction in the complexity and time to deploy new applications. During the March 6, 2012 Wikibon Peer Incite, J. Wolfgang Goerlich, an IT practitioner, discussed how the vendor ecosystem need to become enablers of the DevOps trend.
Vendors need to understand the impact their products have on the ecosystem. Vendors need to evolve beyond the thinking that they are done once the product is shipped. Both hardware and software vendors need to be aware of the changes in a DevOps environment. In order to minimize disruptions to operations, vendors should strive to deliver code that can be managed by and be part of the existing infrastructure and management tools. For software and software-as-a-service companies, there is no “done”, the vendors must commit to continuous improvements. Another change in mindset for vendors that building into the existing environment is more important than piling up a list of features that add to the burden of managing the solution. Leveraging existing tools, plugging into frameworks and working with industry APIs are preferred to trying to build new ones.
Wolfgang pointed out that support from the vendor community needs to match the needs of a DevOps environment. This is an overhaul for most vendors who have a wall between development and operations. For many suppliers today, the customer knows that they have hit a very bad issue when they hear, “we’re getting the development team involved”. The goal is to create closer coordination so that the software development team can feel the pain of the customer.
Wolfgang did have a positive experience with the support of storage solutions from Compellent (prior to its acquisition by Dell). Compellent pairs up Copilot Support with their development team, so that going from support to development is right across the cube wall. The development team knows how its solution impacts support and therefore is more in tune with the needs of the customer.
Action item: Vendors that embrace the DevOps movement can deliver more strategic value to customers. Closer alignment to DevOps will allow vendors to keep close tabs on customer adopting this trend, perform superior service and have early signals of potential new revenue opportunities. This new methodology pairs well with the ITaaS and SaaS solutions that vendors have been rolling out over the last few years.
Efficient DevOps Requires Shedding Aging Applications
Part of the problem of achieving high efficiency with DevOps is what to do with all those old applications. Some may not be in use at all, others are on maintenance and, while still used, are no longer sources of competitive advantage.
During the recent Peer Incite discussion of DevOps, Wolfgang Goerlich offered succinct advice, “Automate it, minimize it, or move it to a third party.”
This means biting a bullet that many IT organizations seem hesitant to touch. When he took over operations at the Midwestern financial firm where he has redesigned IT, one of the first things he did was to identify and shut down old applications that were no longer used. Some of these had been special requests from people who no longer worked at the company, while others had been rendered redundant by newer applications, and others supported business functions that themselves had been outmoded. But in IT no one had thought to shut them down, or people feared that if they did someone would complain, or someone's job was tied to that application.
Once those were eliminated, Goerlich started identifying applications that while still used were no longer centers of competitive advantage. Website development, for instance, was a hot area a decade ago but today is a commodity skill that is only a distraction to Goerlich's lean DevOps team.
Goerlich classifies these applications under “not my problem” and is actively outsourcing them. This is complicated by the special regulations financial companies operate under, but he has turned several applications over to traditional managed service outsourcers and a few to SaaS companies, and he hopes to move more to these third parties as the regulatory situation becomes clearer.
He has strong words for enterprises that use the excuse that their staff “lacks the skills to manage SaaS” to avoid moving older applications to the cloud. That, he says, is tantamount to saying that their staff training is inadequate to keep their organizations abreast of the technology in this fast-evolving IT environment, and that means that their IT organizations are becoming obsolete.
Action item: IT shops commonly devote as much as 80% of their operating budgets to maintaining old applications rather than developing new ones or revolutionizing their environments with virtualization and cloud computing. Today IT has options that can free it from these boat anchors and let them retrain and rededicate staff to technologies that will make their companies more competitive rather than tying them to a dying past.
