Ed. Note: This is a transcription of an interview of Bill Haggard, Director of Enterprise Infrastructure for the Dallas Cowboys, conducted by David Vellante of Wikibon,org and John Furrier, founder of SiliconAngle.com, on SiliconAngle.tv at VMworld 2010 in September. It is provided for research purposes.
John Furrier: Sports is a huge business, and Dallas is a great football town. Tell us about the Dallas Cowboys and your journey.
Bill Haggard: What a lot of people don't understand & we have had to evangelize our message on somewhat is that from a technology perspective we don't just support the football team and the stadium. We also support 30-some businesses that the Jones family owns. So from a technology perspective, everything we did in the stadium we had to make sure was flexible enough & scalable enough to support all those businesses.
And from the sports world you have to worry about how you can market & advertise your brand, not only in the stadium so you can get that “Wow” factor when the fans are there, but how you can get that brand advertising out to other sectors of the market.
John Furrier: So we want to get some Jerry Jones stories later, because I was told by Steve Harrod that you probably have some good ones. First I have to get out of the way, the M.S. Smith Hall of Fame speech, wasn't that fantastic?
Bill Haggard: Absolutely. It was a great speech.
John Furrier: Classy, very well done. M.S. Smith a legend. And he can dance.
So talk about virtualization. You have a story there around how you are using IT in the business to be competitive, and you have to be profitable.
Bill Haggard: From a virtualization standpoint what we had to worry about was – for instance our concession stands. All of our concession stands in the stadium will take credit cards, debit cards, or cash. That is a huge change of mentality from the old Texas Stadium. At Old Texas Stadium we had the mentality of 70% break/fix and 30% innovation. What we had to do going into the new stadium was worry more about what we could do with technology and how we could use it to advance the fan experience in the building. So from a concession stand, standpoint, for instance, we have 212 concession stands. Each concession stand is treated like its own restaurant. So each concession has to have its own Windows 2003 server on the back end to support that. That's 254 of those management servers just to run the concession stands. With that and a few ancillary servers we would have been in the neighborhood of 300 physical servers just for concession stands without virtualization technology. Being able to virtualize that on the back end we were able to put that on two HP C7000 chassis on 16 blades. So instead of taking 10 full racks for servers we have them in half a rack. So it cuts down not only the amount of cooling and power and cabling costs and things like that, it gives us the ability to run it in a much smaller footprint, so we have more floorspace to use for other technology purposes.
David Vellante: It really simplifies things for you, right?
Bill Haggard: Absolutely. It gives us the ability to manage much more with less in less time. It also gives us a DR or resiliency factor that from a completely physical perspective if we had had to do all that physically we wouldn't have had that ability. So if we have issues with VMware's vMotion and other technologies they have the ability to move those around either manually or automatically to make sure we don't have an outage during an event.
David Vellante: Football is an interesting business. A lot of the revenue comes, of course, from TV. But the Dallas Cowboys' brand being what it is as a real innovator – the Yankees are another one – is doing some interesting stuff in diversifying the revenue stream. You mentioned concessions and other merchandising, taking a page out of Walt Disney and other Jerry Jones enterprises. Has virtualization helped you to go beyond just cutting costs to real revenue generation?
Bill Haggard: It gives us the ability not only to support our retail business – let's talk about that a little bit. We have 10 pro shops in the stadium itself; we have 35 outside, because we control our own merchandise rather than the NFL doing that. So it gives us the ability to have more flexibility as we open more pro shops or change things within the stadium. So that technology gives us the ability to do some things that from a physical perspective we wouldn't have had.
John Furrier: How many people come to a game roughly? The new stadium's big.
Bill Haggard: That works a little differently, too, because at the stadium we'll have 40 or so events there this year. There are 11 pro football games; four will be college football games; we'll have 8 or 10 high school football games. We'll have another professional fight there, we'll have an MMA fight there probably. Last year the largest football game we had was opening night regular season against the Giants, 105,000 fans.
John Furrier: So in technology we talk about policy – QOS, moving packets around, etc. Steve was alluding to business policy. If the Dallas Cowboys win maybe I'll pull out my wallet and buy some shirts. So you can maybe adjust pricing? Do you do those kinds of things with virtualization? Steve was alluding to you may have that kind of capability.
Bill Haggard: We have the capability from not only the retail side but the concession side. Because we can control all the menus from prices at the retail stores to dynamic menus at concession stands. So we can control and adjust the prices while the event goes on.
David Vellante: Can you talk a little about the BI system and data warehouse you use to support that? And is that virtualized?
Bill Haggard: The back-end pieces are not virtualized.
David Vellante: Which is fairly common, right?
Bill Haggard: Right. We're looking into that now that vSphere has been out for awhile. Truthfully, when we went to implement all the new technology in the new stadium, vSphere was not out yet. We had to get all the technology up & running in a short amount of time. So we didn't have the luxury of waiting, even though we knew that vSphere would give us the ability to virtualize more of the SQL back-end things. So we couldn't do that then. We're looking at now, probably in this coming off season, to virtualize a lot more of that back end than we have now.
David Vellante: I had a client tell me the other day that their data warehouse BI system was like a snake swallowing a basketball, and so they didn't want to virtualize because they just had to fence it off and drive all the performance possible. But you feel you will be able to virtualize that and share the infrastructure with other applications?
Bill Haggard: We think so. What we'll probably do is take a subset of that and put it up in a test & development environment to make sure that if we do virtualize we won't have problems & can still get the performance out of it on the front end to get the information back to the users in the different departments as quickly as they need it.
David Vellante: So how long have you been at this virtualization game?
Bill Haggard: With the Cowboys, two years is what we've really been at from implementation, making it right, moving forward. In my previous career I had probably worked with them four years before that.
David Vellante: So that's a good time frame, the two years with the Cowboys. And you've had experience with virtualization prior to that. Thinking about the Cowboys experience – what would you do differently if you had it to do over again?
Bill Haggard: (long pause) Truthfully from the way we implemented the virtualization technology, I don't know that we would make very many changes.
David Vellante: So my follow-up is what advice would you give to your peers so they could have a similar experience?
Bill Haggard: Plan it out. Don't rush into it. A lot of companies will look at a new technology and their mentality is we have to do it now, very quickly, on the knowledge that they currently have. For a transformation like we went through it was about a 14-month-long process to plan out what we wanted to do, get the design right, and then start the implementation phase. So the biggest piece of advice is don't rush into it. You absolutely need to look at virtualization technology. It can save you countless amounts of money. But it's something you have to plan out and do right.
David Vellante: How about the people & process pieces around that. Was that part of the planning?
Bill Haggard: You have to have a great staff, whether you get your existing staff to work with you or you bring in consultants from outside. You have to make sure you are getting a top-notch group of people to work with who can take your vision that you are trying to move the technology through that they understand what you need it to do. Because you can run into issues with some consulting companies and different groups – even your own staff. Somebody may have an ulterior motive who thinks that the direction the organization needs to move in is not the right move, and they want to do it in a different way. So you have to get everybody on the same page and buying into the same philosophy and plan.
David Vellante: How about the vendor community? Talk a little about the roll they played in your implementation.
Bill Haggard: Sure. We talked to all of the major players in the market. What we had to do is get them to open up to us, knowing what we were going to go through, and show us what their technology road map was not for tomorrow, not for a month down the road, we wanted to see what that technology vision was seven years down the road, ten years down the road. So as we looked at their platforms, we could be assured that in three years we wouldn't have to do forklift upgrades on everything. We wanted to make sure that what we were going to implement was going to be upgradable & flexible but that the upgrades were not going to cost the same amount of money as the original implementation.
David Vellante: We've heard a lot from the cloud service providers. Did you consider outsourcing some of this to the cloud, or do you do that as part of your infrastructure?
Bill Haggard: We don't currently. We looked at some of that, and we will probably look at that again, but what we would probably use the cloud for is some of our DR services and strategy, not probably a day-to-day working pieces of that. So with all the virtualization we have implemented, between our different locations, we kind of have our own private cloud, but we really wouldn't classify it that way. We have three data centers, one at the stadium, one at our merchandise facility, and one at Valley Ranch. We have a DR strategy that passes data among the three, so that if we have problem with a mission-critical system or a disaster at any of those locations we can bring those up at another location. But from a day-to-day perspective we are not presently looking at taking that to a cloud environment.
David Vellante: Do you have some RPO Zero applications?
Bill Haggard: Very close. I wouldn't quite classify them as RPO Zero, other than during an event, things that run the concession stands; the retail stores. Once the people are in the facility, they've paid for their ticket, they've paid for their parking. Where any revenue is going to be generated at that point is going to be the concession stands and the retail center.
David Vellante: And you can do that locally?
Bill Haggard: Absolutely.
John Furrier: Looking forward, you have some flexibility, you have some agile facilities, you can do things on the business side to make it more of a satisfying environment for your consumers. What's around the corner for you?
Bill Haggard: To answer your questions: for Jerry's external businesses, they are currently attached back to Valley Ranch, which is our corporate office. 95% of their applications are probably virtualized. If they are in a physical platform now, it's because the software vendor does not support them being virtualized. In some of the businesses we have to deal with MRI centers. So you get into HIPPA regulations and things like that software package the vendor does not support in a virtualized world. So it's unique. So what we could virtualize when we put this together we did. What we couldn't virtualize due to vendor not supporting a virtualized environment is what we couldn't.
John Furrier: Are there new things you are seeing now emerging from developers that you never would have contacted in the past that could build on top of this platform?
Bill Haggard: Different apps and things moving forward that we will be able to offer to fans coming into the stadium. Last year we did not offer WiFi to the public in the stadium. This year we will. So if you have smart phones, PDAs, different things like that, we will offer WiFi. We are also looking at other ways to do some target marketing using the infrastructure in the stadium. Maybe we'll send discount coupons to smart phones or push a text number that will pop up on the screen and in return we will send you a discount either in the pro shop or at the concession stands. So different technologies like that are things we are looking at to make it more beneficial or a better experience for the fans when they are in the facility.