Shortly after the Palo Alto plane crash that killed three Tesla employees, took out power for the entire city for 10 hours, and brought down communications for VMware, the following dialogue occurred on Twitter:
- Beaker: @Texiwill If they can get to Gmail...C'mon, DR planning assumes you lose comms. VMware is a mature enough Co. to figure this out.
- Beaker: @Texiwill So VMware has no DR plans for a power outage other than sending email to GMail!? Have they heard of virtualization/cloud/VoIP?
- Beaker: @Texiwill Why would I send production down/sev. 1 issues re: VMware to a gmail address, exactly? This doesn't sound kosher.
- gbrunett: RT @Beaker: @Texiwill So VMware has no DR plans for a power outage other than sending email to GMail!? Have they heard of virtualization/cloud/VoIP?
- dawoo: Not sure VMware using a gmail acct for Sev 1 issues is a top idea given Google's privacy policy on gmail accounts #VMware
The following is a series of posts from the official VMware Twitter account @vmware on February 17, 2010. Times are Eastern Standard Time:
- 12:10 PM Palo Alto plane crash & power outage - VMware Support still available & phones working - 1 877 4VMWARE - Follow @vmwarecares for news
- 1:54 PM RT @vmwarecares #VMWARE ALERT: For absolute production down severity 1 issues, email supportvmware@gmail.com with Company/Contact/Phone
- 11:58 PM All VMware systems have been restored. Support is available online and at 1-877-4VMWARE. We apologize for the inconvenience.
VMware employee, Rick Blythe, who describes himself as a "Social Media Specialist for Global Services and Support at VMware," maintains a Twitter account, @vmwarecares, which provided earlier notification, than VMware's corporate Twitter account. The following are a portion of the tweets as the disaster evolved:
- 11:19 AM Network outage here caused by small plane crash in Palo Alto
- 11:44 AM BREAKING NEWS: Palo Alto plane crash - power outage - VMware Support: still available - phones working - 1 877 4VMWARE
- 12:13 PM #VMWARE SUPPORT UPDATE: Phones are NOT working. No ETA at this time
- 1:04 PM #VMware ALERT: Due to Palo Alto plane crash, we've had a phone outage. We are busy re-routing phones and expect to be back up soon
- 1:49 PM #VMWARE ALERT: For absolute production down severity 1 issues, please email supportvmware@gmail.com with Company/Contact/Phone#
- 2:26 PM #VMWARE ALERT: For critical license issues, please email vmwarelicensesupport@gmail.com with Company/Contact/Phone# < Please RT
- 2:53 PM Due to technical difficulties #VMware Support is currently accepting ONLY Production Down Service Requests through supportvmware@gmail.com
- 10:04 PM Happy to share that all network operations are up and running. Power has been restored - back to business as usual.
Similar, but less frequent Tweets were posted by @VMwareKB
VMware primarily uses the Knowledge Base Twitter account, @VMwareKB, to disseminate educational alerts that have been posted on the company's VMware Knowledge Base Blog.
There are a number of lessons learned from the disaster:
- Alternative methods of communications, such as e-mail from Gmail and social networking from Twitter, can be extremely valuable in facilitating communications with customers during disasters. These channels of communication remained available throughout the power outage.
- Even if communications are not coordinated, options, such as Twitter and Gmail, can be adopted and adapted quickly to a variety of uses.
- Customers may be suspicious and concerned that alternative communications systems, such as Gmail, are not "official," unless the alternative communication system is well-evangelized prior to first use.
- Companies need to have multiple methods of disseminating communications during disasters. Had VMware chosen Facebook, instead of Twitter, the alternate communication option would also have been down, as Facebook's data center is also in Palo Alto.
- While social media is by its nature somewhat uncoordinated, companies still need to develop coordinated alternative communications plans. Clearly the support account, @vmwarecares, provided earlier notification than the official corporate site, @vmware.
- Companies should develop a list of functions, (e.g., sales, customer support, education)and secure social media accounts with those extensions. Using Wikibon and Twitter as an example, Wikibon could secure @wikibonsales, @wikibonsupport, @wikibontraining.
In order to alleviate concerns regarding the veracity of the source, one possibility is to use a verified Twitter user name to disseminate contingency planning information. That way, customers, employees, and partners will know that this is official communication and not malicious dissemination of misinformation from opportunistic scammers.
Action Item: CIOs and disaster recovery professionals should consider cloud-based alternative communications plans in their disaster recovery plans. At the same time, they need to both evangelize the contingency plan and offer a method of authentication, much like Twitter does to authenticate individuals with "star power." Evangelization is critical. As of February 19, 2010, @VMwareKB only had 1,831 Followers on Twitter and the more frequent news source, @vmwarecares, had fewer, with 878 Followers, far fewer than the number of VMware customers. Built into comprehensive business continuity planning processes, companies such as VMware could leverage the full power of these options.
Footnotes:
