How StarPoint Advantage continued to service its clients
during the largest power outage in San Diego County history.
September 8, 2011
Just a Normal Work Day
The weather was picture perfect as it almost always is in San Diego on this Thursday, September 8, 2011. We had almost completed our regularly scheduled 3 o’clock HR policies and initiatives meeting when the power went out in our building just before 4 PM. As with most power outages, we expected the power to be back on any moment. With plenty of sunlight still shining through our office windows, we refocused our attention back to getting our work done. In the meantime, our service team began to research what had happened using their cell phones to access the Internet for any news. We wrapped up the meeting about 4:15 PM and, the power was still out.
As we stepped out of the HR office and into the hallway, our service team discovered that the power outage was widespread and affecting most, if not all, of San Diego county and parts of Arizona and Mexico. There had also been a report of an explosion near the Salton Sea, which we later discovered to be unfounded. Of course, with the 9/11 anniversary a few days away, it was easy to think something terrible might be happening. In any event, we knew we had to be well prepared for the worst since most, if not all, of our clients would be calling in for support.
The Response Plan
We were prepared to continue operations and answer phone calls since our key line of business systems (client management, ticketing system, system monitoring, data backup & storage systems, Email and phones) are hosted in multiple protected data center facilities that are designed to continue to operate in the event of a power outage for days or weeks, if needed. American Internet Services (AIS), our business partner that hosts our systems in their data center, never lost power as their generators kicked in as planned. This continuous up time enabled our team to continue to provide services and give our clients real time updates on the true extent of the outage.
The majority of our clients still host IT systems (servers & phones) at their offices (on premise). We knew that the average battery backup time would not exceed 10 minutes and their servers would subsequently shut down until the power was restored. As predicted, the calls started coming in. Most of our clients still had not realized the extent of the outage and were calling in (most of them from their cell phones) to report power outages in their respective buildings.
Within 30 minutes of the outage our service team created a response plan to this unprecedented outage. We assigned clients (that were geographically close together) to specific engineers to ensure the quickest response possible in the event we had to travel to the client’s office to restore their systems. Because our systems were effectively “immune” from the power outage, our team was not only able to respond to our clients’ calls within the minutes after the outage, but they also had real time data on the status of our clients systems when the power was restored about 8-9 hours later. These systems told us exactly which of our clients were still down, which were partially down and which ones had come back online. This information enabled our team to prioritize our response starting at 5 AM the following morning.
Power Comes Back Just After Midnight
The power for most of San Diego came back shortly after midnight on September 9, 2011. At 5 AM, we immediately swung into action, prioritizing our recovery plan based on the latest information from our monitoring systems. Our plan shifted to addressing those clients that were still “hard down” and getting our “boots on the ground” where needed while addressing the rest of the issues remotely from our network control center based on severity. Resources were deployed immediately to address those most in need.
On occasion, power outages such as this one, can impair mission-critical equipment like routers and switches, consequently we keep spare components available for an easy removal and re-installation. In some cases, the part is not on hand and that’s when we must find (whether locally or nationally) the part as quickly as possible. Our ability to coordinate with suppliers is part of our response plan. More importantly, our service commitment to our customers is to work non-stop in tracking the arrival and installation of system components until resolution. Once the part arrived, our engineers completed the installation and had systems back up and running Friday evening. The Museum of Contemporary Art San Diego, our customer, had planned a major gala for Saturday and looked to StarPoint Advantage to get them back online for the weekend.
In conclusion, our ability to get real time information and provide proactive service to our clients during and after the power outage is due in large part because we maintain our critical systems in secured, highly redundant data center environments. The data center resources and redundancy became critical to our operations enabling us to access critical information and continue servicing our clients to ultimately minimized downtime for our clients.
For more information on how your business can reduce risk of downtime due to unforeseen circumstances, contact StarPoint Advantage at firstname.lastname@example.org or 619-272-4238.
Under the leadership of founder, David Davis, StarPoint Advantage opened its doors in May 2008 as a spinoff from a larger company David founded in 2003. StarPoint Advantage is the IT department for small and medium size businesses in San Diego County with half of them comprised of law firms. We help ensure IT systems are reliable, secure and available 24/7 . Customers can rely on the StarPoint Advantage team for outstanding service built on trust, integrity and experience.