October 15, 2015
TO: Campus Community
FR: Matthew Hall, Associate Vice Chancellor and Chief Information Officer
David McHale, Director of Facilities Management
RE: Southern California Edison Power Outage
Around 6:03 p.m. on Wednesday evening, a power outage disrupted campus telecommunications. Power was restored within three to five seconds, but it impacted our telecommunications equipment. Telecommunications staff responded immediately. Because 911 and other calls were ringing fast and busy on campus and going to voice mail if off campus, emergency service calls were routed to Santa Barbara County. Around 7:15 pm on Wednesday evening, campus telephone service was restored. At that time 911 services were still being serviced by the County. The return to campus dispatch is at the discretion of our Police Department.
Investigation continues into the root cause, though current understanding is that several line cards were damaged during the power spike and the switch is still showing internal faults. The line cards were replaced from spare line cards onsite.
Our North Hall Data Center successfully cut over to generator power and worked with Facilities Management to overcome a serious air conditioning issue. Their teamwork paid dividends and a shut-down of core services was averted.
At approximately 1:00 am this morning, Southern California Edison (SCE) contacted the campus to alert us that lightning had struck their 66kV transmission system and they had to de-energize the transmission line that serves the campus. The campus requested SCE to delay for one hour so that Facilities Management could recall trade staff and alert critical users of the pending outage. At approximately 2:15 am, SCE shut down service to the campus to make repairs to their line. SCE estimate that the repair would take two hours to complete, but as the work was underway they discovered there was additional damage and provided a new time estimate of 8:30 am.
FM crews refueled campus generators to ensure operation during the event and additionally installed a temporary generator at Marine Biology (Building 555) as their generator failed.
We would like to express our sincere appreciation to our research colleagues for their amazing efforts during this power outage. Also, thank you to our Enterprise Technology Services (including North Hall Data Center and Telecommunications) and Facilities Management colleagues for their efforts throughout Wednesday evening and Thursday morning in restoring power back to the campus.
Matthew Hall, Associate Vice Chancellor and Chief Information Officer, 805-729-7504 (m)
David McHale, Director of Facilities Management, 805-893-4610 (o), 805-698-3078 (m)