December 19, 2018
TO: Campus Community
FR: Matthew Hall, Associate Vice Chancellor and Chief Information Officer
RE: Google Service Disruption Incident Report Incident Summary
On Tuesday, December 18, from 7:19 am to 12:15 pm, the UC Santa Barbara campus experienced a disruption of all Google services, including our Connect email, calendar, and collaboration services.
UC Santa Barbara was one of many universities and colleges nationwide affected by a disruption to Google’s services.
We recommend that any email sent during the disruption in service be resent to ensure delivery.
We continue to work with Google and our other network partners to help them identify the root cause of this disruption in service with the goal of preventing similar incidents in the future.
Our Service Desk at ext. 5000 is available to answer questions.
For more detailed information, see the incident report below.
Incident Report
On Tuesday, December 18, from 7:19 am to 12:15 pm, the UC Santa Barbara campus experienced a disruption of all Google services, including our Connect email, calendar, and collaboration services.
Affected Services
- Gmail
- Drive
- Calendar
- Hangouts
- Search
Google Mail
Google Mail was not available to campus, and new email sent from Google -- or through the use of Google services -- to ucsb.edu email addresses was generally undeliverable and bounced back to the sender with a “no such domain” message.
After the 12:03 resolution, delivery completed for most email from non-Google sites queued during the event.
The possible outcomes for email sent during yesterday's outage depend on the specific configuration of the sending mail systems. These are the general possibilities:
- Outbound emails (i.e. UCSB email sent from phones or from home computers) There was no impact to emails sent to non-UCSB addresses.
- Email sent between UCSB Google accounts Delivery failed for most email sent during the outage. However, some messages sent during the early stages of the outage achieved delivery following the resumption of services. To ensure message receipt, any email sent between UCSB Google accounts during the incident should be resent
- Email sent from a Gmail or G Suite hosted account to UCSB Google accounts Most messages sent during the outage failed delivery and senders should have received a bounce-back notification.
- Email sent from non-Google sources but which used Google's DNS servers. Early during the outage, delivery of these occurred normally. Later (as DNS information expired) messages failed delivery and senders should have received a bounce-back notification.
- Email sent from non-Google sources using non-Google DNS Messages from non-Google sources accumulated in mail queues on the UCSB mail exchange servers. Upon service restoration, message delivery completed. The backlog cleared by 5 pm yesterday (12/18).
Because most email servers discard their copies of email as soon as their role in the message flow is complete, it is not possible to restore any of the undelivered messages. Messages sent during the outage should be resent to ensure receipt.
Networks and Access Disruption
Certain devices, such as Android phones, were unable to use the campus wireless network. These devices reverted to using the cellular data network.
Various UCSB-hosted and third-party websites were difficult to access due to their inclusion of Google content within their pages. Examples of Google content include Google Analytics and Google-hosted fonts.
There are various systems, both on-campus and off-campus, that make use of Google’s public Domain Name System (DNS) servers. These DNS servers were unavailable to on-campus clients, resulting in either delayed or failed access to network services. Off-campus clients would generally fail to access UCSB systems due to lack of information via Google DNS