Google Workspace Status Dashboard
Incident affecting Google Docs
Incident began at 2024-06-17 16:06 and ended at 2024-06-17 22:46 (times are in Coordinated Universal Time (UTC)).
Date | Time | Description | |
---|---|---|---|
| Jun 21, 2024 | 4:13 PM UTC | Incident ReportSummaryOn 17 June 2024, a subset of our customers using Google Workspace products (Google Chat, Google Meet, Google Drive, Google Calendar, Google Cloud Directory and Google Docs) experienced elevated error rates and latency of varying impact between the period of 6 hours and 40 minutes between 09:06 and 15:46 US/Pacific. To our Google Workspace customers who were impacted during this disruption, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability. Root CauseThe root cause is a latent issue that caused contention in getting access in a cache used for profile information. The issue was masked by overprovisioned quotas for some of the tables and only surfaced when quota adjustments were made to match the traffic patterns. Remediation and PreventionGoogle engineers were alerted to the issue from our internal monitoring at 09:21 US/Pacific on 17 June 2024 and immediately started working on the incident. Once the nature and scope of the issue was identified, our engineers mitigated the issue by increasing quotas and disabling the suspected feature in the key/value serving system. Google is committed preventing a repeat of this issue in the future and is completing the following actions :
Detailed Description of ImpactOn Monday, 17 June 2024, six of our Workspace products were impacted by varying degrees between 09:06 and 15:46 US/Pacific. The details and timeframes for the impact to each product have been outlined below. Google Calendar : Impacted users saw email addresses instead of names while using Calendar for a duration of 6 hours 21 minutes from 09:06 to 15:27 US/Pacific on 17 June 2024. Google Chat : Impacted users experienced intermittent issues loading chat, opening chat rooms, and sending messages for a duration of 6 hours 21 minutes from 09:06 to 15:27 US/Pacific on 17 June 2024. Google Meet : Impacted users sparsely experienced latency or failures while joining scheduled meetings over a period of 4 hours, between 11:00 to 15:00 US/Pacific on 17 June 2024. Google Docs : Impacted users encountered the error "Name loading" in comments or other areas where authors’ names were expected for a duration of 6 hours 21 minutes from 09:06 to 15:27 US/Pacific on 17 June 2024. Google Drive : Impacted users experienced elevated errors/latency while using third party APIs, for a duration of 6 hours 30 minutes from 09:06 to 15:36 US/Pacific on 17 June 2024. Google Cloud Directory : Affected customers were unable to access the “User” section in the admin console. The initial dashboard displayed the error "Data can't be fetched due to system” for a period of 6 hours 11 minutes from 09:05 to 15:16 US/Pacific on 17 June 2024. |
| Jun 18, 2024 | 5:54 PM UTC | Mini Incident ReportWe apologize for the inconvenience this service disruption may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced an impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213. (All Times US/Pacific) Incident Start: 17 June, 2024 09:23 Incident End: 17 June, 2024 15:46 Duration: 6 hours, 23 minutes Affected Services and Features: Google Calendar, Google Chat, Google Docs, Google Drive & Google Meet. Regions/Zones: Global Description: Multiple Google Workspace products experienced elevated error rates and latency. From preliminary analysis, the root cause is a latent issue that caused lock contention in a cache used for profile information. The issue was masked by overprovisioned quotas for some of the tables and only surfaced when quota adjustments were made to match the traffic patterns. Our engineers mitigated the issue by increasing quotas and disabling the suspected feature in the key/value serving system. Google will complete a full Incident Report in the following days that will provide a full root cause. Customer Impact:
|
| Jun 17, 2024 | 10:46 PM UTC | Summary: Multiple Workspace products are experiencing elevated error rates and latency Description: The issue with Google Calendar, Google Chat, Google Docs, Google Drive, Google Meet has been resolved for all affected users as of Monday, 2024-06-17 15:27 US/Pacific. We thank you for your patience while we worked on resolving the issue. |
| Jun 17, 2024 | 9:34 PM UTC | Summary: Multiple Workspace products are experiencing elevated error rates and latency Description: The mitigation applied previously did not yield results as expected. Our Engineering team continues to investigate and ascertain a mitigation strategy to fix all known issues We will provide more information by Monday, 2024-06-17 16:30 US/Pacific. Customer Symptoms:
Workaround: None at this time. |
| Jun 17, 2024 | 8:52 PM UTC | Summary: Multiple Workspace products are experiencing elevated error rates and latency Description: Mitigation work is currently underway by our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Monday, 2024-06-17 14:45 US/Pacific. Customer Symptoms:
Workaround: None at this time. |
| Jun 17, 2024 | 8:06 PM UTC | Summary: Multiple Workspace products are experiencing elevated error rates and latency Description: We are experiencing an issue with Google Chat, Google Drive, Google Docs, Google Calendar, Google Meet beginning at Monday, 2024-06-17 09:00 US/Pacific. Our engineering team has identified the root cause of the issue and are currently working on a mitigation. We will provide an update by Monday, 2024-06-17 14:00 US/Pacific with current details. We apologize to all who are affected by the disruption. Customer Symptoms:
Workaround: None at this time. |
- Times are listed in Coordinated Universal Time (UTC)