Google Workspace Status Dashboard

This page provides status information on the services that are part of Google Workspace. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://workspace.google.com/. For incidents related to Google Analytics, visit the Google Ads Status Dashboard.

Incident affecting Gmail

Incident began at 2024-07-30 14:05 and ended at 2024-07-30 17:10 (times are in Coordinated Universal Time (UTC)).

Date Time Description
Aug 2, 2024 4:18 PM UTC

Incident Report

Summary

On Tuesday, 30 July 2024, Gmail experienced increased error rates and delivery delays for a duration of 3 hours and 5 minutes.

To our Gmail users who were impacted by these issues, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.

Root Cause

Google engineers have determined that the increased error rates and delivery delays were due to unrelated root causes.

Email Delivery Delays

On Tuesday, 30 July 2024, at 07:05 US/Pacific, a series of fiberlines which carry Gmail in-flight traffic were unintentionally disabled between Google's US and European data centers as part of routine maintenance.

Increased Error Rates

On Tuesday, 30 July 2024, at 08:00 US/Pacific, a non-critical process responsible for batch data processing started generating an increased volume of traffic which overloaded Gmail’s back-end system.

Remediation and Prevention

Email Delivery Delays

Google engineers were alerted of the incident through automated alert systems on Tuesday, 30 July 2024 at 07:14 US/Pacific and immediately began an investigation. By 07:34 US/Pacific, engineers performed a change to mitigate congestion/throttling which rerouted traffic to available servers for processing.

Increased Error Rates

Google engineers were alerted of the incident through automated alert systems on Tuesday, 30 July 2024 at 08:00 US/Pacific and immediately began an investigation. The issue self-resolved on Tuesday, 30 July 2024 at 10:10 US/Pacific, and errors were trending down to the baseline. The engineers identified and made the necessary changes to avoid a repeat of the issue at 10:30 US/Pacific.

Google is committed to preventing a repeat of this issue in the future and is completing the following actions:

  • Audit and test our traffic priority labels and storage system behavior to ensure that background traffic cannot overload the system.
  • Audit/test error propagation to make sure the routing layer responds adequately to error conditions.

Google is committed to quickly and continually improving our technology and operations to prevent service disruptions. We appreciate your patience and apologize again for the impact to your organization. We thank you for your business.

Detailed Description of Impact

Email Delivery Delays On July 30, 2024 07:05 to 08:00 US/Pacific, some emails were delayed between 10-20 minutes.

Increased Error Rates On July 30, 2024 07:46 to 10:10 US/Pacific, some users received potentially visible errors:

  • An error message banner with a valid retry button for users to manually retry the next Gmail web server - “Oops… the system encountered a problem (#2014)”
Jul 30, 2024 8:40 PM UTC

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support or to Google Workspace Support using help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 30 July, 2023 07:05

Incident End: 30 July, 2023 10:10

Duration: 3 hours, 5 minutes

Affected Services and Features:

Google Gmail

Regions/Zones: Global

Description:

Gmail experienced an overload scenario leading to delayed email delivery as well as serving user visible errors "Unable to reach Gmail---Trying to reconnect".

From the preliminary analysis, our teams identified a potential root cause and blocked traffic contributing towards the overload.

Customer Impact:

  • Approximately 40M emails, sent by approximately 2.6M unique senders, were delayed by 10 minutes or more.
  • At peak, 70k unique Gmail users received error messages. e.g. "Unable to reach Gmail---Trying to reconnect"

Additional details:

Our engineering team is making the required changes to prevent such issues. We apologize to all who were affected by the disruption.
Jul 30, 2024 5:49 PM UTC

Status: RESOLVED

The issue with Gmail has been resolved for all affected users as of Tuesday, 2024-07-30 10:00 US/Pacific.

We thank you for your patience while we worked on resolving the issue.
Jul 30, 2024 4:51 PM UTC

Summary:

A subset of Gmail users might be experiencing delayed delivery of their emails.

Description:

We are experiencing an intermittent issue with Gmail beginning on Tuesday, 2024-07-30 07:05 US/Pacific.

This mainly caused delayed email delivery and a small number of users receiving errors when sending/receiving emails.

Our engineering team continues to investigate the issue.

We will provide an update by Tuesday, 2024-07-30 11:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis:

Some customers might see a yellow banner with the "Unable to reach Gmail---Trying to reconnect" message.

Workaround:

None at this time.