Incident Report

Summary

On Monday, 19 August 2024, at 11:10 AM US/Pacific, Gmail experienced an increase in the number of server unavailability (500/502) errors for a duration of 20 minutes. Impacted users may have encountered errors when attempting to access Gmail via IMAP / web / mobile clients.

To our affected customers, we sincerely apologize for the inconvenience this disruption may have caused. Our engineers have been focused on the investigation and analysis that followed, in order to ensure this issue does not recur.

Root Cause

Our engineering team has conducted a thorough investigation of the underlying cause and has identified a combination of factors that led to the issue:

• A previously unknown condition existed in our system that, when triggered, could result in performance slowdowns and errors when the service instances are restarted during high-load situations.
• A recent change that was introduced to improve the efficiency of our system interacted negatively with an ongoing unrelated canary-testing of another system configuration. This combination resulted in a high load situation.
• At the same time, a routine rollout of a new software version caused a restart of the canary test area during a peak usage period, further increasing the load.

Remediation and Prevention

Google engineers were alerted to the service disruption on Monday, 19 August 2024 at 11:19 AM US/Pacific via our proactive monitoring system and immediately started an investigation. The team quickly concluded that the errors were caused by instance restarts and immediately halted all deployments to mitigate the issue at 11:30 US/Pacific.

Google is committed to preventing a repeat of the issue in the future and is completing the following actions:

• Fix the latent issue that causes performance slowdowns and system errors when service instances are restarted under high load.
• Introduce a step to force a system restart under high load as part of our efficiency rollouts process.

If your service or application was affected, we apologize — this is not the level of quality and reliability we strive to offer you, and we have taken and are taking immediate steps to improve the platform’s performance and availability.

Detailed Description of Impact

On Monday, 19 August, 2024 at 11:10 AM US/Pacific, a small percentage of Gmail customers experienced increased server unavailability errors for a duration of 20 minutes. Customers who were impacted experienced Gmail account unavailability errors on their IMAP, web, and mobile app. The impacted customers also experienced Internal server (500) errors along with other in app errors.

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Workspace Support using help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 19 August, 2024 11:10

Incident End: 19 August, 2024 11:30

Duration: 20 minutes

Affected Services and Features:

Gmail

Regions/Zones: Global

Description:

On 19 August, 2024 11:10 US/Pacific, Gmail experienced increased server unavailability errors for a duration of 20 minutes. From preliminary analysis, the root cause of the issue was resource contention issues on the Gmail backend.

Google will complete a full Incident Report in the following days that will provide a full root cause.

Customer Impact:

Customers may have been unable to access Gmail via IMAP, web or mobile clients.

The issue with Gmail has been resolved for all affected users as of Monday, 2024-08-19 12:03 US/Pacific.

We will publish an analysis of this incident once we have completed our internal investigation.

We thank you for your patience while we worked on resolving the issue.