From 7:50 AM PDT, events sent to Streaming Import were experiencing the delay before it become queryable. The maximum average delay was 71 minutes at 9:15 AM. By applying the remediation to the Backend system at 9:00 AM and with additional provisioned capacity, the delay was resolved at 10:20 AM.
Posted Jun 11, 2020 - 10:20 PDT
Monitoring
After the remediation, all of our systems are working as expected except for queued Streaming Import requests. We provisioned additional processing capacity to the production system. The current delay of Streaming Import is 50 minutes. We estimate the delay will be solved around 10:00 AM PDT.
We will send an additional update when the Streaming Import delay is resolved.
Posted Jun 11, 2020 - 09:42 PDT
Update
Our Backend team applied the remediation to the production system at 9:00 AM PDT. The delay of Streaming Import started shrinking. To accelerate the pace of the recovery, the team is provisioning additional processing capacity to the production system.
We will send an additional update in 30 minutes.
Posted Jun 11, 2020 - 09:13 PDT
Identified
Our Backend team identified the cause of core database bottleneck. The team is now evaluating a remediation at staging environment before applying it to the production system.
We will send an additional update when we apply the remediation to the production system.
Posted Jun 11, 2020 - 08:56 PDT
Investigating
We're experiencing an elevated delay of Streaming Import. Currently it takes approximately 30 minutes to process Streaming import records. Our Backend team is currently investigating issues related to Core database performance.
We will send an additional update in 30 minutes.
Posted Jun 11, 2020 - 08:26 PDT
This incident affected: US (Streaming Import REST API).