From 1:30 AM PDT (5:30 PM JST), Result Export processing workers were under disk capacity saturation. Some customers who were running Result Export job experienced slower job execution after that. At 2:30 AM PDT (6:30 AM JST), we implemented the fix. At 3:00 AM PDT (7:00 AM JST), the system finished catching up the queued Result Export jobs.
The incident is resolved.
Posted Jul 21, 2020 - 03:11 PDT
Monitoring
We have applied the fix and the system gets back to normal. We are now monitoring closely.
Posted Jul 21, 2020 - 02:38 PDT
Identified
We identify the cause of the problem, and we are working on applying a fix for this Result export queuing issue to improve our Result Export cluster performance now.
Posted Jul 21, 2020 - 02:28 PDT
Investigating
We are investigating a possible system problem currently affecting Result Export job. Any customer impact has not been determined at this time, but we will provide an update as soon as we know more.