[Tokyo Region] Workflow performance degradation

Incident Report for Treasure Data

Resolved

Based on our investigation, a sudden increase in the load on the workflow cluster caused delays in the start and processing of workflow tasks.
The task delay started from 2022-10-11 07:20 UTC to 2022-10-11 10:15 UTC for about 3 hours.
This incident was resolved by increasing the capacity of the workflow cluster.

Posted Oct 11, 2022 - 03:20 PDT

Monitoring

Initial remediation was completed. Now we are monitoring continuously.

Posted Oct 11, 2022 - 02:50 PDT

Investigating

We are experiencing performance degradation from Workflow. The scheduled workflow and/or tasks can be delayed.
We are currently investigating the issue.

Posted Oct 11, 2022 - 01:06 PDT

This incident affected: Tokyo (Workflow).