Hive query jobs are now being processed at a normal rate, the system is restored to normal operations.
Between Monday, 21 JUN 2021 13:10 PDT to Monday, 21 JUN 2021 14:01 PDT, all customers experienced a delay in hive query processing related to a Hadoop cluster outage. We will be carrying out a root cause analysis to determine the source of the cluster outage.
We apologize for any inconvenience caused. If you have any questions about it, please contact support@treasure-data.com
Posted Jun 21, 2021 - 14:04 PDT
Monitoring
Hadoop team have created a new cluster and we are seeing jobs starting to be processed.
We are monitoring to ensure the backlog of queries begin to be processed by the new cluster.
We will monitor the new cluster and we expect to see recovery to normal operations in about 30 minutes.
Posted Jun 21, 2021 - 13:41 PDT
Identified
We're experiencing a backlog of hive query processing due to a cluster outage.
Login is unaffected. Hive query jobs are delayed.
Our Hadoop team are working to resolve the incident by replacing the cluster, which will be followed by a root cause analysis to determine the cause of the cluster outage.
At present all users in Korea region are affected.
We expect to see the results of the new cluster spin-up within 30 minutes.
Posted Jun 21, 2021 - 13:35 PDT
This incident affected: Korea (Hadoop / Hive Query Engine).