[US Region] Hive job on one Hadoop cluster slow down

Incident Report for Treasure Data

Resolved

From 1:14am PT (5:14pm JST) to 2:30am PT (6:30pm JST), Hive jobs took longer than usual. We confirmed our system is operating normally. This incident was resolved.

Posted May 14, 2019 - 02:41 PDT

Monitoring

All Hive jobs are running normally on the new cluster. Currently, our system is operating normally. We are monitoring the situation.

Posted May 14, 2019 - 02:30 PDT

Identified

We have identified the master node of one of our Hadoop clusters has been dead. We are working on replacing the cluster. Once we create the new cluster, new cluster will consume Hive jobs.

Posted May 14, 2019 - 01:34 PDT

Investigating

We are investigating the issue where Hive job on one of our Hadoop clusters slows down.

Posted May 14, 2019 - 01:30 PDT

This incident affected: US (Hadoop / Hive Query Engine).