[US Region] Hadoop query cluster degradation
Incident Report for Treasure Data
Resolved
Hive jobs are now being processed as normally by new cluster.
Posted Oct 21, 2018 - 21:42 PDT
Update
We are continuing to monitor for any further issues.
Posted Oct 21, 2018 - 21:17 PDT
Monitoring
A replacement cluster started and is processing queries now. We're closely monitoring the new cluster if it's working as it should be.
Posted Oct 21, 2018 - 21:15 PDT
Identified
At 12:36 JST / 03:36 UTC a single cluster in the US Region began experiencing degraded performance.
At 12:42 JST / 03:42 UTC this cluster experienced a failure and new query processing stopped.
At 12:52 JST / 03:52 UTC a replacement cluster was launched.
We expect to fully shift affected queries to the replacement cluster shortly.
At 13:11 JST / 04:11 UTC a replacement cluster has started processing queries. We're closely monitoring the new cluster if it's working as it should be.
Posted Oct 21, 2018 - 21:01 PDT
Investigating
We are currently investigating issues with a single cluster in the US Region. We will update shortly.
Posted Oct 21, 2018 - 20:54 PDT
This incident affected: US (Hadoop / Hive Query Engine).