[US Region] Hadoop performance degradation

Incident Report for Treasure Data

Resolved

All jobs have been redirected to an unaffected cluster. Queries are progressing as normal.

Posted Mar 22, 2018 - 00:26 PDT

Update

We are decommissioning the affected cluster and forcing jobs to retry on an unaffected cluster to improve performance.

Posted Mar 22, 2018 - 00:17 PDT

Identified

We identified that one of the new Hadoop clusters created at 22:30 shows slow performance. At 23:37 we changed the Hive job routing to use the previous cluster.

Posted Mar 21, 2018 - 23:52 PDT

Investigating

We are experiencing Hadoop performance degradation from 22:30 PDT.

Posted Mar 21, 2018 - 23:32 PDT

This incident affected: US (Hadoop / Hive Query Engine).