[Tokyo Region] Infrastructure trouble

Incident Report for Treasure Data

Resolved

This incident has been resolved.
Posted Aug 25, 2019 - 23:35 PDT

Update

The impaired 4 Datatanks were restored from backups. They have been back to normal.
Posted Aug 25, 2019 - 23:31 PDT

Update

The general incident is considered resolved but the functionalities for 4 customers' Datatanks remain impaired.

We continue to work on fully restoring the functionality and we will keep the incident open until that has happened.
Posted Aug 23, 2019 - 18:19 PDT

Update

The network issues in the underlying infrastructure have been resolved, and, as we have reported here already, the majority of affected machines have already been restored to normal status. We are going to close this incident as resolved as our overall system is operational now.
Posted Aug 23, 2019 - 05:23 PDT

Monitoring

The network issues in the underlying infrastructure have been mitigated and the majority of affected machines have been restored to normal status.
We continue to monitor for any outstanding issues.
Posted Aug 23, 2019 - 03:30 PDT

Identified

We have trouble on underlying infrastructure right now. So far, we've detected some of symptoms derived from the network issues, 1) delay in job executions, 2) some of DataTanks are experiencing connectivity issues
Posted Aug 22, 2019 - 22:33 PDT

Investigating

We are observing infrastructure trouble at some of our services. We are investigating.
Posted Aug 22, 2019 - 21:35 PDT
This incident affected: Tokyo (Hadoop / Hive Query Engine, Presto Query Engine).