[US Region] Job Queuing Issue
Incident Report for Treasure Data
Resolved
The issue of our partition metadata system has been fixed.
Posted Jul 31, 2019 - 16:25 PDT
Update
Stream import API is working normally.
Posted Jul 31, 2019 - 16:15 PDT
Update
We are continuing to monitor for any further issues.
Posted Jul 31, 2019 - 16:03 PDT
Update
We are continuing to monitor for any further issues.
Posted Jul 31, 2019 - 15:58 PDT
Monitoring
A fix has been made. We are checking affected components and adding more capacity for processing queued jobs.
Posted Jul 31, 2019 - 15:49 PDT
Update
The system is now gradually recovering.
Posted Jul 31, 2019 - 15:44 PDT
Update
We are continuing to work on a fix for this issue.
Posted Jul 31, 2019 - 15:37 PDT
Update
We are applying a fix to the metadata system.
Posted Jul 31, 2019 - 15:36 PDT
Identified
We've found performance degradation of our partition metadata system. It is causing delay of data import and query processing. We are now working on fixing the issue.
Posted Jul 31, 2019 - 15:31 PDT
Update
We have an increased number of queued jobs and delay of data import. We are now investigating the cause.
Posted Jul 31, 2019 - 15:25 PDT
Investigating
We are currently investigating this issue.
Posted Jul 31, 2019 - 15:21 PDT
This incident affected: US (Streaming Import REST API, Hadoop / Hive Query Engine, Presto Query Engine, Presto JDBC/ODBC Gateway, Workflow).