[US Region] Job Queuing Issue
Incident Report for Treasure Data
Resolved
The issue of our partition metadata system has been fixed.
Posted 3 months ago. Jul 31, 2019 - 16:25 PDT
Update
Stream import API is working normally.
Posted 3 months ago. Jul 31, 2019 - 16:15 PDT
Update
We are continuing to monitor for any further issues.
Posted 3 months ago. Jul 31, 2019 - 16:03 PDT
Update
We are continuing to monitor for any further issues.
Posted 3 months ago. Jul 31, 2019 - 15:58 PDT
Monitoring
A fix has been made. We are checking affected components and adding more capacity for processing queued jobs.
Posted 3 months ago. Jul 31, 2019 - 15:49 PDT
Update
The system is now gradually recovering.
Posted 3 months ago. Jul 31, 2019 - 15:44 PDT
Update
We are continuing to work on a fix for this issue.
Posted 3 months ago. Jul 31, 2019 - 15:37 PDT
Update
We are applying a fix to the metadata system.
Posted 3 months ago. Jul 31, 2019 - 15:36 PDT
Identified
We've found performance degradation of our partition metadata system. It is causing delay of data import and query processing. We are now working on fixing the issue.
Posted 3 months ago. Jul 31, 2019 - 15:31 PDT
Update
We have an increased number of queued jobs and delay of data import. We are now investigating the cause.
Posted 3 months ago. Jul 31, 2019 - 15:25 PDT
Investigating
We are currently investigating this issue.
Posted 3 months ago. Jul 31, 2019 - 15:21 PDT
This incident affected: US (Streaming Import REST API, Hadoop / Hive Query Engine, Presto Query Engine, Presto JDBC/ODBC Gateway, Workflow).