[US Region] Presto Queueing Issue

Incident Report for Treasure Data

Resolved

This incident has been resolved.
Posted Sep 19, 2019 - 18:04 PDT

Update

We have applied a fix. The number of queued queries is decreasing.
Posted Sep 19, 2019 - 17:23 PDT

Update

A limited number of accounts still have this issue. We are working on the fix.
Posted Sep 19, 2019 - 16:52 PDT

Update

We are continuing to monitor for any further issues.
Posted Sep 19, 2019 - 15:35 PDT

Update

We will keep monitoring the system.
Posted Sep 19, 2019 - 15:32 PDT

Monitoring

We have applied a fix to this issue.
Posted Sep 19, 2019 - 15:23 PDT

Update

We have increased the resources for processing queued queries. We are also trying to fix an issue around result export task. Currently, some users who have large number of queued jobs cannot start new result export processes.
Posted Sep 19, 2019 - 15:13 PDT

Identified

We have identified the cause of the queueing issue. Queries that use result export functionality are taking long time to finish. We are working on the fix.
Posted Sep 19, 2019 - 15:02 PDT

Update

We found an issue around result export tasks. Still investigating the cause.
Posted Sep 19, 2019 - 14:56 PDT

Investigating

We have observed an increased number of queued queries for Presto in our system. Investigating the cause.
Posted Sep 19, 2019 - 14:48 PDT
This incident affected: US (Presto Query Engine).