Long queueing time of Presto queries

Incident Report for Treasure Data

Resolved

The incident has been resolved.

Posted Sep 26, 2016 - 12:15 PDT

Monitoring

Queries are now being processed gradually. We will continue monitoring.

Posted Sep 26, 2016 - 09:59 PDT

Identified

We have identified the cause; Presto queries that require large amount of memory have been blocked because of the increase of memory consuming queries. This type of queries often include count(distinct x), order by, UNION (duplicate elimination), etc. These operations cannot be distributed well and end up consuming single node memory. For the cluster stability, we are limiting the number of such memory consuming jobs that can run at the same time.

If you have noticed any delay of query execution, please refer to the following guidelines for reducing the memory usage:
https://docs.treasuredata.com/articles/presto-query-faq#q-exceeded-max-local-memory-xxgb-error

Posted Sep 26, 2016 - 09:25 PDT

Investigating

Some presto queries are queued for a long time. We are investigating the cause.

Posted Sep 26, 2016 - 09:03 PDT