Elevated API error rate
Incident Report for Treasure Data
Resolved
This incident was resolved.

At 21:48 PDT we updated DB configuration for a part of API servers. The change was to isolate streaming import DB from query job DB at old streaming import API, but the wrong configuration caused DB connectivity issue to the old streaming import API. In addition to it the problem was propagated to other REST API. We reverted the change at 21:58 PDT and all problems were resolved. Queued streaming imports were already processed after 21:58 PDT. Please re-execute CLI when you observed API error 500 during the incident.

We are now working to isolate API clusters per purpose in this quarter to prevent this kind of error propagation. Please accept our apologies for any inconvenience this has caused.
Posted Apr 25, 2017 - 22:39 PDT
Monitoring
We found the wrong DB endpoint configuration in today's API release and reverted the application at 21:58 PDT. Now API is working normally.
Posted Apr 25, 2017 - 22:05 PDT
Identified
We are observing elevated API error rate from 21:50 PDT. We identified the cause and fixing now.
Posted Apr 25, 2017 - 21:57 PDT
This incident affected: US (REST API).