API and Import outage

Incident Report for Treasure Data

Postmortem

The incident was caused by unexpected table lock by one-time operation tool. We fixed the tool and made sure that the same trouble not happen.

Streaming import via td-agent and SDKs were recovered and the event sent to Treasure Data at that time were properly imported after API server restored. In terms of bulk import via td command, uploads should have been properly uploaded and imported as well but if you find that your import session is not yet performed or committed, you can re-submit 'perform' and/or 'commit' command to restore the import.

Posted Apr 26, 2015 - 20:45 PDT

Resolved

API access and Imports have been working fine for the past hour so we are marking this incident as resolved.

Posted Apr 24, 2015 - 14:04 PDT

Monitoring

The API servers seem to have recovered gradually from this outage.
The performance is still not optimal but API and Import access is recovered now.

Posted Apr 24, 2015 - 12:16 PDT

Investigating

We are experience an outage for our API servers.
All API accesses and imports are currently not available.
We are investigating and working to mitigate the problem as soon as possible and avoid any loss of data.

Posted Apr 24, 2015 - 12:12 PDT