Treasure Data
All Systems Operational

About This Site

This is Treasure Data's status page. Our belief: "Trust starts with full transparency".

Web Interface Operational
REST API Operational
REST API (Import) Operational
Event Collector Operational
Presto Operational
Operational
Degraded Performance
Partial Outage
Major Outage
System Metrics Month Week Day
REST API Response Time ?
Fetching
REST API (Import) Response Time ?
Fetching
REST API Error Rates ?
Fetching
Web Interface Response Time ?
Fetching
Number of Queued Imports ?
Fetching
Past Incidents
Feb 28, 2015

No incidents reported today.

Feb 27, 2015

No incidents reported.

Feb 26, 2015

No incidents reported.

Feb 25, 2015
Resolved - Now the problem was completely solved. Sorry for your inconvenience, and thank you for your patience.
Feb 25, 08:40 PST
Monitoring - We have created the patch, reviewed, tested, and deployed into production. The performance problem we've identified was solved, but we'll keep monitoring the system.
Feb 25, 08:09 PST
Update - As a next approach, we'll modify Web interface application to issue the different SQL query. This will allow MySQL optimizer to user appropriate index. We're testing the change in staging environment.
Feb 25, 07:30 PST
Investigating - We have tried to add index with online schema migration tool (pt-online-schema-change), but it locked the table for 2 minutes. We cancelled the operation at this point.
Feb 25, 07:03 PST
Update - We've identified what index we need to add to the database. We'll add the index to the table, which contains 20 million rows. This could cause read-lock for a while for API calls involving jobs as well. We'll keep you posted.
Feb 25, 06:49 PST
Identified - Starting from around 1AM PST, we started observing MySQL queries not using the index. These queries scan the entire table, which stores multiple gigabytes of entries. This started causing the performance degradation of our Web interface. We're working on resolving the issue. At this point, we do NOT have performance impacts on API side.
Feb 25, 01:33 PST
Feb 24, 2015

No incidents reported.

Feb 23, 2015

No incidents reported.

Feb 22, 2015

No incidents reported.

Feb 21, 2015
Resolved - This incident has been resolved.
Feb 21, 14:59 PST
Monitoring - One of our MySQL databases failed over to a slave around 2:07pm PST, and our API servers and the Console were unavailable for 1 or 2 minutes. The service is now operating normally. If you're using the latest Treasure Agent / Bulk Loader / JDBC Driver / Java REST API libray, the underlying mechanism does the retry properly. We're monitoring the system carefully.
Feb 21, 14:31 PST
Feb 20, 2015

No incidents reported.

Feb 19, 2015
Completed - The scheduled maintenance has been completed.
Feb 19, 23:00 PST
Verifying - The database rebooted successfully, but the down time of our console was longer than expected. The actual down time of the API endpoint was 2 - 3 mins and the Console was 9 mins. Currently all of our services are operating normally.
Feb 19, 22:32 PST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Feb 19, 22:25 PST
Scheduled - Our infrastructure provider announced that one of our core databases needs to be rebooted at the following time range, to mitigate the potential security risk. We could have 1 - 2 minutes of API access downtime during that time period.

Feb. 19 10:00pm PST - 11:00pm PST

If you're using Teasure Agent / Bulk Loader / JDBC Driver / Java REST API libray, the underlying mechanism does the retry properly.
Feb 18, 04:24 PST
Feb 18, 2015

No incidents reported.

Feb 17, 2015

No incidents reported.

Feb 16, 2015

No incidents reported.

Feb 15, 2015

No incidents reported.

Feb 14, 2015

No incidents reported.