All Systems Operational

About This Site

This is Treasure Data's status page.
We believe that trust starts with full transparency.

US Operational
90 days ago
100.0 % uptime
Today
Web Interface   Operational
90 days ago
100.0 % uptime
Today
REST API   Operational
90 days ago
100.0 % uptime
Today
Streaming Import REST API   Operational
90 days ago
100.0 % uptime
Today
Mobile/Javascript REST API   Operational
90 days ago
100.0 % uptime
Today
Data Connector Integrations   Operational
90 days ago
100.0 % uptime
Today
Hadoop / Hive Query Engine   Operational
90 days ago
100.0 % uptime
Today
Presto Query Engine   Operational
90 days ago
100.0 % uptime
Today
Presto JDBC/ODBC Gateway   Operational
90 days ago
100.0 % uptime
Today
Workflow   ? Operational
90 days ago
100.0 % uptime
Today
Tokyo Operational
90 days ago
100.0 % uptime
Today
Web Interface   Operational
90 days ago
100.0 % uptime
Today
REST API   Operational
90 days ago
100.0 % uptime
Today
Streaming Import REST API   Operational
90 days ago
100.0 % uptime
Today
Mobile/Javascript REST API   Operational
90 days ago
100.0 % uptime
Today
Data Connector Integrations   Operational
90 days ago
100.0 % uptime
Today
Hadoop / Hive Query Engine   Operational
90 days ago
100.0 % uptime
Today
Presto Engine Query Engine   Operational
90 days ago
100.0 % uptime
Today
Presto JDBC/ODBC Gateway   Operational
90 days ago
100.0 % uptime
Today
Workflow   ? Operational
90 days ago
100.0 % uptime
Today
Global Operational
90 days ago
100.0 % uptime
Today
Reporting   ? Operational
90 days ago
100.0 % uptime
Today
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
US - REST API - Response Time ?
Fetching
US - REST API - Error Rates ?
Fetching
US - Streaming Import REST API - Response Time ?
Fetching
US - Queued Streaming Import requests ?
Fetching
US - Mobile/Javascript REST API - Response Time ?
Fetching
US - Web Interface - Response Time ?
Fetching
Tokyo - REST API - Response Time ?
Fetching
Tokyo - Streaming Import REST API - Response Time ?
Fetching
Tokyo - Queued Streaming Import requests ?
Fetching
Tokyo - Web Interface - Response Time ?
Fetching
Past Incidents
May 20, 2018

No incidents reported today.

May 19, 2018

No incidents reported.

May 18, 2018

No incidents reported.

May 17, 2018
Resolved - The service is now operating normally.
May 17, 01:54 PDT
Monitoring - We found some query tasks that are causing major GCs. We have fixed the issue. Now we are monitoring the service.
May 17, 01:39 PDT
Identified - we are investigating increased failure rates in a single presto compute cluster in our US region.
May 16, 23:56 PDT
May 16, 2018
Resolved - Clusters are back to normal state.
May 16, 01:40 PDT
Update - Presto clusters are gradually recovering to a normal state. We will keep monitoring.
May 16, 01:27 PDT
Monitoring - We have increased the capacity of Presto clusters. Keep monitoring
May 16, 01:20 PDT
Identified - We have distributed the query workload to mitigate the impact of memory consuming queries.
May 16, 01:06 PDT
Update - We detected one of our Presto clusters is having frequent major GCs. We are now working on the fix.
May 16, 00:51 PDT
Investigating - We have detected an increase of internal failure rate of Presto queries. Investigating.
May 16, 00:25 PDT
May 15, 2018

No incidents reported.

May 14, 2018

No incidents reported.

May 13, 2018

No incidents reported.

May 12, 2018

No incidents reported.

May 11, 2018

No incidents reported.

May 10, 2018

No incidents reported.

May 9, 2018

No incidents reported.

May 8, 2018
Completed - *Maintenance* and system *recovery* processes have been fully executed.
Streaming import and job execution delay was around 9 minutes from 6:19 pm PDT.

The scheduled Metadata Database maintenance is now complete.
May 8, 18:45 PDT
Verifying - The scheduled Metadata Database maintenance is complete.

We are monitoring the system closely to ensure all systems successfully complete their *recovery* and return to full functionality and throughput as quickly as possible.
May 8, 18:33 PDT
In progress - The scheduled Metadata Database maintenance is starting now.

The database will become unreachable for the duration of the *maintenance* procedure, which should last no longer than 15 minutes. We expect this to be followed by a *recovery* period of around 15 minutes during which the system will gradually reach back to full throughput. During the maintenance and recovery, customers may experience delays with Streaming (td-agent, fluentd) imports, Mobile (Android, iOS, Unity SDK) imports, Browser (JavaScript SDK) imports, and job execution as well as Presto JDBC / ODBC request failures.
May 8, 18:15 PDT
Update - In about an hour, at 6:15 PM PDT (10:15 AM JST), we will start the scheduled maintenance on our PlazmaDB Metadata database.

The maintenance is necessary to upgrade the PostgreSQL database to keep up with feature advancement and take advantage of the latest performance improvements.

The database will become unreachable for the duration of the *maintenance* procedure, which should last no longer than 15 minutes. We expect this to be followed by a *recovery* period of around 15 minutes during which the system will gradually reach back to full throughput. During the maintenance and recovery, customers may experience the following:

* Streaming, Mobile, and JavaScript/Browser imports delay
Streaming import (through td-agent or fluentd) requests will continue to be accepted as usual but the requests will remain queued until after the database maintenance is complete. We expect stream import processing to be further delayed during recovery. Thus we expect the stream import delay to extend to up to 30 minutes.
The same will apply to import requests from Browsers (Javascript SDK) and Mobiles (Android, iOS, and Unity SDKs).

* Jobs execution delay
All jobs (Presto, Hive, Result Export, Data Connector, Bulk Import, Export, and Partial Delete jobs submitted from Console, API, Workflow or triggered by our system according to the configured schedule) will remain queued during maintenance. During recovery, we expect jobs to begin processing slowly: within 30 minutes job processing should reach back to full throughput.

* Presto JDBC / ODBC Gateway errors
The Presto JDBC / ODBC Gateway will report errors during maintenance due to the unreachability of the Metadata database: errors will be propagated to the clients. We expect processing of Presto JDBC / ODBC jobs to follow the same recovery pattern as all other jobs (see above).

Beyond this notice, we will provide updates at the start and completion of the operation and once the verification has completed: at that time, all systems will have returned to full functionality and this Scheduled Maintenance will be closed.
May 8, 17:07 PDT
Scheduled - On Tuesday, May 8th from 6 to 7 PM PDT (May 9th from 10 to 11 AM JST) we will be performing maintenance on the PlazmaDB Metadata database.

The maintenance is necessary to upgrade the PostgreSQL database to keep up with feature advancement and take advantage of the latest performance improvements.

# Impact

NOTE: This maintenance notice only interests *AWS customers using the US region* and not AWS customers using the Tokyo region.

The database will become unreachable for the duration of the *maintenance* procedure, which should last no longer than 15 minutes. We expect this to be followed by a *recovery* period of around 15 minutes during which the system will gradually reach back to full throughput.

During the maintenance and recovery, customers may experience the following:

* Streaming, Mobile, and JavaScript/Browser imports delay
Streaming import (through td-agent or fluentd) requests will continue to be accepted as usual but the requests will remain queued until after the database maintenance is complete. We expect stream import processing to be further delayed during recovery. Thus we expect the stream import delay to extend to up to 30 minutes.
The same will apply to import requests from Browsers (Javascript SDK) and Mobiles (Android, iOS, and Unity SDKs).

* Jobs execution delay
All jobs (Presto, Hive, Result Export, Data Connector, Bulk Import, Export, and Partial Delete jobs submitted from Console, API, Workflow or triggered by our system according to the configured schedule) will remain queued during maintenance. During recovery, we expect jobs to begin processing slowly: within 30 minutes job processing should reach back to full throughput.

* Presto JDBC / ODBC Gateway errors
The Presto JDBC / ODBC Gateway will report errors during maintenance due to the unreachability of the Metadata database: errors will be propagated to the clients. We expect processing of Presto JDBC / ODBC jobs to follow the same recovery pattern as all other jobs (see above).

# Communication

Beyond this notice, we will provide updates approximately 1 hour before the beginning of the maintenance window, at the start and completion of the operation, and once the verification is completed. At that time, all systems will have returned to full functionality and the Scheduled Maintenance will be closed.

If you have any question or concern about this upgrade, please feel free to reach out to our Support team at support@treasuredata.com.
Apr 6, 13:42 PDT
Completed - *Maintenance* and system *recovery* processes have been fully executed.
Streaming import and job execution delay was less than 1 minute at 6:26 pm PDT.

The scheduled Metadata Database maintenance is now complete.
May 8, 18:42 PDT
Verifying - The scheduled Metadata Database maintenance is complete.

We are monitoring the system closely to ensure all systems successfully complete their *recovery* and return to full functionality and throughput as quickly as possible.
May 8, 18:34 PDT
In progress - The scheduled Metadata Database maintenance is starting now.

We expect the operation to cause *no downtime*: the database will be online after the upgrade within less than 2 minutes. This will be followed by a *recovery* period of around 5 minutes during which the system will gradually reach back to full throughput. During the maintenance and recovery, customers may experience delays with Streaming (td-agent, fluentd) imports, Mobile (Android, iOS, Unity SDK) imports, Browser (JavaScript SDK) imports, and job execution as well as Presto JDBC / ODBC request failures.
May 8, 18:15 PDT
Update - In about an hour, at 6:15 PM PDT (10:15 AM JST), we will start the scheduled maintenance on our PlazmaDB Metadata database.

The maintenance is necessary to upgrade the PostgreSQL database engine with the latest patches (including security related ones).

We expect the operation to cause *no downtime*: the database will be online after the upgrade within less than 2 minutes. This to be followed by a *recovery* period of around 5 minutes during which the system will gradually reach back to full throughput. During the maintenance and recovery, customers may experience the following:

* Streaming, Mobile, and JavaScript/Browser imports delay
Streaming import (through td-agent or fluentd) requests will continue to be accepted as usual but the requests will remain queued until after the database maintenance is complete. We expect stream import processing to be further delayed during recovery. Thus we expect the stream import delay to extend to up to 10 minutes.
The same will apply to import requests from Browsers (Javascript SDK) and Mobiles (Android, iOS, and Unity SDKs).

* Jobs execution delay
All jobs (Presto, Hive, Result Export, Data Connector, Bulk Import, Export, and Partial Delete jobs submitted from Console, API, Workflow or triggered by our system according to the configured schedule) will remain queued during maintenance. During recovery, we expect jobs to begin processing slowly: within 10 minutes job processing should reach back to full throughput.

* Presto JDBC / ODBC Gateway errors
The Presto JDBC / ODBC Gateway will report errors during maintenance due to the unreachability of the Metadata database: errors will be propagated to the clients. We expect processing of Presto JDBC / ODBC jobs to follow the same recovery pattern as all other jobs (see above).

Beyond this notice, we will provide updates at the start and completion of the operation and once the verification has completed: at that time, all systems will have returned to full functionality and this Scheduled Maintenance will be closed.
May 8, 17:08 PDT
Scheduled - On Tuesday, May 8th from 6 to 7 PM PDT (May 9th from 10 to 11 AM JST) we will be performing maintenance on the PlazmaDB Metadata database.

The maintenance is necessary to upgrade the PostgreSQL database engine with the latest patches (including security related ones).

We expect the operation to cause *no downtime*: the database will be online after the upgrade within less than 2 minutes.

# Impact

NOTE: This maintenance notice only interests *AWS customers using the Tokyo region* and not AWS customers using the US region.

The database will become unreachable for the duration of the *maintenance* procedure, which should last no longer than 2 minutes. We expect this to be followed by a *recovery* period of around 5 minutes during which the system will gradually reach back to full throughput.

All components reading/writing the Metadata Database have built-in fault tolerance that will allow them to retry the request in case of failures: when the connection incurs into an error, the request is retried several times and for long enough to ensure maintenance and recovery periods are successfully completed.

During the maintenance and recovery, customers may experience bie following:

* Streaming, Mobile, and JavaScript/Browser imports delay
Streaming import (through td-agent or fluentd) requests will continue to be accepted as usual but the requests will remain queued until after the database maintenance is complete. We expect stream import processing to be further delayed during recovery. Thus we expect the stream import delay to extend to up to 10 minutes.
The same will apply to import requests from Browsers (Javascript SDK) and Mobiles (Android, iOS, and Unity SDKs).

* Jobs execution delay
All jobs (Presto, Hive, Result Export, Data Connector, Bulk Import, Export, and Partial Delete jobs submitted from Console, API, Workflow or triggered by our system according to the configured schedule) will remain queued during maintenance. During recovery, we expect jobs to begin processing slowly: within 10 minutes job processing should reach back to full throughput.

* Presto JDBC / ODBC Gateway errors
The Presto JDBC / ODBC Gateway will report errors during maintenance due to the unreachability of the Metadata database: errors will be propagated to the clients. We expect processing of Presto JDBC / ODBC jobs to follow the same recovery pattern as all other jobs (see above).

# Communication

Beyond this notice, we will provide updates approximately 1 hour before the beginning of the maintenance window, at the start and completion of the operation, and once the verification is completed. At that time, all systems will have returned to full functionality and the Scheduled Maintenance will be closed.

If you have any question or concern about this upgrade, please feel free to reach out to our Support team at support@treasuredata.com.
Apr 6, 13:57 PDT
May 7, 2018

No incidents reported.

May 6, 2018

No incidents reported.