All Systems Operational

About This Site

This is Treasure Data's status page.
We believe that trust starts with full transparency.

US Operational
90 days ago
99.99 % uptime
Today
Web Interface   Operational
90 days ago
100.0 % uptime
Today
REST API   Operational
90 days ago
100.0 % uptime
Today
Streaming Import REST API   Operational
90 days ago
100.0 % uptime
Today
Mobile/Javascript REST API   Operational
90 days ago
100.0 % uptime
Today
Data Connector Integrations   Operational
90 days ago
100.0 % uptime
Today
Hadoop / Hive Query Engine   Operational
90 days ago
100.0 % uptime
Today
Presto Query Engine   Operational
90 days ago
100.0 % uptime
Today
Presto JDBC/ODBC Gateway   Operational
90 days ago
99.98 % uptime
Today
Workflow   ? Operational
90 days ago
100.0 % uptime
Today
Tokyo Operational
90 days ago
100.0 % uptime
Today
Web Interface   Operational
90 days ago
100.0 % uptime
Today
REST API   Operational
90 days ago
100.0 % uptime
Today
Streaming Import REST API   Operational
90 days ago
100.0 % uptime
Today
Mobile/Javascript REST API   Operational
90 days ago
100.0 % uptime
Today
Data Connector Integrations   Operational
90 days ago
100.0 % uptime
Today
Hadoop / Hive Query Engine   Operational
90 days ago
100.0 % uptime
Today
Presto Engine Query Engine   Operational
90 days ago
100.0 % uptime
Today
Presto JDBC/ODBC Gateway   Operational
90 days ago
100.0 % uptime
Today
Workflow   ? Operational
90 days ago
100.0 % uptime
Today
Global Operational
90 days ago
100.0 % uptime
Today
Reporting   ? Operational
90 days ago
100.0 % uptime
Today
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Scheduled Maintenance
Database Upgrades Oct 23, 18:00-18:15 PDT
On Monday, October 23rd starting at 6 PM PDT (October 24th starting at 10 AM JST), we will be performing the upgrade of some of our databases.

The upgrade is necessary to bring our production databases to the latest and greatest version supported (MySQL v5.7.19), allowing us to stay on par with the latest improvements, fixes, and security patches.

The procedure will **last up to 10 minutes**: during this time, our REST API will primarily be unavailable and respond with error codes 500 or similar. In particular:

* The Web Console will not be functional.

* Streaming import requests will fail:
Where fluentd / td-agent is being used (as recommended), event collection will continue locally on each device/server and will recover automatically once the maintenance is complete due to the built-in buffering and retry mechanisms.

* Javascript SDK and Mobile SDK (Android, iOS, and Unity) event collection will continue undisturbed but events will not be available for querying until the maintenance is completed.

* Real-time CDP event ingestion will continue undisturbed but events will be delayed and become available only shortly after the maintenance is completed.

* td command­-line (CLI) commands will encounter failures and execution will be delayed until the maintenance is complete. In some cases, the commands will be returning errors.

* Scheduled queries and connector jobs will be delayed. Already running scheduled jobs will be completed successfully or retried internally until they are.

* Workflows using Treasure Data operators (e.g. td>) will retry and regain full functionalities again after the upgrade. In case of workflow sessions failures, the customer could elect to resume them manually.

* The Presto JDBC / ODBC Gateway will report authentication failures.

Beyond this notice, we will provide updates approximately 1 hour before the beginning of the upgrade window, at the start and completion of the operation, and once the verification is completed. At that time, all systems will have returned to full functionality and the Scheduled Maintenance will be closed.

If you have any question or concern about this upgrade, please feel free to reach out to our Support team at support@treasuredata.com.
Posted on Oct 16, 21:53 PDT
System Metrics Month Week Day
US - REST API - Response Time ?
Fetching
US - REST API - Error Rates ?
Fetching
US - Streaming Import REST API - Response Time ?
Fetching
US - Queued Streaming Import requests ?
Fetching
US - Mobile/Javascript REST API - Response Time ?
Fetching
US - Web Interface - Response Time ?
Fetching
Tokyo - REST API - Response Time ?
Fetching
Tokyo - Streaming Import REST API - Response Time ?
Fetching
Tokyo - Queued Streaming Import requests ?
Fetching
Tokyo - Web Interface - Response Time ?
Fetching
Past Incidents
Oct 21, 2017
Resolved - The API response has been healthy for the past hour.
We are closing this incident accordingly.
Oct 21, 12:16 PDT
Monitoring - The additional capacity we brought online was sufficient to mitigate the access flood we detected and at the moment functionality and performance is restored to normal levels.
We will be monitoring the performance for the next 30 minutes before closing the incident.
Oct 21, 11:21 PDT
Update - We deployed additional API servers at 10:00, which should reduce the API error rate. We are still working for fixing the original cause.
Oct 21, 10:20 PDT
Identified - We identified the cause of elevated API error rate. We are working for the fix.
Oct 21, 09:49 PDT
Investigating - We are observing elevated error rate from 9:00 am PDT. We are investigating the trouble.
Oct 21, 09:18 PDT
Oct 20, 2017

No incidents reported.

Oct 19, 2017
Completed - The Treasure Data Reporting system has been upgraded and verified.

Access to the system is fully restored, including Single Sign On from the TD Console (available to TD account owners only).

The upgrade process took 63 minutes in total.

Took a few minutes more due to a misconfiguration of the reporting server.
Oct 19, 23:08 PDT
Verifying - The upgrade of the Treasure Data Reporting system is now complete. We are now monitoring the upgraded system and verifying access and functionalities.

We have restored *direct access* to the system and will restore Single Sign On from the TD Console shortly.
Oct 19, 22:51 PDT
In progress - The upgrade of the Treasure Data Reporting system is starting now.

The upgrade process is expected to take approximately 1 hour and last until 11 PM PDT or 3 PM JST. Until then:

* Single Sign On (SSO) from the TD Console to the Reporting site will be disabled for all
* Access to the reporting.treasuredata.com site will be mostly unavailable or spotty at best
Oct 19, 22:05 PDT
Update - In about 40 minutes (at 10 PM PDT or 2 PM JST), we will start the upgrade of the Treasure Data Reporting system.

As a reminder (from the previous communications), during this time:

* Single Sign On (SSO) from the TD Console to the Reporting site will be disabled for all
* Access to the reporting.treasuredata.com site will be mostly unavailable or spotty at best

Beyond this notice, we will provide updates at the start and completion of the operation and once the verification of the new system is completed: at that time, all systems will have returned to full functionality and the Scheduled Maintenance will be closed.
Oct 19, 21:20 PDT
Scheduled - On Thursday, October 19th PDT, the Treasure Data Reporting system will be upgraded.
The upgrade will be performed between 10 and 11 PM PDT (October 20th between 2 and 3 PM JST).

During this time:
* Single Sign On (SSO) from the TD Console to the Reporting site will be disabled for all
* Access to the reporting.treasuredata.com site will be mostly unavailable or spotty at best

Under the covers, this planned upgrade will bring many enhanced and new functionalities to the TD Reporting system, powered by Slemma https://slemma.com/:
* Overall UI redesign for better usability
* Added the ability to share Dashboard templates
* Added ‘Dynamic filtering’ capability to Dashboards, allowing the same dashboard to be shared, customized, and reutilized for multiple people and teams based on custom fields, configurable on a per-user or per-group basis
* Better User management control
* Many new Data Sources and integration capabilities will become gradually available over time

For more detailed information please refer to the Slemma Release Notes https://support.slemma.com/knowledge_base/topics/release-notes/ (upgrading from version v7.20 to v8.19).

Beyond this notice, we will provide updates approximately 1 hour before the beginning of the upgrade window, at the start and completion of the operation, and once the verification of the new system is completed. At that time, all systems will have returned to full functionality and the Scheduled Maintenance will be closed.

If you have any question or concern about this upgrade, please feel free to contact our Support team at support@treasuredata.com.
Oct 5, 21:40 PDT
Resolved - The service is now operating normally. We apologize any inconvenience caused by the issue.
Oct 19, 03:06 PDT
Monitoring - Memory starvation on some Presto worker nodes was observed, and additional capacity was brought.
Oct 19, 02:29 PDT
Update - We are currently investigating this issue.
Oct 19, 02:09 PDT
Investigating - We are investigating query delays in one presto cluster in our US region.
Oct 19, 02:02 PDT
Oct 18, 2017
Resolved - Incident has been resolved. Presto JDBC/ODBC gateway returned 5xx errors while presto cluster migration was in progress. Some in-progress queries during the migration failed and should be retried by clients.
Oct 18, 22:59 PDT
Investigating - Jobs through Presto ODBC gateway fails. We are now investigating the cause.
Oct 18, 21:48 PDT
Oct 17, 2017

No incidents reported.

Oct 16, 2017
Resolved - The fix has been rolled out to production and verified. This incident has been resolved.
Oct 16, 10:59 PDT
Monitoring - We applied a fix to address the issues with saved queries again.
Oct 16, 10:52 PDT
Identified - This is a follow-up notification for the previous status page: https://status.treasuredata.com/incidents/4ty27p7rrzq9
We found that the previous fix had another problem. We rollbacked the fix and re-implementing the fix.
Oct 16, 09:25 PDT
Resolved - The temporary fix has been rolled out to production and verified.
While we continue to work on a more long term solution for the problems, we are going to call this incident resolved.
Oct 16, 08:52 PDT
Update - We applied a temporary fix to address the issues with saved queries and will be rolling it out to production shortly.
Oct 16, 08:39 PDT
Identified - We identified the problem and we are looking for a proper fix.

Since resolving this issue will likely take a while, we suggest using the legacy Console's query editor to author and save queries at this point:
https://console.treasuredata.com/query_forms/new
https://console.treasuredata.co.jp/query_forms/new
Oct 16, 02:13 PDT
Investigating - We are observing issues with saving queries when inputted in the Console SQL editor.
Oct 16, 00:09 PDT
Oct 15, 2017

No incidents reported.

Oct 14, 2017

No incidents reported.

Oct 13, 2017

No incidents reported.

Oct 12, 2017

No incidents reported.

Oct 11, 2017

No incidents reported.

Oct 10, 2017

No incidents reported.

Oct 9, 2017

No incidents reported.

Oct 8, 2017

No incidents reported.

Oct 7, 2017

No incidents reported.