All Systems Operational

About This Site

This is Arm Treasure Data's status page.
We believe that trust starts with full transparency.

Uptime over the past 90 days. View historical uptime.
US ? Operational
90 days ago
99.99 % uptime
Today
Web Interface Operational
90 days ago
99.99 % uptime
Today
REST API Operational
90 days ago
99.98 % uptime
Today
Streaming Import REST API Operational
90 days ago
99.98 % uptime
Today
Mobile/Javascript REST API Operational
90 days ago
100.0 % uptime
Today
Data Connector Integrations Operational
90 days ago
99.98 % uptime
Today
Hadoop / Hive Query Engine Operational
90 days ago
99.99 % uptime
Today
Presto Query Engine Operational
90 days ago
99.99 % uptime
Today
Presto JDBC/ODBC Gateway Operational
90 days ago
100.0 % uptime
Today
Workflow ? Operational
90 days ago
100.0 % uptime
Today
Tokyo ? Operational
90 days ago
99.99 % uptime
Today
Web Interface Operational
90 days ago
100.0 % uptime
Today
REST API Operational
90 days ago
100.0 % uptime
Today
Streaming Import REST API Operational
90 days ago
100.0 % uptime
Today
Mobile/Javascript REST API Operational
90 days ago
100.0 % uptime
Today
Data Connector Integrations Operational
90 days ago
99.98 % uptime
Today
Hadoop / Hive Query Engine Operational
90 days ago
100.0 % uptime
Today
Presto Query Engine Operational
90 days ago
99.99 % uptime
Today
Presto JDBC/ODBC Gateway Operational
90 days ago
99.99 % uptime
Today
Workflow ? Operational
90 days ago
100.0 % uptime
Today
EU ? Operational
90 days ago
99.99 % uptime
Today
Web Interface Operational
90 days ago
100.0 % uptime
Today
REST API Operational
90 days ago
100.0 % uptime
Today
Streaming Import REST API Operational
90 days ago
100.0 % uptime
Today
Mobile/Javascript REST API Operational
90 days ago
100.0 % uptime
Today
Data Connector Integrations Operational
90 days ago
99.98 % uptime
Today
Hadoop / Hive Query Engine Operational
90 days ago
99.99 % uptime
Today
Presto Query Engine Operational
90 days ago
100.0 % uptime
Today
Presto JDBC/ODBC Gateway Operational
90 days ago
100.0 % uptime
Today
Workflow ? Operational
90 days ago
100.0 % uptime
Today
Global ? Operational
90 days ago
100.0 % uptime
Today
Reporting ? Operational
90 days ago
100.0 % uptime
Today
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Major outage
Partial outage
No downtime recorded on this day.
had a major outage
had a partial outage

No incidents or maintenance related to this downtime.

System Metrics Month Week Day
US - REST API - Response Time ?
Fetching
US - REST API - Error Rates ?
Fetching
US - Streaming Import REST API - Response Time ?
Fetching
US - Queued Streaming Import requests ?
Fetching
US - Mobile/Javascript REST API - Response Time ?
Fetching
US - Web Interface - Response Time ?
Fetching
Tokyo - REST API - Response Time ?
Fetching
Tokyo - Streaming Import REST API - Response Time ?
Fetching
Tokyo - Queued Streaming Import requests ?
Fetching
Tokyo - Web Interface - Response Time ?
Fetching
EU - REST API - Response Time ?
Fetching
EU - Streaming Import REST API - Response Time ?
Fetching
EU - Queued Streaming Import requests ?
Fetching
EU - Web Interface - Response Time ?
Fetching
Past Incidents
Apr 24, 2019

No incidents reported today.

Apr 23, 2019

No incidents reported.

Apr 22, 2019

No incidents reported.

Apr 21, 2019

No incidents reported.

Apr 20, 2019

No incidents reported.

Apr 19, 2019

No incidents reported.

Apr 18, 2019
[US, JP, EU Regions] Data Connector Job Failures
Resolved - We have confirmed that Data Connector / Result Output jobs which have been submitted to us during 20:46 and 21:58 were affected by this incident. We have confirmed that the fix has already been applied to our job executer and newly submitted jobs are working as they were supposed to be. Some jobs might be stuck at queued status for long, and we will notify customers with affected jobs, as needed.
Apr 18, 00:25 PDT
Monitoring - We have confirmed that newly submitted Data Connector / Result Output jobs are running as they were supposed to be. We are still verifying the impact of the incident, and will reach out the owner of the affected jobs as needed.
Apr 17, 22:51 PDT
Update - We have identified that this Data Connector incident affects Result Output jobs as well. We will update the status once we confirmed that the fix has been applied.
Apr 17, 21:37 PDT
Identified - We have investigated and identified an issue with a Data Connector release causing jobs to fail. A fix is in deployment and we expect full resolution in under 30 minutes. Jobs will be automatically retried subject to normal retry limits.
Apr 17, 21:31 PDT
Investigating - We have investigated and identified an issue with Data Connector jobs to fail. A fix is in deployment and we expect to be resolved in approximately 30 minutes.
Apr 17, 21:27 PDT
Apr 17, 2019
[US Region] hive job execution delay
Resolved - We are verifying the list of affected Hive jobs. We will contact to customers who were affected by this incident as needed.
Apr 17, 02:59 PDT
Update - We have removed fault job executer nodes from service cluster. We confirmed that newly submitted Hive jobs are running normally now.
Apr 17, 02:50 PDT
Monitoring - A fix has been implemented and we are monitoring the results.
Apr 17, 01:46 PDT
Investigating - We've detected that our Hive job executer service is not running as it was supposed to be. We're investigating about the impact of the issue.
Apr 17, 01:19 PDT
Apr 16, 2019
[US Region] Workflow Database maintenance
Completed - *Maintenance* and system *recovery* have been fully completed.
Scheduled session execution delay was 1 minute at 6:22 pm PDT.

The Workflow database maintenance is now complete.
Apr 16, 18:50 PDT
Verifying - The scheduled Workflow database maintenance is complete.

We are monitoring the system closely to ensure all systems successfully complete their *recovery* and return as quickly as possible to full functionality and throughput.
Apr 16, 18:35 PDT
In progress - We expect this to be followed by a *recovery* period of around 5 minutes during which the system will gradually reach back to full throughput. During the maintenance and recovery, customers may experience workflow REST API unavailability, delays with scheduled user-defined Workflow session executions, CDP Master Segments, Batch Segments, API Tokens, and Predictive Scoring creation and update failures, and delays in refresh of Master Segments, Batch Segments, and Predictive Scoring.
Apr 16, 18:11 PDT
Update - In about an hour, from 6:00 PM PDT (10:00 AM JST), we will start the scheduled maintenance on the Workflow database.

The maintenance is necessary to upgrade the PostgreSQL database instance to higher grade.

We expect the operation to cause 6 minutes of downtime, the *maintenance* period, followed by a *recovery* period of around 5 minutes during which the system will gradually reach back to full throughput.

During the maintenance and recovery, customers may experience the following:

* Workflow REST API unavailable
All workflow REST API will be unavailable during the maintenance and will respond with a 500 error code.

* Workflow scheduled session execution delayed
The execution of all scheduled Workflow sessions will be delayed during the *maintenance* period and remain queued. During *recovery*, we expect sessions to begin processing slowly: within 5 minutes the processing of queued sessions should reach regular capacity and the queued backlog should be depleted shortly after.

* CDP Master Segments, Batch Segments, API Tokens, and Predictive Scoring creation and update unavailable
Throughout the *maintenance* period, creating and updating Master Segments and , Segments, API Tokens and Predictive Scoring will not be possible: these functionalities creation and update, relying on Workflow to execute an ETL, will not be available. The ability to create and update them will remain be impaired during the *recovery* period and will be restored shortly after.

* CDP Master Segments, Batch Segments, and Predictive Scoring refresh delayed
Refresh of CDP Master Segments, and Batch Segments, and Predictive Scoring relies on the Workflow sub-system to execute. As forLike Workflow session executions, refresh will be delayed during the *maintenance* and *recovery* periods.

Beyond this notice, we will provide updates at the start and completion of the operation and once the verification has completed: at that time, all systems will have returned to full functionality and this Scheduled Maintenance will be closed.
Apr 16, 17:19 PDT
Scheduled - On Tuesday, April 16th from 6 to 7 PM PDT (Wednesday, April 17th from 10 to 11 AM JST) we will be performing maintenance on the Workflow database.

The maintenance is necessary to upgrade the PostgreSQL database instance to higher grade.

We expect the operation to cause *6 minutes of downtime*.

# Impact

The database will become unreachable for the duration of the *maintenance* procedure, expected to last 6 minutes. We expect this to be followed by a *recovery* period of around 5 minutes during which the system will gradually reach back to full throughput.

All internal components reading/writing the Workflow Database have built-in fault tolerance that will allow them to retry the request in case of failures: when the connection incurs into an error, the request is retried several times and for long enough to ensure maintenance and recovery periods are successfully completed.

For customer facing components, connection/requests failures may be reflected back directly on the caller: they should be interpreted as a suggestion to retry the connection/request later, most practically after the maintenance window is announced to have been closed.

During the maintenance and recovery, customers may experience the following:

* Workflow REST API unavailable
All workflow REST API will be unavailable during the maintenance and will respond with a 500 error code.

* Workflow scheduled session execution delayed
The execution of all scheduled Workflow sessions will be delayed during the *maintenance* period and remain queued. During *recovery*, we expect sessions to begin processing slowly: within 5 minutes the processing of queued sessions should reach regular capacity and the backlog should be depleted shortly after.

* CDP Master Segments, Batch Segments, API Tokens, and Predictive Scoring creation and update unavailable
Throughout the *maintenance* period, creating and updating Master Segments, Segments, API Tokens and Predictive Scoring will not be possible: these functionalities , rely on Workflow to execute an ETL. The ability to create and update will remain impaired during the *recovery* period and will be restored shortly after.

* CDP Master Segments, Batch Segments, and Predictive Scoring refresh delayed
Refresh of CDP Master Segments, Batch Segments, and Predictive Scoring relies on the Workflow sub-system to execute. As for Workflow session executions, refresh will be delayed during the *maintenance* and *recovery* periods.

# Communication

Beyond this notice, we will provide updates approximately 1 hour before the beginning of the maintenance window, at the start and completion of the operation, and once the verification is completed. At that time, all systems will have returned to full functionality and the Scheduled Maintenance will be closed.

If you have any question or concern about this upgrade, please feel free to reach out to our Support team at support@treasuredata.com.
Mar 25, 13:10 PDT
[US Region] Degraded performance in import processing
Resolved - This incident was resolved.
Apr 16, 08:40 PDT
Monitoring - We identified Multi-AZ failover by one of our database cluster. From 8:05 am PT to 8:12 am, US Region streaming import requests experienced 15 minutes import delay in average. The delay was resolved at 8:32am. Our system is operating normally now.
Apr 16, 08:36 PDT
Investigating - We're facing delays with streaming import. Our team is working to restore service at this time.
Apr 16, 08:15 PDT
Apr 15, 2019

No incidents reported.

Apr 14, 2019

No incidents reported.

Apr 13, 2019

No incidents reported.

Apr 12, 2019

No incidents reported.

Apr 11, 2019

No incidents reported.

Apr 10, 2019
[US Region] Degraded performance in import processing
Resolved - This incident was resolved.
Apr 10, 08:56 PDT
Monitoring - Queued import requests has been processed. From 8:00 am PT to 8:20 am, US Region streaming import requests experienced 10 minutes import delay in average. The delay was resolved at 8:50am.
Our system is operating normally now.
Apr 10, 08:52 PDT
Identified - We identified the network throughput degradation in one of our database cluster. The symptom started happen at 8:00 am and was resolved at 8:20 am PT. Queued import jobs are processed quickly. We will update once the delay is resolved.
Apr 10, 08:44 PDT
Investigating - Currently investing an issue where import streaming is processing slowly
Apr 10, 08:35 PDT
← Incident History Powered by Statuspage