Scheduled session execution delay was 30 seconds at 11:16am JST (19:16 PDT).
Posted Apr 26, 2021 - 19:34 PDT
Verifying
The scheduled Worker database maintenance is complete.
We are monitoring the system closely to ensure all systems successfully complete their *recovery* and return as quickly as possible to full functionality and throughput.
Posted Apr 26, 2021 - 19:27 PDT
In progress
The scheduled Worker database maintenance is starting now.
During the maintenance and recovery, customers may experience the following:
- The REST API to submit and cancel jobs will be unavailable
The REST API endpoints for job submission and cancellations will be unavailable during the maintenance and will respond with a 500 error code.
- Jobs execution delay
All jobs (Presto, Hive, Data Connector Integration Import and Export, Bulk Import, Table Export, and Partial Delete jobs submitted from Console, API, Workflow, CDP or triggered by our system according to the configured schedule) will be delayed and will be automatically retried until they succeed after the downtime period. During recovery, we expect jobs to begin processing slowly and within 30 minutes processing should reach back to full throughput.
If customers experience errors of Data Connector Integration Export jobs, please reach out to our Support team at support@treasuredata.com to recover the jobs. Integration Export jobs won’t be retried by default to avoid exporting query results to external destinations twice, potentially causing data duplication downstream.
Posted Apr 26, 2021 - 19:01 PDT
Update
In about an hour, from 19:00 PDT (11:00 JST), we will start the scheduled maintenance on the Worker database.
During the maintenance and recovery, customers may experience the following:
- The REST API to submit and cancel jobs will be unavailable
The REST API endpoints for job submission and cancellations will be unavailable during the maintenance and will respond with a 500 error code.
- Jobs execution delay
All jobs (Presto, Hive, Data Connector Integration Import and Export, Bulk Import, Table Export, and Partial Delete jobs submitted from Console, API, Workflow, CDP or triggered by our system according to the configured schedule) will be delayed and will be automatically retried until they succeed after the downtime period. During recovery, we expect jobs to begin processing slowly and within 30 minutes processing should reach back to full throughput.
If customers experience errors of Data Connector Integration Export jobs, please reach out to our Support team at support@treasuredata.com to recover the jobs. Integration Export jobs won’t be retried by default to avoid exporting query results to external destinations twice, potentially causing data duplication downstream.
Posted Apr 26, 2021 - 18:01 PDT
Scheduled
On Tuesday, April 27th from 11 to 11:30 JST (Monday, April 26th from 19 to 19:30 PDT) we will be performing short-notice emergency maintenance on the Worker database. This maintenance window is required to perform maintenance of the underlying hardware and it will replace the instance on which the primary database is running on.
Although the maintenance window duration is 30 minutes, the database will only become unreachable for no longer than 3 minutes in total. We expect this to be followed by a recovery period of around 5 minutes during which the system will gradually return to full throughput.
# Impact
During the maintenance and recovery, customers may experience the following:
- The REST API to submit and cancel jobs will be unavailable
The REST API endpoints for job submission and cancellations will be unavailable during the maintenance and will respond with a 500 error code.
- Jobs execution delay
All jobs (Presto, Hive, Data Connector Integration Import and Export, Bulk Import, Table Export, and Partial Delete jobs submitted from Console, API, Workflow, CDP or triggered by our system according to the configured schedule) will be delayed and will be automatically retried until they succeed after the downtime period. During recovery, we expect jobs to begin processing slowly and within 30 minutes processing should reach back to full throughput.
If customers experience errors of Data Connector Integration Export jobs, please reach out to our Support team at support@treasuredata.com to recover the jobs. Integration Export jobs won’t be retried by default to avoid exporting query results to external destinations twice, potentially causing data duplication downstream.
# Communication
Beyond this notice, we will provide updates approximately 1 hour before the beginning of the maintenance window, at the start and completion of the operation, and once the verification is completed. At that time, all systems will have returned to full functionality and the Scheduled Maintenance will be closed.
If you have any question or concern about this upgrade, please feel free to reach out to our Support team at support@treasuredata.com.
Posted Apr 23, 2021 - 13:43 PDT
This scheduled maintenance affected: Tokyo (REST API, Data Connector Integrations, Hadoop / Hive Query Engine, Presto Query Engine, Workflow, CDP API).