We are monitoring the system closely to ensure all systems successfully complete their *recovery* and return as quickly as possible to full functionality and throughput.
Posted May 27, 2023 - 18:41 PDT
In progress
The Worker database maintenance is starting now.
During the maintenance and recovery, customers may experience the following:
- The REST API to submit and cancel jobs will be unavailable
The REST API endpoints for job submission and cancellations will be unavailable during the maintenance and will respond with a 500 error code.
- Jobs execution delay
All jobs (Presto, Hive, Data Connector Integration Import and Export, Bulk Import, Table Export, and Partial Delete jobs submitted from Console, API, Workflow, CDP or triggered by our system according to the configured schedule) will be delayed and will be automatically retried until they succeed after the downtime period. During recovery, we expect jobs to begin processing slowly and within 30 minutes processing should reach back to full throughput.
If customers experience errors of Data Connector Integration Export jobs, please reach out to our Support team at support@treasuredata.com to recover the jobs. Integration Export jobs won’t be retried by default to avoid exporting query results to external destinations twice, potentially causing data duplication downstream.
Posted May 27, 2023 - 17:56 PDT
Update
In about an hour, we will start the maintenance on the Worker database.
During the maintenance and recovery, customers may experience the following:
- The REST API to submit and cancel jobs will be unavailable
The REST API endpoints for job submission and cancellations will be unavailable during the maintenance and will respond with a 500 error code.
- Jobs execution delay
All jobs (Presto, Hive, Data Connector Integration Import and Export, Bulk Import, Table Export, and Partial Delete jobs submitted from Console, API, Workflow, CDP or triggered by our system according to the configured schedule) will be delayed and will be automatically retried until they succeed after the downtime period. During recovery, we expect jobs to begin processing slowly and within 30 minutes processing should reach back to full throughput.
If customers experience errors of Data Connector Integration Export jobs, please reach out to our Support team at support@treasuredata.com to recover the jobs. Integration Export jobs won’t be retried by default to avoid exporting query results to external destinations twice, potentially causing data duplication downstream.
Posted May 27, 2023 - 16:56 PDT
Scheduled
On Sunday, May 28th from 09:59 to 10:29 JST (Saturday, May 27 from 17:59 to 18:29 PST) we will be performing short-notice maintenance on the Worker database. This maintenance window is required for database minor version upgrade and will replace the instance the primary MySQL database is running on.
Although the maintenance window duration is 30 minutes, the database will only become unreachable for no longer than 3 minutes in total. We expect this to be followed by a recovery period of around 5 minutes during which the system will gradually return to full throughput.
# Impact
During the maintenance and recovery, customers may experience the following:
- The REST API to submit and cancel jobs will be unavailable
The REST API endpoints for job submission and cancellations will be unavailable during the maintenance and will respond with a 500 error code.
- Jobs execution delay
All jobs (Presto, Hive, Data Connector Integration Import and Export, Bulk Import, Table Export, and Partial Delete jobs submitted from Console, API, Workflow, CDP or triggered by our system according to the configured schedule) will be delayed and will be automatically retried until they succeed after the downtime period. During recovery, we expect jobs to begin processing slowly and within 30 minutes processing should reach back to full throughput.
If customers experience errors of Data Connector Integration Export jobs, please reach out to our Support team at support@treasuredata.com to recover the jobs. Integration Export jobs won’t be retried by default to avoid exporting query results to external destinations twice, potentially causing data duplication downstream.
# Communication
Beyond this notice, we will provide updates approximately 1 hour before the beginning of the maintenance window, at the start and completion of the operation, and once the verification is completed. At that time, all systems will have returned to full functionality and the maintenance will be closed.
If you have any question or concern about this upgrade, please feel free to reach out to our Support team at support@treasuredata.com.
Posted May 26, 2023 - 13:21 PDT
This scheduled maintenance affected: Tokyo (REST API, Data Connector Integrations, Hadoop / Hive Query Engine, Presto Query Engine, Workflow, CDP API).