*Maintenance* and system *recovery* have been fully completed.
The scheduled database maintenance is now complete.
Posted Mar 12, 2024 - 18:57 PDT
Verifying
The scheduled database maintenance is complete.
We are monitoring the system closely to ensure all systems successfully complete their *recovery* and return as quickly as possible to full functionality.
Posted Mar 12, 2024 - 18:45 PDT
In progress
The scheduled database maintenance window is starting now.
# Impact
During the database downtime and recovery period, customers may experience the following:
- Streaming, Mobile, and JavaScript/Browser imports delay
Streaming import (through td-agent or fluentd) requests will continue to be accepted as usual, but the requests will remain queued until after the database maintenance is complete. We expect stream import processing to be further delayed during recovery. The same impact will apply to import requests from Browsers (Javascript SDK), Mobiles (Android, iOS, and Unity SDKs), Data Ingestion APIs, and Postback APIs.
- Jobs execution delay
All jobs (Presto, Hive, Data Connector Integrations, Bulk Import, Bulk Export, Result Export, and Partial Delete jobs submitted from Console, API, Workflow or triggered by our system according to the configured schedule) will be delayed and automatically retried until they succeed after the downtime period. We expect job processing to ramp up over 30 minutes after the maintenance is complete before returning to full throughput.
- Table creation errors while maintenance is underway
During the maintenance period, any REST API requests to create tables will result in an error. SQL or Workflow requests to create tables will be delayed and automatically retried after the maintenance period.
- Console execution delay
Data Workbench and Audience Studio will show that jobs are running longer due to job execution delays. These delays will show because the Parent Segments, Segments, and Workflows jobs will be unavailable. After maintenance is complete, we expect processing of Data Workbench and Audience Studio jobs to follow the same recovery pattern as all other jobs (see above).
- Data Access API (beta) return errors during downtime
Data Access API (Plazma Public API) for Spark will return errors during the maintenance period.
- Treasure Workflow delay and failure
We will attempt to retry any workflows, but some long-running workflows may be terminated if they run more than a few minutes. We encourage you to check any failed workflows and re-run if necessary.
- Workflow REST API will be unavailable
It won’t be possible to create or update Workflow Projects, Settings related to Audience Studio (including Parent Segment, Segment, Customer Journey Orchestration, etc.), and Predictive Scoring during the maintenance window. If you face errors using the Workflow REST API during maintenance, please wait for an update on our status page and re-execute your tasks when we’ve indicated maintenance is complete.
- Workflow operations in Console and CLI will be unavailable
Workflow-related operations in the Console and CLI may fail during the maintenance. Once we’ve indicated service has returned to normal on our status page, you can resume regular Workflow-related operations.
- Presto JDBC/ODBC queries and CDP segmentation queries failure
JDBC/ODBC queries which access “api-presto.treasuredata.com” will result in an error. Additionally, queries running from the CDP Segmentation UI in Console will be unavailable. We suggest retrying queries after the database maintenance is complete. Scheduled Presto queries in Workflow or Console won't be affected and will automatically run once the maintenance has been completed.
- Console Table preview update delay
The Table preview (sample data and records counter) in Data Workbench may show old information while maintenance is underway. This will return to normal once maintenance is complete.
- The REST API to submit and cancel jobs will be unavailable
The REST API endpoints for job submission and cancellations will be unavailable during the maintenance and will respond with a 500 error code.
If customers experience errors running Data Connector Integration Export jobs, please contact our Support team at support@treasuredata.com to recover the jobs. To avoid data duplication in your systems from Treasure Data exports, Integration Export jobs won’t be automatically retried.
Posted Mar 12, 2024 - 18:00 PDT
Update
In about an hour, from 1:00 UTC (10:00 JST), the maintenance window for Treasure Data databases will commence.
# Impact
During the database downtime and recovery period, customers may experience the following:
- Streaming, Mobile, and JavaScript/Browser imports delay
Streaming import (through td-agent or fluentd) requests will continue to be accepted as usual, but the requests will remain queued until after the database maintenance is complete. We expect stream import processing to be further delayed during recovery. The same impact will apply to import requests from Browsers (Javascript SDK), Mobiles (Android, iOS, and Unity SDKs), Data Ingestion APIs, and Postback APIs.
- Jobs execution delay
All jobs (Presto, Hive, Data Connector Integrations, Bulk Import, Bulk Export, Result Export, and Partial Delete jobs submitted from Console, API, Workflow or triggered by our system according to the configured schedule) will be delayed and automatically retried until they succeed after the downtime period. We expect job processing to ramp up over 30 minutes after the maintenance is complete before returning to full throughput.
- Table creation errors while maintenance is underway
During the maintenance period, any REST API requests to create tables will result in an error. SQL or Workflow requests to create tables will be delayed and automatically retried after the maintenance period.
- Console execution delay
Data Workbench and Audience Studio will show that jobs are running longer due to job execution delays. These delays will show because the Parent Segments, Segments, and Workflows jobs will be unavailable. After maintenance is complete, we expect processing of Data Workbench and Audience Studio jobs to follow the same recovery pattern as all other jobs (see above).
- Data Access API (beta) return errors during downtime
Data Access API (Plazma Public API) for Spark will return errors during the maintenance period.
- Treasure Workflow delay and failure
We will attempt to retry any workflows, but some long-running workflows may be terminated if they run more than a few minutes. We encourage you to check any failed workflows and re-run if necessary.
- Workflow REST API will be unavailable
It won’t be possible to create or update Workflow Projects, Settings related to Audience Studio (including Parent Segment, Segment, Customer Journey Orchestration, etc.), and Predictive Scoring during the maintenance window. If you face errors using the Workflow REST API during maintenance, please wait for an update on our status page and re-execute your tasks when we’ve indicated maintenance is complete.
- Workflow operations in Console and CLI will be unavailable
Workflow-related operations in the Console and CLI may fail during the maintenance. Once we’ve indicated service has returned to normal on our status page, you can resume regular Workflow-related operations.
- Presto JDBC/ODBC queries and CDP segmentation queries failure
JDBC/ODBC queries which access “api-presto.treasuredata.com” will result in an error. Additionally, queries running from the CDP Segmentation UI in Console will be unavailable. We suggest retrying queries after the database maintenance is complete. Scheduled Presto queries in Workflow or Console won't be affected and will automatically run once the maintenance has been completed.
- Console Table preview update delay
The Table preview (sample data and records counter) in Data Workbench may show old information while maintenance is underway. This will return to normal once maintenance is complete.
- The REST API to submit and cancel jobs will be unavailable
The REST API endpoints for job submission and cancellations will be unavailable during the maintenance and will respond with a 500 error code.
If customers experience errors running Data Connector Integration Export jobs, please contact our Support team at support@treasuredata.com to recover the jobs. To avoid data duplication in your systems from Treasure Data exports, Integration Export jobs won’t be automatically retried.
Beyond this notice, we will provide updates on our status page (https://status.treasuredata.com/) at the start and completion of the operation and once the verification of the new system is completed. At that time, all systems will have returned to normal operations, and this Scheduled Maintenance will be closed.
Posted Mar 12, 2024 - 17:00 PDT
Scheduled
On Wednesday, March 13 from 01:00 to 02:00 UTC (March 13 from 10:00 to 11:00 JST), we will be performing routine maintenance on the backend databases that power the Treasure Data Platform.
During this maintenance, we will apply patches and updates to databases used by the Treasure Data Platform. These updates are necessary to ensure we have the latest security and performance improvements available. We expect downtime to be brief, lasting only a few minutes. The remainder of this time will be used for our team to validate that the maintenance was successful. Customers should expect some attempts to reach Treasure Data may time out or produce errors while we are deploying database improvements. See below for detailed information about the impact of this change on each Treasure Data service.
During the database downtime and recovery period, customers may experience the following:
- Streaming, Mobile, and JavaScript/Browser imports delay
Streaming import (through td-agent or fluentd) requests will continue to be accepted as usual, but the requests will remain queued until after the database maintenance is complete. We expect stream import processing to be further delayed during recovery. The same impact will apply to import requests from Browsers (Javascript SDK), Mobiles (Android, iOS, and Unity SDKs), Data Ingestion APIs, and Postback APIs.
- Jobs execution delay
All jobs (Presto, Hive, Data Connector Integrations, Bulk Import, Bulk Export, Result Export, and Partial Delete jobs submitted from Console, API, Workflow or triggered by our system according to the configured schedule) will be delayed and automatically retried until they succeed after the downtime period. We expect job processing to ramp up over 30 minutes after the maintenance is complete before returning to full throughput.
- Table creation errors while maintenance is underway
During the maintenance period, any REST API requests to create tables will result in an error. SQL or Workflow requests to create tables will be delayed and automatically retried after the maintenance period.
- Console execution delay
Data Workbench and Audience Studio will show that jobs are running longer due to job execution delays. These delays will show because the Parent Segments, Segments, and Workflows jobs will be unavailable. After maintenance is complete, we expect processing of Data Workbench and Audience Studio jobs to follow the same recovery pattern as all other jobs (see above).
- Treasure Workflow delay and failure
We will attempt to retry any workflows, but some long-running workflows may be terminated if they run more than a few minutes. We encourage you to check any failed workflows and re-run if necessary.
- Workflow REST API will be unavailable
It won’t be possible to create or update Workflow Projects, Settings related to Audience Studio (including Parent Segment, Segment, Customer Journey Orchestration, etc.), and Predictive Scoring during the maintenance window. If you face errors using the Workflow REST API during maintenance, please wait for an update on our status page and re-execute your tasks when we’ve indicated maintenance is complete.
- Workflow operations in Console and CLI will be unavailable
Workflow-related operations in the Console and CLI may fail during the maintenance. Once we’ve indicated service has returned to normal on our status page, you can resume regular Workflow-related operations.
- Presto JDBC/ODBC queries and CDP segmentation queries failure
JDBC/ODBC queries which access “api-presto.treasuredata.com” will result in an error. Additionally, queries running from the CDP Segmentation UI in Console will be unavailable. We suggest retrying queries after the database maintenance is complete. Scheduled Presto queries in Workflow or Console won't be affected and will automatically run once the maintenance has been completed.
- Console Table preview update delay
The Table preview (sample data and records counter) in Data Workbench may show old information while maintenance is underway. This will return to normal once maintenance is complete.
- The REST API to submit and cancel jobs will be unavailable
The REST API endpoints for job submission and cancellations will be unavailable during the maintenance and will respond with a 500 error code.
If customers experience errors running Data Connector Integration Export jobs, please contact our Support team at support@treasuredata.com to recover the jobs. To avoid data duplication in your systems from Treasure Data exports, Integration Export jobs won’t be automatically retried.
# Communication
Beyond this notice, we will provide updates on our status page (https://status.treasuredata.com) approximately 1 hour before the beginning of the maintenance window, at the start and completion of the operation, and once the verification is complete. At that time, all systems will have returned to normal operations, and the Scheduled Maintenance will be closed.
If you have any questions or concerns about this upgrade, please feel free to contact our Support team at support@treasuredata.com.