*Maintenance* and system *recovery* have been fully completed. Scheduled session execution delay was 1 minute at 6:22 pm PDT.
The Workflow database maintenance is now complete.
Posted Apr 16, 2019 - 18:50 PDT
Verifying
The scheduled Workflow database maintenance is complete.
We are monitoring the system closely to ensure all systems successfully complete their *recovery* and return as quickly as possible to full functionality and throughput.
Posted Apr 16, 2019 - 18:35 PDT
In progress
We expect this to be followed by a *recovery* period of around 5 minutes during which the system will gradually reach back to full throughput. During the maintenance and recovery, customers may experience workflow REST API unavailability, delays with scheduled user-defined Workflow session executions, CDP Master Segments, Batch Segments, API Tokens, and Predictive Scoring creation and update failures, and delays in refresh of Master Segments, Batch Segments, and Predictive Scoring.
Posted Apr 16, 2019 - 18:11 PDT
Update
In about an hour, from 6:00 PM PDT (10:00 AM JST), we will start the scheduled maintenance on the Workflow database.
The maintenance is necessary to upgrade the PostgreSQL database instance to higher grade.
We expect the operation to cause 6 minutes of downtime, the *maintenance* period, followed by a *recovery* period of around 5 minutes during which the system will gradually reach back to full throughput.
During the maintenance and recovery, customers may experience the following:
* Workflow REST API unavailable All workflow REST API will be unavailable during the maintenance and will respond with a 500 error code.
* Workflow scheduled session execution delayed The execution of all scheduled Workflow sessions will be delayed during the *maintenance* period and remain queued. During *recovery*, we expect sessions to begin processing slowly: within 5 minutes the processing of queued sessions should reach regular capacity and the queued backlog should be depleted shortly after.
* CDP Master Segments, Batch Segments, API Tokens, and Predictive Scoring creation and update unavailable Throughout the *maintenance* period, creating and updating Master Segments and , Segments, API Tokens and Predictive Scoring will not be possible: these functionalities creation and update, relying on Workflow to execute an ETL, will not be available. The ability to create and update them will remain be impaired during the *recovery* period and will be restored shortly after.
* CDP Master Segments, Batch Segments, and Predictive Scoring refresh delayed Refresh of CDP Master Segments, and Batch Segments, and Predictive Scoring relies on the Workflow sub-system to execute. As forLike Workflow session executions, refresh will be delayed during the *maintenance* and *recovery* periods.
Beyond this notice, we will provide updates at the start and completion of the operation and once the verification has completed: at that time, all systems will have returned to full functionality and this Scheduled Maintenance will be closed.
Posted Apr 16, 2019 - 17:19 PDT
Scheduled
On Tuesday, April 16th from 6 to 7 PM PDT (Wednesday, April 17th from 10 to 11 AM JST) we will be performing maintenance on the Workflow database.
The maintenance is necessary to upgrade the PostgreSQL database instance to higher grade.
We expect the operation to cause *6 minutes of downtime*.
# Impact
The database will become unreachable for the duration of the *maintenance* procedure, expected to last 6 minutes. We expect this to be followed by a *recovery* period of around 5 minutes during which the system will gradually reach back to full throughput.
All internal components reading/writing the Workflow Database have built-in fault tolerance that will allow them to retry the request in case of failures: when the connection incurs into an error, the request is retried several times and for long enough to ensure maintenance and recovery periods are successfully completed.
For customer facing components, connection/requests failures may be reflected back directly on the caller: they should be interpreted as a suggestion to retry the connection/request later, most practically after the maintenance window is announced to have been closed.
During the maintenance and recovery, customers may experience the following:
* Workflow REST API unavailable All workflow REST API will be unavailable during the maintenance and will respond with a 500 error code.
* Workflow scheduled session execution delayed The execution of all scheduled Workflow sessions will be delayed during the *maintenance* period and remain queued. During *recovery*, we expect sessions to begin processing slowly: within 5 minutes the processing of queued sessions should reach regular capacity and the backlog should be depleted shortly after.
* CDP Master Segments, Batch Segments, API Tokens, and Predictive Scoring creation and update unavailable Throughout the *maintenance* period, creating and updating Master Segments, Segments, API Tokens and Predictive Scoring will not be possible: these functionalities , rely on Workflow to execute an ETL. The ability to create and update will remain impaired during the *recovery* period and will be restored shortly after.
* CDP Master Segments, Batch Segments, and Predictive Scoring refresh delayed Refresh of CDP Master Segments, Batch Segments, and Predictive Scoring relies on the Workflow sub-system to execute. As for Workflow session executions, refresh will be delayed during the *maintenance* and *recovery* periods.
# Communication
Beyond this notice, we will provide updates approximately 1 hour before the beginning of the maintenance window, at the start and completion of the operation, and once the verification is completed. At that time, all systems will have returned to full functionality and the Scheduled Maintenance will be closed.
If you have any question or concern about this upgrade, please feel free to reach out to our Support team at support@treasuredata.com.
Posted Mar 25, 2019 - 13:10 PDT
This scheduled maintenance affected: US (Web Interface, REST API, Workflow).