Completed -
*Maintenance* and system *recovery* processes have been fully executed.
Streaming import and job execution delay was less than 1 minute at 6:26 pm PDT.
The scheduled Metadata Database maintenance is now complete.
May 8, 18:42 PDT
Verifying -
The scheduled Metadata Database maintenance is complete.
We are monitoring the system closely to ensure all systems successfully complete their *recovery* and return to full functionality and throughput as quickly as possible.
May 8, 18:34 PDT
In progress -
The scheduled Metadata Database maintenance is starting now.
We expect the operation to cause *no downtime*: the database will be online after the upgrade within less than 2 minutes. This will be followed by a *recovery* period of around 5 minutes during which the system will gradually reach back to full throughput. During the maintenance and recovery, customers may experience delays with Streaming (td-agent, fluentd) imports, Mobile (Android, iOS, Unity SDK) imports, Browser (JavaScript SDK) imports, and job execution as well as Presto JDBC / ODBC request failures.
May 8, 18:15 PDT
Update -
In about an hour, at 6:15 PM PDT (10:15 AM JST), we will start the scheduled maintenance on our PlazmaDB Metadata database.
The maintenance is necessary to upgrade the PostgreSQL database engine with the latest patches (including security related ones).
We expect the operation to cause *no downtime*: the database will be online after the upgrade within less than 2 minutes. This to be followed by a *recovery* period of around 5 minutes during which the system will gradually reach back to full throughput. During the maintenance and recovery, customers may experience the following:
* Streaming, Mobile, and JavaScript/Browser imports delay
Streaming import (through td-agent or fluentd) requests will continue to be accepted as usual but the requests will remain queued until after the database maintenance is complete. We expect stream import processing to be further delayed during recovery. Thus we expect the stream import delay to extend to up to 10 minutes.
The same will apply to import requests from Browsers (Javascript SDK) and Mobiles (Android, iOS, and Unity SDKs).
* Jobs execution delay
All jobs (Presto, Hive, Result Export, Data Connector, Bulk Import, Export, and Partial Delete jobs submitted from Console, API, Workflow or triggered by our system according to the configured schedule) will remain queued during maintenance. During recovery, we expect jobs to begin processing slowly: within 10 minutes job processing should reach back to full throughput.
* Presto JDBC / ODBC Gateway errors
The Presto JDBC / ODBC Gateway will report errors during maintenance due to the unreachability of the Metadata database: errors will be propagated to the clients. We expect processing of Presto JDBC / ODBC jobs to follow the same recovery pattern as all other jobs (see above).
Beyond this notice, we will provide updates at the start and completion of the operation and once the verification has completed: at that time, all systems will have returned to full functionality and this Scheduled Maintenance will be closed.
May 8, 17:08 PDT
Scheduled -
On Tuesday, May 8th from 6 to 7 PM PDT (May 9th from 10 to 11 AM JST) we will be performing maintenance on the PlazmaDB Metadata database.
The maintenance is necessary to upgrade the PostgreSQL database engine with the latest patches (including security related ones).
We expect the operation to cause *no downtime*: the database will be online after the upgrade within less than 2 minutes.
# Impact
NOTE: This maintenance notice only interests *AWS customers using the Tokyo region* and not AWS customers using the US region.
The database will become unreachable for the duration of the *maintenance* procedure, which should last no longer than 2 minutes. We expect this to be followed by a *recovery* period of around 5 minutes during which the system will gradually reach back to full throughput.
All components reading/writing the Metadata Database have built-in fault tolerance that will allow them to retry the request in case of failures: when the connection incurs into an error, the request is retried several times and for long enough to ensure maintenance and recovery periods are successfully completed.
During the maintenance and recovery, customers may experience bie following:
* Streaming, Mobile, and JavaScript/Browser imports delay
Streaming import (through td-agent or fluentd) requests will continue to be accepted as usual but the requests will remain queued until after the database maintenance is complete. We expect stream import processing to be further delayed during recovery. Thus we expect the stream import delay to extend to up to 10 minutes.
The same will apply to import requests from Browsers (Javascript SDK) and Mobiles (Android, iOS, and Unity SDKs).
* Jobs execution delay
All jobs (Presto, Hive, Result Export, Data Connector, Bulk Import, Export, and Partial Delete jobs submitted from Console, API, Workflow or triggered by our system according to the configured schedule) will remain queued during maintenance. During recovery, we expect jobs to begin processing slowly: within 10 minutes job processing should reach back to full throughput.
* Presto JDBC / ODBC Gateway errors
The Presto JDBC / ODBC Gateway will report errors during maintenance due to the unreachability of the Metadata database: errors will be propagated to the clients. We expect processing of Presto JDBC / ODBC jobs to follow the same recovery pattern as all other jobs (see above).
# Communication
Beyond this notice, we will provide updates approximately 1 hour before the beginning of the maintenance window, at the start and completion of the operation, and once the verification is completed. At that time, all systems will have returned to full functionality and the Scheduled Maintenance will be closed.
If you have any question or concern about this upgrade, please feel free to reach out to our Support team at
support@treasuredata.com.
Apr 6, 13:57 PDT
