After the release of the Presto performance improvements, we have been monitoring the system for 2 hours and observed no anomaly.
We will now mark this scheduled release as closed but we are committed to continue closely monitoring for the next 3 days.
Posted Jul 27, 2020 - 21:22 PDT
Verifying
The scheduled Presto performance improvement release is complete.
We are monitoring the system closely to ensure all systems successfully complete their *recovery* and return as quickly as possible to full functionality.
Posted Jul 27, 2020 - 19:40 PDT
In progress
The scheduled Presto performance improvement release is starting now.
The release should have no impact on running queries, which will transparently be transferred to the new Presto query engine version.
The release rolls forward the changes that were introduced on May 7th to improve the query performance degradation experienced by queries in specific scenarios.
The May 7th release was rolled back on May 22nd because it had a bug causing sporadic write inconsistencies in CREATE TABLE AS, INSERT INTO, and DELETE FROM queries. This new version, based on the May 7th release, contains an additional fix for the write inconsistency issue.
The release should have no impact on running queries, which will transparently be transferred to the new Presto query engine version.
As communicated in the last postmortem, due to the Presto write inconsistency having affected the data integrity of the platform, we have taken the following additional precautions:
* Remediation We removed the code responsible for the aggressive write optimization that caused the write inconsistency bug.
* Verification We have reproduced the write inconsistency issue and we built a reliable set of tests to confirm the fix is effective.
* Detection We implemented additional application logic to detect eventual race conditions (e.g. throw an exception, alert about the anomaly, etc...) when writing into our Plazma storage. Should a race condition even occur (not expected), it will raise an exception forcing the query to error out and an alert to be sent to our staff. Upon receiving an alert, our staff will investigate the situation and when warranted, reach out to the customer to recommend data recovery.
* Monitoring Our Presto monitoring and alerting was improved. Our team will follow an on-call duty rotation to monitor the health of the system post release for an extended amount of time (96 hours or 3 days) and catch any anomalies.
# Communication
Beyond this notice, we will provide updates at the start and completion of the operation and once the verification of the new system is completed: at that time, all systems will have returned to full functionality and this Scheduled Maintenance will be closed.
If you have any question or concern about this maintenance, please feel free to reach out to our Support team at support@treasure-data.com.
Posted Jul 27, 2020 - 16:13 PDT
This scheduled maintenance affected: US (Presto Query Engine) and Tokyo (Presto Query Engine).