[US Region] Scheduled Metadata Database maintenance

Scheduled Maintenance Report for Treasure Data

Completed

*Maintenance* and system *recovery* have been fully completed.

The scheduled database maintenance is now complete. The downtime was 106 seconds (1 minute and 46 seconds) in total.

Posted Feb 01, 2021 - 17:32 PST

Verifying

The scheduled database maintenance is complete.

We are monitoring the system closely to ensure all systems successfully complete their *recovery* and return as quickly as possible to full functionality.

Posted Feb 01, 2021 - 17:20 PST

In progress

The scheduled database maintenance window is starting now.

During the maintenance and recovery, customers may experience the following:

- Streaming, Mobile, and JavaScript/Browser imports delay

Streaming import (through td-agent or fluentd) requests will continue to be accepted as usual but the requests will remain queued until after the database maintenance is complete. We expect stream import processing to be further delayed during recovery.
The same will apply to import requests via event-collector from Browsers (Javascript SDK) and Mobiles (Android, iOS, and Unity SDKs).

- Jobs execution delay

All jobs (Presto, Hive, Result Export, Data Connector Integrations, Bulk Import, Export, and Partial Delete jobs submitted from Console, API, Workflow or triggered by our system according to the configured schedule) will fail and continue to retry during maintenance. During recovery, we expect jobs to begin processing slowly: within 30 minutes job processing should reach back to full throughput.

- Presto JDBC / ODBC Gateway return errors during downtime

The Presto JDBC / ODBC Gateway will return errors during downtime period.

- Console execution delay

Data Workbench and Audience Studio will show that jobs are running longer due to underlying job execution delay due to failure of the underlying Parent Segments, Segments, and Workflows jobs that overlap with the maintenance period. During recovery, we expect processing of Data Workbench and Audience Studio jobs to follow the same recovery pattern as all other jobs (see above).

- Data Access API (beta) return errors during downtime

Data Access API (Plazma Public API) for Spark will return errors during downtime period.

Posted Feb 01, 2021 - 17:00 PST

Update

In about an hour, from 17:00 PST (10:00 JST, 02:00 CET), the maintenance window for the PlazmaDB Metadata database will commence.

During the maintenance and recovery, customers may experience the following:

- Streaming, Mobile, and JavaScript/Browser imports delay

Streaming import (through td-agent or fluentd) requests will continue to be accepted as usual but the requests will remain queued until after the database maintenance is complete. We expect stream import processing to be further delayed during recovery.
The same will apply to import requests via event-collector from Browsers (Javascript SDK) and Mobiles (Android, iOS, and Unity SDKs).

- Jobs execution delay

All jobs (Presto, Hive, Result Export, Data Connector Integrations, Bulk Import, Export, and Partial Delete jobs submitted from Console, API, Workflow or triggered by our system according to the configured schedule) will fail and continue to retry during maintenance. During recovery, we expect jobs to begin processing slowly: within 30 minutes job processing should reach back to full throughput.

- Presto JDBC / ODBC Gateway return errors during downtime

The Presto JDBC / ODBC Gateway will return errors during downtime period.

- Console execution delay

Data Workbench and Audience Studio will show that jobs are running longer due to underlying job execution delay due to failure of the underlying Parent Segments, Segments, and Workflows jobs that overlap with the maintenance period. During recovery, we expect processing of Data Workbench and Audience Studio jobs to follow the same recovery pattern as all other jobs (see above).

- Data Access API (beta) return errors during downtime

Data Access API (Plazma Public API) for Spark will return errors during downtime period.

Beyond this notice, we will provide updates at the start and completion of the operation, and once the verification of the new system is completed. At that time, all systems will have returned to full functionality and this Scheduled Maintenance will be closed.

Posted Feb 01, 2021 - 16:00 PST

Scheduled

On Monday, February 1st from 17:00 to 18:00 PST (Tuesday, February 2nd from 10:00 to 11:00 JST and from 02:00 to 03:00 CET) we will be performing maintenance on the PlazmaDB Metadata database.

The PlazmaDB Metadata database maintenance we performed last week on January 19th, was able to free up enough the minimum required amount of capacity to allow us to continue operating the system without immediate risk of downtime short term. With this maintenance we will scale up the database instance to allocate additional capacity that will be sufficient to comfortably accommodate the peak load, now and in the foreseeable future.
We expect the scaled up instance will lead to better performing customer queries and reduced import delay during data ingestion.

Although the maintenance window is 60 minutes, the database will only become unreachable for a portion of the migration which should last no longer than 10 minutes in total. We expect this to be followed by a recovery period of around 30 minutes during which the system will gradually reach back to full throughput.

# Impact

During the maintenance and recovery, customers may experience the following:

- Streaming, Mobile, and JavaScript/Browser imports delay

Streaming import (through td-agent or fluentd) requests will continue to be accepted as usual but the requests will remain queued until after the database maintenance is complete. We expect stream import processing to be further delayed during recovery.
The same will apply to import requests via event-collector from Browsers (Javascript SDK) and Mobiles (Android, iOS, and Unity SDKs).

- Jobs execution delay

All jobs (Presto, Hive, Result Export, Data Connector Integrations, Bulk Import, Export, and Partial Delete jobs submitted from Console, API, Workflow or triggered by our system according to the configured schedule) will fail and continue to retry during maintenance. During recovery, we expect jobs to begin processing slowly: within 30 minutes job processing should reach back to full throughput.

- Presto JDBC / ODBC Gateway return errors during downtime

The Presto JDBC / ODBC Gateway will return errors during downtime period.

- Console execution delay

Data Workbench and Audience Studio will show that jobs are running longer due to underlying job execution delay due to failure of the underlying Parent Segments, Segments, and Workflows jobs that overlap with the maintenance period. During recovery, we expect processing of Data Workbench and Audience Studio jobs to follow the same recovery pattern as all other jobs (see above).

- Data Access API (beta) return errors during downtime

Data Access API (Plazma Public API) for Spark will return errors during downtime period.

# Communication

Beyond this notice, we will provide updates approximately 1 hour before the beginning of the maintenance window, at the start and completion of the operation, and once the verification is complete. At that time, all systems will have returned to full functionality and the Scheduled Maintenance will be closed.

If you have any question or concern about this upgrade, please feel free to reach out to our Support team at support@treasuredata.com.

Posted Jan 27, 2021 - 22:59 PST

This scheduled maintenance affected: US (Web Interface, REST API, Streaming Import REST API, Mobile/Javascript REST API, Data Connector Integrations, Hadoop / Hive Query Engine, Presto Query Engine, Presto JDBC/ODBC Gateway, Workflow, CDP API, Data Access API (beta)).