*Maintenance* and system *recovery* have been fully completed.
From 10:18 to 10:21, Streaming Imports request from fluentd / td-agent received 5XX API errors. Instances of fluentd / td-agent should handle the request errors automatically by buffering the data and retrying the uploads.
The scheduled database maintenance is now complete.
Posted Jul 16, 2019 - 18:38 PDT
Verifying
The scheduled database failover are complete.
We are monitoring the system closely to ensure all systems successfully complete their *recovery* and return as quickly as possible to full functionality.
Posted Jul 16, 2019 - 18:31 PDT
In progress
The scheduled database maintenance window is starting now.
During the maintenance window, the database will experience failover, after which the respective systems are expected to recover in approximately 10 minutes. During the failover period, customers will observe the following symptoms:
- Streaming Imports error
During the database failover Streaming import (through td-agent or fluentd) requests will receive HTTP 5XX errors. Instances of fluentd / td-agent will handle the request errors automatically by buffering the data and retrying the uploads once the import service will have recovered. Customers interfacing with the Streaming Import REST API directly may have to keep track of the failures and execute the request again after the import service will have recovered.
- Mobile and JavaScript/Browser imports delay
Import requests from Browsers (Javascript SDK) and Mobiles (Android, iOS, and Unity SDKs) will continue to be accepted as usual but the requests will remain queued until after the database failover is complete. This will protract until after the import service will have recovered and shortly thereafter, up to 10 minutes.
Posted Jul 16, 2019 - 18:06 PDT
Update
In about an hour, from 10:00 AM JST, the maintenance window for the databases supporting the Streaming Import REST API will commence.
During the maintenance window, the database will experience failover, after which the respective systems are expected to recover in approximately 10 minutes. During the failover period, customers will observe the following symptoms:
- Streaming Imports error
During the database failover Streaming import (through td-agent or fluentd) requests will receive HTTP 5XX errors. Instances of fluentd / td-agent will handle the request errors automatically by buffering the data and retrying the uploads once the import service will have recovered. Customers interfacing with the Streaming Import REST API directly may have to keep track of the failures and execute the request again after the import service will have recovered.
- Mobile and JavaScript/Browser imports delay
Import requests from Browsers (Javascript SDK) and Mobiles (Android, iOS, and Unity SDKs) will continue to be accepted as usual but the requests will remain queued until after the database failover is complete. This will protract until after the import service will have recovered and shortly thereafter, up to 10 minutes.
Beyond this notice, we will provide updates at the start and completion of the operation and once the verification of the new system is completed: at that time, all systems will have returned to full functionality and this Scheduled Maintenance will be closed.
Posted Jul 16, 2019 - 17:04 PDT
Scheduled
On Wednesday, July 17th from 10 AM to 10:30 AM JST the databases supporting the Streaming Import REST API will undergo a short maintenance.
During the maintenance window, the database will experience failover, after which the respective systems are expected to recover in approximately 10 minutes. During the failover period, customers will observe the following symptoms:
- Streaming Imports error
During the database failover Streaming import (through td-agent or fluentd) requests will receive HTTP 5XX errors. Instances of fluentd / td-agent will handle the request errors automatically by buffering the data and retrying the uploads once the import service will have recovered. Customers interfacing with the Streaming Import REST API directly may have to keep track of the failures and execute the request again after the import service will have recovered.
- Mobile and JavaScript/Browser imports delay
Import requests from Browsers (Javascript SDK) and Mobiles (Android, iOS, and Unity SDKs) will continue to be accepted as usual but the requests will remain queued until after the database failover is complete. This will protract until after the import service will have recovered and shortly thereafter, up to 10 minutes.
Beyond this notice, we will provide updates approximately 1 hour before the beginning of the maintenance window, at the start and completion of the operation, and once the verification is completed. At that time, all systems will have returned to full functionality and the Scheduled Maintenance will be closed.
If you have any question or concern about this upgrade, please feel free to reach out to our Support team at support@treasure-data.com.
Posted Jul 02, 2019 - 21:21 PDT
This scheduled maintenance affected: Tokyo (Streaming Import REST API, Mobile/Javascript REST API).