Monitoring - We have rolled out a fix and are observing our backlog recovering at a safe rate. We are currently processing a backlog of historical data, which is progressing at a controlled rate. Estimated time to a full recover of all data is approximately 32 hours, and we will be monitoring and provide updates through this process.
Apr 21, 2025 - 11:59 PDT
Update - We are now attempting to restore the pipeline from a timestamp rather than a checkpoint. There is no risk of user data loss. Downstream systems may observe duplicate events from the affected time window, and this will be limited in scope.
Apr 21, 2025 - 10:43 PDT
Identified - We have applied a fix and are observing it.
We have identified a window between Thursday 3AM UTC and 11AM Monday UTC where data was not being processed. We have implemented a remediation and new incoming data is not impacted. We are actively working to resume sending of this data, which will arrive out of order along with new incoming data.
Apr 21, 2025 - 09:40 PDT
Update - We are continuing to investigate this issue.
Apr 21, 2025 - 09:25 PDT
Investigating - Our Ingest API is experiencing a performance issue. We are investigating the cause.
Apr 21, 2025 - 09:10 PDT
On 2025-04-25 from 05:00 to 06:00 UTC (2025-04-24 from 22:00 to 23:00 PDT), we will be performing maintenance on the Data Tank 2.0 Management Console. This maintenance window is required to perform a security update. Although the maintenance window duration allocated is 1 hour, we expect the management console will become unavailable for no longer than 30 minutes in total.
# Impact
During the maintenance, customers may experience the following: • Access to Data Tank 2.0 Management Console will be unavailable.
# Data Tank 2.0 core service availability
We want to reiterate that this change *will not* impact the availability of the Data Tank 2.0 core services, only the management console.
# Communication
Beyond this notice, we will provide updates approximately 1 hour before the beginning of the maintenance window, at the start and completion of the operation, and once the verification is completed. At that time, all systems will have returned to full functionality and the scheduled maintenance will be closed.
If you have any question about this maintenance, please feel free to reach out to our Support team at support@treasuredata.com.
We appreciate your understanding in this matter. Posted on
Apr 09, 2025 - 00:42 PDT
Resolved -
We have confirmed that Hive jobs are now working properly, and the related Result Export jobs are functioning as expected. The impact occurred between April 8, 16:00 UTC and April 9, 01:35 UTC. If you had any jobs that failed during the affected time window, please re-run them as needed. We sincerely apologize for the inconvenience this my have caused.
Apr 8, 20:22 PDT
Monitoring -
We have identified the root cause and applied a hotfix.
The issue was that Hive Jobs were unable to start properly, which in turn caused some Result Export jobs triggered from those Hive Jobs to fail. We are currently monitoring the system to ensure that the situation continues to improve.
Apr 8, 18:49 PDT
Investigating -
We are currently investigating an issue where Hive Jobs and the Result Export jobs triggered from Hive Jobs are not functioning properly.
Our team is actively looking into the cause of the issue. We will provide updates as soon as more information becomes available.
Apr 8, 18:24 PDT