We confirmed the catch-up is complete at 6:44 am PT. From 2024-08-06 03:20 am to 2024-08-07 06:44 am PT, the events arrived at us01.records.in.treasuredata.com and c360-ingest-api.treasuredata.com experienced maximum 8 hours of delay in batch data ingestion. There was no impact in real-time system.
Posted Aug 07, 2024 - 07:23 PDT
Update
We are continuing to monitor for any further issues.
Posted Aug 07, 2024 - 06:15 PDT
Update
We are continuing to monitor the recovery. As of now, 99% of events become visible within 45 minutes. We will resolve the incident when the catch-up is complete.
Posted Aug 07, 2024 - 06:13 PDT
Update
We are continuing to monitor for any further issues.
Posted Aug 07, 2024 - 06:09 PDT
Update
We are continuing to monitor our systems' recovery as we work through the backlog of messages sent in the last 12 hours. We have added more resources to reduce the impact of this issue. At this time, we expect all messages to be processed, but customers may continue to see multi-hour delays as we continue to process messages to our Ingest API for the next few hours.
We will continue to monitor this issue, and we appreciate your patience as we work through it.
Posted Aug 06, 2024 - 18:12 PDT
Monitoring
We have rolled out a fix and observed that processing delays are no longer increasing. Customers may continue to see delayed message processing over the next 3-4 hours as the backlog is processed. We continue exploring options to shorten this time and will monitor for any issues.
Posted Aug 06, 2024 - 11:18 PDT
Identified
We have identified the source of the problem and are applying a solution now. Customers may still see processing delays as we catch up on the request backlog. We will continue to explore options to accelerate our recovery, and we will continue to monitor the situation.
Posted Aug 06, 2024 - 10:41 PDT
Update
We are observing slower processing time for messages sent to our Ingest API. Users may see a delay up to two hours in message processing. We are continuing to investigate the root cause and exploring options to catch up on our backlog of messages, and will provide an update once we know more.
Posted Aug 06, 2024 - 10:21 PDT
Update
We are still investigating the cause of this issue.
Posted Aug 06, 2024 - 09:04 PDT
Investigating
Our Ingest API is experiencing a performance issue. We are investigating the cause.
Posted Aug 06, 2024 - 08:32 PDT
This incident affected: US (Streaming Import REST API, CDP Personalization - Ingest API).