[US region] Ingest API - Performance Degradation

Incident Report for Treasure Data

Resolved

This incident has been resolved.

Duration:
~35 minutes of processing delay. Data Ingestion delay between June 1 - 7:35 AM to June 1 - 10:35 AM PST.

Affected Customers:
All customers ingesting data to AWS US region during the incident window

Impact:
Delayed data availability in Plazma (up to 40 mins)
No data loss occurred
Posted Jun 02, 2025 - 10:43 PDT

Update

We are continuing to monitor for any further issues.
Posted Jun 02, 2025 - 10:37 PDT

Update

Latest Update:
Our team is still actively monitoring and assessing the processing of the data.

We will leave this status page in a Monitoring state until we are certain everything has been processed.

Thank you for your continued patience.
Posted Jun 02, 2025 - 09:26 PDT

Monitoring

Latest Update:
Our internal graphs show that the overwhelming majority of the data has been processed. There is small amount of residual data that is taking longer than expected to process.

We will leave this status page in a Monitoring state until we are certain everything has been processed.

Thank you for your continued patience.
Posted Jun 01, 2025 - 21:47 PDT

Update

Latest Update:
Our team is still actively monitoring and assessing the processing of the data backlog.

Thank you for your continued patience.
Posted Jun 01, 2025 - 18:49 PDT

Update

Latest Update:
Our team has deployed a fix, and the system is now processing the backlog of data at a controlled rate.

Estimated Time to Full Recovery:
~7 hours

We’re actively monitoring the recovery and will provide further updates as progress continues. Thank you for your patience.
Posted Jun 01, 2025 - 10:18 PDT

Update

Current Status:
A fix has been applied and is currently under observation.

Impact:
An issue was identified with data ingestion(delayed ingestion) between Sunday 8:00 AM PST - 8:30 AM PST.

Remediation:
A fix has been implemented, and new incoming data is now processing normally.

Next Steps:
We are actively working to resume sending the affected data, which will arrive out of order along with new incoming data.

Updates to Follow:
Further details will be provided as the situation progresses.
Posted Jun 01, 2025 - 09:35 PDT

Investigating

Our Ingest API is experiencing a performance issue. We are investigating the cause.
Posted Jun 01, 2025 - 09:17 PDT
This incident affected: US (Streaming Import REST API).