[All Regions] Elevated error rate for CDP KVS

Incident Report for Treasure Data

Resolved

We would like to inform you that the issue has been fully resolved.

Incident Impact Details:
- Profiles API experienced an increased frequency of errors and timeouts.
- The latest logs were not reflected in real-time segments.

Incident Impact Time by Region:
us:
- Start: November 13, 04:14 UTC
- End: November 13, 08:55 UTC

aws-tokyo:
- Start: November 13, 04:14 UTC
- End: November 13, 08:54 UTC

eu01:
- Start: November 13, 04:17 UTC
- End: November 13, 08:51 UTC

ap02
- Start: November 13, 04:15 UTC
- End: November 13, 09:01 UTC

ap03
- Start: November 13, 04:17 UTC
- End: November 13, 08:52 UTC

We apologize for any inconvenience this may have caused and thank you for your patience and understanding.
Posted 6 months ago. Nov 13, 2024 - 02:16 PST

Monitoring

Through our investigation, we identified the cause of the issue as a recent release operation. We have reverted all changes from this release, and normal functionality has been restored. Our team is closely monitoring the system to ensure continued stability.
Posted 6 months ago. Nov 13, 2024 - 01:17 PST

Investigating

Since approximately 4:00 UTC, we have been experiencing an issue with requests to CDP KVS, which may be affecting Profiles API functionality, causing delays in KVS data synchronization and updates to real-time segment information. Our team is actively investigating and working to resolve the issue as quickly as possible.

Please note that Realtime 2.0 is not affected.
Posted 6 months ago. Nov 12, 2024 - 23:09 PST
This incident affected: US (CDP Personalization - Lookup API, CDP Personalization - Ingest API), Tokyo (CDP Personalization - Lookup API, CDP Personalization - Ingest API), EU (CDP Personalization - Lookup API, CDP Personalization - Ingest API), Korea (CDP Personalization - Lookup API, CDP Personalization - Ingest API), and AP03 (CDP Personalization - Lookup API, CDP Personalization - Ingest API).