[All Regions] Elevated error rate for CDP KVS
Incident Report for Treasure Data
Resolved
We would like to inform you that the issue has been fully resolved.

Incident Impact Details:
- Profiles API experienced an increased frequency of errors and timeouts.
- The latest logs were not reflected in real-time segments.

Incident Impact Time by Region:
us:
- Start: November 13, 04:14 UTC
- End: November 13, 08:55 UTC

aws-tokyo:
- Start: November 13, 04:14 UTC
- End: November 13, 08:54 UTC

eu01:
- Start: November 13, 04:17 UTC
- End: November 13, 08:51 UTC

ap02
- Start: November 13, 04:15 UTC
- End: November 13, 09:01 UTC

ap03
- Start: November 13, 04:17 UTC
- End: November 13, 08:52 UTC

We apologize for any inconvenience this may have caused and thank you for your patience and understanding.
Posted Nov 13, 2024 - 02:16 PST
Monitoring
Through our investigation, we identified the cause of the issue as a recent release operation. We have reverted all changes from this release, and normal functionality has been restored. Our team is closely monitoring the system to ensure continued stability.
Posted Nov 13, 2024 - 01:17 PST
Investigating
Since approximately 4:00 UTC, we have been experiencing an issue with requests to CDP KVS, which may be affecting Profiles API functionality, causing delays in KVS data synchronization and updates to real-time segment information. Our team is actively investigating and working to resolve the issue as quickly as possible.

Please note that Realtime 2.0 is not affected.
Posted Nov 12, 2024 - 23:09 PST
This incident affected: US (CDP Personalization - Lookup API, CDP Personalization - Ingest API), Tokyo (CDP Personalization - Lookup API, CDP Personalization - Ingest API), EU (CDP Personalization - Lookup API, CDP Personalization - Ingest API), Korea (CDP Personalization - Lookup API, CDP Personalization - Ingest API), and AP03 (CDP Personalization - Lookup API, CDP Personalization - Ingest API).