Between Thursday, 23 Jan 2025 07:20 UTC to 11:40 UTC, customers experienced elevated error rates and increased latency related to Profiles API. A fix has been implemented, and the issue has been resolved.
If you experience any delays or abnormal errors, please reach out to our support team. Thank you for your patience and understanding during this incident. We will share an incident retrospective soon.
Posted Jan 23, 2025 - 04:18 PST
Monitoring
The response team found problematic real-time segment configurations of one customer's Parent Segment that possibly contributed to consuming the concurrency capacity. The team updated the real-time event routing configuration to mitigate the high latency issue. Combined with capacity addition operations, the team stabilized the Profiles API cache cluster.
If you experience any delays or abnormal errors, please reach out to our support team. Thank you for your patience and understanding during this incident. We will update the postmortem with further remediation plan as promised.
Posted Jan 23, 2025 - 04:16 PST
Update
We successfully provisioned 2x capacity in 30 minutes. New resources improved the latency, but the error rate is still high. The response team is planning to implement another remediation instead of adding resources. We will update you in 30 minutes.
Posted Jan 23, 2025 - 03:08 PST
Update
We provisioned additional capacity at 10:00 am UTC to support the increasing workload. It improved the latency, but we still observed errors and long latency for a small amount of requests. The response team started providing another concurrency capacity. Unlike the previous methods, the new process should not take longer for provisioning. We will update the result in 30 minutes.
Posted Jan 23, 2025 - 02:37 PST
Identified
The response team confirmed the symptom is from the same cause as the previous incidents. We are provisioning additional concurrency capacity to the environment. We will update you when it is completed.
Posted Jan 23, 2025 - 00:05 PST
Investigating
We are currently observing errors or performance degradation for the personalization API. We are investigating the cause of the issue now.
Posted Jan 22, 2025 - 23:34 PST
This incident affected: EU (CDP Personalization - Lookup API, CDP Personalization - Ingest API).