Between Wednesday, 29 Jan 2025, 15:47 UTC to 16:51 UTC, customers experienced elevated error rates and increased latency related to Profiles API. The cause was a slightly but non-visible elevated error rate monitor kicked a system recovery operation. Then, the recovery operation caused the same incident due to a configuration problem we had on Friday. https://status.treasuredata.com/incidents/jyqjpyscvjzh
The response team re-deployed the safe version to recover the system. Also, as a short-term mitigation, we updated the recovery operation until we complete the root cause analysis and permanent fix we described in "Further Actions" in the previoius postmortem: https://status.treasuredata.com/incidents/jyqjpyscvjzh
At the moment, if you experience any delays or abnormal errors, please reach out to our support team. Thank you for your patience and understanding during this incident.
Posted Jan 29, 2025 - 09:15 PST
Update
We are continuing to monitor for any further issues.
Posted Jan 29, 2025 - 09:01 PST
Update
Currently, we can see a lot of improvement in the monitoring of health status. We continue to carefully monitor the health status.
Posted Jan 29, 2025 - 08:57 PST
Monitoring
We started to apply a remediation and we are observing that the service is recovering. However, we closely monitor the service health status
Posted Jan 29, 2025 - 08:49 PST
Investigating
We detected degraded performance of personalization API and an error rate increase. We are currently investigating this issue.
Posted Jan 29, 2025 - 08:13 PST
This incident affected: EU (CDP API, CDP Personalization - Lookup API, CDP Personalization - Ingest API).