[US Region] Custom Script execution failure
Incident Report for Treasure Data
Resolved
We haven’t found any issue or suspicious behavior in our platform during last 1 hour monitoring. This incident has been resolved.
Posted Nov 26, 2020 - 01:19 PST
Update
We have not been detecting any issue on our platform since the immediate actions to prevent done by our infrastructure partner. Our infrastructure partner has restored all requests and completed the remediations. The issue has been resolved. We will continue to monitor closely for any further issue.
Posted Nov 26, 2020 - 00:08 PST
Update
Our infrastructure partner is continuing to incrementally restore requests. We continue to monitor for any further issues and updates.
Posted Nov 25, 2020 - 22:55 PST
Update
Tasks are continuing to succeed and the system is slowly recovering. Our infrastructure partner is bringing traffic in more slowly than anticipated after recovery to ensure continued stability. It is expected to be completed over the next few hours. We continue to monitor for any further issues and updates.
Posted Nov 25, 2020 - 21:34 PST
Update
Tasks are continuing to succeed and the system is slowly recovering. Our infrastructure partner continues working towards recovery. During this recovery period their services are not at full throughput, which may result in delays or errors. We continue to monitor for any further issues and updates.
Posted Nov 25, 2020 - 20:27 PST
Update
Tasks are continuing to succeed and the system is slowly recovering. Our infrastructure partner has mitigated the core issue and is working on recovery. During this recovery period their services are not at full throughput, which may result in delays or errors. We continue to monitor for any further issues and updates.
Posted Nov 25, 2020 - 19:30 PST
Update
Tasks are continuing to succeed and the system is slowly recovering. Our infrastructure partner is still reporting issues and working through remediations. We continue to monitor for any further issues.
Posted Nov 25, 2020 - 18:32 PST
Update
We are continuing to monitor for any further issues.
Posted Nov 25, 2020 - 17:44 PST
Update
We are continuing to monitor for any further issues.
Posted Nov 25, 2020 - 16:59 PST
Update
We are continuing to monitor for any further issues.
Posted Nov 25, 2020 - 16:31 PST
Update
We are continuing to see successful task launches and continue to work with with our infrastructure partner to further remediate.
Posted Nov 25, 2020 - 16:04 PST
Monitoring
We are starting to see recovery from our infrastructure provider, and some tasks are successfully launching. We are continuing to work with our infrastructure partner to remediate.

In addition, task logs may be delayed due to the same root cause.
Posted Nov 25, 2020 - 15:35 PST
Update
We have confirmed the root cause is the result of an ongoing incident at our infrastructure provider. We're working with the partner to resolve the incident.
Posted Nov 25, 2020 - 15:04 PST
Identified
Custom Scripts on Treasure Workflow are failing to create a new task. Tasks may show an error with the following message.

- java.lang.RuntimeException: Submitted task could not be found
- Unable to unmarshall exception response with the unmarshallers provided.

We suspect the root cause is the result of the ongoing incident at our infrastructure provider.
We're working with the partner to resolve the incident.
Posted Nov 25, 2020 - 14:31 PST
This incident affected: US (Workflow).