The Event Collector's queue depth is now back to normal and all the pending uploads have been caught up without loss of records.
This is incident is now resolved.
Posted Jan 21, 2015 - 16:21 PST
Monitoring
We fixed the Event collector problem and restarted the instances.
The instances are now full dedicated to consume the queued events and upload them to Treasure Data.
We will send a final update when the Event Collector will be fully caught up.
Posted Jan 21, 2015 - 15:57 PST
Identified
We are experiencing problems with our Event Collector SDK Backend.
Due to several consecutive failures in attempts to upload data through our API Backend, starting at ~ 10:50 AM PST today, the Event Collector has not been able to upload records to Treasure Data.
All user data is currently queued therefore customers may experience significant delays in getting their records uploaded to Treasure Data; however no record has been lost in the process.
We are fixing the problem and restarting our Event Collector instances: the collector will begin to catch up on the queued uploads immediately and we will monitor the queue depth until the missing uploads are all caught up.