The error late became low as before. This incident was resolved.
Due to upstream DNS provider problem, 3 out of 42115 job submissions were failed (0.007%) across all Treasure Data customers. 703 out of 15785807 (0.004%) streaming import requests were failed, but td-agent retries the request automatically.
Posted Oct 21, 2016 - 08:06 PDT
Monitoring
Upstream DNS provider resolved the DNS problem (http://status.aws.amazon.com/) but we keep observing slightly higher error rate at the specific data center. We keep monitoring for a while.
Posted Oct 21, 2016 - 07:20 PDT
Identified
We're observing elevated error rate of Job submission and Streaming Import API due to infrastructure DNS resolution problem. You may receive "500 Internal Server Error" from job submission API cal. In that case please check job list and resubmit after confirming the job is not yet executed. For Streaming Import, td-agent retries the request automatically.