All systems are working normally after 04:57 PDT. This incident is resolved.
From 04:43 to 04:57 PDT our API server showed high error rate due to slow backend DB response. Mobile import chunks were queued at our server and processed quickly after API recovered. td-agent has retry mechanism therefore no data loss should have happened. Scheduled jobs also has retry mechanism and all queued scheduled jobs were processed, too.
td command may fail due to API error response during this incident. Please retry the command execution.
Sorry for the inconvenience this incident has caused.
Posted Apr 21, 2017 - 05:34 PDT
Monitoring
From 04:43 PDT one of our backend DB server underwent slow storage IO. At 04:57 the IO problem was solved.
Posted Apr 21, 2017 - 05:08 PDT
Investigating
We are observing high error rate of API. We are investigating now.
Posted Apr 21, 2017 - 04:58 PDT
This incident affected: US (Web Interface, REST API, Streaming Import REST API, Mobile/Javascript REST API).