Bulk import problem

Incident Report for Treasure Data

Postmortem

Today we released several bug fixes to Hadoop clusters but it caused Bulk import failure in a few clusters. From 10pm to 7am, Bulk import ran on the clusters failed to import records and reported error to Toolbelt CLI.

Hive/Presto query with "Result Export to Treasure Data" also failed to import query result to the specified table, because it uses Bulk import internally. More importantly "Result Export to Treasure Data" finished successfully with "valid records: 0" like this.

15/07/02 06:25:35 INFO result.TreasureDataResult:   perform job ID: 12345678
15/07/02 06:25:35 INFO result.TreasureDataResult:   valid parts: 0
15/07/02 06:25:35 INFO result.TreasureDataResult:   error parts: 0
15/07/02 06:25:35 INFO result.TreasureDataResult:   valid records: 0
15/07/02 06:25:35 INFO result.TreasureDataResult:   error records: 0

These queries should be re-executed so that query result records are properly imported. To assist it we are now creating a list of affected queries. Please wait for a while until our support team contact to you.

We deeply regret the inconvenience this has caused. Please don't hesitate to contact support@treasure-data.com if there's anything we can do to help resolve this problem for you.

Posted Jul 02, 2015 - 10:37 PDT

Resolved

This incident was resolved. We're going to contact to affected customers. For more question, please contact to support@treasure-data.com.

Posted Jul 02, 2015 - 08:49 PDT

Monitoring

We fixed this Bulk import issue. It seems to have recovered already. Now we're monitoring it.

Posted Jul 02, 2015 - 07:46 PDT

Identified

We found our today's release affected Bulk Import of some users and it can finish with "Valid records: 0". We're recovering our system now. Really sorry for inconvenient.

Posted Jul 02, 2015 - 06:47 PDT

Investigating

We've been reported that Bulk imports don't work now. We're investigating the issue.

Posted Jul 02, 2015 - 05:35 PDT