I'm trying to process a single large XML file with many records and then parse each record in parallel using AWS EMR.


I can read in a small XML and loop with Pig program step in EMR:

DEFINE XPath org.apache.pig.piggybank.evaluation.xml.XPath();

A = LOAD '$INPUT' using org.apache.pig.piggybank.storage.XMLLoader('record') as (x:chararray);
B = FOREACH A GENERATE XPath(x, 'record/id'),XPath(x,'record/name');
dump B;

I can also read in the file, broken into records, using a Streaming program step with -inputreader StreamXmlRecordReader,begin=<record>,end=</record> as an argument.

Currently the Pig script is breaking with the large XML, it works fine on a smaller sample. What I really need is to break this large XML into single records, then parse each record in parallel using a chain of Pig (for the XML/XPath) => further custom steps to parse each subsequent output in parallel, how would I do this? How do you organise AWS EMR to run one step, then split into many steps?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.