I'm trying to process a single large XML file with many records and then parse each record in parallel using AWS EMR.
<records> <record> <id>1</id> <name>Jane</name> </record> <record> <id>2</id> <name>John</name> </record> </records>
I can read in a small XML and loop with
Pig program step in EMR:
DEFINE XPath org.apache.pig.piggybank.evaluation.xml.XPath(); A = LOAD '$INPUT' using org.apache.pig.piggybank.storage.XMLLoader('record') as (x:chararray); B = FOREACH A GENERATE XPath(x, 'record/id'),XPath(x,'record/name'); dump B;
I can also read in the file, broken into records, using a
Streaming program step with
-inputreader StreamXmlRecordReader,begin=<record>,end=</record> as an argument.
Currently the Pig script is breaking with the large XML, it works fine on a smaller sample. What I really need is to break this large XML into single records, then parse each record in parallel using a chain of Pig (for the XML/XPath) => further custom steps to parse each subsequent output in parallel, how would I do this? How do you organise AWS EMR to run one step, then split into many steps?