I'm trying to process a single large XML file with many records and then parse each record in parallel using AWS EMR.
<records>
<record>
<id>1</id>
<name>Jane</name>
</record>
<record>
<id>2</id>
<name>John</name>
</record>
</records>
I can read in a small XML and loop with Pig program
step in EMR:
DEFINE XPath org.apache.pig.piggybank.evaluation.xml.XPath();
A = LOAD '$INPUT' using org.apache.pig.piggybank.storage.XMLLoader('record') as (x:chararray);
B = FOREACH A GENERATE XPath(x, 'record/id'),XPath(x,'record/name');
dump B;
I can also read in the file, broken into records, using a Streaming program
step with -inputreader StreamXmlRecordReader,begin=<record>,end=</record>
as an argument.
Currently the Pig script is breaking with the large XML, it works fine on a smaller sample. What I really need is to break this large XML into single records, then parse each record in parallel using a chain of Pig (for the XML/XPath) => further custom steps to parse each subsequent output in parallel, how would I do this? How do you organise AWS EMR to run one step, then split into many steps?