0
$\begingroup$

I have created the following function which converts an XML File to a DataFrame. This function works good for files smaller than 1 GB, for anything greater than that the RAM(13GB Google Colab RAM) crashes. Same happens if I try it locally on Jupyter Notebook (4GB Laptop RAM). Is there a way to optimize the code?

Code

#Libraries
import pandas as pd
import xml.etree.cElementTree as ET

#Function to convert XML file to Pandas Dataframe    
def xml2df(file_path):

  #Parsing XML File and obtaining root
  tree = ET.parse(file_path)
  root = tree.getroot()

  dict_list = []

  for _, elem in ET.iterparse(file_path, events=("end",)):
      if elem.tag == "row":
        dict_list.append(elem.attrib)      # PARSE ALL ATTRIBUTES
        elem.clear()

  df = pd.DataFrame(dict_list)
  return df

Part of an XML File ('Badges.xml')

<badges>
  <row Id="82946" UserId="3718" Name="Teacher" Date="2008-09-15T08:55:03.923" Class="3" TagBased="False" />
  <row Id="82947" UserId="994" Name="Teacher" Date="2008-09-15T08:55:03.957" Class="3" TagBased="False" />
  <row Id="82949" UserId="3893" Name="Teacher" Date="2008-09-15T08:55:03.957" Class="3" TagBased="False" />
  <row Id="82950" UserId="4591" Name="Teacher" Date="2008-09-15T08:55:03.957" Class="3" TagBased="False" />
  <row Id="82951" UserId="5196" Name="Teacher" Date="2008-09-15T08:55:03.957" Class="3" TagBased="False" />
  <row Id="82952" UserId="2635" Name="Teacher" Date="2008-09-15T08:55:03.957" Class="3" TagBased="False" />
  <row Id="82953" UserId="1113" Name="Teacher" Date="2008-09-15T08:55:03.957" Class="3" TagBased="False" />

This conversion in needed so that I can perform furthur Data Analysis.

I have asked this on StackOverflow (Link) but the answers did not solve my query. I hope to find a solution here.

$\endgroup$
0
$\begingroup$
import dask
import dask.bag as db
import dask.dataframe as dd
from dask.dot import dot_graph
from dask.diagnostics import ProgressBar

dask.set_options(get=dask.multiprocessing.get)
tags_xml = db.read_text('data/Tags.xml', encoding='utf-8')
tags_xml.take(10)

Refer this link for complete tutorial Dask with XML

| improve this answer | |
$\endgroup$
  • $\begingroup$ BTW 4GB is not sufficient enough for DS. And if its provided by your organization, please do let them know to give a good computation power if they really want you to work on Data Analysis/Data Science. Most Organization just go stingy when it comes to shelling out $$ for good computing rack or cloud services and force a DS/DA guy to work on 4gb-6GB Laptops. if they cant, look for better organization or request them to shut their AI shops. $\endgroup$ – Syenix Aug 6 at 21:23
  • $\begingroup$ Is it possible to create a pandas dataframe from xml considering the given file has more than 500k rows? $\endgroup$ – Ishan Dutta Aug 7 at 4:08
  • $\begingroup$ Did you even try the link which I posted? 🤦🏼‍♂️ $\endgroup$ – Syenix Aug 7 at 5:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.