I have a question about memory usage.

I want to do 4 things:

1) make a dataframe from one of several columns from a datasource, say a json string
2) make the third column of the original dataset the index to the dataframe
3) change the name of another column
4) change the series i've created to a dataframe

My question is about memory efficiency. It seems that for step 1), I am first loading a whole dataframe, then run a concat command to concatenate the columns I want.

For step 2, I again need to resave the new dataframe as another object.

For step 3, it seems to stick so nothing there.

Please advise on a more efficient way to go about this, if that exists.


   df = pd.DataFrame(jsonobject)
   df = df.set_index("columnC")
   df.index.names= ["foo"]
   df1 = df["foo"].map(lambda x:x["id"])
   df2 = pd.DataFrame(df1)
  • $\begingroup$ Sorry, what you are doing here df1 = df["foo"].map(lambda x:x["id"])? And also could you post few lines of sample data? $\endgroup$ Nov 16 '15 at 7:43

Browse other questions tagged or ask your own question.