2
$\begingroup$

I am going to build a 1,000,000 * 2 dataframe in Python and the input I have is in the form of 1*2. So obviously the most straightforward method is to use loop and append a 1*2 pair into the existing dataframe for 1,000,000 times... However, it really takes forever and as a Matlab user, I am almost forbidden in Matlab to do this and always told to initiate the final size of the matrix (or dataframe in Python) first and put elements into the matrix... Is this the same in Python? Thanks in advance!

$\endgroup$
8
  • $\begingroup$ do you want to read from file? $\endgroup$ Mar 29, 2018 at 10:04
  • $\begingroup$ my inputs are from a web-based xml file... I get input from there which is 1*2 pairs $\endgroup$
    – TingTing
    Mar 29, 2018 at 10:06
  • $\begingroup$ Do you get them all at once, or separately? $\endgroup$ Mar 29, 2018 at 10:08
  • $\begingroup$ i get them separately $\endgroup$
    – TingTing
    Mar 29, 2018 at 10:37
  • 1
    $\begingroup$ This is referred to as pre-allocation and is preferred stackoverflow.com/questions/25649788/… $\endgroup$ Mar 29, 2018 at 12:54

1 Answer 1

1
$\begingroup$

Yes it will be just as slow as in matlab. You’re going to be copying the whole data frame every time you append something.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.