I am going to build a 1,000,000 * 2 dataframe in Python and the input I have is in the form of 1*2. So obviously the most straightforward method is to use loop and append a 1*2 pair into the existing dataframe for 1,000,000 times... However, it really takes forever and as a Matlab user, I am almost forbidden in Matlab to do this and always told to initiate the final size of the matrix (or dataframe in Python) first and put elements into the matrix... Is this the same in Python? Thanks in advance!
8
-
$\begingroup$ do you want to read from file? $\endgroup$– Green FalconMar 29, 2018 at 10:04
-
$\begingroup$ my inputs are from a web-based xml file... I get input from there which is 1*2 pairs $\endgroup$– TingTingMar 29, 2018 at 10:06
-
$\begingroup$ Do you get them all at once, or separately? $\endgroup$– Green FalconMar 29, 2018 at 10:08
-
$\begingroup$ i get them separately $\endgroup$– TingTingMar 29, 2018 at 10:37
-
1$\begingroup$ This is referred to as pre-allocation and is preferred stackoverflow.com/questions/25649788/… $\endgroup$– Sam WeisenthalMar 29, 2018 at 12:54
|
Show 3 more comments
1 Answer
$\begingroup$
$\endgroup$
Yes it will be just as slow as in matlab. You’re going to be copying the whole data frame every time you append something.
