Background: We are trying to build a customized ML library in Python 3 to tackle analysis we often repeat, in a general fashion. But it would not be nearly as general as sklearn. In fact, we are prepared to break some interfaces if that give us enough performance boost in return.

The basic starting point would be constructing a Learner by feeding it X and y, and predict on new input X0

learner = Learner(X, y)
y0 = learner.predict(X0)

One of the design decision is what data type to use for X and y, here are 3 choices, along with some rudimentary brainstorm advantages for each respectively:

  1. native Python list: X being a list of lists, y being a list. Would this have better performance for being 'closer to metal'?
  2. numpy: X being an ndarray of (n, p), y being an ndarray of (n, ). This would benefit from the richer functionalities in numpy/scipy. This is also the data type choice of sklearn.
  3. pandas: X being a DataFrame, y being a Series. This can utilize more Data Analysis (read: dirty work) functionalities from pandas. This way we can also refer to variables with their names instead of just integer indices. But performance would be the worst?

Please share your thinking of pros and cons for each choice from both tech and math perspectives. Thanks in advance!

PS: I thought about whether this should be a StackOverflow question, but still feel this is more Data Science.


Pandas does normally a decent job allowing dataframes to behave as numpy arrays.

My recommendation is to use numpy types, the reason is that, for consistency with pretty much what the industry is doing, you are much safer with numpy.

I love pandas, and I love the dataframes, but they provide extra functionality that the model does NOT need, the same way in general programming you will not use a String to represent a boolean (even though tou could do it with a String), simply because you should use whatever data types provides you the functionality you need... and nothing else.

So, numpy is the way to go. As for python lists, you do not get the mathematical operations that you get with numpy, so do not consider them.

  • $\begingroup$ Thanks! The extra functionalities in pandas we consider may be useful are: 1) column names, so that we can say we selected 'rain' and 'wind_speed' as the two variables in our regression model instead of X3 and X8. 2) all the SQL like functionalities, such as df[df.rain > 0.5], group by, or apply. What are your thoughts? $\endgroup$ – Indominus Jan 14 '19 at 3:40
  • $\begingroup$ But the algorithm should NOT need to care about column names, just about matrices. The same goes for filtering, that should all be done BEFORE feeding the data into the algorithm, $\endgroup$ – Juan Antonio Gomez Moriano Jan 14 '19 at 3:43
  • 1
    $\begingroup$ A lot of numpy is written in C and Fortran (it also allows interaction with C, C++, Fortran) so also has the added speed benefit. $\endgroup$ – bradS Jan 14 '19 at 11:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.