# Machine Learning library in Python, list or numpy or pandas

Background: We are trying to build a customized ML library in Python 3 to tackle analysis we often repeat, in a general fashion. But it would not be nearly as general as sklearn. In fact, we are prepared to break some interfaces if that give us enough performance boost in return.

The basic starting point would be constructing a Learner by feeding it X and y, and predict on new input X0

learner = Learner(X, y)
y0 = learner.predict(X0)


One of the design decision is what data type to use for X and y, here are 3 choices, along with some rudimentary brainstorm advantages for each respectively:

1. native Python list: X being a list of lists, y being a list. Would this have better performance for being 'closer to metal'?
2. numpy: X being an ndarray of (n, p), y being an ndarray of (n, ). This would benefit from the richer functionalities in numpy/scipy. This is also the data type choice of sklearn.
3. pandas: X being a DataFrame, y being a Series. This can utilize more Data Analysis (read: dirty work) functionalities from pandas. This way we can also refer to variables with their names instead of just integer indices. But performance would be the worst?

Please share your thinking of pros and cons for each choice from both tech and math perspectives. Thanks in advance!

PS: I thought about whether this should be a StackOverflow question, but still feel this is more Data Science.

• Thanks! The extra functionalities in pandas we consider may be useful are: 1) column names, so that we can say we selected 'rain' and 'wind_speed' as the two variables in our regression model instead of X3 and X8. 2) all the SQL like functionalities, such as df[df.rain > 0.5], group by, or apply. What are your thoughts? – Indominus Jan 14 at 3:40
• A lot of numpy is written in C and Fortran (it also allows interaction with C, C++, Fortran) so also has the added speed benefit. – bradS Jan 14 at 11:01