Background: We are trying to build a customized ML library in Python 3 to tackle analysis we often repeat, in a general fashion. But it would not be nearly as general as
sklearn. In fact, we are prepared to break some interfaces if that give us enough performance boost in return.
The basic starting point would be constructing a
Learner by feeding it
y, and predict on new input
learner = Learner(X, y)
y0 = learner.predict(X0)
One of the design decision is what data type to use for X and y, here are 3 choices, along with some rudimentary brainstorm advantages for each respectively:
- native Python list:
Xbeing a list of lists,
ybeing a list. Would this have better performance for being 'closer to metal'?
(n, ). This would benefit from the richer functionalities in numpy/scipy. This is also the data type choice of
Series. This can utilize more Data Analysis (read: dirty work) functionalities from pandas. This way we can also refer to variables with their names instead of just integer indices. But performance would be the worst?
Please share your thinking of pros and cons for each choice from both tech and math perspectives. Thanks in advance!
PS: I thought about whether this should be a StackOverflow question, but still feel this is more Data Science.