# Feature Selection, Machine learning and Time Series analysis, for large financial timeseries

I have m( around O(millions) ) of rows of type

timestamp | val | ind1 | ind2 | ind3 | .... k entries


My task is to predict the value of "val" for any future given time. How should I model this as a machine learning problem. Note that we want the value of "val" given the past data without knowing the current value of

ind1 | ind2 | ind3 | .... etc.


of the same row. This is different in that sense.

Formally, assume that I have first i rows(each entry is known) of the table. And I want to predict the value of $val_j$ (j>i, i.e. near future). How will I do that? I have not done these kind of predictions using ML or TensorFlow earlier . If we were given $ind1_j,ind2_j,...$ etc but not $val_j$. This kind of prediction would have been easy.

Even if I model my val as a function of ind1, ind2, etc of previous indices. How will I get it for different future times ?

About data I do not know much about data. All I know is that it is related to stocks and exchanges. Maybe "val" is the entity which is dependent on ind1, ind2, etc. You assume whatever is suitable apart from that.

I have m( around O(millions) ) of rows of type

timestamp | val | ind1 | ind2 | ind3 | .... k entries


My task is to predict the value of "val" for any future given time. How should I model this as a machine learning problem. Note that we want the value of "val" given the past data without knowing the current value of

ind1 | ind2 | ind3 | .... etc.


of the same row. This is different in that sense.

Formally, assume that I have first i rows(each entry is known) of the table. And I want to predict the value of $val_j$ (j>i, i.e. near future). How will I do that? I have not done these kind of predictions using ML or TensorFlow earlier . If we were given $ind1_j,ind2_j,...$ etc but not $val_j$. This kind of prediction would have been easy.

Even if I model my val as a function of ind1, ind2, etc of previous indices. How will I get it for different future times ?

About data I do not know much about data. All I know is that it is related to stocks and exchanges. Maybe "val" is the entity which is dependent on ind1, ind2, etc. You assume whatever is suitable apart from that.

And please give any idea, How will system learn iteratively as new data rows are added. I do not want to do the whole training again and again.

• please convey your reasons first before down-voting this question. – v78 Sep 15 '16 at 11:27
• Cross-posted: datascience.stackexchange.com/q/14011/8560, stackoverflow.com/q/39509446/781723. Please don't post the same question on multiple SE sites; that violates SE rules. – D.W. Sep 16 '16 at 3:43
• Is your target variable 'val', and are your features 'ind*' mostly integer, numerical or categorical? Roughly what is k, the number of features: 100? 1,000? Roughly how many distinct integer or categorical values do your features take? (I know you said "I do not know much about data. All I know is that it is related to stocks and exchanges.", but is it prices and volumes of trades, or what?) – smci Nov 23 '16 at 12:39
• Are your timestamps contiguous? Are there gaps? In the time-domain is it one big long dataset, or chunks? If chunks, what sizes are they? – smci Nov 23 '16 at 12:40

For instance, you could use your ind# columns to predict the values of your ind#${_j}$ rows, then use the ind#${_j}$s to predict the val${_j}$. But this would only make sense if there is a predictive relationship between the ind#${_j}$s. And you would definite want to have some idea of what that relationship might be in order to choose an appropriate algorithm for each particular column.
Furthermore, the same would apply for $j_{th}$ row val${_j}$ prediction. At the very least you would want to make an educated guess about the relationship between your ind#${_j}$s and your val${_j}$ column so that an appropriate algorithm could be chosen.
As another example, you may be able to use the previous row as a predictor for the next value in the time series. i.e. ind#${_{j-1}}$s => val${_j}$. But again, this only makes sense if you are able to make reasonable assumptions about the relationship of the data, because you ultimately need to choose the algorithm you use to make the predictions.