2
$\begingroup$

I have m( around O(millions) ) of rows of type

timestamp | val | ind1 | ind2 | ind3 | .... k entries

My task is to predict the value of "val" for any future given time. How should I model this as a machine learning problem. Note that we want the value of "val" given the past data without knowing the current value of

ind1 | ind2 | ind3 | .... etc.

of the same row. This is different in that sense.

Formally, assume that I have first i rows(each entry is known) of the table. And I want to predict the value of $val_j$ (j>i, i.e. near future). How will I do that? I have not done these kind of predictions using ML or TensorFlow earlier . If we were given $ind1_j,ind2_j,...$ etc but not $val_j$. This kind of prediction would have been easy.

Even if I model my val as a function of ind1, ind2, etc of previous indices. How will I get it for different future times ?

About data I do not know much about data. All I know is that it is related to stocks and exchanges. Maybe "val" is the entity which is dependent on ind1, ind2, etc. You assume whatever is suitable apart from that.

I have m( around O(millions) ) of rows of type

timestamp | val | ind1 | ind2 | ind3 | .... k entries

My task is to predict the value of "val" for any future given time. How should I model this as a machine learning problem. Note that we want the value of "val" given the past data without knowing the current value of

ind1 | ind2 | ind3 | .... etc.

of the same row. This is different in that sense.

Formally, assume that I have first i rows(each entry is known) of the table. And I want to predict the value of $val_j$ (j>i, i.e. near future). How will I do that? I have not done these kind of predictions using ML or TensorFlow earlier . If we were given $ind1_j,ind2_j,...$ etc but not $val_j$. This kind of prediction would have been easy.

Even if I model my val as a function of ind1, ind2, etc of previous indices. How will I get it for different future times ?

About data I do not know much about data. All I know is that it is related to stocks and exchanges. Maybe "val" is the entity which is dependent on ind1, ind2, etc. You assume whatever is suitable apart from that.

And please give any idea, How will system learn iteratively as new data rows are added. I do not want to do the whole training again and again.

| improve this question | |
$\endgroup$
  • $\begingroup$ please convey your reasons first before down-voting this question. $\endgroup$ – v78 Sep 15 '16 at 11:27
  • 1
    $\begingroup$ Cross-posted: datascience.stackexchange.com/q/14011/8560, stackoverflow.com/q/39509446/781723. Please don't post the same question on multiple SE sites; that violates SE rules. $\endgroup$ – D.W. Sep 16 '16 at 3:43
  • $\begingroup$ Is your target variable 'val', and are your features 'ind*' mostly integer, numerical or categorical? Roughly what is k, the number of features: 100? 1,000? Roughly how many distinct integer or categorical values do your features take? (I know you said "I do not know much about data. All I know is that it is related to stocks and exchanges.", but is it prices and volumes of trades, or what?) $\endgroup$ – smci Nov 23 '16 at 12:39
  • $\begingroup$ Are your timestamps contiguous? Are there gaps? In the time-domain is it one big long dataset, or chunks? If chunks, what sizes are they? $\endgroup$ – smci Nov 23 '16 at 12:40
1
$\begingroup$

Your model should be based on what your data represents. If you want meaningful results, you should get some idea of what your data is representing before trying to answer this question.

For instance, you could use your ind# columns to predict the values of your ind#${_j}$ rows, then use the ind#${_j}$s to predict the val${_j}$. But this would only make sense if there is a predictive relationship between the ind#${_j}$s. And you would definite want to have some idea of what that relationship might be in order to choose an appropriate algorithm for each particular column.

Furthermore, the same would apply for $j_{th}$ row val${_j}$ prediction. At the very least you would want to make an educated guess about the relationship between your ind#${_j}$s and your val${_j}$ column so that an appropriate algorithm could be chosen.

As another example, you may be able to use the previous row as a predictor for the next value in the time series. i.e. ind#${_{j-1}}$s => val${_j}$. But again, this only makes sense if you are able to make reasonable assumptions about the relationship of the data, because you ultimately need to choose the algorithm you use to make the predictions.

| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ you can assume , it is stocks data, i.e. closing prices at the end of days for some m stocks(i.e. m columns). And also assume that val_j is a function of all previous data. I just want ideas of models I can use for this problem. $\endgroup$ – v78 Sep 17 '16 at 10:57
  • $\begingroup$ This Wikipedia page lists several candidates for models. link Your choices should be based on the type of function you think bests relates the data. For example linear or non-linear, etc. Doing some basic data analyses should point you in the right direction. Find out how these data are distributed to start: Normal, Poisson, etc. From there research predictive models for that type of distribution. $\endgroup$ – grldsndrs Sep 17 '16 at 18:06
  • $\begingroup$ @ee2: Please edit all those clarifying details into the question. They matter hugely. $\endgroup$ – smci Nov 23 '16 at 12:43
  • $\begingroup$ When you say "closing prices at the end of day", do you mean you only have closing price, not the other timestamps? Is it perhaps the last hour or 30min of trading? $\endgroup$ – smci Nov 23 '16 at 12:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.