0
$\begingroup$

I am building a normal feed-forward neural network to predict the value of a masked time point using regression, e.g. I have values for x at times 1, 2, and 4, and I want to predict its value at time 3. How do I make it so that time points 2 and 4 have more weight on the prediction than time point 1? The case would be similar if I wanted to mask and predict the second time point etc...

I thought about multiplying the input by a time-decay factor based on how far the point is from the masked point and training the model (input to the model would have time embeddings with it), but I haven't seen any literature doing this. What are strategies I can use?

I am trying to stay away from models like RNNs because I want to use all the time points to predict other time points.

$\endgroup$

1 Answer 1

1
$\begingroup$

I think the core idea of neural networks is that they find these weights by themselves. If you want the model to have certain properties with these weights, you should start with a statistical model like AR. It can also be a tool to verify your assumption that these points are supposed to have higher weights.

However, I'm not sure if you really need some model for your problem or just a nice interpolation method like splines. It's always better to start from a simpler method and try more complicated when the former is insufficient.

$\endgroup$
5
  • $\begingroup$ Thanks for your answer! I kinda do need a more sophisticated model than splines; I oversimplified my problem and there's more to it than just predicting 1 time point from a couple of others, but I wanted to see if there exists a way to dynamically "add importance" to input, like with some sort of decay or attention mechanism. $\endgroup$ Commented Apr 25 at 6:40
  • $\begingroup$ 1) A common trick to add importance to a certain output is to train on more data where this input is particularly important. 2) If you are going to build a more sophisticated model, you might find Positional Encoding helpful in your problem. Maybe some modifications to it will give exactly what you want. 3) Again, you can use attention weights, but it will learn weights by itself. You can try fitting it and using its value to test your hypothesis about the importance of close positions. $\endgroup$ Commented Apr 25 at 7:03
  • $\begingroup$ Thanks a lot for this, I'll look into all these options. What do you think of using a time-decay factor? Namely, when training, multiply the inputs by a time-decay factor (say, exp(-|i-t|), where i is the time point we're at and t is the time point we want to predict) and then train the NN. That way, the further time points decay and go to zero. Is this a thing? $\endgroup$ Commented Apr 25 at 7:21
  • $\begingroup$ I don't see how it could work, multiplying features by some factor might just spoilt them. However, if you do an experiment on a simple model and show it works, we both will learn from that. $\endgroup$ Commented Apr 25 at 7:26
  • $\begingroup$ If all else fails, maybe I will. Thanks again! $\endgroup$ Commented Apr 25 at 7:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.