0
$\begingroup$

I need to train a regression algorithm with multiple features and a single label (predicted value). The problem is that this algorithm has to be able to do on-line learning and the number of features it will receive will vary. Let me give a clear example:

The algorithm is trained on a dataset of shape:

[--------Features-----------------] [Label]
[-- context11 -- | -- context12 --] [label1]

Then, for the next training example, one of the contexts might be missing, so the training example might either be:

[--------Features-----------------] [Label]
[-- context21 -- ]                  [label2]

or 

[--------Features-----------------] [Label]
                  [-- context22 --] [label2]

How can I deal with this situation? So far, I thought about two possibilities:

  1. Replace the missing part of the features with zeros. I am not sure how this will affect the algorithm though.
  2. Use a decision tree or random forest? Do these have a more natural way of dealing with a variable number of features?

Any other ideas?

Improve this question
$\endgroup$
0
3
$\begingroup$

I think in the specific case that you have given as an example, you can use approach 1 which you have mentioned above i.e.

either replace the missing features with a zero value

or otherwise you can also

implement your algorithm in a sparse manner so that it doesn't need to always have all the features for an instance.

Using a sparse format will not affect the runtime (or any other aspect) of your algorithm. Whereas, if you use the zeroing features approach, then your algorithm will have to do the unnecessary multiply by zero operations and this will increase your runtime ( How much? that depends on how sparse is your actual data).

Hope that helps.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Could you be a bit more explicit about how to implement an algorithm in a sparse manner? Got a link with an example that might help? $\endgroup$ – Qubix Jun 8 '18 at 12:09
  • $\begingroup$ Yes sure. Can you just clarify if you know about sparsity and also about which language are you using? So that I can send you more germane links. $\endgroup$ – Sujay_K Jun 8 '18 at 17:17
  • $\begingroup$ But to answer that question in a more general sense, you can exploit sparsity in any algorithm by storing/manipulating vectors or matrices in a sparse form. This means that all your vector-vector, matrix-vector, matrix-matrix multiplications can be done as sparse operations which can save you a lot of time. Some good reference links would be: en.wikipedia.org/wiki/Sparse_matrix , se.mathworks.com/help/matlab/math/sparse-matrix-operations.html $\endgroup$ – Sujay_K Jun 8 '18 at 17:24
1
$\begingroup$

Replace the missing part of the features with zeros. I am not sure how this will affect the algorithm though.

I don't suggest this. This input may have special meaning as the input in the real data. A simple an most common behavior is to use the expected value for that entry if you have the distribution of that specific feature which is missed. If you don't know the distribution which is a common issue, you can find the mean of that specific entry among your training data and replace it although there are numerous ways to deal with missing data, that's a simple approach due to regression task.

Use a decision tree or random forest? Do these have a more natural way of dealing with a variable number of features?

It is also a common practice but due to having regression task, it may have so many branches. In data mining, it's a common approach.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.