1
$\begingroup$

Let's say I have a huge database with 100K records and 60 columns. Let's say one of the column is "min_p". What I do is apply some logic/rule to determine the output label for this record. Basically I look at previous two records and next two records of this min_p. If the condition is satisfied, I will mark the label as 1 else I will mark it as 0.

Now my question, since I have directly derived the label from this called "min_p", should I retain it as one of my predictors in my final dataset? Since I have used that derive the label, I didn't include them in my dataset as a input variable thinking that it is incorrect

Can you help me with this?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

[edited, I misread the question in the first version]

The fact that the label is determined from a combination of values from this feature is not a problem in itself: if it makes sense, it's always better to give the best indicators to the learning algorithm. So the only questions are:

  • whether it makes sense for your problem to have the feature provided as input for any new instance: if yes, then there's no reason to remove it.
  • whether it's useful to apply ML to your problem: if the label can be determined directly from a single feature, it's simply not useful to train a model.

You mention that the label is based on information from the previous/next two records. Keep in mind that the model needs to predict its target for any individual instance as input, unless you're using a sequential model (for instance with times series).

Improve this answer
$\endgroup$
7
  • $\begingroup$ Thanks for the response. Upvoted. Just to make sure I got it right. Am I right to understand that use of "min_p" column which was used to derive the output label can be used as an input feature. The fact that it was used to derive the label has nothing to do with it being used as a predictor of not. $\endgroup$
    – The Great
    Commented Dec 20, 2019 at 1:58
  • $\begingroup$ I understand your second point about sequence. Let's keep that aside. $\endgroup$
    – The Great
    Commented Dec 20, 2019 at 2:00
  • $\begingroup$ @SSMK sorry I realize that I read your question too fast, I see now why my answer is not so clear to you! I'm going to edit it. $\endgroup$
    – Erwan
    Commented Dec 20, 2019 at 2:10
  • $\begingroup$ Sure. got it. marked as an answer. Looking for your inputs on other post as well. Will be of great help $\endgroup$
    – The Great
    Commented Dec 20, 2019 at 2:32
  • $\begingroup$ Hi, one more question here. regarding the 2nd bullet point. Yes, my label can be determined from a single feature (let's say it is min_bp). Now what I am trying to do is, find out the factors that can influence the outcome (label). In this case, do along with other variables, should I also include this variable? Meaning I know that min_bp is a strong predictor, along with that am trying to see whether there is any other factors that can help us predict the outcome. If I do a feature Importance through tree based model, I am getting a list of features which are important along with min_bp $\endgroup$
    – The Great
    Commented Dec 20, 2019 at 8:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.