I have a feature which has specific categorical values ex(Technology, Hardware, Software, Marketing, Evnts etc). Based on this and some other features, I am trying to classify the dataset into 2 categories IsSoftwareSystem or NotSoftwareSystem. In this case is this cause a reduce in accuracy because i am feeding the category itself in the data and trying to predict the same.
Using Random Forest/XGB.
$\begingroup$
$\endgroup$
2
-
1$\begingroup$ If "Software" is just a Feature name. Then it will not impact. If it is the "Target" then you are leaking the data. You should get 100% score on training/test. But then why you need the Model. Please add some sample data $\endgroup$– 10xAICommented Mar 10, 2021 at 10:57
-
$\begingroup$ So I have a feature named "Category" which has values like Software, Hardware, Marketing, Events, Services etc. and there are thousands of records along with other features which in the end predict if it is Software or non software. $\endgroup$– tumblewoodCommented Mar 10, 2021 at 13:12
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
4
You have two problems:
- Technical problem: As 10xAI said in a comment, if the target also belongs to the features then the model should very easily predict every instance correctly. So you should obtain perfect performance on the test set. The ML model doesn't care about "cyclic dependency", it uses any good indicator it receives as feature. Since you mention a reduction in accuracy, it means that there is an error somewhere in the process.
- Semantic problem: which problem are you trying to solve with this ML setup? If the goal is to distinguish software vs. non-software and you have as input a category "software" then it's pointless to train a ML model. You can obtain the same result much more efficiently with a simple condition:
for each instance:
if instance.category == "software":
instance.answer="software"
else:
instance.answer="non-software"
-
$\begingroup$ I agree it does look like a rule based thing. But it is not. So a category can be "Hardware" but there are other features due to which it is a "Software". So direct rules can't be applied. My doubt is basically around whether I should even include a feature named "Category" because that is kind of the target that I am trying to predict. It is like partially feeding the target to input $\endgroup$ Commented Mar 11, 2021 at 20:39
-
$\begingroup$ @tumblewood in a way partially feeding the target as input is exactly what we want in a supervised model: the more information the features give about the target, the better. The only question is whether this information "category" is always available as input, i.e. whether the design is consistent with how the model would be used in production. If it is, it would be ridiculous not to use a perfectly good indicator. $\endgroup$– ErwanCommented Mar 11, 2021 at 23:28
-
$\begingroup$ thanks Erwan, yes it is always available in the input. What I am afraid of is that the model may learn/develop a 1:1 kind of relationship between the "Category" and Target, and the case when the "Category" itself is not sent correctly for a particular record it might give a wrong prediction because the model learnt that 1:1 relationship, $\endgroup$ Commented Mar 12, 2021 at 20:17
-
$\begingroup$ @tumblewood if this case can happen then your training data is supposed to contain such a case, so the model would do its best to deal with the issue (it might just trust the value anyway if this is the best option statistically). If 100% of the instances have the correct value for Category in the training set but this proportion is only 99% in the test set, then the problem is that the training data is not a fully representative sample. $\endgroup$– ErwanCommented Mar 12, 2021 at 21:05