-1
$\begingroup$

I am a newbie in Data Science, so please don't blame me for stupid questions.

Here is my problem. I've got a dataset (no empty cells, only numerical values) consisted of 15 columns (1st - user id, last one - response, 2-14 - different features such as eventtype, country, browser etc., actually this dataset describes users actions on some webpage). Response value could be 1 or 0, so it's definitely a classification problem.

In order to have some practice I would like to try main algorithms on this dataset (kNN, Random Forest, SVC). The problem is that in this dataset I've got around 5000 unique id's and the total number of lines is around 17000, so different users have several lines related to their actions on the webpage.

In this case, algorithms listed above gave too good predictions, but I suppose it's not correct to use them on such data, because they make predictions for the object based on this object's historic data. Which options do I have to improve the dataset?

I think I could try to leave only unique id's in dataset, but it will reduce the dataset size from 17k lines to 5k lines, which, I suppose, will have a negative effect on predictions. Which other strategis I can use?

$\endgroup$
3
  • 2
    $\begingroup$ Tried removing the "cheater" column completely? A more advanced option would be to re-design the problem as a frequent pattern miner. $\endgroup$
    – Mephy
    Oct 30, 2017 at 13:31
  • $\begingroup$ @Mephy, what do you mean by "cheater" column? $\endgroup$
    – kisma
    Oct 30, 2017 at 14:35
  • $\begingroup$ The id column, in your case. "Cheater" is the column that transforms you learning algorithm in an information retrieval system, that gives away too much information that should not be used to generalize the model. For an extreme example, if you could use the target as a predictor, why would you even try to learn something. Less obvious cases is using future to predict the past, or using information that the system may not have at production time. In those case, you're "leaking" or "cheating" the train-time model. $\endgroup$
    – Mephy
    Oct 30, 2017 at 14:38

1 Answer 1

2
$\begingroup$

What you have here is some data that's too specific. I don't know if there's a widely accepted name for that, but I've seen it be called "leaking" or "cheating" data. Using this kind of data is potentially dangerous for the algorithm, as you have seen in your experiments.

Here's a quick example: I want to predict whether a costumer prefers to wear white socks or black socks. I have a lot of information from this costumer, like the city he lives or how many TVs he owns (let's pretend there's some relation with preferred sock color). For some reason, the people handling the interview I use as data sorted the 200 interview responses by class - the first 120 responses are for white socks, and the next 80 are for black socks. Each response has an identifier from 1 to 200.

Now I run this data through a decision tree learner. It comes up with a stump (a tree with only a single node) containing the rule:

If Id <= 120 Then
   User prefers white socks
Else
   User prefers black socks

This is completely useless! If I just keep giving new identifiers to people, I'll assume every new costumer prefers black socks! (Except if one of the first 120 comes back).

While may be some advanced ways to deal with this problem to use the fact the user may repeat and that's relevant (it usually is), I wouldn't recommend it. 17k instances isn't that much to try for a more complex model, you could be overfitting. Only go towards some other strategy (like frequent pattern mining, or turning the problem in a time series of sort) if and only if removing the Id column is not good enough.

$\endgroup$
2
  • $\begingroup$ Removing the ID column seems an option, but how can I understand that it is "good enough"? $\endgroup$
    – kisma
    Oct 30, 2017 at 15:05
  • $\begingroup$ @kisma common validation techniques such as cross-validation should give a good guess about the performance at (at least the near) production time. $\endgroup$
    – Mephy
    Oct 30, 2017 at 15:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.