1
$\begingroup$

Background: As part of prediction analysis, I am given a train and test dataset. Both train and test data have numerical and categorical predictor variables and additionally, train data has a numerical target variable. The objective is to predict target in the test.

train = [c1,c2,x3,x4, y] = [Xc,X, y]
test = [c1,c2,x3,x4] = [Xc,X]

Xc, X denotes categorical and numerical predictor variables. I am trying to generate additional features from categorical variables Xc such as count features, count_mean, count_variance and similar features from a combination of categorical variable and a numerical variable (mean, variance etc).

Problem: Is it better to generate features on a combined dataset train+test or is it better to generate features separately on train and test datasets?

What are the implications when the distribution of a categorical variable are different in train and test and what happens when they are similar?

Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
1
$\begingroup$

The idea is to build features based on your training set. In a real-world application, all you will have is a training set to build and deploy a model, and that model will predict the test examples that the application needs. In this setting, it is not realistic to assume that you will have the test data to generate features, as you will already have your model delivered. So, the mechanism to generate the features can only be fed with training information.

About this:

What are the implications when the distribution of a categorical variable are different in train and test and what happens when they are similar?

The implications of having different distributions of the data in the training and test sets are very bad. Your learning algorithm will only learn about things that are on the training set, and for that reason, if your test set has a different distribution you should consider changing your training set. However, if they follow the same distribution, then your algorithms will generalize properly if you do things right.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ You always have the option to withhold data for holdout or / k fold cross validation, etc with real world data. $\endgroup$
    – The Lyrist
    Jun 5, 2018 at 23:27
  • $\begingroup$ That is correct. $\endgroup$
    – David Masip
    Jun 7, 2018 at 6:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.