0
$\begingroup$
    Word ID  X            Y   Page IDFont Style  Color  size    bold    an uppercse all uppercase   single digit    multiple digit  special character   email   hperlink    calender month
0   234378  247.400000  27.058  p1  nimbussanl  #000000 14.0    0   True    True    False   False   False   False   False   False
1   234378  293.539231  27.058  p1  nimbussanl  #000000 14.0    0   True    True    False   False   False   False   False   False
2   71195   376.700000  54.470  p1  nimbussanl  #000000 10.0    0   True    False   False   False   False   False   False   False
3   201816  395.973846  54.470  p1  nimbussanl  #000000 10.0    0   False   False   False   False   False   False   False   False
4   166593  406.110769  54.470  p1  nimbussanl  #000000 10.0    0   True    True    False   False   False   False   False   False

enter image description here

Word ID : I have used NLTK’s english word corpus to convert the words to unique ID’s.

I have data set that is given above contains data types that have classes(Page ID,Font), Boolean and Numeric values.

I have successfully pre-processed (Normalized X and Y;One hot encoding other columns) all the columns except Word ID.

Word ID is the ID given to each word in the dataset. Can somebody help me to preprocess the Word ID column that contains value as high as 234378 and as low as 230 in the original dataset. I've this doubt because Word ID(numerical) as feature must not numerically influence the final answer.

$\endgroup$
4
  • $\begingroup$ What IS the Word ID? What are you trying to do? Predict something? $\endgroup$
    – Mephy
    Dec 11, 2017 at 20:52
  • $\begingroup$ Word ID : I have used NLTK’s english word corpus to convert the words to unique ID’s.I am trying to predict that if the word is content heading or content word... $\endgroup$
    – Berry
    Dec 11, 2017 at 20:56
  • $\begingroup$ ID variables are not features. Particularly and absolutely if they are unique. $\endgroup$
    – HEITZ
    Dec 11, 2017 at 23:19
  • $\begingroup$ They are unique, but there are multiple occurrences of each ID and that too have different value for respective feature(you can refer the image) $\endgroup$
    – Berry
    Dec 12, 2017 at 18:18

1 Answer 1

1
$\begingroup$

I would agree with what @Berry said.

I think using a Unique Identifier is not suggestible as it is might be Over-fitting/ Over-train your model.

Before deciding to go forward and use a Unique Identifier I would suggest you to check the model by taking some random data by giving some random identifier and see what are its results. If my guess is right, it would worsen the model. BY that you can prove yourself that it is not right.

The possible reasons for that to happen are: you can go through the link, it is specified clearly.

Finally, even if you try to normalize and use an unique identifier it is going to worsen the model. Unless you have a method which says that the Unique Identifier is assigned based on some value(explained in 1st para by Ryan in the link attached).

Edit: If the target variable is skewed towards one feature then we need to balance the training set and train the model with the new balance data. using the test data you can predict.

To over come the imbalance of data, we can use SMOTE,ROSE packages. Generally SMOTE is rated better over ROSE but suggestible to try both.

Implementation of SMOTE, a link to go through different type of over and under-sampling techniques

You can go through this link for better understanding on ROSE and SMOTE

Do let me know if you have any additional questions!

$\endgroup$
9
  • $\begingroup$ I trained model without using Word ID and achived 99% test accuracy for binary class classification using SVM $\endgroup$
    – Berry
    Dec 13, 2017 at 19:16
  • $\begingroup$ I think you are over fitting the data(data leakage) make sure that the data used for testing is unique. Try using Train, Test, Validation sets. Try doing some feature engineering. Also remove the feature which is directly proportional to the target variable. These are some of the possible reasons for this issue. $\endgroup$
    – Toros91
    Dec 14, 2017 at 2:28
  • $\begingroup$ I used separate data for testing. The data which is used for testing is unseen by the classifier. Am I still messing up somewhere? $\endgroup$
    – Berry
    Dec 14, 2017 at 17:50
  • $\begingroup$ Please share the predictor importance chart, SVM with 90-95% accuracy is acceptable but anything more than that is most likely to be over fitting. But here I think because of some important variable, do share your predictor importance chart $\endgroup$
    – Toros91
    Dec 15, 2017 at 0:24
  • $\begingroup$ It was giving that much accuracy because the dataset was not evenly classified. Most of the datapoints belong to a single class that is why it was giving that much accuracy. $\endgroup$
    – Berry
    Dec 15, 2017 at 19:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.