6
$\begingroup$

Most of the material I have read in the past usually assumes that the training set is flawless. However that doesn't seem to be the case here with what I am given. The data that is meant to send into the training set is often questionable (I don't even know how to start separating good from bad ones). I have spent much more time trying to pre-process the files in order to increase the validity of the data, than actually building the prototype.

I have only dumped a part of the data that is claimed to be better. There are still data from other sources to complement the missing part in the first set of training data. Given the amount of data, it is practically impossible for me to spend too much time on them.

So the question is, how do people deal with really horribly, inconsistent data in real life (assuming I am working with addresses, where people often write the wrong postcode due to bad city planning or lack of clear instruction, misspell the name of neighbourhood, plus data got "autocorrected" by excel such that "1-2" becomes "1-Feb", etc. etc.)? Specifically, how do I ensure the quality of the train classifier given the training set (and cross validation set) has questionable quality?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
6
$\begingroup$

Welcome to the real world of data science. Here, the data sets are not as clean as you thought while doing those courses/tutorials online. Those are super polished and refined. But, the real world data is not so.

The step where one does the cleaning and scrubbing is called the data pre-processing step.

So, some nice data cleaning techniques, in addition to @jknappen's excellent answer are:

  1. Elimination of zero variance columns/predictors: These columns are not important, and they cause the model and the fit to crash and leak errors. So, eliminating them would make complete sense.
  2. Correlated Predictors: Reducing the level of correlation between the predictors would be a very nice step in the pre-processing process.
  3. Scaling: You must be knowing why scaling is important during pre-process.
  4. Predictor Transformations

A nice reference from Kaggle forums where the pre-processing and cleaning of data sets is discussed.

Improve this answer
$\endgroup$
1
  • $\begingroup$ TIL about kaggle, looks interesting (: I am still trying to process the source data which is a collection of unstructured text (e.g. address) with components often wrongly parsed (may be due to wrongly entered postcode in source, typing error, crappy mock data, or wrong value like city in postcode column). $\endgroup$
    – Jeffrey04
    Nov 18, 2015 at 2:34
5
$\begingroup$

You can use techniques of semi-supervised learning where you have a small clean training set and some dirty data. You than extend your data base by judging how good the other data are and incorporate the "best" data points into your training set.

Improve this answer
$\endgroup$
1
  • $\begingroup$ eh, semi-supervised learning, let me do some reading on that $\endgroup$
    – Jeffrey04
    Nov 17, 2015 at 2:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.