2
$\begingroup$

I am working on a classification task. The dataset is a UCI data set about machine learning with 200 observations and 2 classes.

Part of my model includes the following preprocessing steps:

  1. remove missing values
  2. normalize between 0 and 1
  3. remove outlier
  4. smoothing
  5. remove trend from data
  6. SMOTE

I would like to use a clustering method to remove noisy data points. The question is, at which step should this happen?

$\endgroup$
2
  • $\begingroup$ To make your question clearer, can you maybe describe a bit more what kind of data you're dealing with? And what you intend to do after all your preprocessing? $\endgroup$ Commented Dec 5, 2019 at 8:46
  • 1
    $\begingroup$ @ValentinCalomme its an UCI data set about machine learning with 200 observation and 2 classes and after preprocessing I want to use classification technique $\endgroup$
    – Davood
    Commented Dec 5, 2019 at 9:40

1 Answer 1

3
$\begingroup$

Looking at your different steps, the important thing to do is check which step would be affected by outliers.

  1. Removing missing values is not affected because this step is not dependent on other data points present (or not) in the dataset.
  2. However, normalizing your data is. Indeed, let's say your outliers contain extreme values, this will affect the normalized values of the non-outlier data points.

Therefore, intuitively, I would perform your noise removal at the very start or after step 1.

Ultimately, you should see what works better for your task. Perhaps removing outliers doesn't help as much as you'd expect. Same with your pre-processing. Feel free to experiment!

$\endgroup$
2
  • 1
    $\begingroup$ Thanks for your time, you mean I should do it after step 1 and before step 2, ha? $\endgroup$
    – Davood
    Commented Dec 5, 2019 at 10:19
  • 1
    $\begingroup$ Indeed, it could be done before step 1 as well $\endgroup$ Commented Dec 5, 2019 at 10:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.