Do you use automatic cleaning tools for data?
I mean something similar to h2o.ai's auto ml function but applied to preprocessing data. Or do you always clean data 'by hand'.
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Cleaning data largely varies from data to data. Considering you are talking mostly about unstructured data, it can be of two types, namely Image data and text data. The process of cleaning both these data depends on the type of goal required to be achieve using these. A little brief process could be explained as follows:
Image data: While performing convolutional neural network on the data, the process of feature extraction from the data is automated. Specific pre-processing need to be carried out aiming it to the target. This includes:
- Keeping the data set close to the actual data on which the model is to be tested.
- Removing any unnecessary images that do not co-ordinate with the corresponding label.
- Resizing the images as per the model requirement.
Text data: Text data has a robust procedure for of pre-processing consisting of Removing stop words, tokenizing, lemmetizing the tokens etc. At times when we are using the word to vec approach, we need to convert the text tokens to their corresponding word-vectors.
Generally functions designed for these processes are fed into the pipeline and so individually every entry is not required to be processed in a similar way.
There are some papers and blogs I would suggest you to read: Towards automated patient data cleaning , Turn Messy Data into Tidy Data
