Preprocessing in Data mining?

Question

I am still new to data mining but I really want (and need) to learn it so badly. I know that before I can actually process my data in softwares like WEKA, I need to do some filtering like cleaning the data, integrating, transforming, etc to actually get your data cleaned from any kind of duplicate, missing value, noise, etc. But I only know all of these theoretically.

The problem I have now is I have a very big set of data that I need to filter first before going on to the processing part. But I don't know where to start. Fyi, my dataset is very big that the usual spreadsheet programs like Ms Excel, Libre Office, WPS,etc can't open it. I have to use Linux terminal and commands to count the number of rows, columns etc.

What do I do in the preprocessing start? How do I 'clean' my data? I have been thinking to use Linux commands to do all of these, but I am also wondering how real data scientists clean their data. Do they do these manually or they already have some sort of software to help them? Because seriously I don't know where to start or to do. Every reference I found in the internet only explain things theoretically. ANd I need something more practical to help me understand.

What do I do with my dataset? Help please?

Answer 1

The preprocessing will depend on what your data is like: textual? numerical? If in whatever analysis you want to do the data is not allowed to have duplicates, you'll have to filter those out. If not, they'll get to stay, etc. If you already know exactly what you want to do theoretically and just need to find some software, python pandas is very good. Load the data, clean it, group it, perform elementary analysis.

Answer 2

I think different people probably have varying approaches to this depending upon their background. Often the linux toolset gets over looked, things like awk, sed, grep, cut, paste, sort, uniq and so on, can be combined in many sophisticated ways and are very powerful and scalable, but they aren't for everyone.

I suspect many people use a programming language that they already have familiarity with or that they use for their data science work, R, Python, Matlab, Mathematica, C++, Julia, the list goes on. Others may use commercial products like SAS.

Certainly Python, and it's relevant libraries, and R have large users bases and are popular choices. Ultimately you may have to look at a few options and decide which one suits you and your skill set best.

As you've already mentioned, WEKA provides an accessible user interface to get started quickly and is certainly a good place to start for early experimentation and to produce results quickly. It has a range of filters which can be used to preprocess data which may be of use to you. It exposes much more of its functionality via its API which is available via Java and this gives you the ability to be much more creative as you build your own solutions to your problems, but you will need to learn to program in Java.

Another, more eclectic, approach is that you draw from many toolboxes like R, Python, WEKA, C++, Apache Mahout etc and fuse them using your own framework written in your favourite programming language.

Exactly which data cleansing operations you need to apply, missing value replacement, addition of noise, resampling, etc., will depend more heavily on the specific nature of your data and it's application.

Answer 3

You can opt for Apache spark it has API for languages like JAVA,Python,R .Spark is very good at performing operations on large scale data.It is having lot of inbuilt functions to clean and data is stored in spark Data frames. Machine learning algorithms are also implemented in spark Here is relevant information