I am still new to data mining but I really want (and need) to learn it so badly. I know that before I can actually process my data in softwares like WEKA, I need to do some filtering like cleaning the data, integrating, transforming, etc to actually get your data cleaned from any kind of duplicate, missing value, noise, etc. But I only know all of these theoretically.
The problem I have now is I have a very big set of data that I need to filter first before going on to the processing part. But I don't know where to start. Fyi, my dataset is very big that the usual spreadsheet programs like Ms Excel, Libre Office, WPS,etc can't open it. I have to use Linux terminal and commands to count the number of rows, columns etc.
What do I do in the preprocessing start? How do I 'clean' my data? I have been thinking to use Linux commands to do all of these, but I am also wondering how real data scientists clean their data. Do they do these manually or they already have some sort of software to help them? Because seriously I don't know where to start or to do. Every reference I found in the internet only explain things theoretically. ANd I need something more practical to help me understand.
What do I do with my dataset? Help please?