What initial steps should I use to make sense of large data sets, and what tools should I use?

Question

Caveat: I am a complete beginner when it comes to machine learning, but eager to learn.

I have a large dataset and I'm trying to find pattern in it. There may / may not be correlation across the data, either with known variables, or variables that are contained in the data but which I haven't yet realised are actually variables / relevant.

I'm guessing this would be a familiar problem in the world of data analysis, so I have a few questions:

1. The 'silver bullet' would be to throw this all this data into a stats / data analysis program and for it to crunch the data looking for known / unknown patterns trying to find relations. Is SPSS suitable, or are there other applications which may be better suited.

2. Should I learn a language like R, and figure out how to manually process the data. Wouldn't this comprimise finding relations as I would have to manually specify what and how to analyse the data?

3. How would a professional data miner approach this problem and what steps would s/he take?

Answer 1

I will try to answer your questions, but before I'd like to note that using term "large dataset" is misleading, as "large" is a relative concept. You have to provide more details. If you're dealing with bid data, then this fact will most likely affect selection of preferred tools, approaches and algorithms for your data analysis. I hope that the following thoughts of mine on data analysis address your sub-questions. Please note that the numbering of my points does not match the numbering of your sub-questions. However, I believe that it better reflects general data analysis workflow, at least, how I understand it.

1. Firstly, I think that you need to have at least some kind of conceptual model in mind (or, better, on paper). This model should guide you in your exploratory data analysis (EDA). A presence of a dependent variable (DV) in the model means that in your machine learning (ML) phase later in the analysis you will deal with so called supervised ML, as opposed to unsupervised ML in the absence of an identified DV.

2. Secondly, EDA is a crucial part. IMHO, EDA should include multiple iterations of producing descriptive statistics and data visualization, as you refine your understanding about the data. Not only this phase will give you valuable insights about your datasets, but it will feed your next important phase - data cleaning and transformation. Just throwing your raw data into a statistical software package won't give much - for any valid statistical analysis, data should be clean, correct and consistent. This is often the most time- and effort-consuming, but absolutely necessary part. For more details on this topic, read this nice paper (by Hadley Wickham) and this (by Edwin de Jonge and Mark van der Loo).

3. Now, as you're hopefully done with EDA as well as data cleaning and transformation, your ready to start some more statistically-involved phases. One of such phases is exploratory factor analysis (EFA), which will allow you to extract the underlying structure of your data. For datasets with large number of variables, the positive side effect of EFA is dimensionality reduction. And, while in that sense EFA is similar to principal components analysis (PCA) and other dimensionality reduction approaches, I think that EFA is more important as it allows to refine your conceptual model of the phenomena that your data "describe", thus making sense of your datasets. Of course, in addition to EFA, you can/should perform regression analysis as well as apply machine learning techniques, based on your findings in previous phases.

Finally, a note on software tools. In my opinion, current state of statistical software packages is at such point that practically any major software packages have comparable offerings feature-wise. If you study or work in an organization that have certain policies and preferences in term of software tools, then you are constrained by them. However, if that is not the case, I would heartily recommend open source statistical software, based on your comfort with its specific programming language, learning curve and your career perspectives. My current platform of choice is R Project, which offers mature, powerful, flexible, extensive and open statistical software, along with amazing ecosystem of packages, experts and enthusiasts. Other nice choices include Python, Julia and specific open source software for processing big data, such as Hadoop, Spark, NoSQL databases, WEKA. For more examples of open source software for data mining, which include general and specific statistical and ML software, see this section of a Wikipedia page.

UPDATE: Forgot to mention Rattle, which is also a very popular open source R-oriented GUI software for data mining.

Answer 2

1. SPSS is a great tool, but you can accomplish a great deal with resources that you already have on your computer, like Excel, or that are free, like the R-project. Although these tools are powerful, and can help you identify patterns, you'll need to have a firm grasp of your data before running analyses (I'd recommend running descriptive statistics on your data, and exploring the data with graphs to make sure everything is looking normal). In other words, the tool that you use won't offer a "silver bullet", because the output will only be as valuable as the input (you know the saying... "garbage in, garbage out"). Much of what I'm saying has already been stated in the reply by Aleksandr - spot on.

2. R can be challenging for those of us who aren't savvy with coding, but the free resources associated with R and its packages are abundant. If you practice learning the program, you'll quickly gain traction. Again, you'll need to be familiar with your data and the analyses you want to run anyway, and that fact remains regardless of the statistical tools you utilize.

3. I'd begin by getting super familiar with my data (follow the steps outlined in the reply from Aleksandr, for starters). You might consider picking up John Foreman's book called Data Smart. It's a hands-on book, as John provides datasets and you follow along with his examples (using Excel) to learn various ways of navigating and exploring data. For beginners, it's a great resource.

Answer 3

Aleksandr has given a very thorough explanation, but briefly, these are the steps that are followed:

Extracting data

Cleaning data

Feature extraction

Building models

Inferring results

Publishing results

Repeat steps 3,4,5 in loop till you get the right accuracy.

Answer 4

R has pnc dialogue GUIs like SPSS. They print R code so you can learn and combine their efforts. I would recommend BlueSky for it's dialogues for everything and rattle. While these software are great for EDA, statistics and visualization, machine learning they don't do well.