2
$\begingroup$

I am absolutely new in R, and my problem is that I do not have any Real World experience in it. I mean, I have learnt a lot but I am always struggling when I get a new task to deal with. Generally speaking I am talking about, how to start to deal with a new task.

Sometimes the dataset is so big (surprisingly :)) that I am not able to get the panorama about it and the usually used functions such as str(), summarise(), head(), tail() maybe sample_n from package dplyr are not enough to fill me in satisfactorily.

Almost every example that I found on the net, were about datasets which were almost perfect. If we need to clean the data at all, we can relatively easily identify the basic problems because the problems are unambiguous and you can realize them when you check the head() or something.

What about the real world data? What if the columns shifted in the middle of your dataset, or there are some rows where the values consists an inappropriate symbol or space or something (salary, price, phone-number etc)?

In summary: - What is your general method to getting familiar with your dataset (lets assume that we are already know what is the meaning of the variables because we have a description about it)? - Do you have a general examining method?

I know that there are no two similar projects, I am really interested in YOUR basic workflow (with some examples or explanations) though.

$\endgroup$
3
  • $\begingroup$ I highly recommend you goto Kaggle. Look at what people have been doing. $\endgroup$ – SmallChess Apr 17 '18 at 2:22
  • 1
    $\begingroup$ "columns shifted in the middle of your dataset" is indeed a significant problem, and one that occurs before serious "data-science" can begin. I often have unit-tests on the data to check certain assumptions, ranging from checking class to verifying ordinality and rank to comparing with expected ranges. But if you have doubts about the legitimacy of the CSV or XLS(x) data format, then the data-validation stage is critical to have any assurance in follow-on summarization/viewing/panoramic exploration. $\endgroup$ – r2evans Apr 23 '18 at 18:16
  • $\begingroup$ ... and for data unit-tests, I literally use the testthat package, as it does a good job of summarizing which and how assumptions/tests fail. The trick is coming up with meaningful and robust "tests". $\endgroup$ – r2evans Apr 23 '18 at 18:17
2
$\begingroup$

There are several packages on R that provides you full reports on the status of your dataframe. Not only the number of missings, but also the type of the columns, the number of outliers (if it is continuous), the distribution along categories (if it is categorical),...

If I have to choose one of them, a great help is for sure DataExplorer, give it a try! I usually combine it with a set of checks prepared by myself, once I know a bit about the data frame and the information I expect from a "business" side.

$\endgroup$
1
$\begingroup$

Here is a great recent video of Hadley Wickham doing exploratory analysis of a medium-sized dataset. It contains many simple but clever hacks, and demonstrates strategies such as "doing a single case first, then doing something for all instances" (here: one city -> all cities)

https://www.youtube.com/watch?v=go5Au01Jrvs

Briefly showing the "whole game" of data analysis. See code and data at https://github.com/hadley/building-permits

$\endgroup$
0
$\begingroup$

Go with Analytics vidya where there are many competitions are ongoing, and I recommend you to work on a problem which is still live (Practice problem on Time Series). I hope you like it.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.