I want to know which language/packages performs better & faster on wrangling big data? R and Python both has packages and libraries for wrangling and cleaning data. But which packages and libraries performs the best on wrangling and cleaning Big datasets?
4 Answers
I think this may help you: Is the R language suitable for Big Data
I think it really depends on what you are comfortable with and what your objectives are. In Python, I can execute SQL queries directly into Pandas data frames. From there, data cleaning and visualizing (I like seaborn) is fairly straightforward. I can easily perform matrix manipulations using Numpy, and the sklearn is a well documented ML package.
That being said, I still like R and there are definitely times when I will use R. Learn both!
It really depends on what you mean by 'big data'. A truly big dataset cannot fit in memory, in which case local python and R really only work for smaller scale experimentation and prototyping. For the purpose of data wrangling, you'll want a picture of the whole dataset by either slicing based on cuts, sampling, or aggregation. In any case, you'll need to work on a distributed computational platform. In my opinion, python has a big leg up in this regard, as its interface with Apache Spark is quite robust (whereas the R interface seems to be wanting). With that said, if you are skillful with impala, pig, or hive, you can do partitioning in a distributed query language, and create data that can be examined locally.
Wrong question. Big data is not a question of this or that language, but cluster computing. For me it's implicit in the definition; if you can find a way to process your data on your laptop it just isn't big data.
Spark is the de facto standard today for cluster computing. It comes with many of its own munging primitives, borrowed from numerous languages (think dataframes and functional programming), and bindings to them. Scala is the best language in terms of Spark API coverage, followed by python, then R. If you want to experiment with Spark you can rent managed instances from Google on DataProc or spin up your own.
This is an interesting question. I also see people mentioning Spark as de-facto. Here are my two cents on the same
- Big Data:
The size of the data and the goal is the key here. Goal could be defined as
- Decrease computation time for data wrangling
- Efficient storage
- Handling multiple file formats coming from different sources
- Ability to provide data summarization so one can keep track of the data being wrangled. Its very easy to make mistakes and handling missing data the right way is more difficult than it seems.
R for BigData: A lot of wrangling can be done using R itself, if one has been able to resolve the problem of storage. May NAS or our good friend HDFS. The number of aggregators R provide is very detailed and mature(less buggy). But, one problem with R is that the syntax is not consistent and it becomes difficult to mange the code over a peroid of time. Also, there is a SparkR interface if you decide to go down the route of Spark.
python for BigData: Pandas and Numpy provide a pretty good package for handling data in an efficient way. Types can be guaranteed through
type-casting and not just depending on duck-typing. There is a binding with Spark called pyspark which could be useful if one decides to move from python based data-structure to spark dataframes/RDDs.data wrangling in pythonThrowing in another one scala for BigData: Spark is natively written in
Scala and is a good candidate for data wrangling and data modeling. Unlike python and R it support functional paradigm as well OOP as a first class citizens hence allowing the ease to write manageable code with static typing. Twitter guys have a bunch of libraries to help one in data-wrangling e.g. algebird, scala collection api, shapeless, slick
And then as other have mentioned there are other offering such as Pig, Hive(is batch oriented). Checkout Apache Drill( May be that suits your need better ). And if you land up using Spark then Spark SQL is an option too.
To sum it up, would suggest to start with python to be safe and then move to pyspark if you feel that the size is not manageable anymore and you need more sophisticated data-structure. And once you realize your needs may be use Scala for production code. Let me know if I could be of help or you need more information or guidance in anyway. Thanks
Big datasets
? What is the their average size? $\endgroup$