Hot answers tagged

114

Two Categorical Variables Checking if two categorical variables are independent can be done with Chi-Squared test of independence. This is a typical Chi-Square test: if we assume that two variables are independent, then the values of the contingency table for these variables should be distributed uniformly. And then we check how far away from uniform the ...


106

Some real important differences to consider when you are choosing R or Python over one another: Machine Learning has 2 phases. Model Building and Prediction phase. Typically, model building is performed as a batch process and predictions are done realtime. The model building process is a compute intensive process while the prediction happens in a jiffy. ...


42

Actually this is coming around. In the book R in a Nutshell there is even a section on using R with Hadoop for big data processing. There are some work arounds that need to be done because R does all it's work in memory, so you are basically limited to the amount of RAM you have available to you. A mature project for R and Hadoop is RHadoop RHadoop has ...


42

Whenever I work with xgboost I often make my own homebrew parameter search but you can do it with the caret package as well like KrisP just mentioned. Caret See this answer on Cross Validated for a thorough explanation on how to use the caret package for hyperparameter search on xgboost. How to tune hyperparameters of xgboost trees? Custom Grid Search I ...


32

Python is more "general purpose" while R has a clear(er) focus on statistics. However, most (if not all) things you can do in R can be done in Python as well. The difference is that you need to use additional packages in Python for some things you can do in base R. Some examples: Data frames are base R while you need to use Pandas in Python. ...


31

The main problem with using R for large data sets is the RAM constraint. The reason behind keeping all the data in RAM is that it provides much faster access and data manipulations than would storing on HDDs. If you are willing to take a hit on performance, then yes, it is quite practical to work with large datasets in R. RODBC Package: Allows connecting to ...


29

High validation scores like accuracy generally mean that you are not overfitting, however it should lead to caution and may indicate something went wrong. It could also mean that the problem is not too difficult and that your model truly performs well. Two things that could go wrong: You didn't split the data properly and the validation data also occured in ...


27

R On Cloud provides a browser-embedded R-console. Jupyter.org evolved from the IPython Project (the language-agnostic parts of IPython); supports Python 3, Julia, R, Haskell, Ruby, etc.


27

Has anyone done any benchmarks? Yes, the 2014's benchmark in question has turned into foundation for db-benchmark project. Initial step was to reproduce 2014's benchmark on recent version of software, then to make it a continuous benchmark, so it runs routinely and automatically upgrades software before each run. Over time many things have been added. Below ...


26

There is nothing like "python is better" or "R is much better than x". The only fact I know is that in the industry allots of people stick to python because that is what they learned at the university. The python community is really active and have a few great frameworks for ML and data mining etc. But to be honest, if you get a good c programmer he can ...


26

For instance: rs<-c("copyright @ The Society of mo","I want you to meet me @ the coffeshop") s<-gsub("@.*","",rs) s [1] "copyright " "I want you to meet me " Or, if you want to keep the @ character: s<-gsub("(@).*","\\1",rs) s [1] "copyright @" "I want you to meet me @" EDIT: If what you want is to remove ...


22

R contains some standard functions for data manipulation, which can be used for data cleaning, in its base package (gsub, transform, etc.), as well as in various third-party packages, such as stringr, reshape/reshape2, and plyr/dplyr. Examples and best practices of usage for these packages and their functions are described in the following paper: http://vita....


22

Just use predict_proba instead of predict. You can leave the objective as binary:logistic.


21

RIDE - R-Brain IDE (RIDE) for R & Python, Other Data Science R IDEs, Other Data Science Python IDEs. Flexible layout. Multiple language support. Jupyter notebook - The Jupyter Notebook App is a server-client application that allows editing and running notebook documents via a web browser. The Jupyter Notebook App can be executed on a local desktop ...


20

IntelliJ supports R via this plugin: https://plugins.jetbrains.com/plugin/6632 It's a recent project, so RStudio is still more powerful, including its focus on data-friendly environment (plots and data are always in sight).


19

Most common types of decision trees you encounter are not affected by any monotonic transformation. So, as long as you preserve orde, the decision trees are the same (obviously by the same tree here I understand the same decision structure, not the same values for each test in each node of the tree). The reason why it happens is because how usual impurity ...


17

From my point of view, this question is suitable for a two-step answer. The first part, let us call it soft preprocessing, could be taken as the usage of different data mining algorithms to preprocess data in such a way that makes it suitable for further analyses. Notice that this could be the analysis itself, in case the goal is simple enough to be tackled ...


17

Some good answers here. I would like to join the discussion by adding the following three notes: The question's emphasis on the volume of data while referring to Big Data is certainly understandable and valid, especially considering the problem of data volume growth outpacing technological capacities' exponential growth per Moore's Law (http://en.wikipedia....


17

Some additional thoughts. The programming language 'per se' is only a tool. All languages were designed to make some type of constructs more easy to build than others. And the knowledge and mastery of a programming language is more important and effective than the features of that language compared to others. As far as I can see there are two dimensions ...


17

Here some resources that might be helpful: Recommenderlab - a framework and open source software for developing and testing recommendation algorithms. Corresponding R package recommenderlab. The following blog post illustrates the use of recommenderlab package (which IMHO can be generalized for any open source recommendation engine) for building movie ...


17

Know I'm a bit late, but to get probabilities from xgboost you should specify multi:softmax objective like this: xgboost(param, data = x_mat, label = y_mat,nround = 3000, objective='multi:softprob') From the ?xgb.train: multi:softprob same as softmax, but output a vector of ndata * nclass, which can be further reshaped to ndata, nclass matrix. The ...


16

I think that Shiny is an overkill in this situation and doesn't match your requirement of dashboard reports to be static. I guess, that your use of the term "dashboard" is a bit confusing, as some people might consider that it has more emphasis of interactivity (real-time dashboards), rather than information layout, as is my understanding (confirmed by the "...


16

A colleague and I have conducted some preliminary studies on the performance differences between pandas and data.table. You can find the study (which was split into two parts) on our Blog (You can find part two here). We figured that there are some tasks where pandas clearly outperforms data.table, but also cases in which data.table is much faster. You can ...


15

As for a complete machine learning package on GPU's, no such package exists. However, there are actually a handful of R packages that can use GPU's. You can see these packages on the CRAN High Performance Computing page. You should note that most of these packages do require you to have a NVIDIA card. Of the packages available, there are three packages ...


15

R and SQL are two completely different beasts. SQL is a language that you can use to query data that is stored in databases as you already experienced. The benefits of SQL versus R lays mostly in the fact of the database server (MS SQL, Oracle, PostgreSQL, MySQL, etc.). Most, if not all, modern database servers permit multiple users to query data from the ...


14

I would add to what others have said till now. There is no single answer that one language is better than other. Having said that, R has a better community for data exploration and learning. It has extensive visualization capabilities. Python, on the other hand, has become better at data handling since introduction of pandas. Learning and development time ...


14

Benchmarking mlr (default) learners on OpenML The entire openml database of ML results. Test from RStudio suggests SVM. Mlmastery suggests LDA and Trial and Error. Do we Need Hundreds of Classifiers to Solve Real World Classification Problems? by Fern ́andez-Delgado et al. Paper concludes parallel random forest (parRF_t) is best followed by random ...


13

There is another choice which popular recently: docker(https://www.docker.com). Docker is a container and let you create/maintain a working environment very easily and fast. install essential tools for data science in python https://registry.hub.docker.com/u/ceshine/python-datascience/ use r language to do data science https://github.com/rocker-org/...


13

I have used R, SAS Base and H2O. First, I do not think that H2O seeks to be either R or SAS. H2O provides data mining algorithms that are highly efficient. You can interface with H2O using several APIs such as their R API. The benefit of combining R and H2O is that H2O is very good at exploiting multi-cores or clusters with minimal effort of the user. It ...


13

Python being more widely used is an important consideration. This will especially become important when applying for a job. Also Python has as many if not more key statistical and ML/AI tools as R, and a larger open-source base to utilize. Python is designed for programmers, R is designed for statisticians. Originally I was a R programmer, but most of my ...


Only top voted, non community-wiki answers of a minimum length are eligible