91

Two Categorical Variables Checking if two categorical variables are independent can be done with Chi-Squared test of independence. This is a typical Chi-Square test: if we assume that two variables are independent, then the values of the contingency table for these variables should be distributed uniformly. And then we check how far away from uniform the ...


39

Or must I loose most of the efficiency gained by programming in C by calling on R scripts or other languages? Do the opposite: learn C/C++ to write R extensions. Use C/C++ only for the performance critical sections of your new algorithms, use R to build your analysis, import data, make plots etc. If you want to go beyond R, I'd recommend learning python. ...


38

This answer is on the general side of cost functions, not related to TensorFlow, and will mostly address the "some explanation about this topic" part of your question. In most examples/tutorial I followed, the cost function used was somewhat arbitrary. The point was more to introduce the reader to a specific method, not to the cost function specifically. It ...


33

Another alternative is to use the heatmap function in seaborn to plot the covariance. This example uses the Auto data set from the ISLR package in R (the same as in the example you showed). import pandas.rpy.common as com import seaborn as sns %matplotlib inline # load the R package ISLR infert = com.importr("ISLR") # load the Auto dataset auto_df = com....


31

Python is more "general purpose" while R has a clear(er) focus on statistics. However, most (if not all) things you can do in R can be done in Python as well. The difference is that you need to use additional packages in Python for some things you can do in base R. Examples: Data frames are base R while you need to use Pandas in Python. Linear ...


28

I suggest some sort of play on the following: Using the UCI Abalone data for this example... import matplotlib import numpy as np import matplotlib.pyplot as plt %matplotlib inline # Read file into a Pandas dataframe from pandas import DataFrame, read_csv f = 'https://archive.ics.uci.edu/ml/machine-learning-databases/abalone/abalone.data' df = read_csv(f) ...


27

R On Cloud provides a browser-embedded R-console. Jupyter.org evolved from the IPython Project (the language-agnostic parts of IPython); supports Python 3, Julia, R, Haskell, Ruby, etc.


21

A GLM is absolutely a statistical model, but statistical models and machine learning techniques are not mutually exclusive. In general, statistics is more concerned with inferring parameters, whereas in machine learning, prediction is the ultimate goal.


21

Introductory: Machine Learning: The Art and Science of Algorithms that Make Sense of Data (Flach) Learning From Data (Abu-Mostafa et al.) Introduction to Statistical Learning (James et al.) Digging deeper: Elements of Statistical Learning (Hastie et al.) Pattern Recognition and Machine Learning (Bishop) Some special interest examples: Convex ...


18

I think in the original paper they suggest using $\log_2(N +1$), but either way the idea is the following: The number of randomly selected features can influence the generalization error in two ways: selecting many features increases the strength of the individual trees whereas reducing the number of features leads to a lower correlation among the trees ...


16

A possible solution is to calculate the information gain associated to each attribute: Initially you have the whole dataset, and compute the information gain of each item. The item with the best information gain is the one you should use to partition the dataset (considering the item's values). Then, perform the same computations for each item (but the ones ...


15

Regarding prediction, statistics and machine learning sciences started to solve mostly the same problem from different perspectives. Basically statistics assumes that the data were produced by a given stochastic model. So, from a statistical perspective, a model is assumed and given various assumptions the errors are treated and the model parameters and ...


15

From this sample set I would expect a histogram of receipt that shows two occurrences of receipt 102857 (since that person bought two items in one transaction) and one occurrence respectively of receipt 102856 and of receipt 102858. Then you want: df.groupby('receipt').receipt.count() receipt 102856 1 102857 2 102858 1 Name: receipt, ...


13

If I could only recomend one to you, it would be: The Elements of Statistical Learning and Prediction by Hastie, Tibshirani and Friedman. It provides the math/statistics behind a lot of commonly used techniques in data science. For Bayesian Techniques, Bayesian Data Analysis by Gelman, Carlin, Stern, Dunson, Vehtari and Rubin is excellent. Statistical ...


13

It depends what question you are trying to answer. You are looking at the rate of change of a time series, and it sounds like you are trying to show how that changed over time. The mean gives the reader one intuitive insight: they can trivially estimate the number of followers at any date $d$ days since the start by multiplying by the mean rate of change. ...


13

Python being more widely used is an important consideration. This will especially become important when applying for a job. Also Python has as many if not more key statistical and ML/AI tools as R, and a larger open-source base to utilize. Python is designed for programmers, R is designed for statisticians. Originally I was a R programmer, but most of my ...


12

For the record, I think this is the type of question that's perfect for the data science Stack Exchange. I hope we get a bunch of real world examples of data problems and several perspectives on how best to solve them. I would encourage you not to use p-values as they can be pretty misleading (1, 2). My approach hinges on you being able to summarize traffic ...


12

The one thing that you can say for sure is: Nobody can say this for sure. And it might indeed be opinion-based to some extent. The introduction of terms like "Big Data" that some people consider as "hypes" or "buzzwords" don't make it easier to flesh out an appropriate answer here. But I'll try. In general, interdisciplinary fields often seem to have the ...


12

This is a pretty massive question, so this is not intended to be a full answer, but hopefully this can help to inform general practice around determining the best tool for the job when it comes to data science. Generally, I have a relatively short list of qualifications I look for when it comes to any tool in this space. In no particular order they are: ...


12

I agree that the current trend is to use Python/R and to bind it to some C/C++ extensions for computationally expensive tasks. However, if you want to stay in C/C++, you might want to have a look at Dlib: Dlib is a general purpose cross-platform C++ library designed using contract programming and modern C++ techniques. It is open source software and ...


11

The downvotes are because of the topic, but I'll attempt to answer your question as best I can since it's here. Data science is a term that is thrown around as loosely as Big Data. Everyone has a rough idea of what they mean by the term, but when you look at the actual work tasks, a data scientist's responsibilities will vary greatly from company to ...


11

This is very broad question, which I think it's impossible to cover comprehensively in a single answer. Therefore, I think that it would be more beneficial to provide some pointers to relevant answers and/or resources. This is exactly what I will do by providing the following information and thoughts of mine. First of all, I should mention the excellent and ...


11

As Andre Holzner has said, extending R with C/C++ extension is a very good way to take advantage of the best of both sides. Also you can try the inverse , working with C++ and ocasionally calling function of R with the RInside package o R. Here you can find how http://cran.r-project.org/web/packages/RInside/index.html http://dirk.eddelbuettel.com/code/...


11

To answer your question on Cross entropy, you'll notice that both of what you have mentioned are the same thing. $-\frac{1}{n} \sum(y\_train * \log(y\_output) + (1 - y\_train) \cdot \log(1 - y\_output))$ that you mentioned is simply the binary cross entropy loss where you assume that $y\_train$ is a 0/1 scalar and that $y\_output$ is again a scalar ...


10

Back in my data analyst days this type of problem was pretty typical. Basically, everyone in marketing would come up with a crazy idea that the sold to higher ups as the single event that would boost KPI's by 2000%. The higher ups would approve them and then they would begin their "test". Results would come back, and management would dump it on the data ...


10

The arithmetic mean is denoted as $\bar{x}$ $$\bar{x} = \frac{1}{n} \sum_{i=1}^n x_i $$ where each $x_i$ represent an unique observation. The arithmetic mean measures the average value for a given set of numbers. In contrast to this, the median is the value which falls directly in the middle of your dataset. The median is especially useful when you are ...


9

You are asking about Data Dredging, which is what happens when testing a very large number of hypotheses against a data set, or testing hypotheses against a data set that were suggested by the same data. In particular, check out Multiple hypothesis hazard, and Testing hypotheses suggested by the data. The solution is to use some kind of correction for ...


9

Well it depends on what kind of "Data Science" you wish to get in to. For basic analytics and reporting statistics will certainly help, but for Machine Learning and Artificial Intelligence then you'll want a few more skills Probability theory - you must have a solid background in pure probability so that you can decompose any problem, whether seen before ...


8

Just head to kaggle.com; it'll keep you busy for a long time. For open data there's the UC Irvine Machine Learning Repository. In fact, there's a whole Stackexchange site devoted to this; look there.


8

There are basic things you can do with any set of data: Validate values (String length tolerance, data type, formatting masks, required field presence, etc.) Range correctness (Does this seemingly correct data fall within expected ranges of values) Preliminary processing (If I attempt to analyze this data, can I perform the basics without running into ...


Only top voted, non community-wiki answers of a minimum length are eligible