29
Python is more "general purpose" while R has a clear(er) focus on statistics. However, most (if not all) things you can do in R can be done in Python as well. The difference is that you need to use additional packages in Python for some things you can do in base R.
Examples:
Data frames are base R while you need to use Pandas in Python.
Linear ...
13
Python being more widely used is an important consideration. This will especially become important when applying for a job. Also Python has as many if not more key statistical and ML/AI tools as R, and a larger open-source base to utilize. Python is designed for programmers, R is designed for statisticians. Originally I was a R programmer, but most of my ...
7
Practically everything related to statistics (including Machine Learning) has to do with studying chance, i.e. trying to determine to what extent an observation is due to chance or not.
For example one might want to know whether a drug actually helps with a particular disease or not. If we observe that one patient improves after taking the drug, there's ...
6
The shaded area likely shows the dark green line plus or minus some error/uncertainty estimate. Common error estimates may be based on the standard deviation, a confidence interval, or the interquartile range depending on the data and the analysis being done. Without more information, we cannot know what the shaded area represents.
5
It might be plus or minus one standard deviation. But it could be anything really. Without more context you can't be sure.
4
The statement:
We most often use k=10 because evidence shows it's the best value for k. Smaller values don't give as good estimates, and larger values don't provide much better results either.
Is just categorically false. The reason people default to K=10 is because they don't know how changing K effects their estimates of the generalization error and ...
4
Yes, you are correct. Rolling of dice is (almost always) a random phenomenon. In case of biased v unbiased dice, the difference is solely in the probability distributions. For unbiased the distribution must be uniform, for biased it can be anything we want.
The only case where biased dice stop being random phenomena is when we load them in such a way that ...
4
I'd like to add two points to the existing answers:
There is excellent interaction between R and python, with various possibilities for either direction.
To me, it's not that much of a decision python vs. R. The decision is to choose the main language appropriately for the project at hand, and then do parts in the other language if that is better for some ...
3
Without more information it is hard to say for certain, but my best guess is that the shaded region is a confidence interval (say ± 1 standard deviation) around the predicted values which are represented by the line. For example, have a read through this article where the author creates a time series prediction for stock market prices with a confidence ...
3
Moving averages will give you a smoother time series so that a trend is easier to see by eye. This approach makes sense when you’re exploring the data.
The next step is to try to comment on where the time series will go next. Based on the tags you have chosen for your question, you are comfortable writing python. You might consider Facebook’s open source ...
3
Yes it makes sense, a moving average makes the curve "smoother" in the sense that it's less sensitive to short variations. This usually makes it easier to observe the general tendency.
You could also try different time periods for the average, e.g. 10 days or 15 days.
It looks to me like there's a moderate increase trend in your data, but the ...
3
It sometimes is called "ensemble learning" where several "weak learners" make a prediction. These predictions are "combined" by some meta-model. A simplistic approach would be that you just use the majority vote. You can also use logistic regression. You can (and should!) of course check the performance of the stacked model(s) ...
3
I eventually do plan on moving more towards ML
One aspect that I would like to add based on what I observed.
Things are moving with more focus towards Deep Learning e.g. Neural Networks and in this space, most of the dominating Libraries supports Python as first choice.
Companies manage a separate Python version to open-source, just to maintain the user ...
3
One thing that can be a gotcha coming from R to Python is that the Python stats ecosystem tends to be more machine learning-ey oriented rather than inferential stats-ey oriented.
This can create some hiccups, because some of the defaults in R that are the defaults because people who do inferential stats like in the social sciences always use them, are not ...
3
You could think of histogram as one way of plotting a distribution of values. Another way of plotting such distribution is a KDE (Kernel Density Estimate), but after all, these inform about the same concept, which is how frequent is each value (or values interval) of your values series (x-axis).
I like this picture from seaborn distplot to have a one shot ...
3
In addition to Erwan's answer, which gives great general advice, consider the following questions when you are deciding rather to keep data.
What question(s) are you trying to answer? What are you trying to learn from the data?
If you are trying to build a model that will predict patient recovery based on drug administered and a variety of other ...
2
Excerpt from ESL -
Ideally, if we had enough data, we would set aside a validation set and use
it to assess the performance of our prediction model. Since data are often
scarce, this is usually not possible. To finesse the problem, K-fold cross-validation uses part of the available data to fit the model
So, key goal is to get the variance of full ...
1
One definition for "modality of data" is how many different types of data are included in the dataset.
For example: Images along with tags and text.
Different modalities usually have very different statistical properties, which can make the dataset more complex to work with.
1
Cardinality and Modality are the two data modelling concepts used for understanding the information domain of the problem. For analysing the data objects, data attributes and relationships structures, the terms given above are very important.
The major difference between cardinality and modality is that, the cardinality is defined as the metric used to ...
1
If you draw a sample equally favoring each ball in the bag - disregarding the number on the ball - it will be a random sample.
If the expectation is that you want the random sample to have a similar distribution as the balls in the bag, then sampling with replacement is better.
But, if the number of draws is small (and the bag is big), the weather you ...
1
Yes it does, you resample from the whole population without replacement $N_A$ samples, which are associated to group $A$, and the rest of the samples are associated to group $B$. As I understand, you compute the statistic for every possible permutation of the total population and you measure how the actual statistic compares in the distribution of all the ...
1
You always choose the statement that you want to disprove as Null hypothesis. You can either reject it in favour of alternative hypothesis or not reject it. We don't say we accept null hypothesis. Because we don't have enough evidence for the null hypothesis to be true. It might be true, might be not. But with significant t/p value we can reject null ...
1
There are many ways in which Ensembling can be done and each one has a different foundation logic to gain improvement.
Key variations can be -
1. Nature(High Bias/High Variance) of models in the ensemble
2. How we put models into work i.e. same model type, different model type, parallel, sequential, sample data, full data etc.
3. How we combine individual ...
1
For the love of the flying spaghetti monster, use anaconda to install the needed packages for data science. I have seen both Python and R being used in the data science setting and both needed additional packages to execute any data science capabilities. Conda made it way easier to install them.
From my point of view, Python has a better support for all ...
1
Performance of a model can be judged along several dimensions:
Accuracy - Is the train-test validation performing well?
Overfitting - Is the difference between training score and the validation score minimal?
Efficiency - Is the model light-weight, does it compute and calculate fast?
Complexity - Is the model easy to explain, does it use minimal ...
1
BM25 is usually used in information retrieval. In this task, you have a query and a lot of documents(maybe millions), and then you want to find a subset of these documents that are most relevant to your query. A ranking of a set of documents will be provided from the most relevant to the least.
If by efficient you mean fast in a computational way. I would ...
1
The distribution of a variable is an abstract concept which represents how the variable is "distributed", that is it represents the chances that the variable has any particular value.
For example if the variable is the outcome of a regular dice, then any of the values 1 to 6 has the same chances to appear (1/6). This is a uniform distribution. If you ...
1
About what ANOVA is used for: it can answer whether the difference between the mean values for the data samples I have is due to randomness or is it statistcally significant. Then it is a significance-test that gives you an idea about whether your mean values are (statistically significantly) the same or not. A drawback is that it does not tell you which ...
Only top voted, non community-wiki answers of a minimum length are eligible
