33
votes
Accepted
Is Python a viable language to do statistical analysis in?
Python is more "general purpose" while R has a clear(er) focus on statistics. However, most (if not all) things you can do in R can be done in Python as well. The difference is that you need ...
- 7,207
13
votes
Is Python a viable language to do statistical analysis in?
Python being more widely used is an important consideration. This will especially become important when applying for a job. Also Python has as many if not more key statistical and ML/AI tools as R, ...
- 1,869
11
votes
Math PhD (Nonlinear Programming) switching to Data Science?
Data science jobs cover a wide range of different activities so any answer is likely to be subjective. I'm in academia so my knowledge of the job market is limited, but from what I can see:
The ...
- 24.5k
7
votes
Accepted
Math PhD (Nonlinear Programming) switching to Data Science?
I don't disagree with the other answers, but here's a different perspective you should bear in mind. Also, I can offer answers to your specific questions as someone who left academia (applied math/CS) ...
- 276
7
votes
Accepted
Plot Two Categorical Variables
Well, there are a few ways to do the job. Here are some I thought of:
Scatterplots with noise:
Normally, if you try to use a scatter plot to plot two categorical features, you would just get a few ...
- 198
6
votes
Additive vs Multiplicative model in Time Series Data
I want to know which model between additive and multiplicative best suits the above data.
It is hard to tell just by looking at it.
A multiplicative decomposition roughly corresponds to an additive ...
- 388
6
votes
Accepted
How to interpret standard deviation calculated by excel
You should interpret it as 12.5%.
The percentage values that you are seeing in your spreadsheet (93.33%, ...) are only formats, internally Excel is dealing with ...
- 2,371
5
votes
Additive vs Multiplicative model in Time Series Data
Calculate one day returns.
Plot histogram of daily returns.
Calculate $log(\frac{price_{i+1}}{price_i})$.
Plot histogram of above logarithm.
If second plot is more likely to be normally distributed ...
- 161
5
votes
Is Python a viable language to do statistical analysis in?
I'd like to add two points to the existing answers:
There is excellent interaction between R and python, with various possibilities for either direction.
To me, it's not that much of a decision ...
5
votes
Is Python a viable language to do statistical analysis in?
One thing that can be a gotcha coming from R to Python is that the Python stats ecosystem tends to be more machine learning-ey oriented rather than inferential stats-ey oriented.
This can create some ...
- 181
5
votes
Accepted
What advantages does Data Visualization have in EDA?
First, visualization is just an easy and intuitive way to understand underlying patterns in your data. Everything that you can achieve through this, can also be achieved through painstakingly printing ...
- 518
5
votes
What advantages does Data Visualization have in EDA?
The simplest example is Anscombe's Quartet
. These four data sets are quite different, which doesn't appear just by looking at the summary stats.
- 294
5
votes
Accepted
Is it Possible to plot Scatter Plot + Histogram + Correlation Values in a single plot (in python)?
Corrmorant is based on ggplot, but it seems that there is no equivalent in Python.
However, you can redo it thanks to this code:
...
- 4,299
4
votes
experimental design in R project
I suggest you take a look at the TidyTuesday repo, where every week they post a raw dataset, a chart or article related to that dataset, and ask you to explore the data. The repo also contains other ...
- 19.2k
3
votes
Accepted
ValueError: pos_label=1 is not a valid label: array(['N', 'Y'], dtype='<U1')
pos_label is an argument of scikit-learn's precision_score (docs); its purpose is, well, to indicate which label is the positive ...
- 1,859
3
votes
Math PhD (Nonlinear Programming) switching to Data Science?
Erwan nailed it (+1). But I think my addition is a little too long for a comment.
You seem to be well ahead of where I was when I landed my DS job. I was in pure math, a couple of postdocs in, and ...
- 10.8k
3
votes
Math PhD (Nonlinear Programming) switching to Data Science?
Just Don't take this advice blindly:
The subjects you have mentioned in mathematics are core to solving problems using machine learning/Deep learning, programming is a tool to implement all this ...
- 142
3
votes
Accepted
How to view entire 5788 rows and 7 columns.?
pandas has a max rows setting - https://pandas.pydata.org/pandas-docs/stable/user_guide/options.html
Though perhaps looking at a 5,000+ row csv in an editor, or a spreadsheet or some IDEs have a csv ...
- 904
3
votes
Is Python a viable language to do statistical analysis in?
I eventually do plan on moving more towards ML
One aspect that I would like to add based on what I observed.
Things are moving with more focus towards Deep Learning e.g. Neural Networks and in this ...
- 5,404
3
votes
Accepted
Compare two tends with big difference in absolute value
I think it would be better to use a standard scaler that removes the mean and divides by the standard deviation. See here for more info and an implementation using ...
- 1,335
3
votes
Labelling a Time series dataset
Some feedback/tips/tricks/opinions here:
Problem setup
Including requirement analysis. Gotta decide how the system/solution should work, how to know ho how well we are doing, and then how to get there....
- 1,292
2
votes
Data cleaning and data transformation before EDA?
Although not very helpful, the answer is probably "it depends".
I like to do data cleaning and some EDA together since EDA can highlight appropriate treatments to clean the data - e.g. influencing ...
- 1,527
2
votes
Accepted
Are Hadoop and Python SciPy used for the same?
I think you're quite confused.
Hadoop is a collection of software that contains a a distributed file system called HDFS. Essentially HDFS is a way to store data cross a cluster. You can access ...
- 2,330
2
votes
Accepted
Tools to explore various datasets
There is a very cool active Python package called pandas-profiling, is exactly what you want. With a simple pandas_profiling.ProfileReport(df) it returns a lot of ...
- 4,108
2
votes
What is the difference between 'if the data is of good quality' and 'if the data is tidy'?
Let me try to explain by intuitively. First let me take the easy one.
Data being tidy
As per definition Tidy means Arranged in Order, Neat, Uncluttered.
All of these explain the physical aspects ...
- 178
2
votes
Accepted
How far can one go with excel?
TL;DR
If you have unlimited time and use a 64-bit version of Excel, you can get as far with Excel as any other data analysis tool.
Time
I mention time as my first factor, because Excel only has ...
- 14.4k
2
votes
Building predictive model with low correlated data
It seems like a challenging problem. If it were my task, I would start with a probabilistic approach like apriori, but you may want to check out ...
- 978
2
votes
Building predictive model with low correlated data
It is really hard to answer your question because there is too little information. Try to do some EDA and attach it to your question because EDA defines the model.
Anyway, for weakly correlated data, ...
- 49
2
votes
Accepted
Plotting two pandas data frame columns against each other
You need to groupby to deal with multiple vote counts:
df.groupby('timestamp').sum().plot(x='timestamp', y='vote_count')
- 10.5k
2
votes
Covariance as inner product
Definition
A inner product (AKA dot product and scalar product) can be define on two vectors $\mathbf{x}$ and $\mathbf{y}$ $\in \mathcal{R^n} $ as
$$ \mathbf{x.x^T} = \langle \mathbf{x},\mathbf{y}\...
- 1,586
Only top scored, non community-wiki answers of a minimum length are eligible