33 votes
Accepted

Is Python a viable language to do statistical analysis in?

Python is more "general purpose" while R has a clear(er) focus on statistics. However, most (if not all) things you can do in R can be done in Python as well. The difference is that you need ...
  • 6,962
13 votes

Is Python a viable language to do statistical analysis in?

Python being more widely used is an important consideration. This will especially become important when applying for a job. Also Python has as many if not more key statistical and ML/AI tools as R, ...
  • 1,819
11 votes

Math PhD (Nonlinear Programming) switching to Data Science?

Data science jobs cover a wide range of different activities so any answer is likely to be subjective. I'm in academia so my knowledge of the job market is limited, but from what I can see: The ...
  • 22.8k
7 votes
Accepted

Math PhD (Nonlinear Programming) switching to Data Science?

I don't disagree with the other answers, but here's a different perspective you should bear in mind. Also, I can offer answers to your specific questions as someone who left academia (applied math/CS) ...
  • 276
6 votes
Accepted

Plot Two Categorical Variables

Well, there are a few ways to do the job. Here are some I thought of: Scatterplots with noise: Normally, if you try to use a scatter plot to plot two categorical features, you would just get a few ...
  • 188
6 votes
Accepted

How to interpret standard deviation calculated by excel

You should interpret it as 12.5%. The percentage values that you are seeing in your spreadsheet (93.33%, ...) are only formats, internally Excel is dealing with ...
  • 2,276
5 votes

Additive vs Multiplicative model in Time Series Data

I want to know which model between additive and multiplicative best suits the above data. It is hard to tell just by looking at it. A multiplicative decomposition roughly corresponds to an additive ...
  • 378
5 votes

Is Python a viable language to do statistical analysis in?

I'd like to add two points to the existing answers: There is excellent interaction between R and python, with various possibilities for either direction. To me, it's not that much of a decision ...
5 votes

Is Python a viable language to do statistical analysis in?

One thing that can be a gotcha coming from R to Python is that the Python stats ecosystem tends to be more machine learning-ey oriented rather than inferential stats-ey oriented. This can create some ...
4 votes

Additive vs Multiplicative model in Time Series Data

Calculate one day returns. Plot histogram of daily returns. Calculate $log(\frac{price_{i+1}}{price_i})$. Plot histogram of above logarithm. If second plot is more likely to be normally distributed ...
4 votes

experimental design in R project

I suggest you take a look at the TidyTuesday repo, where every week they post a raw dataset, a chart or article related to that dataset, and ask you to explore the data. The repo also contains other ...
  • 16.3k
3 votes
Accepted

ValueError: pos_label=1 is not a valid label: array(['N', 'Y'], dtype='<U1')

pos_label is an argument of scikit-learn's precision_score (docs); its purpose is, well, to indicate which label is the positive ...
  • 1,820
3 votes

Math PhD (Nonlinear Programming) switching to Data Science?

Erwan nailed it (+1). But I think my addition is a little too long for a comment. You seem to be well ahead of where I was when I landed my DS job. I was in pure math, a couple of postdocs in, and ...
  • 10.1k
3 votes

Math PhD (Nonlinear Programming) switching to Data Science?

Just Don't take this advice blindly: The subjects you have mentioned in mathematics are core to solving problems using machine learning/Deep learning, programming is a tool to implement all this ...
3 votes
Accepted

How to view entire 5788 rows and 7 columns.?

pandas has a max rows setting - https://pandas.pydata.org/pandas-docs/stable/user_guide/options.html Though perhaps looking at a 5,000+ row csv in an editor, or a spreadsheet or some IDEs have a csv ...
  • 864
3 votes

Is Python a viable language to do statistical analysis in?

I eventually do plan on moving more towards ML One aspect that I would like to add based on what I observed. Things are moving with more focus towards Deep Learning e.g. Neural Networks and in this ...
  • 5,224
3 votes
Accepted

Compare two tends with big difference in absolute value

I think it would be better to use a standard scaler that removes the mean and divides by the standard deviation. See here for more info and an implementation using ...
  • 1,155
3 votes

Labelling a Time series dataset

Some feedback/tips/tricks/opinions here: Problem setup Including requirement analysis. Gotta decide how the system/solution should work, how to know ho how well we are doing, and then how to get there....
  • 1,012
2 votes

Data cleaning and data transformation before EDA?

Although not very helpful, the answer is probably "it depends". I like to do data cleaning and some EDA together since EDA can highlight appropriate treatments to clean the data - e.g. influencing ...
  • 1,517
2 votes
Accepted

Are Hadoop and Python SciPy used for the same?

I think you're quite confused. Hadoop is a collection of software that contains a a distributed file system called HDFS. Essentially HDFS is a way to store data cross a cluster. You can access ...
  • 2,225
2 votes
Accepted

Tools to explore various datasets

There is a very cool active Python package called pandas-profiling, is exactly what you want. With a simple pandas_profiling.ProfileReport(df) it returns a lot of ...
  • 4,046
2 votes

What is the difference between 'if the data is of good quality' and 'if the data is tidy'?

Let me try to explain by intuitively. First let me take the easy one. Data being tidy As per definition Tidy means Arranged in Order, Neat, Uncluttered. All of these explain the physical aspects ...
2 votes
Accepted

How far can one go with excel?

TL;DR If you have unlimited time and use a 64-bit version of Excel, you can get as far with Excel as any other data analysis tool. Time I mention time as my first factor, because Excel only has ...
  • 14.1k
2 votes

Building predictive model with low correlated data

It seems like a challenging problem. If it were my task, I would start with a probabilistic approach like apriori, but you may want to check out ...
  • 958
2 votes

Building predictive model with low correlated data

It is really hard to answer your question because there is too little information. Try to do some EDA and attach it to your question because EDA defines the model. Anyway, for weakly correlated data, ...
  • 39
2 votes
Accepted

Plotting two pandas data frame columns against each other

You need to groupby to deal with multiple vote counts: df.groupby('timestamp').sum().plot(x='timestamp', y='vote_count')
  • 10.4k
2 votes

Covariance as inner product

Definition A inner product (AKA dot product and scalar product) can be define on two vectors $\mathbf{x}$ and $\mathbf{y}$ $\in \mathcal{R^n} $ as $$ \mathbf{x.x^T} = <\mathbf{x},\mathbf{y}>_\...
2 votes

How is DS used in the case of Payment Gateways?

Common use cases include: Fraud detection Transactions volume prediction Next transaction date Fraud detection This is usually tackled with anomaly detection. It requires information on the two ...
2 votes
Accepted

How to analyse player and enemy position for data analysis

Either do unsupervised learning with something like k-means clustering or DBScan where you attempt to segment students into groups and see if you can discern any insights based on the cluster ...
  • 371
2 votes
Accepted

No statistical significance but observable trends

There is high variance within each group. Even though there is a mean difference between the groups, there is a high amount of spread within just treatment A or just treatment B. From a statistical ...

Only top scored, non community-wiki answers of a minimum length are eligible