33 votes
Accepted

Is Python a viable language to do statistical analysis in?

Python is more "general purpose" while R has a clear(er) focus on statistics. However, most (if not all) things you can do in R can be done in Python as well. The difference is that you need ...
Peter's user avatar
  • 7,426
13 votes

Is Python a viable language to do statistical analysis in?

Python being more widely used is an important consideration. This will especially become important when applying for a job. Also Python has as many if not more key statistical and ML/AI tools as R, ...
Donald S's user avatar
  • 1,939
11 votes

Math PhD (Nonlinear Programming) switching to Data Science?

Data science jobs cover a wide range of different activities so any answer is likely to be subjective. I'm in academia so my knowledge of the job market is limited, but from what I can see: The ...
Erwan's user avatar
  • 25.2k
8 votes
Accepted

Plot Two Categorical Variables

Well, there are a few ways to do the job. Here are some I thought of: Scatterplots with noise: Normally, if you try to use a scatter plot to plot two categorical features, you would just get a few ...
MartinM's user avatar
  • 308
7 votes
Accepted

Math PhD (Nonlinear Programming) switching to Data Science?

I don't disagree with the other answers, but here's a different perspective you should bear in mind. Also, I can offer answers to your specific questions as someone who left academia (applied math/CS) ...
RHC's user avatar
  • 276
6 votes

Additive vs Multiplicative model in Time Series Data

I want to know which model between additive and multiplicative best suits the above data. It is hard to tell just by looking at it. A multiplicative decomposition roughly corresponds to an additive ...
naive's user avatar
  • 388
6 votes
Accepted

How to interpret standard deviation calculated by excel

You should interpret it as 12.5%. The percentage values that you are seeing in your spreadsheet (93.33%, ...) are only formats, internally Excel is dealing with ...
Multivac's user avatar
  • 2,949
5 votes

Additive vs Multiplicative model in Time Series Data

Calculate one day returns. Plot histogram of daily returns. Calculate $log(\frac{price_{i+1}}{price_i})$. Plot histogram of above logarithm. If second plot is more likely to be normally distributed ...
Michał Kardach's user avatar
5 votes

Is Python a viable language to do statistical analysis in?

I'd like to add two points to the existing answers: There is excellent interaction between R and python, with various possibilities for either direction. To me, it's not that much of a decision ...
cbeleites unhappy with SX's user avatar
5 votes

Is Python a viable language to do statistical analysis in?

One thing that can be a gotcha coming from R to Python is that the Python stats ecosystem tends to be more machine learning-ey oriented rather than inferential stats-ey oriented. This can create some ...
Paul Gowder's user avatar
5 votes
Accepted

What advantages does Data Visualization have in EDA?

First, visualization is just an easy and intuitive way to understand underlying patterns in your data. Everything that you can achieve through this, can also be achieved through painstakingly printing ...
liakoyras's user avatar
  • 636
5 votes

What advantages does Data Visualization have in EDA?

The simplest example is Anscombe's Quartet . These four data sets are quite different, which doesn't appear just by looking at the summary stats.
Richard Careaga's user avatar
5 votes
Accepted

Is it Possible to plot Scatter Plot + Histogram + Correlation Values in a single plot (in python)?

Corrmorant is based on ggplot, but it seems that there is no equivalent in Python. However, you can redo it thanks to this code: ...
Nicolas Martin's user avatar
4 votes

experimental design in R project

I suggest you take a look at the TidyTuesday repo, where every week they post a raw dataset, a chart or article related to that dataset, and ask you to explore the data. The repo also contains other ...
noe's user avatar
  • 25.6k
4 votes

Pandas Profiling Not Working

Something to note is that the package name will soon change to ydata-profiling, so we should use the new name. This is the announcement on their Pypi site: ⚠️ ...
noe's user avatar
  • 25.6k
3 votes
Accepted

ValueError: pos_label=1 is not a valid label: array(['N', 'Y'], dtype='<U1')

pos_label is an argument of scikit-learn's precision_score (docs); its purpose is, well, to indicate which label is the positive ...
desertnaut's user avatar
  • 1,958
3 votes

Math PhD (Nonlinear Programming) switching to Data Science?

Erwan nailed it (+1). But I think my addition is a little too long for a comment. You seem to be well ahead of where I was when I landed my DS job. I was in pure math, a couple of postdocs in, and ...
Ben Reiniger's user avatar
  • 11.6k
3 votes

Math PhD (Nonlinear Programming) switching to Data Science?

Just Don't take this advice blindly: The subjects you have mentioned in mathematics are core to solving problems using machine learning/Deep learning, programming is a tool to implement all this ...
khwaja wisal's user avatar
3 votes
Accepted

How to view entire 5788 rows and 7 columns.?

pandas has a max rows setting - https://pandas.pydata.org/pandas-docs/stable/user_guide/options.html Though perhaps looking at a 5,000+ row csv in an editor, or a spreadsheet or some IDEs have a csv ...
Craig's user avatar
  • 944
3 votes

Is Python a viable language to do statistical analysis in?

I eventually do plan on moving more towards ML One aspect that I would like to add based on what I observed. Things are moving with more focus towards Deep Learning e.g. Neural Networks and in this ...
10xAI's user avatar
  • 5,574
3 votes
Accepted

Compare two tends with big difference in absolute value

I think it would be better to use a standard scaler that removes the mean and divides by the standard deviation. See here for more info and an implementation using ...
etiennedm's user avatar
  • 1,385
3 votes

Labelling a Time series dataset

Some feedback/tips/tricks/opinions here: Problem setup Including requirement analysis. Gotta decide how the system/solution should work, how to know ho how well we are doing, and then how to get there....
Jon Nordby's user avatar
  • 1,482
3 votes

Solve tough clustering problem with overlapping clusters

The Problem is that many clustering algorithms focus on distances (between points, clusters and so on). Especially at the connection between the two desired clusters, distances between points are ...
Broele's user avatar
  • 1,352
2 votes

Data cleaning and data transformation before EDA?

Although not very helpful, the answer is probably "it depends". I like to do data cleaning and some EDA together since EDA can highlight appropriate treatments to clean the data - e.g. influencing ...
bradS's user avatar
  • 1,565
2 votes
Accepted

Are Hadoop and Python SciPy used for the same?

I think you're quite confused. Hadoop is a collection of software that contains a a distributed file system called HDFS. Essentially HDFS is a way to store data cross a cluster. You can access ...
Tophat's user avatar
  • 2,400
2 votes
Accepted

Tools to explore various datasets

There is a very cool active Python package called pandas-profiling, is exactly what you want. With a simple pandas_profiling.ProfileReport(df) it returns a lot of ...
TwinPenguins's user avatar
  • 4,219
2 votes
Accepted

How far can one go with excel?

TL;DR If you have unlimited time and use a 64-bit version of Excel, you can get as far with Excel as any other data analysis tool. Time I mention time as my first factor, because Excel only has ...
n1k31t4's user avatar
  • 14.8k
2 votes

Metrics/Methods for deciding duration of video retention for on-demand websites

Clustering can be an ideal choice here. From the question, seems the data will most probably be in continuous format. Essentially, clustering is a method of finding groups of similar objects. The ...
mnm's user avatar
  • 411
2 votes

What is the difference between 'if the data is of good quality' and 'if the data is tidy'?

Let me try to explain by intuitively. First let me take the easy one. Data being tidy As per definition Tidy means Arranged in Order, Neat, Uncluttered. All of these explain the physical aspects ...
Karthik Sunil's user avatar
2 votes

How to get the survival duration prediction for each individual in the data by using the Kaplan-Meier method?

The Kaplan Meier curve is a summary statistic, similar to the average. Therefore, it is an unconditional statistic. If you are interested in the conditional survival, we can use the Kaplan Meier curve ...
Cam.Davidson.Pilon's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible