32

Python is more "general purpose" while R has a clear(er) focus on statistics. However, most (if not all) things you can do in R can be done in Python as well. The difference is that you need to use additional packages in Python for some things you can do in base R. Some examples: Data frames are base R while you need to use Pandas in Python. ...


13

Python being more widely used is an important consideration. This will especially become important when applying for a job. Also Python has as many if not more key statistical and ML/AI tools as R, and a larger open-source base to utilize. Python is designed for programmers, R is designed for statisticians. Originally I was a R programmer, but most of my ...


11

Data science jobs cover a wide range of different activities so any answer is likely to be subjective. I'm in academia so my knowledge of the job market is limited, but from what I can see: The current context is very favorable to data scientists looking for a job, so anybody with some basic knowledge of ML has a chance. You're already above this level so ...


7

I don't disagree with the other answers, but here's a different perspective you should bear in mind. Also, I can offer answers to your specific questions as someone who left academia (applied math/CS) for data science. In short, understanding the underlying needs and use cases for a business problem is paramount to any project, and so developing a strong ...


5

I want to know which model between additive and multiplicative best suits the above data. It is hard to tell just by looking at it. A multiplicative decomposition roughly corresponds to an additive decomposition of the logarithms. The additive decomposition is the most appropriate if the magnitude of the seasonal fluctuations, or the variation around the ...


5

I'd like to add two points to the existing answers: There is excellent interaction between R and python, with various possibilities for either direction. To me, it's not that much of a decision python vs. R. The decision is to choose the main language appropriately for the project at hand, and then do parts in the other language if that is better for some ...


5

One thing that can be a gotcha coming from R to Python is that the Python stats ecosystem tends to be more machine learning-ey oriented rather than inferential stats-ey oriented. This can create some hiccups, because some of the defaults in R that are the defaults because people who do inferential stats like in the social sciences always use them, are not ...


4

I suggest you take a look at the TidyTuesday repo, where every week they post a raw dataset, a chart or article related to that dataset, and ask you to explore the data. The repo also contains other resources, like data science books. Together with the repo, I suggest the TidyTuesday videos by David Robinson, where he creates screencasts of complete data ...


3

Calculate one day returns. Plot histogram of daily returns. Calculate $log(\frac{price_{i+1}}{price_i})$. Plot histogram of above logarithm. If second plot is more likely to be normally distributed then choose multiplicative model. Else, choose additive model. You can also perform statistical test for normal distribution and check, which one has higher p-...


3

Erwan nailed it (+1). But I think my addition is a little too long for a comment. You seem to be well ahead of where I was when I landed my DS job. I was in pure math, a couple of postdocs in, and had only a short time of self-study when I was applying for industry data science. On the other hand, I had actuarial exams in my undergrad, which probably ...


3

Just Don't take this advice blindly: The subjects you have mentioned in mathematics are core to solving problems using machine learning/Deep learning, programming is a tool to implement all this theory that you learn and on that basis you create your hypotheses and then test by implementing it in code for that you do not need coding skills of a coder you ...


3

pandas has a max rows setting - https://pandas.pydata.org/pandas-docs/stable/user_guide/options.html Though perhaps looking at a 5,000+ row csv in an editor, or a spreadsheet or some IDEs have a csv editor would be more useful.


3

I eventually do plan on moving more towards ML One aspect that I would like to add based on what I observed. Things are moving with more focus towards Deep Learning e.g. Neural Networks and in this space, most of the dominating Libraries supports Python as first choice. Companies manage a separate Python version to open-source, just to maintain the user ...


3

I think it would be better to use a standard scaler that removes the mean and divides by the standard deviation. See here for more info and an implementation using sklearn. Why? At least you should be aware that dividing by the maximum could hide smaller effects. In the case you have an outlier that has a very high value, you would loose the small changes in ...


3

Some feedback/tips/tricks/opinions here: Problem setup Including requirement analysis. Gotta decide how the system/solution should work, how to know ho how well we are doing, and then how to get there. Model evaluation. It is very desirable to have a quantitative way to evaluate our model performance. For that we want some labeled data. It is very quick to ...


2

Although not very helpful, the answer is probably "it depends". I like to do data cleaning and some EDA together since EDA can highlight appropriate treatments to clean the data - e.g. influencing how to handle missing values. I think data transformation should be done just prior to modelling; whether or not you need to do any transformation at all depends ...


2

I think you're quite confused. Hadoop is a collection of software that contains a a distributed file system called HDFS. Essentially HDFS is a way to store data cross a cluster. You can access file stores as you would in a local file store (with some modification) and modify things via Java API. Furthermore, ON TOP OF the file system there exist a ...


2

There is a very cool active Python package called pandas-profiling, is exactly what you want. With a simple pandas_profiling.ProfileReport(df) it returns a lot of important statistical information about your data, the official documentation says: For each column the following statistics - if relevant for the column type - are presented in an interactive ...


2

Let me try to explain by intuitively. First let me take the easy one. Data being tidy As per definition Tidy means Arranged in Order, Neat, Uncluttered. All of these explain the physical aspects of the data representation. For example, data arranged in proper columns, with good headings, with relevance etc. You can think of this being syntactic in nature ...


2

TL;DR If you have unlimited time and use a 64-bit version of Excel, you can get as far with Excel as any other data analysis tool. Time I mention time as my first factor, because Excel only has basic funcitonality built in, such as summing, random number generation, lookups etc. These correspond to a kind of standard library, which Python and R also have. ...


2

It seems like a challenging problem. If it were my task, I would start with a probabilistic approach like apriori, but you may want to check out Naive Bayes based approach. There are some differences in these approaches, but, either one may produce decent results. More generally, I think the analysis you want to perform is $Association\ Rule\ Learning$. ...


2

really hard answer for u question cause there is to little information. Try to make EDA and attach it to question. Cause EDA define the model Anywhere, for low corellated data, try to use k-NN. If u use Python: scikit-learn have implementation. Also u can try decision trees. Sorry, but i'm also new in DS and can be wrong:)


2

You need to groupby to deal with multiple vote counts: df.groupby('timestamp').sum().plot(x='timestamp', y='vote_count')


2

Definition A inner product (AKA dot product and scalar product) can be define on two vectors $\mathbf{x}$ and $\mathbf{y}$ $\in \mathcal{R^n} $ as $$ \mathbf{x.x^T} = <\mathbf{x},\mathbf{y}>_\mathcal{R^n}=<\mathbf{y},\mathbf{x}>_\mathcal{R^n} = \sum_{i=1}^{n} x_i \times y_i $$ The inner product can be seem as the length of the projection of a ...


2

pos_label is an argument of scikit-learn's precision_score (docs); its purpose is, well, to indicate which label is the positive one and, if not given explicitly (like in your case here), it assumes the default value of 1 (again, check the docs). Since it seems that the positive label in your case is 'Y', replace the last line with: print("Precision:",...


2

Common use cases include: Fraud detection Transactions volume prediction Next transaction date Fraud detection This is usually tackled with anomaly detection. It requires information on the two transaction parties and using machine learning to figure out when a transaction is out of the norm and flagging as a potential case of fraud. Transactions volume ...


2

Either do unsupervised learning with something like k-means clustering or DBScan where you attempt to segment students into groups and see if you can discern any insights based on the cluster generated or pick a threshold for certain categories, create a class column and label each student, and do a classification model.


2

There is high variance within each group. Even though there is a mean difference between the groups, there is a high amount of spread within just treatment A or just treatment B. From a statistical point of view, the difference between the groups could be due to chance because of the large spread relative to the small mean difference. Due to the amount of ...


2

You can print all your rows by iterating over them and printing. import pandas as pd df = pd.read_csv("BusinessData.csv") print(df.columns) for index, row in df.iterrows(): print(index, row.tolist())


2

You can edit the maximum number of rows displayed by PANDAS with the 'display.max_rows' option. If you want it to show all your rows, you can do: import pandas as pd df = pd.read_csv("BusinessData.csv") pd.set_option('display.max_rows', df.shape[0]) print(df)


Only top voted, non community-wiki answers of a minimum length are eligible