# Tag Info

## Hot answers tagged data-analysis

31

Python is more "general purpose" while R has a clear(er) focus on statistics. However, most (if not all) things you can do in R can be done in Python as well. The difference is that you need to use additional packages in Python for some things you can do in base R. Examples: Data frames are base R while you need to use Pandas in Python. Linear ...

13

Python being more widely used is an important consideration. This will especially become important when applying for a job. Also Python has as many if not more key statistical and ML/AI tools as R, and a larger open-source base to utilize. Python is designed for programmers, R is designed for statisticians. Originally I was a R programmer, but most of my ...

10

Data science jobs cover a wide range of different activities so any answer is likely to be subjective. I'm in academia so my knowledge of the job market is limited, but from what I can see: The current context is very favorable to data scientists looking for a job, so anybody with some basic knowledge of ML has a chance. You're already above this level so ...

5

I want to know which model between additive and multiplicative best suits the above data. It is hard to tell just by looking at it. A multiplicative decomposition roughly corresponds to an additive decomposition of the logarithms. The additive decomposition is the most appropriate if the magnitude of the seasonal fluctuations, or the variation around the ...

4

I don't disagree with the other answers, but here's a different perspective you should bear in mind. Also, I can offer answers to your specific questions as someone who left academia (applied math/CS) for data science. In short, understanding the underlying needs and use cases for a business problem is paramount to any project, and so developing a strong ...

4

I'd like to add two points to the existing answers: There is excellent interaction between R and python, with various possibilities for either direction. To me, it's not that much of a decision python vs. R. The decision is to choose the main language appropriately for the project at hand, and then do parts in the other language if that is better for some ...

4

One thing that can be a gotcha coming from R to Python is that the Python stats ecosystem tends to be more machine learning-ey oriented rather than inferential stats-ey oriented. This can create some hiccups, because some of the defaults in R that are the defaults because people who do inferential stats like in the social sciences always use them, are not ...

3

Calculate one day returns. Plot histogram of daily returns. Calculate $log(\frac{price_{i+1}}{price_i})$. Plot histogram of above logarithm. If second plot is more likely to be normally distributed then choose multiplicative model. Else, choose additive model. You can also perform statistical test for normal distribution and check, which one has higher p-...

3

Erwan nailed it (+1). But I think my addition is a little too long for a comment. You seem to be well ahead of where I was when I landed my DS job. I was in pure math, a couple of postdocs in, and had only a short time of self-study when I was applying for industry data science. On the other hand, I had actuarial exams in my undergrad, which probably ...

3

Just Don't take this advice blindly: The subjects you have mentioned in mathematics are core to solving problems using machine learning/Deep learning, programming is a tool to implement all this theory that you learn and on that basis you create your hypotheses and then test by implementing it in code for that you do not need coding skills of a coder you ...

3

I eventually do plan on moving more towards ML One aspect that I would like to add based on what I observed. Things are moving with more focus towards Deep Learning e.g. Neural Networks and in this space, most of the dominating Libraries supports Python as first choice. Companies manage a separate Python version to open-source, just to maintain the user ...

2

Although not very helpful, the answer is probably "it depends". I like to do data cleaning and some EDA together since EDA can highlight appropriate treatments to clean the data - e.g. influencing how to handle missing values. I think data transformation should be done just prior to modelling; whether or not you need to do any transformation at all depends ...

2

I think you're quite confused. Hadoop is a collection of software that contains a a distributed file system called HDFS. Essentially HDFS is a way to store data cross a cluster. You can access file stores as you would in a local file store (with some modification) and modify things via Java API. Furthermore, ON TOP OF the file system there exist a ...

2

There is a very cool active Python package called pandas-profiling, is exactly what you want. With a simple pandas_profiling.ProfileReport(df) it returns a lot of important statistical information about your data, the official documentation says: For each column the following statistics - if relevant for the column type - are presented in an interactive ...

2

It seems like a challenging problem. If it were my task, I would start with a probabilistic approach like apriori, but you may want to check out Naive Bayes based approach. There are some differences in these approaches, but, either one may produce decent results. More generally, I think the analysis you want to perform is $Association\ Rule\ Learning$. ...

2

really hard answer for u question cause there is to little information. Try to make EDA and attach it to question. Cause EDA define the model Anywhere, for low corellated data, try to use k-NN. If u use Python: scikit-learn have implementation. Also u can try decision trees. Sorry, but i'm also new in DS and can be wrong:)

2

TL;DR If you have unlimited time and use a 64-bit version of Excel, you can get as far with Excel as any other data analysis tool. Time I mention time as my first factor, because Excel only has basic funcitonality built in, such as summing, random number generation, lookups etc. These correspond to a kind of standard library, which Python and R also have. ...

2

Definition A inner product (AKA dot product and scalar product) can be define on two vectors $\mathbf{x}$ and $\mathbf{y}$ $\in \mathcal{R^n}$ as $$\mathbf{x.x^T} = <\mathbf{x},\mathbf{y}>_\mathcal{R^n}=<\mathbf{y},\mathbf{x}>_\mathcal{R^n} = \sum_{i=1}^{n} x_i \times y_i$$ The inner product can be seem as the length of the projection of a ...

2

Common use cases include: Fraud detection Transactions volume prediction Next transaction date Fraud detection This is usually tackled with anomaly detection. It requires information on the two transaction parties and using machine learning to figure out when a transaction is out of the norm and flagging as a potential case of fraud. Transactions volume ...

2

Either do unsupervised learning with something like k-means clustering or DBScan where you attempt to segment students into groups and see if you can discern any insights based on the cluster generated or pick a threshold for certain categories, create a class column and label each student, and do a classification model.

2

There is high variance within each group. Even though there is a mean difference between the groups, there is a high amount of spread within just treatment A or just treatment B. From a statistical point of view, the difference between the groups could be due to chance because of the large spread relative to the small mean difference. Due to the amount of ...

2

pandas has a max rows setting - https://pandas.pydata.org/pandas-docs/stable/user_guide/options.html Though perhaps looking at a 5,000+ row csv in an editor, or a spreadsheet or some IDEs have a csv editor would be more useful.

2

You can edit the maximum number of rows displayed by PANDAS with the 'display.max_rows' option. If you want it to show all your rows, you can do: import pandas as pd df = pd.read_csv("BusinessData.csv") pd.set_option('display.max_rows', df.shape[0]) print(df)

2

I would suggest using a genetic algorithm of some kind. The idea is to assign hypothetical costs to each item, then check how well the hypothesis matches the data you have. An individual represents an "hypothesis", i.e. assignment of costs: starting from random hypotheses, the genetic algorithm might be able to converge to a solution. I gave the following ...

2

Taking the log doesn't result in a normally-distributed target; it would tend to if the target was log-normally distributed, and you have something normalish there, not quite. But, this distribution isn't actually what matters. What taking the log does is change your model of how errors arise when fitting a regressor. You're now saying that the target ...

2

First of all I just want to say that I am not a data engineer and there is definitely someone out there that can answer this better than me. I do think that there is a lot of theory behind data engineering. It is also very interesting. I too thought that it was boring and I was more interested in just data science/ machine learning. I am not sure if I can ...

1

df['date'] = pd.to_datetime(df['created_at']).dt.date to_datetime: Convert argument to datetime. For example, if your column of "created_at" is a string column, it converts it to a datetime column dt: Access object for datetime like properties of the Series values.

1

I am also new to time series forecasting. I used simple lineplot to visualize the time series data. Some of the models used for time series are : ARIMA SARIMA prophet LSTM (deep neural networks) LBATS You can refer to this link to know what things you should consider while building a time series model

1

You can calculate mean target for each categorical variable and compare its values. In pandas this can be done easily: df.groupby('categorical_feature').target.mean() Then you can make a histogram to compare the approach. I also, seaborn has a catplot, where it do the same as above in a bar plot format, showing mean value for target variable based on each ...

1

Clustering can be an ideal choice here. From the question, seems the data will most probably be in continuous format. Essentially, clustering is a method of finding groups of similar objects. The similarity between the objects is determined by the type of distance measure being used. With this background, you can start by finding the videos that are most ...

Only top voted, non community-wiki answers of a minimum length are eligible