Questions tagged [data-analysis]
Data analysis is a process of inspecting, cleansing, transforming, and modeling data with the goal of discovering useful information, informing conclusions, and supporting decision-making.
281 questions
26
votes
7
answers
7k
views
Is Python a viable language to do statistical analysis in?
I originally came from R, but Python seems to be the more common language these days. Ideally, I would do all my coding in Python as the syntax is easier and I've had more real life experience using ...
- 498
10
votes
4
answers
2k
views
Math PhD (Nonlinear Programming) switching to Data Science?
I am a math Ph.D. student who is interested in going to the industry as a Data Scientist after graduation. I will briefly give some background on my education before posing my question, so that it is ...
- 123
8
votes
2
answers
9k
views
Covariance as inner product
Why is covariance considered as inner product if there is no projection of one vector onto another?
Right now I perceive this as just a multiplication of $x$ segment of vector($x_i - \bar{x}$) and $y$...
- 143
8
votes
1
answer
477
views
How far can one go with excel? [closed]
in my business we handle all analytics through Excel. This includes mostly scheduling, production planning and accounting operations. We currently are looking into adding a bit of predictive modelling ...
- 340
6
votes
0
answers
95
views
Fitting model to differenced time series
I have a time series on daily stock price of company(2013 data points).I took a first order difference and the following acf and pacf plots of the differenced series were obtained. However, I am ...
- 231
5
votes
5
answers
56k
views
Additive vs Multiplicative model in Time Series Data
The above time series plot is a daily closing stock index of a company. I want to know which model between additive and multiplicative best suits the above data. I know what the two models are, but i ...
- 231
5
votes
2
answers
144
views
What advantages does Data Visualization have in EDA?
It is not clear to me what advantage the EDA data visualization provides. By advantage I mean what decision I will make according to one or the other visualization.
Could someone give me an example ...
5
votes
2
answers
33k
views
Plot Two Categorical Variables
How to plot two categorical variables in Python or using any library? I want to plot the Playing Role of a Cricketer (Batsman, Bowler, etc.) VS Bought_By (Franchise Names, e.g., CSK, DC, etc.). The ...
5
votes
1
answer
114
views
experimental design in R project
I want to know of any repositories that contain complete experimental design in R covering basic test and analyses?
I want to take a top-buttom approach to learn step by step through a real project ...
- 171
5
votes
4
answers
6k
views
What is the difference between data analysis and machine learning?
It seems to me that machine learning (especially deep learning) can work with thousands (even millions) of different inputs. After training an ML model on the inputs, the computer has hopefully "...
- 51
5
votes
3
answers
150
views
How can data science teams inside businesses measure costs and efficiency of their technical work?
How can data science teams measure and improve costs of their technical work, when they often don't know the monetary value of the datasets and insights they are producing? Are they using industry ...
- 51
5
votes
3
answers
411
views
ETL and Data Engineering - is it purely the knowledge of tools or is there theory behind it?
I would like to better understand what a good Data Englineer must know or what he does. Job descriptions mostly list tools that are required, such as Python.
If it is possible to separate Data ...
- 51
4
votes
1
answer
304
views
How to interpret standard deviation calculated by excel
I have some percentage values in my excel spreadsheet like:
When i use the STDEV function to calculate the standard deviation with these values, i get ...
- 395
4
votes
1
answer
516
views
Solve tough clustering problem with overlapping clusters
I'm having some trouble to solve a hard clustering problem.
I have a 2D dataset characterized by non spherical and partially overlaping clusters with different densities.
I've read a lot about ...
4
votes
2
answers
206
views
find most dense neighborhood of points in high dimensional space
I'm working on a project where I have many high-dimensional points and I want to find the most dense neighborhood of them. Ideally, out of my ~500 points that are each a 4x300 matrix (300 ms time ...
- 41
4
votes
1
answer
6k
views
Monthly trend with fb prophet-Interpreting the graph
I have monthly data with month/year in one column and price on another. I would like to get a yearly trend with fb prophet library in python (how to use monthly data with the library is explained at ...
- 167
4
votes
1
answer
63
views
Spatially constrained geospatial similarity
What's the current methodology for clustering geospatial data by features?
Example: I have some demographic dataset. Let's say this contains average home price and population density.
So, an example ...
3
votes
1
answer
1k
views
Pandas Profiling Not Working
I have been working on EDA lately and i got to know about Pandas Profiling ,
but i am unable to import the module Pandas_Profiling , I could import the module on Google Collab but couldnt import the ...
3
votes
2
answers
441
views
why One-Hot Encoder can avoid the situation that the model will misunderstand the data to be in some kind of order if the data has been Label Encoding
We know that we prefer to using One-Hot Encoding not Label Encoding when processing with non-ordinal data.
And I real a blog which give the difference between Label Encoding and One-Hot Encoding.
So ...
- 287
3
votes
1
answer
333
views
How to get the survival duration prediction for each individual in the data by using the Kaplan-Meier method?
I am trying to learn how to use the Kaplan-Meier survival estimator model in the lifelines package. The documentation says that the KaplanMeierFitter.fit function ...
- 308
3
votes
2
answers
1k
views
Storing Large dataset for processing and analysis of data
I am new to data engineering and wanted to know , what is the best way to store more than 3000 GB of data for further processing and analysis ? I am specifically looking for open source resources . I ...
- 41
3
votes
1
answer
60
views
Can neural networks be considered a method for data analysis?
I am in the process of writing my thesis.
In this thesis I am stating that neural networks can be considered a method for data analysis.
After a bit of though I am now unsure if this claim is ...
- 605
3
votes
1
answer
126
views
Detecting specific points in (noisy) dataset
In my recent work I've came across a problem where I need to find certain points in quickly oscillating data. Let's work with syntethic data instead of real measurements and ignore the noise for the ...
- 131
3
votes
1
answer
190
views
How to figure out if the problem is time series Forecasting or not?
Recently I have encountered a time-series based, where I have a dataset which contains data for call-centers performance. The dataset contains information about the number of calls made by customers ...
3
votes
2
answers
389
views
Possible Challenges for a Data Science Escape Room
Dear Data Science Community,
as my project for my bachelorthesis I am working on a concept for a physical Data Science Escape Room. The goal of the escape room is to create awareness for data science ...
3
votes
1
answer
26
views
Not sure how to visual customer retention vs product line count
I created a customer retention grid that tracks customer counts by territory by year. The sales managers have loved it. The numbers were terrible, but now they know what they didn't know and can ...
- 31
3
votes
2
answers
78
views
is it a good idea to take the derivative or integral of some features and add them as new features in machine learning?
I'm learning how to do feature Engineering and come across some ideas in my head that's why I want to ask if I had some dataset with some features let's say 2 features and I have a timestamp column ...
- 243
3
votes
1
answer
1k
views
Comparing rows in Data Studio
I am using Data Studio for a project and I am connecting to my BigQuery table.
My table contains following columns:
Date
Store_name
Footfall
I'd love to compare the footfall of two stores using Data ...
- 171
3
votes
1
answer
266
views
Pandas: Calculating the best seller to incorporate number of houses sold
I have been analysing seller data and trying to get insights. I have written a groupby statement to get the average price of selling for every seller
...
- 31
2
votes
2
answers
788
views
what qualifies as a data leakage?
I am currently working on a binary classification problem using imbalanced data. The algorithm that I am using is random forest. The problem is about predicting whether each sales project will meet ...
- 2,655
2
votes
1
answer
1k
views
Is it Possible to plot Scatter Plot + Histogram + Correlation Values in a single plot (in python)?
I recently came across corrmorant package in R.
It allows to plot all three basic EDA plots together: Scatter Plot + Histogram + Correlation Values.
Is it possible ...
- 123
2
votes
1
answer
162
views
Problem regarding calculating correlation approach?
There are 3 columns say product_id, product_type, and price_drop. ...
- 23
2
votes
1
answer
45
views
How to solve (or even effectively think about) a complex real-life multivariate problem
In my work, we get estimates.
An estimate may include up to 12 different categories of costs (Development, Legal Travel, etc.) to produce any number of assets/deliverables from dozens of different ...
- 23
2
votes
1
answer
34
views
Metrics/Methods for deciding duration of video retention for on-demand websites
This might be a general question but I thought this might be the best place to brainstorm. If I had a video website that only wants to retain videos on-demand for a certain number of days before ...
- 131
2
votes
1
answer
187
views
What is the difference between 'if the data is of good quality' and 'if the data is tidy'?
I'm doing Data Analyst nanodegree from Udacity. I'm confused between the difference even after going through the lecture a few times.
- 121
2
votes
1
answer
7k
views
2
votes
2
answers
1k
views
Removing features that correlate with the target label
I know that it is better to remove correlated features, but what if the feature correlates with the target label? So, it gives the most information for prediction? Do I need to remove features that ...
- 131
2
votes
1
answer
121
views
Log scaling whole dataset
I am log scaling my whole dataset in log base 10. When I do this I get -infitity for the minimum value.
I am wondering how I can get rid of this -infinity?
I have been advised to add a small value ...
- 21
2
votes
1
answer
47
views
How to account for reduced student capacity when calculating program retention?
I work at a nonprofit youth center that has 2 distinctive programs throughout the year. Our afterschool program runs during the school year, and our summer camp program runs during the summer.
I am ...
2
votes
1
answer
190
views
Recommendations for Data Visualization solution for large CSV file on a non-profit website
I would like your advice and recommendations for a non-profit that is trying to turn a large CSV file into meaningful information.
I am part of a non-profit that would like to turn a 23,000 row x 23 ...
2
votes
1
answer
66
views
Analysis of sales performance
This is my first question on this stackexchange - i hope i'm in the right place and i'm asking the right question.
I work for a business, we sell certain products.
We are running a trial currently on ...
- 121
2
votes
1
answer
447
views
EDA for analysis of nominal variable with high cardinality
I have a nominal variable (car model) with very high cardinality (~8500 labels) and I would like to analyse its relation with a binary target variable. While I can create logical groups and compare ...
- 321
2
votes
2
answers
916
views
Analysis of Time Series data
The below graph is a scatterplot of daily stock price. My aim is to predict future stock price of the company.
From the scatterplot it seems that it is a multiplicative model, so I tried to "...
- 231
2
votes
0
answers
23
views
Can you apply technical analysis indicators to general data?
I collect data, for example, "how many people mentioned 'Bitcoin' on an investing community", I take the data for daily, so it's like "5/1,231;5/2,156;5/3,223" and so on. So ...
- 121
2
votes
0
answers
39
views
How to aggregate Weather Data from county to state level? [closed]
I am doing a prediction of forest fire risk using the weather data and the fire incident data. I have data related to different county and the weather data on the day of the incident. As the incident ...
2
votes
0
answers
143
views
Plotting regression test errors
I am trying out an R exercise where I have to randomly split my data into train/test set 50 times. On each iteration, I use lm() to train polynomial models up to ...
- 21
2
votes
1
answer
68
views
Predicting if a search keyword is lower volume based only on high volume keywords
My friend was asked this question in an interview for analytics and I cannot figure out the answer so I would like to see how could this data science problem be solved. Here's the problem:
Let's say ...
2
votes
0
answers
32
views
Dimension of the manifold on which my data sits
Suppose that I have data points, in the form of vectors with binary entries. We create a metric space, or Vietoris-Rips complex, using the Hamming distance between the data points.
I would like to ...
- 31
2
votes
2
answers
154
views
Predicting missing data. Looking for good data predicting technique
I am analysing data for Countries Trade GDP. Some of the countries have missing GDP value for given a year. However, I have Grand Total for the entire region for that year. Is there a good data ...
1
vote
3
answers
1k
views
How to handle a large number of categories in one column effectively in machine learning?
In my dataset one column has a large no of categories. For model training or building we need to convert these categories to numeric. There are several methods we can use to convert labels to numeric, ...
