Questions tagged [data-analysis]
Data analysis is a process of inspecting, cleansing, transforming, and modeling data with the goal of discovering useful information, informing conclusions, and supporting decision-making.
288
questions
0
votes
1
answer
13
views
Using boxplots of previous day’s percentage change in S&P index to predict today's return based on yesterday's movement
In the book "An Introduction to Statistical Learning with Applications in Python, Trevor Hastie et al., Springer", there's the following paragraph:
The left-hand panel of Figure 1.2 ...
1
vote
1
answer
42
views
Affine 2D mapping in python
I have two sets of 2D data $A$ and $B$ (representing 2D positions on a 2D plane $x,y$) which are related (the first pair of $x,y$ of $A$ is related to the first pair of $x,y$ of $B$ for instance). I ...
0
votes
0
answers
24
views
Interpreting Perplexity, U_mass coherence and Cv score trends for a Latent Dirichlet Allocation Model
I'm running an LDA model through gensim. To my understanding, closer the u_mass coherence score is to zero, higher is the interpretability of the topics that come up. I'm getting the u_mass coherence ...
-2
votes
1
answer
38
views
How to decide your first self-project for Data Science/Analysis [closed]
Suppose a complete beginner(in the field of data analysis/data science), which is my case indeed, decided to make a project for his/her resume.
These are the following skills he have learnt:
Python(...
0
votes
0
answers
169
views
AI tools/LLM models to analyze excel data
I have a bunch of excel files containing information of employee attendance. I want to use NLP-based search to ask questions like: Which employee has taken most leaves?; What dates witness a high ...
0
votes
0
answers
42
views
Heavy right skewed target
I scrapped Data science job postings of almost 1000 jobs for salary prediction. I wanted to bin the salary column and build a classification model from the job descriptions. But the (salary)target is ...
0
votes
0
answers
8
views
Exploring Data Science for Enhanced Reporting
I'm a self thought data analyst who has been primarily focused on generating standard reports. However, this year, my role is evolving. My boss has challenged me to not just report data but to extract ...
0
votes
1
answer
17
views
Best way to analyse change in amplitude of cyclic data?
I have some cyclic data that I'm trying to analyse which I've posted below:
I would like to analyse some overall trends of the data. The peak-to-peak amplitude appears to be decreasing, however, I'm ...
3
votes
1
answer
139
views
Pandas Profiling Not Working
I have been working on EDA lately and i got to know about Pandas Profiling ,
but i am unable to import the module Pandas_Profiling , I could import the module on Google Collab but couldnt import the ...
0
votes
1
answer
24
views
Working with functions [closed]
What are some good sources for learning how to deal with functions in Python and R. I have used them a lot of times and they can be a bit challenging. I like to write out a plan of what I want done ...
0
votes
1
answer
21
views
How should I plot 2D data as 3D plot?
I got data (1503, 38697) (cell(datapoint), genes(features)), each elements has integer values (gene expression).
so something like
...
0
votes
0
answers
19
views
To Become a Data Analyst?
currently I am in year4 and also working as a product owner who would want to shift the occupation path to data analyst in banking, but the problem is I don't know where to start. May I have some ...
0
votes
0
answers
24
views
How to infer the following graphs for dimensionality reduction?
I'm dealing with a high-dimensional(1600 features, 9500 columns) binary classification problem. My current accuracy, and other metrics are not upto the mark. I am trying different feature selection ...
0
votes
1
answer
362
views
ValueError: X has 54 features, but DecisionTreeClassifier is expecting 53 features as input
I am analysing and prediction 2023 Cricket World Cup based on previous given dataset.
This is Exploratary analysis:
Feature selection and Training model:
Applying Random forest classifier algorithm:
...
0
votes
0
answers
34
views
Calculate the curve of a list and apply that to calculate the values of a larger list
I have a list of ~125 sorted ascending values and I am converting those to percentages of total so that they add up to 100% (shortened example would be: 0%, 0%, 15%, 15%, 30%, 40% with the total of ...
0
votes
1
answer
28
views
Trying to write a code that detects a maximum value before a drop off in slope
I'm trying to write code for a force test that will output the maximum force before structural failure occurs. I'm a bit of a novice to python, so the issue here might be something simple that I'm ...
0
votes
0
answers
25
views
Growth (sales) forecast based on season
How to make a forecast based on last year (season)?
I have no experience in the field
Make a forecast of the sales in september 2023 based on last years sales
Example A
Sales august 2022 (last year): ...
1
vote
0
answers
86
views
Is there an elegant way to quantify the mix shift effect?
Imagine there is an online auction business selling 1000+ varieties of fruits. The demand for these fruits change over time, and the prices of these fruits is set by the seller.
Supposed I am ...
3
votes
1
answer
146
views
Solve tough clustering problem with overlapping clusters
I'm having some trouble to solve a hard clustering problem.
I have a 2D dataset characterized by non spherical and partially overlaping clusters with different densities.
I've read a lot about ...
0
votes
1
answer
15
views
Bibliographic references on statistical analysis of classification results in machine learning
I am looking for a good must-have reference about statistical analysis of classification results in machine learning.
What books do you recommend about it?
0
votes
0
answers
12
views
Displaying PowerBI and Microsoft Excel Files for Data Analytics Portfolio
I tried uploading a PowerBI file and an Excel file to GitHub, but it doesn't render them as GitHub would a Jupyter Notebook, for example.
What service can I upload the aforementioned files to for a ...
0
votes
1
answer
194
views
Is it essential to remove NULL values when doing EDA?
I'm doing a Data Analytics project on Youtube Statistics.
Assume that I want to get into the entertainment industry and decided to make a youtube channel. Before that, I need to identify what content ...
0
votes
0
answers
22
views
Find the time and magnitude of abrupt change in time series
I have several time series where there sometimes are abrupt continuous shifts in the data (see example below).
Specifically these are measurements so the reference level (RL) must have changed. E.g., ...
0
votes
1
answer
38
views
Advice on how to approach a recommendation model for marketing
I am working on a project to provide recommendations to the marketing team to launch effective campaigns. The dataset I have has data on existing customers, their demographic and billing details as ...
0
votes
0
answers
73
views
Implicit feedback based recommendation - User - User recommendation
My dataset involves customer_id, item_id and Count of purchases. My datset size is small...we have 15 users and 317 items..
Currently, am trying to build a recommendation system based on user based ...
0
votes
0
answers
21
views
Statistical approach for comparing ranks
I'm looking for a statistical approach to compare ranks produced by 2 versions of an ranking algorithm (A & B) against the actual ranks in the system. This is about ranking hospitals and the ...
0
votes
1
answer
50
views
Analysing 1GB of 200K of data rows having 50 columns
Total Newbie question here :
5 years ago, I wrote a web application in LAMP stack not using a framework, targeting end users, which now has over 200,000 rows where each row has 50+ columns and the ...
1
vote
0
answers
22
views
Most popular location
This feels really simple, but looking for some guidance to find the most popular location for orders.
I have three locations: Location A, Location B, and Location C.
Location A had 43 orders out of a ...
0
votes
0
answers
28
views
I'm looking for a dataset on enviroment monitoring
I'm looking for a dataset related to environmental monitoring, made up of values obtained from various types of sensors (such as temperature, pressure, CO2...etc) for the purpose of a classification ...
0
votes
2
answers
40
views
Data Analysis Process
How do you feed data from a data warehouse to Python for ad-hoc analysis?
My day-to-day work is to answer ad-hoc questions, and 95% of the data I need is in our data warehouse. I often query data from ...
1
vote
2
answers
65
views
Looking for a couple of ideas please
I’ve got some data by postal zone that includes:
Postal zone code
Average rent value per square foot
Brand affinity 1
Brand affinity 2
Brand affinity 3
Brand affinity 4
…and so on
The brand affinity ...
1
vote
2
answers
353
views
what qualifies as a data leakage?
I am currently working on a binary classification problem using imbalanced data. The algorithm that I am using is random forest. The problem is about predicting whether each sales project will meet ...
0
votes
1
answer
317
views
What's the best approach to dealing with missing data in a dataset?
I have a dataset that contains missing values in some columns. I would like to know what is the best approach to deal with this missing data. Should I remove rows with missing data or fill in missing ...
0
votes
0
answers
396
views
TypeError: Could not convert group to numeric
I am going through Krish Naik's "End To End ML Project With Deployment" course on YouTube. Right now, I am in Tutorial 3. Here is the link to the video.
I am facing an issue with BIVARIATE ...
0
votes
3
answers
155
views
Difference between Data analytics and Data Analysis
What we are referring to when we say Data analytics and Data analysis?
0
votes
1
answer
43
views
Study data science at university [closed]
Is it a waste to study data science in university, despite the advancement of artificial intelligence and automation of analysis?
regards
0
votes
0
answers
16
views
Having trouble manipulating data to make better dashboard
I have a test dataset which has has 39 columns. Index is given below:
Sample data is as follows:
Some columns are omitted due to privacy concerns. The colored cells in the index belong to one type. ...
0
votes
0
answers
28
views
What is the steps to generate data from a plot and use them to have a predication using python?
I have a plot that represent a BH curve for magnatic material. The material have a behavior for each H value for two different temperature 25 C and 100 C. Figure 1. I need to extract the data for each ...
0
votes
1
answer
380
views
Regression with time series data
I want to predict temperature when time (datetime type, hourly data for five months) and humidity is given. Before starting in python, I created a regression model in excel. But instead of predicting ...
0
votes
1
answer
52
views
Customer Analysis - How to handle unbalanced data?
I'm taking a data analysis course and decided to work on a customer analysis project. In the data, I have three countries:
USA (539 unique users)
BRA(385 unique users)
TUR (129 unique users)
I'm ...
0
votes
1
answer
38
views
Feature engineering for interest-based age classification
I have a dataset which has users (rows) with the list of their interests (IABs), which looks like this
...
1
vote
1
answer
97
views
Is Data Analysis/Science good for online freelancing (Upwork, Freelancer...)? [closed]
My interest in data analysis/science began about a year ago because it combines statistics, analysis, and programming. Unfortunately, I was surprised that there was almost no demand for it in the job ...
0
votes
1
answer
13
views
What's the word for aggregating by two attributes together, rather than separately?
Let's imagine we have this underlying data:
...
0
votes
1
answer
17
views
Dropping duplicated when visualising different granularity in the same df
I have a dataset about a bunch of users and their comment on Instagram, each row is a comment. Some of the columns of this table are related to the users and some of them have multiple comments.
I'm ...
0
votes
1
answer
32
views
Understanding Box plots
I am new to box plots and I am trying to understand it. Consider the following figures :
According to my understanding, outliers will exist after the max. line and before the min. line. So according ...
0
votes
1
answer
22
views
How do I handle "Greater than X" in a field of integers?
I've been tasked with cleaning a dataset with a "Drive Time" column that lists times taken to drive to a specific location in whole minutes. The values range from 3 to 180 minutes but there ...
0
votes
1
answer
19
views
Total Retention Rate Calculated from Categories
I am calculating retention for 3 categories and then total, and I am trying to double check my total, but my check formula isn't working.
I am comparing the last 14 days (let's call it Period 1) to ...
1
vote
1
answer
94
views
Quantitatively evaluate similarity between two corpus of texts
I want to assess how similar, or different two corpora are, and if the similarity is statistically significant. Something close to a Kolmogorov–Smirnov test in statistics, but for text data.
For ...
1
vote
1
answer
16
views
Complex graphical representation based on monthly groups
I'm still fairly new to R, and am having some real issues turning a data frame into a graphical representation.
I downloaded some data for house prices in the UK and stripped it down to just the ...
0
votes
1
answer
22
views
How do I get a big-picture feel for my database?
Background: I was just granted access to my company's SQL Server (MS SQL 2019 via Azure Data Studio). As I have not yet been assigned any specific projects I want start digging.
I've asked my DBA to ...