Questions tagged [data-analysis]

Data analysis is a process of inspecting, cleansing, transforming, and modeling data with the goal of discovering useful information, informing conclusions, and supporting decision-making.

Filter by
Sorted by
Tagged with
0 votes
1 answer
13 views

Using boxplots of previous day’s percentage change in S&P index to predict today's return based on yesterday's movement

In the book "An Introduction to Statistical Learning with Applications in Python, Trevor Hastie et al., Springer", there's the following paragraph: The left-hand panel of Figure 1.2 ...
Tran Khanh's user avatar
1 vote
1 answer
42 views

Affine 2D mapping in python

I have two sets of 2D data $A$ and $B$ (representing 2D positions on a 2D plane $x,y$) which are related (the first pair of $x,y$ of $A$ is related to the first pair of $x,y$ of $B$ for instance). I ...
patjol's user avatar
  • 11
0 votes
0 answers
24 views

Interpreting Perplexity, U_mass coherence and Cv score trends for a Latent Dirichlet Allocation Model

I'm running an LDA model through gensim. To my understanding, closer the u_mass coherence score is to zero, higher is the interpretability of the topics that come up. I'm getting the u_mass coherence ...
Sarthak's user avatar
-2 votes
1 answer
38 views

How to decide your first self-project for Data Science/Analysis [closed]

Suppose a complete beginner(in the field of data analysis/data science), which is my case indeed, decided to make a project for his/her resume. These are the following skills he have learnt: Python(...
Swayam Jha's user avatar
0 votes
0 answers
169 views

AI tools/LLM models to analyze excel data

I have a bunch of excel files containing information of employee attendance. I want to use NLP-based search to ask questions like: Which employee has taken most leaves?; What dates witness a high ...
Apoorva's user avatar
  • 277
0 votes
0 answers
42 views

Heavy right skewed target

I scrapped Data science job postings of almost 1000 jobs for salary prediction. I wanted to bin the salary column and build a classification model from the job descriptions. But the (salary)target is ...
Sendhan's user avatar
  • 21
0 votes
0 answers
8 views

Exploring Data Science for Enhanced Reporting

I'm a self thought data analyst who has been primarily focused on generating standard reports. However, this year, my role is evolving. My boss has challenged me to not just report data but to extract ...
Sky's user avatar
  • 1
0 votes
1 answer
17 views

Best way to analyse change in amplitude of cyclic data?

I have some cyclic data that I'm trying to analyse which I've posted below: I would like to analyse some overall trends of the data. The peak-to-peak amplitude appears to be decreasing, however, I'm ...
Harrison's user avatar
3 votes
1 answer
139 views

Pandas Profiling Not Working

I have been working on EDA lately and i got to know about Pandas Profiling , but i am unable to import the module Pandas_Profiling , I could import the module on Google Collab but couldnt import the ...
Sahil Rathout's user avatar
0 votes
1 answer
24 views

Working with functions [closed]

What are some good sources for learning how to deal with functions in Python and R. I have used them a lot of times and they can be a bit challenging. I like to write out a plan of what I want done ...
Salem Abdulkerim's user avatar
0 votes
1 answer
21 views

How should I plot 2D data as 3D plot?

I got data (1503, 38697) (cell(datapoint), genes(features)), each elements has integer values (gene expression). so something like ...
containletters's user avatar
0 votes
0 answers
19 views

To Become a Data Analyst?

currently I am in year4 and also working as a product owner who would want to shift the occupation path to data analyst in banking, but the problem is I don't know where to start. May I have some ...
Hunter's user avatar
  • 1
0 votes
0 answers
24 views

How to infer the following graphs for dimensionality reduction?

I'm dealing with a high-dimensional(1600 features, 9500 columns) binary classification problem. My current accuracy, and other metrics are not upto the mark. I am trying different feature selection ...
Tanmay Sharma's user avatar
0 votes
1 answer
362 views

ValueError: X has 54 features, but DecisionTreeClassifier is expecting 53 features as input

I am analysing and prediction 2023 Cricket World Cup based on previous given dataset. This is Exploratary analysis: Feature selection and Training model: Applying Random forest classifier algorithm: ...
Display Name is missing's user avatar
0 votes
0 answers
34 views

Calculate the curve of a list and apply that to calculate the values of a larger list

I have a list of ~125 sorted ascending values and I am converting those to percentages of total so that they add up to 100% (shortened example would be: 0%, 0%, 15%, 15%, 30%, 40% with the total of ...
Disz's user avatar
  • 1
0 votes
1 answer
28 views

Trying to write a code that detects a maximum value before a drop off in slope

I'm trying to write code for a force test that will output the maximum force before structural failure occurs. I'm a bit of a novice to python, so the issue here might be something simple that I'm ...
Miriam Cubstead's user avatar
0 votes
0 answers
25 views

Growth (sales) forecast based on season

How to make a forecast based on last year (season)? I have no experience in the field Make a forecast of the sales in september 2023 based on last years sales Example A Sales august 2022 (last year): ...
clarkk's user avatar
  • 101
1 vote
0 answers
86 views

Is there an elegant way to quantify the mix shift effect?

Imagine there is an online auction business selling 1000+ varieties of fruits. The demand for these fruits change over time, and the prices of these fruits is set by the seller. Supposed I am ...
tanvach's user avatar
  • 111
3 votes
1 answer
146 views

Solve tough clustering problem with overlapping clusters

I'm having some trouble to solve a hard clustering problem. I have a 2D dataset characterized by non spherical and partially overlaping clusters with different densities. I've read a lot about ...
Lorenço Santos's user avatar
0 votes
1 answer
15 views

Bibliographic references on statistical analysis of classification results in machine learning

I am looking for a good must-have reference about statistical analysis of classification results in machine learning. What books do you recommend about it?
user154333's user avatar
0 votes
0 answers
12 views

Displaying PowerBI and Microsoft Excel Files for Data Analytics Portfolio

I tried uploading a PowerBI file and an Excel file to GitHub, but it doesn't render them as GitHub would a Jupyter Notebook, for example. What service can I upload the aforementioned files to for a ...
Faith Alone's user avatar
0 votes
1 answer
194 views

Is it essential to remove NULL values when doing EDA?

I'm doing a Data Analytics project on Youtube Statistics. Assume that I want to get into the entertainment industry and decided to make a youtube channel. Before that, I need to identify what content ...
Leo Tang's user avatar
0 votes
0 answers
22 views

Find the time and magnitude of abrupt change in time series

I have several time series where there sometimes are abrupt continuous shifts in the data (see example below). Specifically these are measurements so the reference level (RL) must have changed. E.g., ...
asher's user avatar
  • 1
0 votes
1 answer
38 views

Advice on how to approach a recommendation model for marketing

I am working on a project to provide recommendations to the marketing team to launch effective campaigns. The dataset I have has data on existing customers, their demographic and billing details as ...
DS_1's user avatar
  • 1
0 votes
0 answers
73 views

Implicit feedback based recommendation - User - User recommendation

My dataset involves customer_id, item_id and Count of purchases. My datset size is small...we have 15 users and 317 items.. Currently, am trying to build a recommendation system based on user based ...
The Great's user avatar
  • 2,565
0 votes
0 answers
21 views

Statistical approach for comparing ranks

I'm looking for a statistical approach to compare ranks produced by 2 versions of an ranking algorithm (A & B) against the actual ranks in the system. This is about ranking hospitals and the ...
museshad's user avatar
  • 101
0 votes
1 answer
50 views

Analysing 1GB of 200K of data rows having 50 columns

Total Newbie question here : 5 years ago, I wrote a web application in LAMP stack not using a framework, targeting end users, which now has over 200,000 rows where each row has 50+ columns and the ...
anjanesh's user avatar
  • 103
1 vote
0 answers
22 views

Most popular location

This feels really simple, but looking for some guidance to find the most popular location for orders. I have three locations: Location A, Location B, and Location C. Location A had 43 orders out of a ...
David Ingledow's user avatar
0 votes
0 answers
28 views

I'm looking for a dataset on enviroment monitoring

I'm looking for a dataset related to environmental monitoring, made up of values obtained from various types of sensors (such as temperature, pressure, CO2...etc) for the purpose of a classification ...
PiEmmeC's user avatar
0 votes
2 answers
40 views

Data Analysis Process

How do you feed data from a data warehouse to Python for ad-hoc analysis? My day-to-day work is to answer ad-hoc questions, and 95% of the data I need is in our data warehouse. I often query data from ...
Bunny Eat Data's user avatar
1 vote
2 answers
65 views

Looking for a couple of ideas please

I’ve got some data by postal zone that includes: Postal zone code Average rent value per square foot Brand affinity 1 Brand affinity 2 Brand affinity 3 Brand affinity 4 …and so on The brand affinity ...
Jake's user avatar
  • 11
1 vote
2 answers
353 views

what qualifies as a data leakage?

I am currently working on a binary classification problem using imbalanced data. The algorithm that I am using is random forest. The problem is about predicting whether each sales project will meet ...
The Great's user avatar
  • 2,565
0 votes
1 answer
317 views

What's the best approach to dealing with missing data in a dataset?

I have a dataset that contains missing values in some columns. I would like to know what is the best approach to deal with this missing data. Should I remove rows with missing data or fill in missing ...
Horbeam's user avatar
0 votes
0 answers
396 views

TypeError: Could not convert group to numeric

I am going through Krish Naik's "End To End ML Project With Deployment" course on YouTube. Right now, I am in Tutorial 3. Here is the link to the video. I am facing an issue with BIVARIATE ...
Shawn's user avatar
  • 1
0 votes
3 answers
155 views

Difference between Data analytics and Data Analysis

What we are referring to when we say Data analytics and Data analysis?
Tauqir Ahmed's user avatar
0 votes
1 answer
43 views

Study data science at university [closed]

Is it a waste to study data science in university, despite the advancement of artificial intelligence and automation of analysis? regards
Amirhossein Qaem's user avatar
0 votes
0 answers
16 views

Having trouble manipulating data to make better dashboard

I have a test dataset which has has 39 columns. Index is given below: Sample data is as follows: Some columns are omitted due to privacy concerns. The colored cells in the index belong to one type. ...
Chinmay Datar's user avatar
0 votes
0 answers
28 views

What is the steps to generate data from a plot and use them to have a predication using python?

I have a plot that represent a BH curve for magnatic material. The material have a behavior for each H value for two different temperature 25 C and 100 C. Figure 1. I need to extract the data for each ...
Will's user avatar
  • 1
0 votes
1 answer
380 views

Regression with time series data

I want to predict temperature when time (datetime type, hourly data for five months) and humidity is given. Before starting in python, I created a regression model in excel. But instead of predicting ...
Scholar7's user avatar
0 votes
1 answer
52 views

Customer Analysis - How to handle unbalanced data?

I'm taking a data analysis course and decided to work on a customer analysis project. In the data, I have three countries: USA (539 unique users) BRA(385 unique users) TUR (129 unique users) I'm ...
noor h's user avatar
  • 103
0 votes
1 answer
38 views

Feature engineering for interest-based age classification

I have a dataset which has users (rows) with the list of their interests (IABs), which looks like this ...
theodre7's user avatar
1 vote
1 answer
97 views

Is Data Analysis/Science good for online freelancing (Upwork, Freelancer...)? [closed]

My interest in data analysis/science began about a year ago because it combines statistics, analysis, and programming. Unfortunately, I was surprised that there was almost no demand for it in the job ...
ahmad popli's user avatar
0 votes
1 answer
13 views

What's the word for aggregating by two attributes together, rather than separately?

Let's imagine we have this underlying data: ...
Richard's user avatar
  • 111
0 votes
1 answer
17 views

Dropping duplicated when visualising different granularity in the same df

I have a dataset about a bunch of users and their comment on Instagram, each row is a comment. Some of the columns of this table are related to the users and some of them have multiple comments. I'm ...
Marc's user avatar
  • 222
0 votes
1 answer
32 views

Understanding Box plots

I am new to box plots and I am trying to understand it. Consider the following figures : According to my understanding, outliers will exist after the max. line and before the min. line. So according ...
AAA's user avatar
  • 145
0 votes
1 answer
22 views

How do I handle "Greater than X" in a field of integers?

I've been tasked with cleaning a dataset with a "Drive Time" column that lists times taken to drive to a specific location in whole minutes. The values range from 3 to 180 minutes but there ...
Leon's user avatar
  • 1
0 votes
1 answer
19 views

Total Retention Rate Calculated from Categories

I am calculating retention for 3 categories and then total, and I am trying to double check my total, but my check formula isn't working. I am comparing the last 14 days (let's call it Period 1) to ...
user485656's user avatar
1 vote
1 answer
94 views

Quantitatively evaluate similarity between two corpus of texts

I want to assess how similar, or different two corpora are, and if the similarity is statistically significant. Something close to a Kolmogorov–Smirnov test in statistics, but for text data. For ...
Pujan Paudel's user avatar
1 vote
1 answer
16 views

Complex graphical representation based on monthly groups

I'm still fairly new to R, and am having some real issues turning a data frame into a graphical representation. I downloaded some data for house prices in the UK and stripped it down to just the ...
Lexifir's user avatar
  • 11
0 votes
1 answer
22 views

How do I get a big-picture feel for my database?

Background: I was just granted access to my company's SQL Server (MS SQL 2019 via Azure Data Studio). As I have not yet been assigned any specific projects I want start digging. I've asked my DBA to ...
Hogan Marhoefer's user avatar

1
2 3 4 5 6