Questions tagged [data-analysis]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1
vote
0answers
17 views

Handling missing data - secondary driver characteristics in insurance data

I have an insurance dataset which includes an indicator that indicates whether the policy insures a secondary driver, and the secondary driver's age/sex. Problem is most policies do not have secondary ...
1
vote
0answers
34 views

whether to use normalization while performing PCA?

I have an excel file containing a table where I registered the frequency of three linguistic phenomena in 72 poems. Since the poems have different lengths I normalized the results dividing each value ...
3
votes
1answer
15 views

Not sure how to visual customer retention vs product line count

I created a customer retention grid that tracks customer counts by territory by year. The sales managers have loved it. The numbers were terrible, but now they know what they didn't know and can ...
5
votes
1answer
76 views

experimental design in R project

I want to know of any repositories that contain complete experimental design in R covering basic test and analyses? I want to take a top-buttom approach to learn step by step through a real project ...
1
vote
1answer
23 views

train test data split up in datasets

In a dataset consisting of 1,000 samples, it has been shown that a 70-30 split (i.e. 70% of the samples used for training, 30% for validation) will provide a good estimation of the test accuracy of ...
1
vote
0answers
11 views

analyze the effect of some new changes to business rules on customers retention and sales

I am trying to analyze the effect of a particular business rule on customer behavior. Background: I have two call centers operating in my company. One is an in-house call center and the other one is a ...
0
votes
0answers
12 views

Preparing and combining datasets for predictions

I am trying to create and train a model to predict the winner of a professional DOTA match. This model is only for "fun" and shouldn't be used for ...
0
votes
1answer
21 views

Classic sport match prediction

So, I am currently learning machine learning and data analysis. I have created for my self a problem that is: Who will win a match of soccer Now, I have narrowed it down as being a ...
1
vote
1answer
34 views

Is there a standardized way to do data analysis?

Is there a standard way to do data analysis? So for example, something like this: 1. Data mining 2. Data cleaning 3. xx 4. Data and result interpretations I ...
0
votes
0answers
6 views

Autocorrelation of two functions multiplied and raised to arbitrary powers

Given a signal $A$ and a signal $B$ with autocorrelation times of $\tau_A$ and $\tau_B$, respectively, where $\tau_A > \tau_B$, is there any general statement that can be made about the ...
0
votes
1answer
26 views

Can I apply Feature Selection even though the number of features is smaller than the number of observations?

I was reading An Introduction to Statistical Learning when I came across the High Dimensional section, they argue that high-dimensional data causes a lot of problems. My question is, it is necesary to ...
0
votes
1answer
27 views

List value in Pandas DataFrame column makes analysis harder

Should I move to database? I have a list of courses data in JSON format that looks like this: ...
1
vote
1answer
20 views

Why does horizontal lines in plt.plot(feature, '.') mean that the data have been properly shuffled?

I am following a Mooc and in this lecture about visualisation in explenatory data analysis the lecturer claims that when plotting the row indexes against feature values, if we have lines on the ...
0
votes
0answers
36 views

Hypothesis Testing - Median Test for linear regression model given standard error and co-efficient

I need help approaching this particular problem on hypothesis testing. A researcher believes that a diet high in raw cornstarch would increase the median blood glucose levels by 20. The diet can ...
0
votes
0answers
18 views

What kind of techniques that I can use to see which type of of entries in the dataset became more popular lately(over a time-span)?

This is for my school project. I've been digging the internet in to find a solution that is backed-up really well but I couldn't find a metric that makes sense. I have a dataset, that has objects say: ...
0
votes
1answer
19 views

Log scaling whole dataset

I am log scaling my whole dataset in log base 10. When I do this I get -infitity for the minimum value. I am wondering how I can get rid of this -infinity? I have been advised to add a small value ...
1
vote
0answers
14 views

How to aggregate Weather Data from county to state level? [closed]

I am doing a prediction of forest fire risk using the weather data and the fire incident data. I have data related to different county and the weather data on the day of the incident. As the incident ...
0
votes
0answers
17 views

Data system that manages aggregates over time intervals

I am looking to know if there is a data system that handles the following use case. To keep it simple, the data is a set of homogeneous enties E. E contains named numeric properties that the app code ...
5
votes
3answers
79 views

How can data science teams inside businesses measure costs and efficiency of their technical work?

How can data science teams measure and improve costs of their technical work, when they often don't know the monetary value of the datasets and insights they are producing? Are they using industry ...
-1
votes
1answer
18 views

How to choose variables to perform Exploratory Data Analysis

I have a dataset with about 110 variables. I have a target variable and I want to do an exploratory data analysis to find out what factors affect this target variable. In such scenarios when we have a ...
1
vote
1answer
25 views

How do I control sensor data readings against (measured) outside influences?

I need some hints in this problem I have. I have a dataset of electro-dermal activity readings that are, by nature, influenced by the movements of the person that is being surveyed. In the same ...
0
votes
0answers
23 views

How to analyze the abnormal trajectory of the vehicle?

I use this code to mark the latitude and longitude of the vehicle with red points on the map: ...
0
votes
1answer
36 views

Analysis of passed/not passed students by enrolled year

I am analysing some data from my University about successful students through years. The dataset looks like as follows ...
1
vote
2answers
46 views

Problem with converting string to dummy variables

I'm new in data science, I have data which want to work on it, I omitted extra columns and convert it to 4 columns ( Product, Date, Market, Demand ) . in this data Product and Market are string, I ...
1
vote
1answer
17 views

How we can Identify Specific Feature from a larger amount of Dataset?

In Machine Learning, we need to play with any kind of datasets. In the Dataset, There are too many records and features, Some datasets had lots of features (sometimes it's called ...
0
votes
1answer
32 views

Interpretation of the data through scatter plot [closed]

I was exploring the data and had observed the data points are forming a triangle on the lower side. x-axis: Total items y-axis: Cancelled items Can someone help me ...
0
votes
0answers
17 views

Selling windows for products

I have sales data of hundreds of products. How they are sold over the year in terms of units sold and revenue. Sales of each product varies largely from one another and their seasonality too. Data ...
0
votes
0answers
28 views

Proximity multi-dimentional arrays. Which algorithms are commonly used?

I'm new to data science so still learning so much. I've been searching for proximity algorithms but I'm not sure which are suited for multi-dimensional arrays. Any guidance would be greatly ...
2
votes
0answers
32 views

Plotting regression test errors

I am trying out an R exercise where I have to randomly split my data into train/test set 50 times. On each iteration, I use lm() to train polynomial models up to ...
0
votes
1answer
36 views

Estimate correlation in Python

I have a dataset with labels and usernames: ...
2
votes
1answer
36 views

How to account for reduced student capacity when calculating program retention?

I work at a nonprofit youth center that has 2 distinctive programs throughout the year. Our afterschool program runs during the school year, and our summer camp program runs during the summer. I am ...
1
vote
1answer
60 views

Labelling a Time series dataset

I was asked this during an interview that I did not crack. You are given data of total (aggregated) power consumption, and the TV power consumption of three households. Each household has data for one ...
0
votes
0answers
27 views

Finding linear regimes in data

I am trying to find all linear regimes in a certain plot, given certain data points. If you are given two vectors of data $y$ and $x$ and $y$ is some possible non-differentiable function of $x$, how ...
2
votes
1answer
77 views

Problem regarding calculating correlation approach?

There are 3 columns say product_id, product_type, and price_drop. ...
0
votes
1answer
15 views

Compare two tends with big difference in absolute value

I'm studying the Spotify streams trend for different nations in order to show the impact of the pandemic on music listeners. So far i retrieved my data, and plotted it. Obviously, since the various ...
0
votes
2answers
36 views

Help deciding the best statistic tool for my data

I'm not from the data science area, and I need to perform some statistical analysis on my data, but I don't know which analytical tools I should use. Number of variables - 6 Independent Variables 3 - ...
0
votes
1answer
17 views

What is the name of the sub-field within spacial Data Science where the data relates in some way to physical objects viewed as 3D (graphics) Models?

What is the name of the specialism / sub-field in spacial Data Science where the data relates in some way to physical objects viewed as 3D (graphics) models?, ... that is, excluding GIS and BIM ...
1
vote
0answers
18 views

Data science Product Question Help [closed]

I was recently asked this in a ds interview pertaining to product thinking - If a product feature is being rolled out, but an a/b test is unable to be performed due to whatever reason, how can we ...
1
vote
2answers
36 views

What is the best way to calculate score based on input features?

Lets say we have certain products. We also have certain input features regarding these products like inventory, sales, price, cost etc. based on these features we want to score these parts. The score ...
0
votes
0answers
32 views

Formulate Confusion Matrix from Precision Score

I have the precision scores from 5 classes. Is it possible to create the confusion matrix from that scores. While running the code, I only saved those precision values by mistake. So now I want to ...
0
votes
1answer
58 views

How to use historic data (granularity at day level) for ML modeling?

There is a scenario where I have to use historic data which is at the day level for the past 5 years.Actually it is water flow data, what quantity of water was flown on that particular day. I have to ...
0
votes
1answer
30 views

Creative Analysis - What are the possible approaches?

We're analyzing some creatives from past advertising campaigns and are looking for the best way to approach. Currently we've broken down each creative into 20+ objective characteristics, for example: ...
0
votes
1answer
25 views

Forecasting using Python

I have very less training observations (15). I need to predict 6 months into the future. What forecasting model is best suited for this scenario? Here is how my dataset looks Month | Response ...
6
votes
2answers
127 views

ETL and Data Engineering - is it purely the knowledge of tools or is there theory behind it?

I would like to better understand what a good Data Englineer must know or what he does. Job descriptions mostly list tools that are required, such as Python. If it is possible to separate Data ...
0
votes
1answer
21 views

I am trying to figure out the stationarity of time series data

Here, this plot shows the number of customers served per day from 1 jan 2018 to 31 dec 2019. I grouped the entire data by each month and calculated the average and variance per month. This is the ...
25
votes
7answers
6k views

Is Python a viable language to do statistical analysis in?

I originally came from R, but Python seems to be the more common language these days. Ideally, I would do all my coding in Python as the syntax is easier and I've had more real life experience using ...
0
votes
2answers
19 views

3D visualisation and post analysis tool

I am looking for a 3D plotting and post analysis tool, specifically which can generate figures like the following. It is preferred the tool has a Linux distribution. The figure is copied from here.
0
votes
2answers
871 views

Discarding non-english words in column

I have some non-english words/sentences in my data. I tokenized my text and tried using nltk.corpus.words.words() but its not really helpful as it also removes the ...
0
votes
1answer
30 views

The OLAP (On-Line Analytical Processing) cube with 4 dimensions [closed]

A typical OLAP cube looks like this: As I can see, this cube can work with 2 or 3 dimensions, but what if I have 4 dimensions to produce facts? Should I use star schema instead when having more than ...
2
votes
1answer
34 views

Recommendations for Data Visualization solution for large CSV file on a non-profit website

I would like your advice and recommendations for a non-profit that is trying to turn a large CSV file into meaningful information. I am part of a non-profit that would like to turn a 23,000 row x 23 ...