Questions tagged [descriptive-statistics]

Descriptive statistics summarize features of a sample, such as mean and standard deviations, median and quartiles, the maximum and minimum. With multiple variables, may include correlations and crosstabs. Can include visual displays - boxplots, histograms, scatterplots and so on.

Filter by
Sorted by
Tagged with
0 votes
0 answers
13 views

Sensor data completeness

We have several sensors collecting data with a specified frequency (usually every 20 minutes but it varies) over a period of many years. The person that programmed them made sure, as a space saving ...
user avatar
0 votes
1 answer
18 views

How to count words in a dataframe?

I would like to count how many Male and female who answer (ex. Biking / Cycling). Below is the sample data:
user avatar
0 votes
1 answer
13 views

PSI where not to use

From what I understand PSI is used for continuous data. Generally, equal sized bins are created to compare two data set, and number of buckets is usually 10. Is that for a reason, why 10 bucket? Also, ...
user avatar
  • 1
0 votes
0 answers
10 views

Making my dataset stationary increases SampleEntropy score

When i make my dataset stationary by the taking difference method, my SampleEntropy score is increasing. It means my data is being "less forecastable". But my results are deffinetly better ...
user avatar
  • 45
0 votes
0 answers
15 views

When do I need Statistical Signifcance testing and when not?

Hi there I have a handful of questions regarding statistical significance testing. As a newcomer I have sometimes topics that I do not really understand entirely. One of them is checking for ...
user avatar
0 votes
0 answers
11 views

Calculate rate from related datasets

I have the monthly sales rate for various products. The products are sold in different countries. I'm looking for a meaningful way to calculate the sales rate at each country. The sales rate indicated ...
user avatar
  • 101
0 votes
0 answers
60 views

Time Series Analysis for Categorical Data Output

Suppose I am having dataset which consist of date as one column and fruits as second column which is categorical data having set of 4 different fruits in that column and my output column has 0's and 1'...
user avatar
  • 36
0 votes
0 answers
24 views

Dividing a data set into segments with consistent inner behavior, using segmentation algorithms and metrics for consistency

Context of the problem: I have signal data which was recorded in a software system and which shows the runtime of multiple processes over time. In total there are more than 900 processes each having ...
user avatar
1 vote
0 answers
10 views

Trying to get an idea of decile system

I was checking a Linkedin profile and I came across this statement in the “Grade” section. It’s a little confusing. It is as follows: “Grade: gpa within the first decile”. Is this statement correct? ...
user avatar
1 vote
1 answer
9 views

what is a collective term for things as screening failures, no response and drop-outs

Hello wondering what is a collective term (in terms of statistics) for things as screening failures, no response and drop-outs?
user avatar
1 vote
0 answers
56 views

Null and 0 Values Handling - Basic Statistics [duplicate]

I am trying to do some descriptive statistics and basic graphs on relatively small data sets derived from environmental sample collection and/or chemical research. It turns out that sample points end ...
user avatar
  • 111
0 votes
2 answers
243 views

A measure of redundancy in mutual information

Mutual information quantifies to what degree $X$ decreases the uncertainty about $Y$. However, to my understanding, it does not quantify "in how many ways" $X$ decreases the uncertainty. E.g....
user avatar
1 vote
0 answers
73 views

Estimating the (normal?) distribution of the grayscale values of a region in camera footage

The problem: Given a small ($25\times25$) region on a $1080$p image extracted from camera footage of a railway system, I am trying to detect whether there is a train present or not. I proceed to take ...
user avatar
  • 111
1 vote
1 answer
36 views

Applying an algorithm on it's own training data for descriptive purposes?

Good Day, I am newer to data science so I am not confident in this. To set up the question I will describe my data and approach. Data I don't want to share specific data examples as I want to try and ...
user avatar
1 vote
1 answer
30 views

Understanding math notation in infoGAN paper

I'm reading this paper about mutual information in infoGAN infoGAN_paper_link and already have the code to run it. I pretty much found code for it which is fine and dandy except for the fact that I ...
user avatar
  • 309
1 vote
1 answer
1k views

Calculating correlation for categorical variables

I am struggling to find out a suitable way to calculate correlation coefficient for categorical variables. Pearson's coefficient is not supported for categorical ...
user avatar
  • 189
1 vote
1 answer
53 views

What is the best way to cluster this kind of data?

I have data that looks like this: The chart on the left is the trend and the smaller chart on the right is the box plot showing the distribution of means. Each color is the output of a particular ...
user avatar
1 vote
1 answer
71 views

Accuracy over different sample sizes from dataset

What I'm trying to do is predict how much more data would help in a classification task. So, what I'm doing is bootstrapping entries in my dataset to get a sample, with a specified size. Then, I fine-...
user avatar
  • 188
2 votes
2 answers
215 views

p-value and effect size

Is it correct to say that the lower the p-value is the higher is the difference between the two means of the two groups in the t-test? For example, if I apply the t-test between two groups of ...
user avatar
1 vote
0 answers
29 views

paired t-test shows no difference between median and Wilcoxon test p value shows that there is a difference between median values ? How to interpret?

I have a dataset. I wanted to do paired t test on it. So I carried out normality test and it showed that it does not follow normal distribution. So I used Wilcoxon test in place of paired t test. The ...
user avatar
  • 151
1 vote
1 answer
35 views

How to bin a distribution data reported with different frequencies ( salaries ), showing mixed linearity and non-linearity?

I am researching on pay-scales, and wish to receive advise to treat data of salaries. Objective My interest is to approximate the salary corresponding to different hierarchical levels in an ...
user avatar
0 votes
0 answers
31 views

Data calculation/normalisation /data summarisation

Clustering is done based on houses data and say suppose cluster 1 contains 5 houses. Appliance 1 is used in 2 houses out of 5 and appliance 2 is used in 4 out of 5 houses.Each appliance produces an ...
user avatar
  • 9
2 votes
0 answers
22 views

Statistical method to find the value which preserves the most information inside "most" of data points. (resize images to a common height)

So I have this data of around 88K images and I found out some interesting properties for my images. ...
user avatar
  • 313
1 vote
1 answer
22 views

How to get a (descriptive) overview of a large database?

I'm facing a data framework with ~ 20 k observations and 151 variables across 2078 subjects At first I am primarily interested in how the data looks like related to a single parameter. But I cannot ...
user avatar
  • 480
2 votes
2 answers
489 views

A/B Testing (Binomial Distribution vs Random Distribution)

When performing an A/B test for the number of clicks for users viewing (each view is an impression) two variants of an ad, a binomial distribution can be assumed where each variant has a constant ...
user avatar
  • 121
1 vote
0 answers
20 views

Normalise the scores by small and large numbers division

I am working on a scoring problem use-case where I will score each task based on few aspects and take an average of scores of tasks in each experiment. For each experiment I will have 1 to N number of ...
user avatar
1 vote
1 answer
52 views

Can a Box plot be used for finding the useful features from the dataset? [closed]

I am reading a book by professor Trevor Hastie and professor Robert Tibshirani called "Introduction to Statistical Learning". In the applied section of the chapter 4, there is a question 11(...
user avatar
0 votes
1 answer
37 views

Analysis of passed/not passed students by enrolled year

I am analysing some data from my University about successful students through years. The dataset looks like as follows ...
user avatar
  • 295
2 votes
1 answer
103 views

Problem regarding calculating correlation approach?

There are 3 columns say product_id, product_type, and price_drop. ...
user avatar
2 votes
3 answers
152 views

How do I interpret the output of linear regression model in R?

I have the following linear regression model and its analysis. There are a few errors, but I am not very sure about the errors. I have not succeeded in finding them so far. First, the 95% confidence ...
user avatar
0 votes
2 answers
150 views

Calculating Median and Mode of numerical variables for different subgroups in R

I have customer call data and I want to get the median and mode for the call success rate for different subgroups. My variables are: Customer ID, Employment Status (Retired, Employed, Unemployed), ...
user avatar
  • 53
3 votes
2 answers
264 views

Distribution vs Histogram [closed]

I am not very good at understanding statistics jargon I tried to read articles on statistics every article says distribution and show a picture of bell curve. I understand that x axis of bell curve ...
user avatar
0 votes
1 answer
12 views

Getting the earliest date (duplicates due to several call ids and agents)

I am trying to get the earliest call interaction, with only the agent that interacted at that call and the customer id. Using the following code, I still get duplicates: ...
user avatar
  • 53
0 votes
0 answers
31 views

Exploratory statistics, how to idenify and remove driver (bias)

I am looking at customer data, and created frequency tables (+histograms) for customers with different professional statuses and what the best time is to reach them. Status ranges here from employed, ...
user avatar
  • 53
1 vote
0 answers
15 views

Coefficient of variance problem

I am doing a project and I am facing a problem I have no idea how to solve. I have a data series of stock prices, and, based on this series I obtained another series, of so-called rentabilities (which ...
user avatar
  • 11
1 vote
1 answer
64 views

Drastic drop in Somers' D ? Why?

I came across to find the correlation between the ratings assigned by two coaches to a same group of 40 players. I have tabulated the results as below: The Somers' D is 50%. However, for the case ...
user avatar
0 votes
1 answer
61 views

Insights betwwen two columns/variables in Dataframe

I have data in two columns one is range of old credit score (Input score range) and new credit score (cvsc100). How do i find insights from both of them ? where the old is range of values and other ...
user avatar
  • 9
0 votes
1 answer
39 views

(Basic) statistics

I got a misunderstanding regarding basic statistics (I think) but I can't get my head around that: I did an online survey regarding the usage of some application. The user could answer 1 (I don't know ...
user avatar
1 vote
0 answers
32 views

How to handle and report patient characteristic statistic with missing data in essay?

I'm now working on an clinical trial essay with some doctors. When generating patient characteristic statistic form, I found basic patient characteristic data has some missing. The missing datas are ...
user avatar
0 votes
1 answer
66 views

How to interpret axis of Histogram and distribution curve?

After doing statistical analysis of data I got the below figures: The data contains a line length in meters which is represented with Line(m). The purpose here is to confirm the distribution behavior ...
user avatar
  • 2,267
1 vote
0 answers
31 views

Seeking advice: How would you define "top performer" with a set of data?

still consider myself a noobie in ds here, so I may be asking a very easy and stupid question, but would really appreciate some guidance. The hypothetical situation is that, you have a website where ...
user avatar
3 votes
2 answers
34 views

evaluation metrics for multiple values per session

I have an application that executes my foo() function several times for each user session. There are 2 alternate algorithms that i can implement as "foo" function and my goal is to evaluate them based ...
user avatar
  • 131
7 votes
2 answers
129 views

Which statistical test tells which classifier performs better than the other?

I have 3 classifiers: A, B and C. According to accuracy, specificity, sensitivity, f-score, and g-mean, say classifier B performs best. Now I want to statistically validate this claim. How should I do ...
user avatar
  • 1,151
2 votes
1 answer
40 views

Generating random numbers from best probability distributions?

I have done some statistical distributions on real-world data. The distribution fitting gives me the below results (lognormal fits the best on data based on chi-squared test): ...
user avatar
  • 2,267
0 votes
4 answers
554 views

What do you call the ratio of positive to negative samples?

I am working with a binary classifier and I want to express the "balance" or "skewness" of the training data using a metric. I want to reflect this ratio in a report, like this: ...
user avatar
1 vote
1 answer
54 views

Analysis for basic weight training analysis?

TL;DR: I'm doing a fairly basic project which involves exercise. It seems that descriptive statistics and basic data vis (ex: line graph) would be most appropriate for this project, but I wonder if ...
user avatar
  • 11
0 votes
3 answers
212 views

Aggregate several attributes into one using PCA

I am working on a soccer dataset. As we all know, soccer has a lot of different metrics related to the game. We have Key passes, Accurate Passes, Shots on Target, Saves, Clearances and so on... I have ...
user avatar
  • 143
1 vote
1 answer
44 views

Normal distribution and QQ plot

I have data that I plotted with a normal distribution and a QQ plot. I was wondering especially in QQ plot it seems that 95.4% of the data is normally distributed. My question is what does numbers ...
user avatar
  • 113
1 vote
2 answers
371 views

Skew and Kurtosis are so similar?

I've been taking the histogram of the optical flow in videos and plotting the kurtosis and skewness of each frame. At the end of the video, I noticed that the skewness and kurtosis follow each other - ...
user avatar
-1 votes
1 answer
36 views

Is it possible to estimate

For example, if I want to know a page views/relative prevalence of a Reddit post, and I have the upvote and downvote data. Is it valid that I use the upvote and downvote to predict that, even if I ...
user avatar
  • 1