Questions tagged [descriptive-statistics]

Descriptive statistics summarize features of a sample, such as mean and standard deviations, median and quartiles, the maximum and minimum. With multiple variables, may include correlations and crosstabs. Can include visual displays - boxplots, histograms, scatterplots and so on.

Filter by
Sorted by
Tagged with
1 vote
2 answers
286 views

How to find a vector representation for each descriptor?

Cubes data is well known data for extreme classification. Each picture has a set of descriptor along with it. Total data set has 312 descriptor. You will find list of descriptior in this file. My ...
0 votes
2 answers
300 views

Missing Values Correlation

Is it worth it to study missingness correlation between columns? If you have strongly correlated missing values (say between two columns, A and B), how will this change or shape the way you look at ...
1 vote
0 answers
25 views

Standard Error interpretation help

I performed a standard error on my machine learning model to predict protein structure. The graph Im showing here is a snippet of the actual data and I deleted some irrelevant info. The y axis is the ...
2 votes
3 answers
1k views

A/B Testing (Binomial Distribution vs Random Distribution)

When performing an A/B test for the number of clicks for users viewing (each view is an impression) two variants of an ad, a binomial distribution can be assumed where each variant has a constant ...
1 vote
2 answers
58 views

Is there a measure to compare features on the basis of normality

I have a dataset of cars and it has many features including 'acceleration’, ‘horsepower’, and ‘mpg'. I am supposed to check which of these features is the most similar to a normal distribution, so I ...
0 votes
1 answer
43 views

What is the best practice to detect constant and nearly constant data over time?

I'm experimenting with the characterization of data over time. I generated some synthesized data with certain periodic patterns over time every 5mins (granularity of 5mins= data generated with the ...
9 votes
6 answers
7k views

When to use mean vs median

I'm new to data science and stats, so this might seems like a beginner question. I'm working on a dataset where I've user's Twitter followers gain per day. I want to measure the average growth he had ...
0 votes
1 answer
100 views

Analysis for basic weight training analysis?

TL;DR: I'm doing a fairly basic project which involves exercise. It seems that descriptive statistics and basic data vis (ex: line graph) would be most appropriate for this project, but I wonder if ...
0 votes
1 answer
108 views

Insights betwwen two columns/variables in Dataframe

I have data in two columns one is range of old credit score (Input score range) and new credit score (cvsc100). How do i find insights from both of them ? where the old is range of values and other ...
1 vote
1 answer
162 views

A dataset has skewness = 1 with missing data. Standard deviation around median is 1.5. How much data will be unaffected?

There is no other description about the data, if it is univariate, bivariate, etc. neither the type of distribution is given. I recently came across this question, I would like to know how skewness ...
1 vote
1 answer
104 views

Drastic drop in Somers' D ? Why?

I came across to find the correlation between the ratings assigned by two coaches to a same group of 40 players. I have tabulated the results as below: The Somers' D is 50%. However, for the case ...
0 votes
1 answer
226 views

PSI where not to use

From what I understand PSI is used for continuous data. Generally, equal sized bins are created to compare two data set, and number of buckets is usually 10. Is that for a reason, why 10 bucket? Also, ...
15 votes
5 answers
2k views

How can I ensure anonymity with queries to small datasets?

I'm building a service that will contain personal data relating to real people. Initially the dataset will be quite small, and as such it may be possible to identify individuals if the search ...
0 votes
1 answer
42 views

what are the best sources to learn about statistics?

am interested in quant analytics. hope statistics is heart for quant analysis. so, what are the best sources to learn core statistics (in depth) and what is the best approach to learn statistics?
0 votes
1 answer
28 views

Best practise XAI: understand features which build up cluster and explain underlaying structure

I want to cluster my data and show which features were used to define the clusters to show the structure in my data. To explain the use case: Imaging I have data from many products and I want show the ...
1 vote
1 answer
48 views

Which test statistic do you recommend for checking the difference between a dynamic for some parameter in control and treatment group?

I have two groups of patients, a control (A) and a treatment (B), and data for patients' weight in the beginning and in the end of a period of treatment in each group. I need to check if the dynamic ...
1 vote
1 answer
21 views

Are there any algorithm to generate a set of data that match some statistic requirements?

I was wondering if there are time-efficient algorithms that can reverse the process of basic statistics computation. What I mean is an algorithm that instead of computing the mean, SD, max-min range, ...
0 votes
1 answer
112 views

What is the difference between accuracy in percentage (classification metric) and MPE (regression metric) as evaluation metrices?

What is the difference between accuracy in percentage (classification metric) and MPE (regression metric) as evaluation metrices?
0 votes
0 answers
32 views

Exploratory statistics, how to idenify and remove driver (bias)

I am looking at customer data, and created frequency tables (+histograms) for customers with different professional statuses and what the best time is to reach them. Status ranges here from employed, ...
0 votes
1 answer
38 views

How to count words in a dataframe?

I would like to count how many Male and female who answer (ex. Biking / Cycling). Below is the sample data:
2 votes
2 answers
3k views

p-value and effect size

Is it correct to say that the lower the p-value is the higher is the difference between the two means of the two groups in the t-test? For example, if I apply the t-test between two groups of ...
2 votes
0 answers
80 views

How do you determine if a value is statistically significant?

I have collected some data I need to analyze. The data is the result of a survey in which I asked approx. 180 sellers at a bazaar, how important a certain buyer's characteristic is in relation to ...
3 votes
2 answers
2k views

How do I handle string feature while performing model generation

I have data which looks like this ...
3 votes
2 answers
44 views

evaluation metrics for multiple values per session

I have an application that executes my foo() function several times for each user session. There are 2 alternate algorithms that i can implement as "foo" function and my goal is to evaluate them based ...
0 votes
1 answer
323 views

Comparing data sets with different measurements

I'm currently writing a thesis based on Cyber Crime, however I'm unsure of the proper to compare/analyse my data sets to talk about them in my thesis. One piece of data (https://www.pandasecurity....
1 vote
1 answer
11 views

what is a collective term for things as screening failures, no response and drop-outs

Hello wondering what is a collective term (in terms of statistics) for things as screening failures, no response and drop-outs?
1 vote
0 answers
10 views

Trying to get an idea of decile system

I was checking a Linkedin profile and I came across this statement in the “Grade” section. It’s a little confusing. It is as follows: “Grade: gpa within the first decile”. Is this statement correct? ...
1 vote
0 answers
57 views

Null and 0 Values Handling - Basic Statistics [duplicate]

I am trying to do some descriptive statistics and basic graphs on relatively small data sets derived from environmental sample collection and/or chemical research. It turns out that sample points end ...
0 votes
2 answers
562 views

A measure of redundancy in mutual information

Mutual information quantifies to what degree $X$ decreases the uncertainty about $Y$. However, to my understanding, it does not quantify "in how many ways" $X$ decreases the uncertainty. E.g....
2 votes
3 answers
430 views

How do I interpret the output of linear regression model in R?

I have the following linear regression model and its analysis. There are a few errors, but I am not very sure about the errors. I have not succeeded in finding them so far. First, the 95% confidence ...
1 vote
0 answers
74 views

Estimating the (normal?) distribution of the grayscale values of a region in camera footage

The problem: Given a small ($25\times25$) region on a $1080$p image extracted from camera footage of a railway system, I am trying to detect whether there is a train present or not. I proceed to take ...
2 votes
1 answer
155 views

Re-bucket weekly sales data and calculate descriptive statistics

I have sales data in weekly buckets like this: ...
1 vote
1 answer
60 views

What is the best way to cluster this kind of data?

I have data that looks like this: The chart on the left is the trend and the smaller chart on the right is the box plot showing the distribution of means. Each color is the output of a particular ...
1 vote
1 answer
39 views

Applying an algorithm on it's own training data for descriptive purposes?

Good Day, I am newer to data science so I am not confident in this. To set up the question I will describe my data and approach. Data I don't want to share specific data examples as I want to try and ...
1 vote
1 answer
57 views

Understanding math notation in infoGAN paper

I'm reading this paper about mutual information in infoGAN infoGAN_paper_link and already have the code to run it. I pretty much found code for it which is fine and dandy except for the fact that I ...
1 vote
1 answer
3k views

Calculating correlation for categorical variables

I am struggling to find out a suitable way to calculate correlation coefficient for categorical variables. Pearson's coefficient is not supported for categorical ...
1 vote
1 answer
119 views

Accuracy over different sample sizes from dataset

What I'm trying to do is predict how much more data would help in a classification task. So, what I'm doing is bootstrapping entries in my dataset to get a sample, with a specified size. Then, I fine-...
7 votes
2 answers
251 views

Which statistical test tells which classifier performs better than the other?

I have 3 classifiers: A, B and C. According to accuracy, specificity, sensitivity, f-score, and g-mean, say classifier B performs best. Now I want to statistically validate this claim. How should I do ...
1 vote
0 answers
38 views

paired t-test shows no difference between median and Wilcoxon test p value shows that there is a difference between median values ? How to interpret?

I have a dataset. I wanted to do paired t test on it. So I carried out normality test and it showed that it does not follow normal distribution. So I used Wilcoxon test in place of paired t test. The ...
1 vote
1 answer
116 views

How to bin a distribution data reported with different frequencies ( salaries ), showing mixed linearity and non-linearity?

I am researching on pay-scales, and wish to receive advise to treat data of salaries. Objective My interest is to approximate the salary corresponding to different hierarchical levels in an ...
0 votes
0 answers
36 views

Data calculation/normalisation /data summarisation

Clustering is done based on houses data and say suppose cluster 1 contains 5 houses. Appliance 1 is used in 2 houses out of 5 and appliance 2 is used in 4 out of 5 houses.Each appliance produces an ...
2 votes
0 answers
23 views

Statistical method to find the value which preserves the most information inside "most" of data points. (resize images to a common height)

So I have this data of around 88K images and I found out some interesting properties for my images. ...
6 votes
1 answer
2k views

How does Seaborn calculate error bars when using estimators other than the arithmetic mean?

If I create a barplot using Seaborn and specify the geometric mean or the median as the estimator, does Seaborn know to use the appropriate standard error formula to create error bars?
1 vote
1 answer
39 views

How to get a (descriptive) overview of a large database?

I'm facing a data framework with ~ 20 k observations and 151 variables across 2078 subjects At first I am primarily interested in how the data looks like related to a single parameter. But I cannot ...
1 vote
0 answers
20 views

Normalise the scores by small and large numbers division

I am working on a scoring problem use-case where I will score each task based on few aspects and take an average of scores of tasks in each experiment. For each experiment I will have 1 to N number of ...
1 vote
1 answer
677 views

Can a Box plot be used for finding the useful features from the dataset? [closed]

I am reading a book by professor Trevor Hastie and professor Robert Tibshirani called "Introduction to Statistical Learning". In the applied section of the chapter 4, there is a question 11(...
0 votes
1 answer
41 views

Analysis of passed/not passed students by enrolled year

I am analysing some data from my University about successful students through years. The dataset looks like as follows ...
2 votes
1 answer
148 views

Problem regarding calculating correlation approach?

There are 3 columns say product_id, product_type, and price_drop. ...
0 votes
2 answers
544 views

Calculating Median and Mode of numerical variables for different subgroups in R

I have customer call data and I want to get the median and mode for the call success rate for different subgroups. My variables are: Customer ID, Employment Status (Retired, Employed, Unemployed), ...
3 votes
2 answers
631 views

Distribution vs Histogram [closed]

I am not very good at understanding statistics jargon I tried to read articles on statistics every article says distribution and show a picture of bell curve. I understand that x axis of bell curve ...