Questions tagged [statistics]

Statistics is a scientific approach to inductive inference and prediction based on probabilistic models of the data. By extension, it covers the design of experiments and surveys to gather data for this purpose.

Filter by
Sorted by
Tagged with
2
votes
1answer
27 views

MOOC - for causal analysis - no statistics background

Am a software guy with no background in causal inference. While I am now familiar with prediction techniques due to plethora of courses available online, I would like to seek recommendations from ...
1
vote
0answers
16 views

Using Z-test score to evaluate model performance

I think I know the answer to this question but I am looking for a sanity check here: Is it appropriate to use z-test scores in order to evaluate the performance of my model? I have a binary model that ...
0
votes
1answer
12 views

Normalize data between 0 and 95 instead of between 0 and 100

I want to normalize the data between 0 and 95 instead of 0 and 100. I am using this formula to normalize between 0 and 100, please let me know how to edit it. ...
1
vote
2answers
27 views

How to choose products based on Number of good, bad and total reviews?

Let us suppose, I have few scenarios for products with good and bad reviews. P1: 1000 Good, 1 bad P2: 100 good, 10 bad P3: 20 Good, 0 bad P4: 10000 good, 500 bad ...
1
vote
1answer
42 views

Neural network linearity and non linearity

Is it right on my part to call: A neural network with only input and output layer (sigmoid) as linear (since it is a logistic regression) A neural network with more than one hidden layer non-linear (...
3
votes
1answer
179 views

SAS Studio seems to imply that apparently non-normal data is normal

I have some data I'm trying to analyze in SAS Studio (university edition). I am using the Distribution Analysis feature to try to test some data for normality. It gives me the following histogram: ...
0
votes
0answers
8 views

What approach do I used to analyze this F&B sales data?

I work as a data analyst at a theme park (kind of like disney) but we operate only during 4-6 months in an year. I was asked to analyze the F&B sales at the different locations in the park and ...
0
votes
0answers
16 views

Can the same data set (dynamic) be described as Chaotic & Pareto?

I'm trying to abstract the mathematical part of the problem as much as possible before the details follow, There's this dynamic data set that's $O(2^{32})$, a recent result described it as a power-law ...
0
votes
1answer
17 views

Real-Time Outlier/Anomaly Detection?

My data is the usage/playing statistics for players of a specific game. One data point for a user is aggregated statistics for one week. The goal is to be able to detect when the account of the player ...
2
votes
2answers
89 views

Which statistical test should I use to show significance in favour of a system?

I have a dataset of the following form: System A Rating System B Rating 4.5 5 3 4 5 3 etc. etc. I have 155 such data points gathered using a survey. Which statistical test should I use to show ...
1
vote
0answers
19 views

Estimating the (normal?) distribution of the grayscale values of a region in camera footage

The problem: Given a small ($25\times25$) region on a $1080$p image extracted from camera footage of a railway system, I am trying to detect whether there is a train present or not. I proceed to take ...
3
votes
2answers
42 views

How can I aggregate/combine 3 columns of a data frame into one column with the sum of the values of the other three in R?

Part of my dataset is shown in this image, I want to combine the columns GR_S01_w1_a, GR_S01_w1_b and ...
0
votes
1answer
23 views

How to compare models and which settings to keep constant? [closed]

I already posted this in another forum but no response. So, posting it here. Currently, in clinical practice, clinicians use a score (as a single feature) to predict the mortality of a patient. Now in ...
0
votes
0answers
10 views

N of Measurements needed for Multitrait Multimethod Matrix

I am taking a statistics subject for psychology and the content has recently covered multitrait multimethod matrices (MTMM). I am trying to wrap my head around it. Just as an example, for a matrix ...
0
votes
0answers
22 views

what is the effect of initial weights on model training for different algorithms

If I do a model training on a dataset with three different algorithms Logistic Regression (L1), SVM(S1) and a Neural Net(N1). If I train the models again with the same data set and same parameters ...
0
votes
1answer
32 views

dividing Mean by standard Deviation meaning

I have played around with logistic regression a little using movement data intervals that are prelabeled as either resting or active. I now found that if I divide the mean movement of the individual ...
0
votes
1answer
31 views

Association between continuous input and categorical output

I have two independent continuous variables like Age, Price and an outcome variable like purchased or not purchased Now, which test should I use to ascertain the association of continuous input ...
1
vote
1answer
21 views

Computing a cumulative distribution function in Python

I'm trying to compute the distribution function of any of the usual distributions in Python... However, all the methods I've seen involve first drawing N samples from said distribution, and then order ...
3
votes
1answer
35 views

How do I know that model performance improvement is significant?

Say I am running a Machine Learning model that produces a certain result (say accuracy of 80%). I now change a minor detail in my model (say, in a Deep Learning model, increase the kernel size in one ...
2
votes
1answer
54 views

is SST=SSE+SSR only in the context of linear regression?

the problem of regression is to minimize the sum of squared errors, i.e. $\sum\limits_{i=1}^n (y_i - \hat{y}_i)^2 = 0$ . But only in linear regression could you use the expression $\hat{y}_i = \beta_0 ...
3
votes
1answer
36 views

How to identify if there is a relationship between 5 categorical independent variables to a binary dependent variable?

My dataset has 5 independent variables, each with a value of either Large, Medium or None and a binary dependent variable. The dataset has 67 rows with a split of 17:50. I would like to identify if ...
1
vote
0answers
30 views

I have a data set of optimal values after simulations, How can I find if this dataset follows a specific pattern or any relation exists?

In the simulation I am conducting, I have a set of triangles and I select the optimal triangle based on my metric. After every simulation, I obtain an optimal triangle and I note down the lengths of ...
1
vote
1answer
46 views

Is there such thing as linear and non-linear data?

While doing machine learning projects we've heard that logistic regression works well with "Linear data" and decision tree works well with "non-linear data" However concept of ...
3
votes
2answers
82 views

How useful is Bayesian Inference

Last few months, I had been exposed to Bayesian Inference in ML course With further investigation, I come to place where there is MCMC technique to simulate the ...
2
votes
0answers
29 views

Clustering and getting threshold to classify data points

I have real world data set (120 data points) about enterprises, containing 4 features. I would like to put these enterprises in exactly 4 categories based on the values of these specific features (an ...
0
votes
1answer
15 views

Non IID variables and SVM Classifier

I am training an SVM model to predict the trend of stock prices (one-day ahead predictions. Classification task). It Had completely slipped from my mind that SVMs assume IID data until I had a ...
0
votes
0answers
10 views

Data generation ( Low KMO index)

I tried to generate a dataset with a respect to a correlation matrix R using cholesky decomposition. The problem is that i always get an KMO index that is very low ( smaller than 0.5 ). Is there any ...
0
votes
0answers
12 views

Imbalanced data identification with Shannon entropy

I was studying about the imbalanced data. Then I had a question , how would someone know that which data is imbalanced or not by looking at its percentage(20,30 or 40). Then I read an answer on stack ...
0
votes
0answers
11 views

Anomaly Detection in Highly Variable Time-Series Data

I am trying to detect anomalies through a column called count. The data is a time-series data and it is present for every 5 minutes for each day. The dataframe looks like this: ...
0
votes
0answers
7 views

Qunatify total time saved by prioritizing tasks based on the failure rate probability of each task

I am trying to solve a problem where I am trying to prioritize the tasks in a job based on the failure rates of each task. For ex: ...
0
votes
0answers
24 views

Why do we use qcut for RFM modelling?

All, I am new to marketing data science. I have seen many literature on RFM modelling and do not understand why we put the aggregated scores/customer into quantiles using qcut. e.g. ...
0
votes
0answers
17 views

ANOVA on one hot encoded feature

Suppose I am training a model for binary classification and I have a categorical feature which has been one-hot encoded. Suppose the feature is Gender and the one ...
4
votes
1answer
84 views

Linear Regression bad results after log transformation

I have a dataset that has the following columns: The variable I'm trying to predict is "rent". My dataset looks a lot similar to what happens in this notebook. I tried to normalize the rent ...
0
votes
0answers
15 views

What is the difference between causal discovery and inverse modeling?

I do not see these words used interchangeably, but they seem to be similar. In inverse modeling we are trying to find causal factors given an effect. In causal discovery, we are also looking for ...
0
votes
0answers
14 views

Create a framework to test effectiveness

I’m going to make up the use case here. Let’s say I’m testing an equipment with software. I can configure the test software to run for any duration - 2 minutes or 8 hours. Hypothesis is that the ...
0
votes
1answer
231 views

How to remove spikes from data with Python using signal.find_peaks

I need to make a regression model to estimate data values in future. Train set contains occasional spikes that make my model less accurate, thus I'm trying to locate and remove them. I've used ...
0
votes
0answers
20 views

Margin of error for imbalanced discrete set

I'm evaluating the performance of a classifier regarding its false negatives. The classifier performed over 9090 samples, from which 9000 were labeled as negative. I randomly chose 800 samples (out of ...
1
vote
2answers
42 views

Does Homoscedasticity applies only for linear regression models?

In statistics, a sequence of random variables is homoscedastic if all its random variables have the same finite variance. This is also known as homogeneity of variance (Wikipedia). Is this assumption ...
1
vote
1answer
21 views

Understanding math notation in infoGAN paper

I'm reading this paper about mutual information in infoGAN infoGAN_paper_link and already have the code to run it. I pretty much found code for it which is fine and dandy except for the fact that I ...
0
votes
0answers
25 views

Imbalanced dataset, finding the statistical significance of a Matthews Correlation Coefficient (MCC) in binary classification (what is a good MCC)?

I have a very imbalanced dataset. Thus, I am using MCC to evaluate the performance of various ML algorithms. It appears that literature is entirely lacking in ways to evaluate how good an MCC score is....
1
vote
0answers
18 views

How can I include these 4 variables into my code in R [closed]

Here's a glimpse of the dataset I'm using. SpeedDating dataset: The data set I'm using is SpeedDating. My goal is to look at whether or people prefer to date each ...
1
vote
1answer
63 views

Similarity between 2 statistical distributions

Is there any index that measures similarity between 2 gaussian distributions of 1-D data (may have slightly different number of points) considering their mean shift, variance shift, difference in ...
1
vote
0answers
28 views

Problem with Median Absolute Deviation

I am using Median Absolute Deviation(MAD) for outlier detection. But the problem with MAD is that if 50% or more values in a sample are identical, then MAD = 0 which is not desirable. Is there any way ...
1
vote
0answers
8 views

How to manage survivorship bias in conversion metrics?

I am trying to understand diminishing returns on additional advertising. Basically, does the 2nd ad convert worse than the 1st, etc. I'm struggling with thinking about how to deal with the ...
0
votes
0answers
30 views

Parallel programming in Python

I have the next code that I am trying to run in parallel: ...
1
vote
0answers
29 views

How do you perform basic statistical analysis on a Binary Dataset?

I have never worked with a Binary dataset (1 and 0) (True or False) so I'm unsure what kind of statistical tests I should run to draw up simple conclusions. I'm doing a data science project and the ...
0
votes
0answers
12 views

Handle unbalanced data by implementing Edited nearest neighbors, smote and Tomek links in r?

Imbalanced data is a big problem in classification problems. I have a binary classification problem with imbalanced data. I have researched and found that a possible method of dealing with this is ...
1
vote
1answer
58 views

Eigenvalues of covariance matrix are negative

I'm working on the PCA of the mnist dataset, and I get a very strange result, I created a matrix whose rows are flattened mnist images, When I try to compute the eigenvalues of the covariance matrix, ...
0
votes
1answer
15 views

Calculation of relative standard deviation with a custom function in R

I have measured concentrations of elements in a number of samples. Each concentration is an average of three measurements. Also the standard deviation of these measurements is recorded. I tried to ...
1
vote
0answers
38 views

Normalizing and joining of independent logistic regression model's prediction

I need to train several Logistic regression models on a different set of data (with a different set of labels): ...

1
2 3 4 5
19