Questions tagged [statistics]

Statistics is a scientific approach to inductive inference and prediction based on probabilistic models of the data. By extension, it covers the design of experiments and surveys to gather data for this purpose.

Filter by
Sorted by
Tagged with
0
votes
0answers
8 views

Understanding math notation in infoGAN paper

I'm reading this paper about mutual information in infoGAN infoGAN_paper_link and already have the code to run it. I pretty much found code for it which is fine and dandy except for the fact that I ...
0
votes
0answers
6 views

Imbalanced dataset, finding the statistical significance of a Matthews Correlation Coefficient (MCC) in binary classification (what is a good MCC)?

I have a very imbalanced dataset. Thus, I am using MCC to evaluate the performance of various ML algorithms. It appears that literature is entirely lacking in ways to evaluate how good an MCC score is....
1
vote
0answers
17 views

How can I include these 4 variables into my code in R [closed]

Here's a glimpse of the dataset I'm using. SpeedDating dataset: The data set I'm using is SpeedDating. My goal is to look at whether or people prefer to date each ...
1
vote
1answer
46 views

Similarity between 2 statistical distributions

Is there any index that measures similarity between 2 gaussian distributions of 1-D data (may have slightly different number of points) considering their mean shift, variance shift, difference in ...
1
vote
0answers
25 views

Problem with Median Absolute Deviation

I am using Median Absolute Deviation(MAD) for outlier detection. But the problem with MAD is that if 50% or more values in a sample are identical, then MAD = 0 which is not desirable. Is there any way ...
1
vote
0answers
8 views

How to manage survivorship bias in conversion metrics?

I am trying to understand diminishing returns on additional advertising. Basically, does the 2nd ad convert worse than the 1st, etc. I'm struggling with thinking about how to deal with the ...
0
votes
0answers
12 views

Parallel programming in Python

I have the next code that I am trying to run in parallel: ...
1
vote
0answers
25 views

How do you perform basic statistical analysis on a Binary Dataset?

I have never worked with a Binary dataset (1 and 0) (True or False) so I'm unsure what kind of statistical tests I should run to draw up simple conclusions. I'm doing a data science project and the ...
0
votes
0answers
10 views

Handle unbalanced data by implementing Edited nearest neighbors, smote and Tomek links in r?

Imbalanced data is a big problem in classification problems. I have a binary classification problem with imbalanced data. I have researched and found that a possible method of dealing with this is ...
1
vote
1answer
36 views

Eigenvalues of covariance matrix are negative

I'm working on the PCA of the mnist dataset, and I get a very strange result, I created a matrix whose rows are flattened mnist images, When I try to compute the eigenvalues of the covariance matrix, ...
0
votes
1answer
13 views

Calculation of relative standard deviation with a custom function in R

I have measured concentrations of elements in a number of samples. Each concentration is an average of three measurements. Also the standard deviation of these measurements is recorded. I tried to ...
1
vote
0answers
34 views

Normalizing and joining of independent logistic regression model's prediction

I need to train several Logistic regression models on a different set of data (with a different set of labels): ...
0
votes
0answers
6 views

Speed up convergence of GLM fitting in custom GLM code (using GD)

So, I've got a little bit of an unusual situation. I do ML on encrypted data, and the ecosystem there is relatively young, so I have to build my own infrastructure, and there are certain restrictions ...
-1
votes
0answers
14 views

What is the difference between statistical learning and deep learning?

statistical learning and deep learning are the sub field of machine learning. But what is the core difference between statistical learning and deep learning and where do they both intersect ? If they ...
1
vote
0answers
46 views

Does Double Median Absolute Deviation work for every distribution for the purpose of outlier detection?

I am using Double Median Absolute Deviation for finding outliers in a 1-D data. As mean with standard deviation gets influenced easily by outliers, that's why I chose median based approach. And the ...
0
votes
0answers
23 views

How to select correlation between object type data for modelling in Data Analysis?

I am analyzing the data with Python. I am at the stage of finding correlation of columns with the target column (price in this example) after data cleaning, to use the columns which have correlations ...
1
vote
2answers
28 views

How do you find the standard deviation of a dataset over a certain period? [closed]

I'm new to R. I could get the std. dev. of the whole data set, ...
0
votes
1answer
15 views

How to compute time-lagged correlation between two variables with many examples at each time t?

I have a dictionary of following form: datetimes = {year : {name : (score1, score2)}} #there are 50+ names/year So, essentially, I'm trying to get an aggregate ...
0
votes
0answers
19 views

A/B Test Machine Learning

I'm thinking about a problem where I have data on different A/B tests and which variant won. The features would be a bunch of one-hot encoded functionalities of each page (i.e., blue button, square ...
1
vote
1answer
32 views

Compare the MAE Vs R^2 output for Machine learning 10-fold CV

I am comparing two 10-fold cv machine learning outputs. I got: For model 1: MAE= 0.252 R^2= 0.447 RMSE= 0.37 For Model 2: ...
0
votes
1answer
16 views

Hot to use the formula model in t.test

I am trying to better understand the formula model for two sample t-tests in R. When I calculate the test in the formula model I get a wrong result. ...
0
votes
1answer
12 views

Is there a difference between Vectorization and Featurization?

I have come across articles that mention Featurization and Vectorization interchangeably. Is there a minute difference between each other or are they exactly referring to the same thing?
1
vote
0answers
25 views

Predicting when a customer will convert

I want to build a model to predict if a customer will convert from a free product to its premium version. In addition, I'm also interested in knowing when she will convert. This is different from the ...
0
votes
0answers
22 views

Can I generate synthetic data after getting approximate distribution parameters using Kolmogorov-Smirnov test?

I have a very limited amount of data. I need to generate more synthetic data . My data shape is (x,21) . Right now , I have used KS (Kolmogorov-Smirnov) test to get the closest matching distribution ...
0
votes
1answer
44 views

Implementing the Trapezoid rule without the formula for the curve

I know that if I have some function f(x) that describes a curve, I can approximate the area under the curve using the trapezoid rule as follows: ...
0
votes
1answer
15 views

An average of averages problem, or two valid results?

From an event log I have derived the days between each user's logins. This leaves me with a dataframe like this: I then took the average of column 'days_between_logins' which gave me a result of 15.9 ...
1
vote
1answer
18 views

Difference between rulefit & skope-rules Python Packages

RuleFit is an implementation of a rule-based prediction algorithm based on the rulefit algorithm from Friedman and Popescu (PDF). skope-rules Skope-rules is a Python machine learning module built on ...
0
votes
0answers
8 views

Finding threshold value for right-skewed data (bivariate)

Intuition of the Problem Suppose you have a dataset of two columns, X and Y, and I plot them using a bar plot. The bar plot shows that there are a lot of values in the first two bars on the left, and ...
0
votes
0answers
13 views

What is the formula used by np.dot?

Dot products are pretty simple for 1- or 2-dimensional arrays, but anything beyond that is incomprehensible to me. I tried looking into numpy‘s dot function but the ...
0
votes
0answers
25 views

Three different errors using external information. Which one makes sense? (Or how to interpret each?)

My goal is to compare clustering methods considering different method and different number of clusters using an external information. Could anyone please give some opinion/ recommend book/paper about ...
1
vote
0answers
45 views

What is the best way to cluster this kind of data?

I have data that looks like this: The chart on the left is the trend and the smaller chart on the right is the box plot showing the distribution of means. Each color is the output of a particular ...
0
votes
0answers
19 views

Precision and Recall grows up together

I evaluate results of an algorithm from analysis of acoustical signal depending on SNR. Algorithm finds right frequencies in spectrum.   The kind of finding of TPs, FPs and FNs (for every time slot) ...
1
vote
1answer
60 views

What is the difference between normalization and re-scaling?

This site does not describe the nature of the normalization tag. How does it differ from re-scaling? Many authors use the two terms interchangeably. I can not understand normalization's operational ...
0
votes
1answer
21 views

Class imbalance and statistics

I have a dataset with 5000 observations with class 0 and 300 observations with class 1. I would like to run some statistical analysis, for example on the average length of strings, the number of words,...
1
vote
0answers
14 views

Is it logical to calculate zscore for list A using the standard deviation and mean from list B?

I am working on an application where we collect the total income of a household. We have a $3^{rd}$ party integration which provides us the household income estimates. I have calculated standard ...
1
vote
1answer
27 views

Combining multiple probabilities from a classifier. Propagating probabilities

Let's say I have trained a classifier that classifies images of animals into 10 different classes. And let's say that I have 20 different images of a particular animal and because I know the ...
0
votes
0answers
10 views

Prove Ridge regression duality of dropout

EDIT: Found one mistake, confused rows and columns in the step of summing over column operations. I'm trying to prove the Ridge regression duality of Dropout, as described in section 9.1 of this paper....
0
votes
0answers
51 views

Is it valid to add MAPE as a margin to prediction output?

I've trained a KNNRegressor on predicting used car prices. A given car's actual selling price is R289,995. My model predicts R260,911. I want to be able to tell the user My knn model predicts the deal ...
0
votes
1answer
21 views
1
vote
0answers
14 views

Derivative of the function of random variable

Suppose we have a function $\phi(X)$ of random variable $X$. Suppose both of $\phi(X)$ and $X$ are random variables. If $\phi$ is differentiable, how to calculate the derivative of $\phi(X)$ w.r.t. $...
0
votes
0answers
23 views

Finding correlation between different columns

I have a dataframe in following shape: ...
0
votes
1answer
25 views

100% Accuracy on test dataset using a previous developed model oputput

My dependent variable is a probability that is sourced from someone else's classification model. I am using this probability as a dependent variable as I don't have the actual data. On building an ...
2
votes
1answer
20 views

Trade-off between variable stability and discriminatory model quality

It is very normal for data scientists and modeling professionals to be concerned with the stability of the model. It basically means that if a variable is important today, it cannot change its ...
0
votes
1answer
49 views

p-value interpretation

Is it correct to say that the lower the p-value is the higher is the difference between the two means of the two groups in t-test? For example, if I apply the t-test between two groups of measurements ...
3
votes
2answers
71 views

How do you use KS-test in a data science report?

I'm writing a data science report, I want to find an exist distribution to fit the sample. I got a good looking result , but when I use KS-test to test the model, I got a low p-value,1.2e-4, ...
0
votes
0answers
6 views

Calculate Variance from Ward distances

The Ward method for clustering minimizes the total within-cluster variance. So I suppose that there is a link between the Ward distances that I got with the linkage function, and the variance of the ...
2
votes
0answers
13 views

Predicting millions of sparse timeseries using them to help each other

This is a very general problem faced by different types of businesses. Predict the future behavior of customers over time. Imagine that we have 1 million customers with their own features over time, ...
0
votes
2answers
44 views

Selecting most important features for multilinear regression

I have a set of 25 features. I would like to choose the best features for my model. Originally, I was looking at the correlation of features with respect to response, and only taking those which are ...
0
votes
1answer
15 views

How to Impute Top-Censored Data

What are the main imputation methods that can be used to deal with top censored panel income data if a big proportion (about 40%) is censored?

1
2 3 4 5
18