Questions tagged [statistics]

Statistics is a scientific approach to inductive inference and prediction based on probabilistic models of the data. By extension, it covers the design of experiments and surveys to gather data for this purpose.

Filter by
Sorted by
Tagged with
0
votes
0answers
8 views

Descriptive statistics shows median=0 and wilcoxon test p value shows median is not equal to 0

I have a dataset. I wanted to do paired t test on it. So I carried out normality test and it showed that it does not follow normal distribution. So I used Wilcoxon test in place of paired t test. The ...
1
vote
1answer
18 views

Machine Learning with intended missing values

I have a dataset relating to humans completing reviews, the target variable is whether the review decision is correct / incorrect and one of my features is a trailing 4 week accuracy score for the ...
1
vote
1answer
21 views

Creating features from raw accelerometer data

I have a dataset containing raw 3-axis accelerometer data collected from a users lower leg and I want to create a classification model (as simple as possible) that detects if the user is sat down or ...
0
votes
1answer
19 views

Terminology of time series

The terms Time Series Analysis, Time Series Forecasting, and Time Series Modeling are widely ...
-1
votes
1answer
26 views

Is 50% accuracy on a 4 label multiclass classifier good?

As the title says, I have a classifier with 4 labels. I am having trouble getting much above 50% accuracy in Predicting labels. I made sure the data and test sets are made up of approximately 25% of ...
0
votes
0answers
13 views

estimate user's satisfaction of a video based on how much of it they watched - normalization

I am trying to estimate how much a user liked a video using how much of the video they watched. Let's say, on the scale of 1 to 10, 1 means that the user didn't like it at all, and 10 means they ...
1
vote
1answer
23 views

train test data split up in datasets

In a dataset consisting of 1,000 samples, it has been shown that a 70-30 split (i.e. 70% of the samples used for training, 30% for validation) will provide a good estimation of the test accuracy of ...
1
vote
0answers
11 views

analyze the effect of some new changes to business rules on customers retention and sales

I am trying to analyze the effect of a particular business rule on customer behavior. Background: I have two call centers operating in my company. One is an in-house call center and the other one is a ...
1
vote
0answers
18 views

Suitable modelling approach to incorporate different signals

I am working with data in the form of content that is being reviewed to decide if it violates the policy standards or not. In the review process there are several methods to increase the chances of ...
2
votes
0answers
22 views

Non-Gaussian like distributions - Classifier of source data fails on target data

I ask you for help on a classification problem (classes are represented by the numbers 0,1 and 2). All features are extracted from time series data (fundamental is sinus shape). I have a source ...
2
votes
0answers
23 views

Showing standard deviation for training curve

I am training a neural network and I wanted to plot the evolution of different metrics (MSE…) during training. To get an idea of the variations between between different trainings, I am using several ...
0
votes
0answers
6 views

Autocorrelation of two functions multiplied and raised to arbitrary powers

Given a signal $A$ and a signal $B$ with autocorrelation times of $\tau_A$ and $\tau_B$, respectively, where $\tau_A > \tau_B$, is there any general statement that can be made about the ...
0
votes
1answer
21 views

Implication of Central Limit theorem and its application in machine Learning

According to central limit theorem, irrespective of the distribution of the original/population dataset, the sampling distribution will follow normal distribution. My Question is if the means of n ...
1
vote
1answer
41 views

Linear transformation from one sample to another

Generate a Sample $\underline{Z_1}$ $\underline{Z_2}\dots \underline{Z_{5000}}$ , while $\underline{Z_i} \sim N_2[(0,0)^T,I_2]$ generate new sample with size of $ n = 5000$ by applying linear ...
4
votes
2answers
88 views

Maximum Likelihood estimation

Given a sample $ X_1,X_2 \dots X_{100}$ and the density function $ f(x;\theta) = \frac{1}{\pi \cdot \left(1+\left(x-\theta \right)^2\right)}$ , find an approximate solution for $\hat{\theta}_{MLE.}$...
0
votes
1answer
22 views

How to generate and visualize results of all possible parameter values in Python?

I have two parameters: demand, ticket_qty. demand is an integer and can have a possible ...
0
votes
2answers
31 views

How to find if any variable which is normally distributed in pandas dataframe [closed]

Im having age column and how can i plot and say this is normally distributed or not
0
votes
0answers
21 views

Ranking metrics: weighted and different

I have 2 ranking lists and I want to see how similar are they between them. setA = {10,9,8 ,7 ,6,5,4,3,2,1} setB = {4,5,12,14,9,8,7,6,2,1} There are the following constraints: 1. High number (or if ...
1
vote
1answer
18 views

Ranking of feature sets which will be used for binary classification

I have some feature sets say X1 and X2 ... Each feature set have some variable amount of features and there is no intersection between different feature sets Say X1 have 100 features and X2 have 500 ...
1
vote
0answers
8 views

How do you calculate how many coefficients are necessary in polynomial regression?

So I can't seem to find much on this by searching so I came here. Let's say I had 3 variables $x_1,x_2,x_3$ and the let's say the degree of the polynomial was $d=2$, I can define the length of a ...
0
votes
0answers
9 views

In a dataset, how to detect simple relationship between two columns (datatype not restricted to numerical values)?

Suppose, I have a dataset with the following features: (FirstName, LastName, FullName, SchoolJoiningDate, SchoolGraduationDate) My aim is to validate the data before feeding it into the ML pipeline. ...
1
vote
0answers
21 views

Show distribution of users affected by outlier response times

My dataset is performance metrics (response time) of a web page over the course of a single day. The data looks roughly like this: ...
2
votes
1answer
17 views

Comparing youtube view counts between historical release dates

I've written a script to use youtube-dl to collect all view-counts and ratings from all listed videos on a youtube channel: https://gist.github.com/philshem/...
0
votes
0answers
14 views

Multi-Collinearity in Classification Problems

I have a "Small data matrix" of scraped data from multiple websites trying to account for sentiment towards certain cellphone models. The matrix includes independent variables such as "...
0
votes
0answers
19 views

Classfication report

I have a simple question about classification reports and validation generator in keras. In my case each number in valid_generator.classes corresponds to an artist. But how do i know which number each ...
0
votes
0answers
16 views

Ideas on handling large quantities of “waste” data

Hello and thank you for taking time out of your day to help me. I am currently working on developing a machine vision application for production monitoring. The application handles images, about 20,...
2
votes
0answers
18 views

What branch of data science to use for revealing preceding conditions of desired outcome

On financial data series, I would like to determine if some conditions precede a desired outcome. For example, let's take as an example the daily data of a stock. On some days, the stock experience ...
0
votes
2answers
24 views

How effective would a data-driven vaccination program be over a simple rules based approach

The rollout of vaccination programs tends to be based on a given set of rules (advice) devised by government UK example. Such a rule-set are generally limited in complexity in order to be consistently ...
1
vote
1answer
46 views

How to implement conditional probability distribution on set-valued Random variables

I'm trying to implement conditional probability distribution when the events of two RVs are sets. If I try to extrapolate concepts from real or categorical variables to sets things become confusing ...
1
vote
0answers
11 views

In classical kernel regression, is there a task which responds almost exclusively to a single kernel choice?

I'm curious if there is any well-known kernel regression/classification task which can only be "solved" using a specific choice of kernel?
0
votes
0answers
38 views

How to set thresholds to automate the management of model drift?

I have a trained ML Model and new data coming in every week. The new data sometimes vary too much (in a statistical sense) and, therefore, the performance of the ML model degrades. The data-set schema ...
0
votes
0answers
16 views

F-Value P-Value T-Value linear regression interpretation

The following is what I found and understood and put in use without really understanding the logic behind it! The t-value measures the size of the difference relative to the variation in your sample ...
1
vote
1answer
24 views

Size of datasets over years

I am looking for statistics, to understand the evolution of the size of the (public) dataset over the years. I just found the following statistics: The poll of KDnuggets that actually shows that over ...
1
vote
1answer
35 views

What is the appropriate statistical significance test for multi-class classification?

I have a multi-class classification problem. I am primarily using macro-average F1 measure to evaluate the performance of models and want to verify if the results are statistically significant. I have ...
1
vote
1answer
24 views

Relationships between groups of features against independent variables

I have several groups of features that I'd like to test against independent variables. The idea is to find which groups tend to be associated with a specific value of an independent variable. Let's ...
0
votes
2answers
44 views

What pandas function is most similar to the transpose procedure in SAS?

I am trying to transpose Amt by Txn_ID across Item_Desc in Python using pandas. Which pandas ...
4
votes
2answers
142 views

Was this dataset analysed correctly? [closed]

There's a Twitter thread going around that claims there are signs of voter fraud due to anomalies in the election vote count data set. You can download the dataset here and find the script to generate ...
1
vote
0answers
21 views

A/B Testing (Binomial Distribution vs Random Distribution)

When performing an A/B test for the number of clicks for users viewing (each view is an impression) two variants of an ad, a binomial distribution can be assumed where each variant has a constant ...
0
votes
1answer
33 views

How to Include Features that Apply to Specific Classes

I'm predicting hours that will be worked for building tasks. Due to the overall low sample size, I've stacked multiple related tasks together into a single model. (There may be 100 total samples in a ...
1
vote
0answers
13 views

Multivariate noise variance in Gaussian process prediction

In GP regression, we predict using $\mu^* = ... (K(X,X)+\sigma^2I)^{-1}...$ This is fine when the noise $\sigma$ is a scalar, but I am confused about what happens when $\sigma$ is Multivariate/...
0
votes
0answers
36 views

Hypothesis Testing - Median Test for linear regression model given standard error and co-efficient

I need help approaching this particular problem on hypothesis testing. A researcher believes that a diet high in raw cornstarch would increase the median blood glucose levels by 20. The diet can ...
1
vote
1answer
31 views

What is a distribution-wise asymmetric measure?

I was trying to understand KL-Divergence, $$D_{KL} \langle P(X) \Vert P(Y) \rangle,$$ and was going through its Wikipedia article. It says the following In contrast to variation of information, it is ...
1
vote
1answer
16 views

Maximum Likelihood with Gradient Descent or Coordinate Descent blows up

Context The maximum likelihood estimators for a Normal distribution with unknown mean and unknown variance are $$ \widehat{\mu} = \frac{1}{n}\sum_{i=1}^n x_i \qquad \text{and} \qquad \widehat{\sigma}^...
0
votes
1answer
16 views

making conclusions after sentiment analysis

After performing some sentiment analysis, I have a dataset that looks like this: For different products, using online reviews, I have obtained some values for positive/negative sentiments. However, ...
0
votes
0answers
9 views

Predicting an annual event – modeling on an annual or monthly basis?

Suppose I'm interested in predicting which of my current customers are likely to renew their insurance at some point in the year. The renewal can happen at any time in the year. I want to proactively ...
0
votes
1answer
31 views

R code that gives results like Wolfram Alpha for the expectation of a function of a random variable? [closed]

When I ask Wolfram Alpha to calculate $E[f(X)]$ where $f(x) = e^{-x^2}$ and $X \sim \mathcal{N}(1,4)$, it gives the result $$ E[f(X)] = \frac{1}{3\sqrt[9]{e}} \approx 0.29828, $$ and the following ...
1
vote
1answer
33 views

Click Through Rate calculation (CTR) calculation problem

So I'm doing a use case for a company interview and one of the questions is to calculate the CTR for a sorting algorithm. My question would be: Should I remove the operations where there were no ...
0
votes
0answers
11 views

Analysing wrongly predicted labels and pick most significant

I try to do behavior analysis of network systems based on network flow. I have a model that predicts if a certain flow pattern is a Windows Server, LX Server or Windows Client. (Highly simplified) I ...
0
votes
0answers
13 views

How many positive responses are good enough for building a marketing response model when the response rate is low(0.5%)

We are planning a marketing campaign to collect data and the response rates for a random sample. Total population size is 10 million and historically, response rates are low (0.5 - 0.65 %). How long ...
1
vote
0answers
15 views

Normalise the scores by small and large numbers division

I am working on a scoring problem use-case where I will score each task based on few aspects and take an average of scores of tasks in each experiment. For each experiment I will have 1 to N number of ...

1
2 3 4 5
17