Questions tagged [statistics]

Statistics is a scientific approach to inductive inference and prediction based on probabilistic models of the data. By extension, it covers the design of experiments and surveys to gather data for this purpose.

Filter by
Sorted by
Tagged with
0
votes
0answers
9 views

Bradley-Terry for pairwise comparison with confidence levels

For a scientific study that I'm currently working on, I will evaluate three different configurations (A - B - C). These configurations will be evaluated in a pairwise comparison fashion (A-B, A-C, B-C,...
0
votes
1answer
22 views

Evaluating likelihood of egg breaking when falling in random container on concrete [closed]

I am working on a project where I would like to predict whether an egg will break if it is put in a container that is then dropped on concrete. I am looking at the different factors that play a role ...
0
votes
0answers
11 views

Structural Risk Minimization: Inequality in “Foundations of Machine Learning”

I am reading the chapter about structural risk minimization in Mohri's book. However, I have trouble understanding the derivation of inequality 4.5 on page 83. More precisely, why is $$\sum_k \...
0
votes
1answer
7 views

Understanding AC_errorRate loss function

I'm reading an article about Rolling Window Regression: a Simple Approach for Time Series Next value Predictions. He explains about 5 different loss functions. I managed to understand the first four, ...
2
votes
1answer
22 views

Generating random numbers from best probability distributions?

I have done some statistical distributions on real-world data. The distribution fitting gives me the below results (lognormal fits the best on data based on chi-squared test): ...
2
votes
1answer
29 views

Diff. in P-value & F-Stat. Multiple linear regression

Even if we have individual p-values for each predictor. Why do we need overall F-statistic? I read this solution but I am not sure if I get it right. Can someone please explain? Source: "An ...
0
votes
1answer
23 views

Customer Intelligence - How to measure success?

we are creating models that aim to filter new leads from our current customer base. We started to create propensity models that calculate a percentage for each customer for a certain product group. I ...
1
vote
1answer
32 views

Is there a common strategy to measure if a difference-significance of two areas under two ROC curves

I conduct sound detection experiments with mice. I have a stimulus sound and a "noise" sound that shoukd be ignored. I want to measure how well the mouse ignors the noise (with respect to, say, ...
4
votes
1answer
37 views

Naive Bayes implementation: why Laplace smoothing is different from theory?

Let's have a Naive Bayes Bernoulli classifier with $n_C$ classes and $n_F$ features. According to the formula in here and here and almost every theory book I could see, Laplacian smoothing means that ...
0
votes
0answers
7 views

What is an appropriate statistical method for determining correlations between variables with likert scale data?

I am conducting an exploratory analysis with a dataset of about 200 records (all likert scales (numeric)). I aim to determine the correlations between the different variables that the responses ...
1
vote
0answers
12 views

should I normalise data of disparate frequencies

The dataset consists of rows of time series data. Most of the time data remains constant and only updates when there are significant changes. Each time series looks something like this where each ...
1
vote
0answers
21 views

Hypothesis Testing- Independent variable Importance

I am learning Data Science and I have a confusion in one topic. I would like to describe my approach. I have understood the problem statement. I have used snowflake schema to join tables that have ...
0
votes
1answer
25 views

How does $\chi^2$ feature selection work?

I can't find the information how $\chi^2$ are used to select numerical features for a model. ...
2
votes
0answers
11 views

Error when drawing random numbers from a custom continuous distribution using scipy.rv_continuous

I am trying to generate a sample of random numbers from a custom distribution $$ p(x) = x^{n}e^{-xtn}. $$ After reading the tutorial on scipy's website, I wrote a subclass which I called kbayes: <...
0
votes
0answers
10 views

Wavelet transform to identify 'meaningful' data changes

Can I use the wavelet transform to find the 'meaningful' changes in a series of 1d data? For example I have a series of integers representing angles of line segments. The majority of these values are ...
0
votes
0answers
33 views

Statistical loss function for categorical distributions

For training an autoencoder model whose outputs (and inputs) are parameters from a categorical distribution $[q_1, q_2, \ldots, q_n]$, I have to define a proper loss function measuring the distance ...
0
votes
1answer
31 views

confusion on outliers

I am not able to distinguish the outliers - When to go with std. dev and When do we need to go with Median. My understanding on std. dev. is - if the data is away from mean by more than 2 std dev. we ...
1
vote
1answer
26 views

Improving the performace of the Naive Bayes classifier by decorrelating the data

I was wondering if it is possible to improve the performance of the Naïve Bayes classifier by decorrelating the data. The Naïve Bayes assumes conditional independence of the features given some class $...
0
votes
1answer
26 views

Stata: Trying to merge two data sets with common year observations

I am currently trying to merge two datasets with a common "year" variable using Stata. One dataset is purely for incorporating CPI with an associated year, and so the only variables are "year" and "...
0
votes
1answer
22 views

Customer Segmentation: Should I use a variable, representing a product, that is unpopular in the dataset for K-Means Clustering?

I am working with a data set that, besides customer age and income, tells the balance a customer has in different type of bank accounts: Checking, Shares, Investment, Savings, Deposit, Mortgage, Loan, ...
0
votes
1answer
13 views

Find changes in variables into two states

I have a dataframe like this: ...
0
votes
0answers
10 views

What are accepted methods for sampling data that will be sampled?

I have a very large (TBs) dataset that I need to sample to a smaller more manageable size. However, I know that the sample will also be sampled to get statistics after it leaves my hands. Are there ...
0
votes
1answer
28 views

What method is recommended after outliers removal?

I have a data of mice reaction times. In every session, there are some trials in which the mouse "decides of a break" and responds after a long time to these specific trials. I was thinking of ...
0
votes
0answers
7 views

Hypothesis Test for Classifier, but without the original model

I just trained a number of DenseNet-based CNN models using an optimization technique that I created for my master dissertation. The problem is, my advisor wants me to perform a hypothesis test to ...
1
vote
0answers
8 views

Hidden markov model to detect how loaded a die is?

I was wondering if you could make a hidden markov model to decide what the optimal loaded ness of a die is. I know normally you would want to do something like take a prior like I know of two types of ...
0
votes
2answers
73 views

ML, Statistics and Mathematics

I have just started getting my hands wet in ML and every time I try delving deeper into the concepts/code, I face the challenges of the mathematics and its cryptic notations. Coming from a Computer ...
0
votes
1answer
18 views

Is consecutive multiple transformation of Y variable a valid approach statistically?

I am going to run a regression analysis. But my response variable is highly right-skewed Firstly I used a log(x+1) transformation and it increased the situation dramatically. But there is still right-...
0
votes
1answer
18 views

Covariates in machine learning classificatoin

I would like to classify a number of people as sick and healthy. I have all the body measures as features. I have to control for age and sex because body features are affected by that. How do I adjust ...
0
votes
0answers
13 views

When to use t-distribution instead of normal distribution?

According to the Student's t-distribution wiki article the t-distribution is used instead of the Normal distribution "when estimating the mean of a normally distributed population in situations where ...
2
votes
1answer
48 views

Analysis of sales performance

This is my first question on this stackexchange - i hope i'm in the right place and i'm asking the right question. I work for a business, we sell certain products. We are running a trial currently on ...
1
vote
1answer
20 views

How to interpret ANOVA results?

I am trying to identify what attributes are not relevant in my dataset to remove them before fitting a classifier. The target is a categorical variable with three different values. I also have a lot ...
0
votes
0answers
33 views

Data Analysis on normally distributed salary with Outliers

I have created normally distributed data with 2 outliers ...
3
votes
2answers
214 views

Need machine learning algorithm to fill in time-series data

I am currently dealing with a time-series data set with cyclical gaps every 30 minutes (30 minutes of data, 30 minutes of no data). Is there a relatively simple way of using scikit-learn (or some ...
0
votes
0answers
21 views

How do I get confidence intervals for an ElasticNet in sklearn?

I need to produce a row for the confidence interval for every field that I am calculating coefficients and scores off of. So here is my code so far- ...
0
votes
0answers
12 views

questions about the positive predictive value from a classifier output

[This is in the context of machine learning applied to drug discovery.] Suppose you have trained a classifier with a set of chemical compounds of known active $(A)$ vs inactive $(I)$ class. The model ...
0
votes
0answers
24 views

Machine learning in the social sciences

I have been working in the transportation sciences(demand forecasting) and it’s social implications for a good 16 years now with the last 3 years focusing on building and deploying machine learners. ...
1
vote
1answer
10 views

suggestion for good online source according to Syllabus

Hi, are there some online courses e.g. some classes in Coursera? I have difficulty in following the professor's teaching because I have a weak statistics background. I want to catch up by reading ...
0
votes
1answer
11 views

Why is the variance going down so much in this weight initialization problem(using pytorch)?

first look at this example >>> x = t.randn(512) >>> w = t.randn(512, 500000) >>> (x @ w).var() tensor(513.9548) it makes sense that ...
1
vote
1answer
40 views

Finding similarity between two datasets

I have two datasets. One is actual percentage of white population in counties in an american state and the other is the simulated percentage of white population in counties in an american state. ...
1
vote
0answers
16 views

How to apply transfer learning on a regression problem?

I am working on predicting mechanical failures and I have trained a model to predict when a component will fail on what type of machine. I would like to now use this prediction model to predict the ...
0
votes
2answers
35 views

A/B testing with non-Gaussian distributions

I have two sets of samples (A, B) with a relatively high number (~10,000) and I want to see if a factor has affected sample B or not. Naturally, I should use A/B testing. The problem is, the ...
0
votes
1answer
46 views

Normalize / Standardize in a Random Forest?

If I have a matrix of co-occurring words in conversations of different lengths, is it appropriate to standardize / normalize the data prior to training? My matrix is set up as follows: one row per ...
0
votes
2answers
60 views

cross validation issues

I have come here from this great answer. I have come across many approaches for using cross validation and the answer to the attached question is by far explaining it the best to me. My dilemma is ...
0
votes
0answers
21 views

Is it correct to say that a high variance feature will be important to my model?

Analysing 3 features I was faced with the following situation: Two of them have almost no variance and the last one have a greater variance than the other (almost two times). Is it correct to assume ...
1
vote
0answers
17 views

How can I measure data anonymity when present summary statistics

I am looking to present users with aggregate statistics of a highly sensitive underlying dataset. See example data below: ...
0
votes
0answers
10 views

Generating correlated time series with different probability distributions

I have the following problem to solve: I need to generate two time series, each modeling a variable (lets call them VarA and VarB). For each of these variables, I have a PDF (PdfA, PdfB) which belongs ...
0
votes
0answers
24 views

Feature Engineering Time Lagged Features

Suppose that you are trying to model the following situation - each of your features is a separate time series, as is the target variable. However, the feature time series occur before the target time ...
2
votes
0answers
27 views

How exactly do probability distributions help modeling/making decisions?

I am an elementary/wanna-be statistician/data scientist from South Korea. I have been studying a variety of theories of mathematical statistics and different probability distributions. (I apologize ...
0
votes
0answers
7 views

Log or StandardScaler for 0.0:1.0 values

If I wanted to normalize a bunch of values that were between zero and 1, where some values were dangerously close to zero such as 0.00001 and others were magnitudes larger like 0.8 - would I take the ...
1
vote
0answers
22 views

How to deal with similar feature values but each indicates to a different information?

If I have a feature with replicated values but each of these values indicates a different piece of information. example: feature 'street name' with value 'A' which some of these 'A's are for Boston ...