Podcast #128: We chat with Kent C Dodds about why he loves React and discuss what life was like in the dark days before Git. Listen now.

Questions tagged [statistics]

Statistics is a scientific approach to inductive inference and prediction based on probabilistic models of the data. By extension, it covers the design of experiments and surveys to gather data for this purpose.

Filter by
Sorted by
Tagged with
0
votes
0answers
6 views

Wavelet transform to identify 'meaningful' data changes

Can I use the wavelet transform to find the 'meaningful' changes in a series of 1d data? For example I have a series of integers representing angles of line segments. The majority of these values are ...
0
votes
0answers
13 views

Statistical loss function for categorical distributions

For training an autoencoder model whose outputs (and inputs) are parameters from a categorical distribution $[q_1, q_2, \ldots, q_n]$, I have to define a proper loss function measuring the distance ...
0
votes
1answer
28 views

confusion on outliers

I am not able to distinguish the outliers - When to go with std. dev and When do we need to go with Median. My understanding on std. dev. is - if the data is away from mean by more than 2 std dev. we ...
1
vote
1answer
25 views

Improving the performace of the Naive Bayes classifier by decorrelating the data

I was wondering if it is possible to improve the performance of the Naïve Bayes classifier by decorrelating the data. The Naïve Bayes assumes conditional independence of the features given some class $...
0
votes
1answer
24 views

Stata: Trying to merge two data sets with common year observations

I am currently trying to merge two datasets with a common "year" variable using Stata. One dataset is purely for incorporating CPI with an associated year, and so the only variables are "year" and "...
0
votes
1answer
20 views

Customer Segmentation: Should I use a variable, representing a product, that is unpopular in the dataset for K-Means Clustering?

I am working with a data set that, besides customer age and income, tells the balance a customer has in different type of bank accounts: Checking, Shares, Investment, Savings, Deposit, Mortgage, Loan, ...
0
votes
1answer
13 views

Find changes in variables into two states

I have a dataframe like this: ...
0
votes
0answers
10 views

What are accepted methods for sampling data that will be sampled?

I have a very large (TBs) dataset that I need to sample to a smaller more manageable size. However, I know that the sample will also be sampled to get statistics after it leaves my hands. Are there ...
0
votes
1answer
24 views

What method is recommended after outliers removal?

I have a data of mice reaction times. In every session, there are some trials in which the mouse "decides of a break" and responds after a long time to these specific trials. I was thinking of ...
0
votes
0answers
6 views

Hypothesis Test for Classifier, but without the original model

I just trained a number of DenseNet-based CNN models using an optimization technique that I created for my master dissertation. The problem is, my advisor wants me to perform a hypothesis test to ...
1
vote
0answers
8 views

Hidden markov model to detect how loaded a die is?

I was wondering if you could make a hidden markov model to decide what the optimal loaded ness of a die is. I know normally you would want to do something like take a prior like I know of two types of ...
0
votes
2answers
58 views

ML, Statistics and Mathematics

I have just started getting my hands wet in ML and every time I try delving deeper into the concepts/code, I face the challenges of the mathematics and its cryptic notations. Coming from a Computer ...
0
votes
1answer
16 views

Is consecutive multiple transformation of Y variable a valid approach statistically?

I am going to run a regression analysis. But my response variable is highly right-skewed Firstly I used a log(x+1) transformation and it increased the situation dramatically. But there is still right-...
0
votes
1answer
18 views

Covariates in machine learning classificatoin

I would like to classify a number of people as sick and healthy. I have all the body measures as features. I have to control for age and sex because body features are affected by that. How do I adjust ...
0
votes
0answers
11 views

When to use t-distribution instead of normal distribution?

According to the Student's t-distribution wiki article the t-distribution is used instead of the Normal distribution "when estimating the mean of a normally distributed population in situations where ...
2
votes
1answer
47 views

Analysis of sales performance

This is my first question on this stackexchange - i hope i'm in the right place and i'm asking the right question. I work for a business, we sell certain products. We are running a trial currently on ...
1
vote
1answer
19 views

How to interpret ANOVA results?

I am trying to identify what attributes are not relevant in my dataset to remove them before fitting a classifier. The target is a categorical variable with three different values. I also have a lot ...
0
votes
0answers
33 views

Data Analysis on normally distributed salary with Outliers

I have created normally distributed data with 2 outliers ...
3
votes
2answers
202 views

Need machine learning algorithm to fill in time-series data

I am currently dealing with a time-series data set with cyclical gaps every 30 minutes (30 minutes of data, 30 minutes of no data). Is there a relatively simple way of using scikit-learn (or some ...
0
votes
0answers
17 views

How do I get confidence intervals for an ElasticNet in sklearn?

I need to produce a row for the confidence interval for every field that I am calculating coefficients and scores off of. So here is my code so far- ...
0
votes
0answers
11 views

questions about the positive predictive value from a classifier output

[This is in the context of machine learning applied to drug discovery.] Suppose you have trained a classifier with a set of chemical compounds of known active $(A)$ vs inactive $(I)$ class. The model ...
0
votes
0answers
23 views

Machine learning in the social sciences

I have been working in the transportation sciences(demand forecasting) and it’s social implications for a good 16 years now with the last 3 years focusing on building and deploying machine learners. ...
1
vote
1answer
10 views

suggestion for good online source according to Syllabus

Hi, are there some online courses e.g. some classes in Coursera? I have difficulty in following the professor's teaching because I have a weak statistics background. I want to catch up by reading ...
0
votes
1answer
10 views

Why is the variance going down so much in this weight initialization problem(using pytorch)?

first look at this example >>> x = t.randn(512) >>> w = t.randn(512, 500000) >>> (x @ w).var() tensor(513.9548) it makes sense that ...
1
vote
1answer
39 views

Finding similarity between two datasets

I have two datasets. One is actual percentage of white population in counties in an american state and the other is the simulated percentage of white population in counties in an american state. ...
1
vote
0answers
16 views

How to apply transfer learning on a regression problem?

I am working on predicting mechanical failures and I have trained a model to predict when a component will fail on what type of machine. I would like to now use this prediction model to predict the ...
0
votes
2answers
27 views

A/B testing with non-Gaussian distributions

I have two sets of samples (A, B) with a relatively high number (~10,000) and I want to see if a factor has affected sample B or not. Naturally, I should use A/B testing. The problem is, the ...
0
votes
1answer
38 views

Normalize / Standardize in a Random Forest?

If I have a matrix of co-occurring words in conversations of different lengths, is it appropriate to standardize / normalize the data prior to training? My matrix is set up as follows: one row per ...
0
votes
2answers
60 views

cross validation issues

I have come here from this great answer. I have come across many approaches for using cross validation and the answer to the attached question is by far explaining it the best to me. My dilemma is ...
0
votes
0answers
20 views

Is it correct to say that a high variance feature will be important to my model?

Analysing 3 features I was faced with the following situation: Two of them have almost no variance and the last one have a greater variance than the other (almost two times). Is it correct to assume ...
1
vote
0answers
16 views

How can I measure data anonymity when present summary statistics

I am looking to present users with aggregate statistics of a highly sensitive underlying dataset. See example data below: ...
0
votes
0answers
10 views

Generating correlated time series with different probability distributions

I have the following problem to solve: I need to generate two time series, each modeling a variable (lets call them VarA and VarB). For each of these variables, I have a PDF (PdfA, PdfB) which belongs ...
0
votes
0answers
24 views

Feature Engineering Time Lagged Features

Suppose that you are trying to model the following situation - each of your features is a separate time series, as is the target variable. However, the feature time series occur before the target time ...
2
votes
0answers
27 views

How exactly do probability distributions help modeling/making decisions?

I am an elementary/wanna-be statistician/data scientist from South Korea. I have been studying a variety of theories of mathematical statistics and different probability distributions. (I apologize ...
0
votes
0answers
6 views

Log or StandardScaler for 0.0:1.0 values

If I wanted to normalize a bunch of values that were between zero and 1, where some values were dangerously close to zero such as 0.00001 and others were magnitudes larger like 0.8 - would I take the ...
1
vote
0answers
19 views

How to deal with similar feature values but each indicates to a different information?

If I have a feature with replicated values but each of these values indicates a different piece of information. example: feature 'street name' with value 'A' which some of these 'A's are for Boston ...
0
votes
1answer
12 views

Statistical test to infer if brand preference depends on income

For the data in image can anyone please suggest the right statistical test. Want to test if brand preference depends on income level. The percentages adds upto 100% row wise meaning the values in a ...
1
vote
4answers
62 views

What is the intuition behind larger number of samples are better for statistics?

It may be well-known that when we take statistics, we essentially need a large number of samples. Because I am taught this fact before studying the math, I have been here without exploring the reason. ...
0
votes
0answers
15 views

Calculating co-variance matrices and plotting the co-variance value

I have an excel file with 10 samples drawn from a two-dimensional normal distribution. There's 20 columns. Sample 1 would be column 1,2. Sample 2 = col 3,4 ect. I'm trying to plot 10 separate ...
0
votes
0answers
9 views

How to estimate claims for warranty analytics where continuous improvement reduces warranty claims?

I have a situation where a company who has warranty claims on its product and they want to know that is it possible for them to extend that. Now the case is that they do improvement on the existing ...
-1
votes
1answer
17 views

How to interact two variables in python?

I have about 8 features as my predictors in a logistic model I am trying to fit in python. One of the features is TotalAward (Financial Aid) and the second is NEED. I am attempting to predict the ...
0
votes
0answers
7 views

Bayes factor values

I have two samples A and B with continuous values such as results students. I want to calculate the baysian t test to get the Bayes values. I get the Bayes value of 0.16,so does it mean the null ...
0
votes
0answers
8 views

pattern recognition tool

From what i've seen while searching google and the site this is most likely way simpler than what normally get's asked but i've not been able to find anything that could help me with it. What i'm ...
0
votes
1answer
35 views

How to test hypothesis?

I have a table named app_satisfaction which has user_id, satisfaction, # of people they've invited. I've grouped by satisfaction and found that on averge. people in satisfaction ="BAD" group invted 2....
0
votes
0answers
11 views

Prediction interval and probability to increase

Let be $y_i$ some observed values of a given time-series. We denote $\hat{y}_i$ the corresponding predicted values. We also assume to have a prediction interval $p_i$ such that $\hat{y}_i \in p_i = [\...
0
votes
0answers
22 views

Bayesian test for classification problems?

I am using two classification algorithms in Weka i.e. Logistic Regression and Naive Bayes and want to know which algorithm has better performance? I need a statistical test like Bayesian, so which ...
0
votes
1answer
27 views

How to perform logistic regression in R with grouped data?

Say I have a categorical variable with 2 levels, and for instance 289 people fall in level 1 and 386 fall in level 2. How would I go about modelling such a variable as a logistic response? All ...
2
votes
0answers
35 views

difference between normal skewed distribution and skewed distribution

From what I've read normal skewed is a distribution that has all the properties of normal distribution and is skewed: according to this resource:https://www.statisticshowto.datasciencecentral.com/...
1
vote
0answers
9 views

What would be a good threshold number to convert some categories to Not Specified

The current topic is derived from this topic: https://community.rstudio.com/t/39722 (optional to read). I have the following dataset: myds : ...
0
votes
0answers
28 views

Exploratory Data Analysis for Big Data with continuous and categorical variables (mixed data types)

I have a dataset with 28 variables (6 continuos + 22 categorical). I'm using ...