Questions tagged [statistics]

Statistics is a scientific approach to inductive inference and prediction based on probabilistic models of the data. By extension, it covers the design of experiments and surveys to gather data for this purpose.

Filter by
Sorted by
Tagged with
0
votes
0answers
18 views

How to detect anomalies (errors and exceptions) in log files?

Is this a good approach? So I'm working on a Root Cause Analysis system which should help find the cause/the root error of failed system builds (packaged in a tarball), through the analysis of log ...
0
votes
0answers
13 views

R: analysis of conditional heteroscedasticity using McLeod Li test

I am using R for McLeod Li test for conditional heteroscedasticity on time series before and after boxcox transformation. The null hypothesis for this test is of no heteroscedasticity. I observed that ...
0
votes
0answers
11 views

Capturing meaningful changes in random variables

Suppose I have random variable A: representing Age of a person and E: representing education level of a person. I wish to construct a metric that could signify how meaningful the change to a given ...
0
votes
0answers
7 views

R: Error using Shapiro Wilk test for multivariate normality

I want to test if my data is multivariate normal. For this, I am using mvnnormtest package and Shapiro-Wilk multivariate normality test. I am passing the data to function as: ...
0
votes
0answers
5 views

R: Breusch-pagan test heteroscedasticity before and after boxcox transformation

I am trying breusch-pagan on a a feature regressed using multiple variables as: bptest(lm(V1~V2+V3+V4+V5+V6+V7+V8+V9+V10+V11+V12+V13+V14,data=data)) It shows BP ...
0
votes
1answer
5 views

R: Breush-pagan test for complete dataset

I am trying breush-pagan test for testing heteroscedasticity in timeseries of 10 features using R package lmtest. This package lets you test heteroscedasticity in two variables of which one is ...
1
vote
1answer
13 views

Word2Vec / Doc2Vec - map change of words overtime

I have text data for 10 years and I want to run a Word2Vec model on the text data each year and calculate the distances between words and I would like to compare them over time. Are there any papers ...
1
vote
0answers
27 views

Using neural nets with just one input variable (without response/feature) [on hold]

Hi i am using following data from R library ftnonpar: ...
1
vote
1answer
39 views

Variance in statistics vs machine learning

In basic statistics, variance is a measure the variability of the data about its mean. In machine learning, variance is a measure of learning the training data too well/capturing the noise in the data/...
1
vote
1answer
27 views

RMSProp - To a layman?

Can someone explain RMSprop in layman's terms? I have tried reading various resources but they don't talk about the intuition on couple of things: 1) Why is is different from SGD with momentum? 2) ...
0
votes
0answers
6 views

Is it worth graphing a correlation plot between your features and your target?

Is it worth graphing the correlation between the most important features and the target variable after you have done either PCA or L1 Regularization to identify the most important features? I guess ...
0
votes
1answer
14 views

Making Inference from a Correlation Heatmap

I have constructed a Heatmap of my dataset for visualization. I have searched on various sites regarding what can I infer from the heatmap and I am unable to get any clear understanding from them. I ...
0
votes
1answer
20 views

Simple EDA: what to visualize?

I want to visualize a data set for EDA having 8 parameters and 2 class labels. I am confused which parameters to select and on what basis? Also is I want to know whether it would be better to apply ...
0
votes
1answer
27 views

Effect of skewness in data

I am preparing classification model. Many of numeric variables are positives skewed. Should I change a distribution of variables to be more Gaussian?
1
vote
0answers
11 views

Testing accuracy has very high accuracy metrics on epoch 1 but decreases rapidly on following epochs

For a binary classification problem my testing accuracy metrics are very high on epoch 1 but it decreases rapidly on further epochs, training trends are similar with epoch 1 having low accuracy ...
0
votes
0answers
28 views

How to fix “'The `start` argument could not be matched to a location related to the index of the data.” Error?

When I try to predict the results using ARIMA for a specific train/test split, its throwing an error like this: "'The start argument could not be matched to a ...
0
votes
1answer
49 views

Why do you need to learn statistics for machine learning?

This maybe a very broad question. Please do not delete. However, I am curious on why do we need to learn statistics for Machine Learning? This is given that the software libraries used already have ...
0
votes
1answer
23 views

Calculate probability vector from sample data

I'd like to compare two distributions using Jensen-Shannon Divergence metric. To do this, I need two probability vectors to plug into distance.jensenshannon(p, q). ...
0
votes
0answers
9 views

why forecast for test/validation set is coming as straight line for ARIMA?

I am trying to forecast values using ARIMA for 14 day data which is in a 5 minute interval [4032 values], the data was stationary hence i plotted seasonal decompose and trend was downwards, moved ...
0
votes
2answers
31 views

Would K-means be Appropriate to Use with Four or More Variables?

Just a general question that I'm trying to mentally visualize. I'm fairly new to using k-means clustering and have used it before on two variables, which creates a 2-D plot of points. I also know, ...
0
votes
0answers
41 views

How to determine percent contribution to classification in logistic regression?

I'm trying to understand the contribution of each feature towards a specific classification. Let's say I have clients as my data and their features are income, debt and age. We're trying to predict ...
0
votes
0answers
26 views

Sum only unique value by group

That's the structur of my data set ...
0
votes
1answer
17 views

are explanatory variables in multiple linear regression slopes?

In simple linear regression the formula is: $y = m \cdot x + b$. When there are multiple independent variables: $y = b_0 + b_1 \cdot x_1 + b_2 \cdot x_2 + \ldots $ I know that $x_1,x_2,\ldots $ ...
1
vote
1answer
49 views

If the entire production dataset is known is advance, should I use it when fitting normalization parameters?

I found in multiple sources the recommendation not to fit the normalization parameters on the combined train/test dataset when evaluating the model, to prevent data leakage. I presume this ...
1
vote
1answer
20 views

identify similarities in a dataset

I have a dataset of customers: age height weight eye_colour 30 174 74 Nan 20 191 71 Nan 28 165 56 Brown ... I would like to ...
1
vote
0answers
17 views

F-test for comparing the mean of two groups

What would be the procedure, in terms of F-test, if I would like to check if the mean of one group is greater than the other group (alternative hypothesis: $\mu_1 > \mu_2$)? And also what would ...
0
votes
0answers
12 views

Grey Information Theory

Grey System Theory was first published by Professor Deng in 1982. As far as I know, the theory encompasses and incorporates the notion of incomplete information into statistics. Statistical methods ...
7
votes
3answers
240 views

Find if the two datasets are close to each other

I have the following three datasets. ...
3
votes
2answers
24 views

Weighted Sum with restricted weights

Given a value $y$ and some values $x_1, ..., x_n$. How do I find weights $w_1, ... w_n$ so that the $error = y - w_1*x_1 + ... w_n*x_n$ is minimal, where the weights have to sum up to 1 or a different ...
3
votes
1answer
75 views

Plotting Weibull distribution on Wind Speed

I have a 720 hourly set of wind speed and wind direction data and I want to fit the Weibull Distribution on it. After searching for some time, I wrote the following code in Python to get my ...
0
votes
1answer
24 views

Is it possible to estimate

For example, if I want to know a page views/relative prevalence of a Reddit post, and I have the upvote and downvote data. Is it valid that I use the upvote and downvote to predict that, even if I ...
0
votes
0answers
18 views

How can you recommend songs based on a user's past listening history by genre (content filtering)?

I'm interested in getting a user's past listening history from Spotify (API call to recently played) and being able to suggest songs from the Charts (another API call for current chat listings) that a ...
1
vote
0answers
42 views

Time Series, Point Detection Python

I have time series data generated from the sensor. Now I know the the actual point I need to find is at the 11:50, as you can see in the graph this point dooes not hold any statistical property which ...
0
votes
0answers
13 views

Can I apply an independent test t on two sets of 30 samples each , even of the normality test fails

In the link t-Test there is a nice explanation about the use of the t-test and how to observe the assumptions of the test: normality and equal variance. If the normality check fails (Shapiro test), ...
3
votes
1answer
52 views

Advice on imputing temperature data with StatsModels MICE

This may be a dumb question but I can't figure out how to actually get the values imputed using StatsModels MICE back into my data. I have a dataframe (dfLocal) with hourly temperature records for ...
1
vote
0answers
33 views

Prerequisites for Elements of Statistical Learning

I am working through Elements of Statistical Learning, and unfortunately have found great difficulty in following the math. I have taken the standard series of Calculus courses (i.e., up to ...
2
votes
1answer
43 views

Elements of Statistical Learning - question on p. 18

In the following expression: What do $E_{X}$ and $E_{Y|X}$ mean, respectively? In case helpful, I note that the authors derived that expression from the following: Based on the above, how would one ...
0
votes
0answers
17 views

Comparing word frequencies between two corpora to find keywords

I have a set of tweets that are either labelled as ironic or non-ironic, and have separated them into two different corpora: ironic and non-ironic. I want to compare these two to see which words are ...
0
votes
0answers
12 views

Bayesian Statistics — Classification Using Probabilistic Inputs

would like to hear people's opinion on the problem. I am working on a project and would like to know if on right path. Essentially, I have a segment of 100k people from a population i'm trying to ...
0
votes
0answers
15 views

Best technique for testing if an event causes a trend?

Example - A major holiday is coming up and the frequency of a product being purchased starts increasing significantly as it gets closer to the day of the holiday. Then after the holiday, the frequency ...
0
votes
1answer
76 views

Why does Naive Forecast,Moving average,Simple Exponential Smothing model predictions flatten?

I have applied simple forecasting models such as Naive Forecast, Moving Average, Simple Exponential Smoothing, Holts Linear Trend Model on 2018 sales data of a salesperson. All the model resulted in ...
0
votes
1answer
16 views

How can you build a model based on non-independent imbalance data?

I am trying to predict customer churn based on the data that I have. I am defining churn as an activity that is not followed by another activity within a week. The customer might come back in two ...
2
votes
0answers
32 views

When is the sum of models the model of the sum?

The response variable in a regression problem, $Y$, is modeled using a data matrix $X$. In notation, this means: $Y$ ~ $X$ However, $Y$ can be separated out into different components that can be ...
0
votes
1answer
22 views

Statistical significance test in deep learning for regression problems

I was reading the tutorial "Statistical Significance Test for comparing ML algorithms", where it suggests to use k-fold and apply the appropriate statistical test. Suppose that I have a train set and ...
1
vote
1answer
29 views

Sales person performance prediction

I am trying to predict/forecast salesperson performance weekly, monthly, quarterly, and yearly based on the products that they sold over 3 years. As part of this effort, I grouped their number of ...
0
votes
2answers
39 views

Statistical methods for Sequence learning

I am bit more traditional and I am looking for a statistical method that can help me also do inference or predictions. I have some table that contains some transitions, where each row of the table ...
0
votes
1answer
46 views

How do I properly write scipy.stats.binom.cdf() details

I need to calculate the probability of my random variable being $\le 0$. It's a binomial distribution, $10000$ trials, probability of success is $\frac{10}{19}$ (roughly $0.53$). How do I properly ...
0
votes
0answers
9 views

How to interpret odds ratio test in logistic regression?

I'm a new learner of DS, and I encounter some problems with logisitc regression. Here is my result of regression, it sets a binary event as response variable and race as only factor. But I don't ...
0
votes
1answer
23 views

In which CRAN mirror is the quadprog package available?

I was trying to find the accuracy of some predicted data and I came to know that I can use the forecast package to do this. When I installed it and gave library(forecast), it said it needs quadprog ...
0
votes
4answers
162 views

How to find the relationship between two categorical variables?

I'm building a model and trying to get the relationship between two multi-level categorical variables. For example, I want to know the relationship between race and likelihood of graduate from ...