Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [statistics]

Statistics is a scientific approach to inductive inference and prediction based on probabilistic models of the data. By extension, it covers the design of experiments and surveys to gather data for this purpose.

0
votes
0answers
15 views

Comparing word frequencies between two corpora to find keywords

I have a set of tweets that are either labelled as ironic or non-ironic, and have separated them into two different corpora: ironic and non-ironic. I want to compare these two to see which words are ...
0
votes
0answers
11 views

Bayesian Statistics — Classification Using Probabilistic Inputs

would like to hear people's opinion on the problem. I am working on a project and would like to know if on right path. Essentially, I have a segment of 100k people from a population i'm trying to ...
0
votes
0answers
15 views

Best technique for testing if an event causes a trend?

Example - A major holiday is coming up and the frequency of a product being purchased starts increasing significantly as it gets closer to the day of the holiday. Then after the holiday, the frequency ...
0
votes
1answer
31 views

Why does Naive Forecast,Moving average,Simple Exponential Smothing model predictions flatten?

I have applied simple forecasting models such as Naive Forecast, Moving Average, Simple Exponential Smoothing, Holts Linear Trend Model on 2018 sales data of a salesperson. All the model resulted in ...
0
votes
1answer
11 views

How can you build a model based on non-independent imbalance data?

I am trying to predict customer churn based on the data that I have. I am defining churn as an activity that is not followed by another activity within a week. The customer might come back in two ...
2
votes
0answers
27 views

When is the sum of models the model of the sum?

The response variable in a regression problem, $Y$, is modeled using a data matrix $X$. In notation, this means: $Y$ ~ $X$ However, $Y$ can be separated out into different components that can be ...
0
votes
1answer
16 views

Statistical significance test in deep learning for regression problems

I was reading the tutorial "Statistical Significance Test for comparing ML algorithms", where it suggests to use k-fold and apply the appropriate statistical test. Suppose that I have a train set and ...
1
vote
1answer
27 views

Sales person performance prediction

I am trying to predict/forecast salesperson performance weekly, monthly, quarterly, and yearly based on the products that they sold over 3 years. As part of this effort, I grouped their number of ...
0
votes
2answers
37 views

Statistical methods for Sequence learning

I am bit more traditional and I am looking for a statistical method that can help me also do inference or predictions. I have some table that contains some transitions, where each row of the table ...
0
votes
0answers
13 views

Nosiy Time Series Data

I have a time sereis data which is from the production power plant. With this data I have to predict when the process is going to complete when actual production is happening. The problem is as there ...
0
votes
1answer
30 views

Feature normalization - proper way

Consider a matrix X where the columns are the attributes and each row is an example. There is no order of occurrence of the examples. The target variables are ...
0
votes
1answer
23 views

How do I properly write scipy.stats.binom.cdf() details

I need to calculate the probability of my random variable being $\le 0$. It's a binomial distribution, $10000$ trials, probability of success is $\frac{10}{19}$ (roughly $0.53$). How do I properly ...
0
votes
0answers
8 views

How to interpret odds ratio test in logistic regression?

I'm a new learner of DS, and I encounter some problems with logisitc regression. Here is my result of regression, it sets a binary event as response variable and race as only factor. But I don't ...
0
votes
1answer
20 views

In which CRAN mirror is the quadprog package available?

I was trying to find the accuracy of some predicted data and I came to know that I can use the forecast package to do this. When I installed it and gave library(forecast), it said it needs quadprog ...
0
votes
3answers
34 views

How to find the relationship between two categorical variables?

I'm building a model and trying to get the relationship between two multi-level categorical variables. For example, I want to know the relationship between race and likelihood of graduate from ...
0
votes
0answers
11 views

What's the correct/preferred way of determining the final class from seq2seq softmax probabilities?

Here's an example, between three classes a, b and c and their softmax probabilities from an imagined seq2seq algorithm. In this case it's pretty obvious that class c is the most likely single label ...
1
vote
1answer
24 views

Question about sklearn's StratifiedShuffleSplit

I'm reading through the book Hands-On Machine Learning with Scikit-Learn and Tensorflow by Aurélien Géron. In a regression project on California Housing Prices, he goes over the concept of stratified ...
0
votes
2answers
41 views

Sum of Least Squares vs Variance

When trying to fit a line to data, using linear regression, we would like it to have the lowest sum of squared differences (least squares method). I am wondering why we use this technique instead of ...
0
votes
2answers
20 views

Determining which entries in input vector have the greatest effect on output vector

I'm looking for a solution in Python. I'm doing a project where I've built a densely packed neural network which takes a 48-dimensional input vector and the last layer is a 2-dimensional linear output....
0
votes
0answers
6 views

Two tailed F test, correct rejection of the null hypothesis

I am performing an F test for equality of two variances by following this website formula https://itl.nist.gov/div898/handbook/eda/section3/eda359.htm, however it states that In the above formulas ...
0
votes
0answers
37 views

Practical use of bootstrap in R

I am a beginner and try to perform a bootstrap in R. It would be nice if somebody could help me with the following situation. I have defined prices and I want to randomly generate quantities (given a ...
0
votes
0answers
23 views

Can McNemar's test be applied to evaluate multiclass models?

Full Disclosure: I did a semi-cross post of this question due to low traffic on Cross Validated. Once I get an answer on any of the two questions, I will link the answer back to the respective other. ...
0
votes
0answers
9 views

Is it possible to calculate precision/recall for a bag-of-words model?

Suppose I have a list of keywords given to a document: {keyword, extract, graph, represent, text, weight, number, document} and then I have the keywords ...
1
vote
1answer
26 views

Different t-test results from R and LibreCalc

while I was trying to apply one-tailed t-test to two vector of data on R, I just wanted to apply same thing on librecalc also and then I noticed that p-values are different. Here is a summary what I ...
0
votes
0answers
9 views

create joint prob distribution or empirical relation for two variables

There are two variables, X1 and X2. The experimental study shows that they are highly correlated. Are there any reliable ways to create an empirical mapping(or equation) between X1 and X2. Assuming ...
0
votes
0answers
5 views

Confidence vs. Count in association rule mining: which one is better?

I am writing a program that mines association rules from a large data set. I have an array of association rules, and I have to decide which ones are more representative of the patterns I am studying. ...
0
votes
1answer
31 views

Is there a safe and simple way to estimate a standard deviation for a next subset?

In case I receive only standard deviation from a sensor of a value $v$ (that is btw normally distributed) each 4th minute but need to provide a standard deviation $\sigma$ for each 15 minutes is there ...
0
votes
1answer
37 views
2
votes
0answers
39 views

Is AUC a good metric for evaluating the performance of a multi-class classification?

Considering the definition of AUC (Area Under Curve), is that a reliable performance metric for a multi-class (30-40 classes) classification problem?
2
votes
2answers
49 views

Possible Challenges for a Data Science Escape Room

Dear Data Science Community, as my project for my bachelorthesis I am working on a concept for a physical Data Science Escape Room. The goal of the escape room is to create awareness for data science ...
0
votes
0answers
16 views

Comparing simulated Lightcurve with measured lightcurve of satellites

With python I want to compare a simulated light curve with the real light curve. It should be mentioned that the measured data contain gaps and outliers and the time steps are not constant. The model, ...
0
votes
1answer
21 views

Comparison between two items

I have table like: Sales 2010 2011 2012 2013 State X Y X Y X Y X Y 1 2 3 4 5 6 7 ...
0
votes
0answers
16 views

How can I transform (pre-process) pure count data for PCA analysis?

If 12 covariates of data are all count data (looks like a Poisson dist with the highest peak at 0), what are reasonable pre-processing methods that might make PCA more effective?
6
votes
1answer
42 views

Covariance as inner product

Why is covariance considered as inner product if there is no projection of one vector onto another? Right now I perceive this as just a multiplication of $x$ segment of vector($x_i - \bar{x}$) and $y$...
2
votes
1answer
62 views

Efficient self study plan

I am hoping for a bit of guidance from experienced practitioners / academics. I want to work through the Bishop ML book, but have minimal background. What is the fastest way to get the pre-...
2
votes
1answer
30 views

How to perform (modified) t-test for multiple variables and multiple models on Python (Machine Learning)

I have created and analyzed around 16 machine learning models using WEKA. Right now, I have a CSV file which shows the models' metrics (such as percent_correct, F-measure, recall, precision, etc.). I ...
2
votes
0answers
18 views

How to create a prediction interval with the fact that the residuals follow a specific distribution (in python)

I am looking at a software development pipeline where I am predicting the lead time of different products flowing through the pipeline. After applying a boxcox transformation on the lead time (...
0
votes
0answers
19 views

How to graphically present the statistical interpretations when using Kolmogorov–Smirnov test (K-S) test?

I am referring to my previous question (Any suitable way to describe the distributions of 2 Pandas Dataframes visually/graphically?) related to visualising the Pandas dataframes to show that they have ...
2
votes
2answers
50 views

How long would it take to become proficient in machine learning for someone with a non-statistical mathematical background?

I am currently a postdoc and my PhD was in applied mathematics in the area of numerical analysis and electromagnetic/acoustic wave propagation. There was no statistical element to my PhD, it was ...
1
vote
0answers
30 views

Multi-class classification as a hypothesis testing problem

I'm diving into the logistic distribution and its applications in classification problem (see my old question for more details about my idea). As discussed in my old post, logistic regression, in ...
0
votes
1answer
23 views

Correlation between Time Series Indicators ( Stock Prices )

I am new to time series analysis and I am currently tackling a stock market prediction problem. I have a set of market indicators (such as Bollinger Bands, ADX etc) which are derived from the time ...
1
vote
1answer
51 views

Grouping probabilities of full model vs fitting model on subset of variables

Suppose I want to predict some event probability with a set of features. Some of the features may be gender or some other categorical variable. Assume the probabilities are well calibrated. Now, say,...
1
vote
1answer
31 views

t - table and degrees of freedom

I am interested if someone can review this process and give me some tips. (I don't have any data science coworkers or friends to collaborate with...) When the script (.py file) below runs, it will ...
1
vote
0answers
20 views

Better way to graph percentage of One Categorical Variable by another in R

Looking through the Brfss 2013 dataset (a study that recorded many elements of an observers health habits and outcomes), I wanted to see if there was a correlation between smoking often and being an ...
0
votes
1answer
15 views

Looking for help calculating a probability formula

How do I put this into a calculator or a excel spreadsheet formula. I have never done this math before, but I want to figure it out.
1
vote
0answers
11 views

Metrics for stain normalization

Are there any metrics or methods for assessing stain normalization techniques?
0
votes
0answers
37 views

How to find out the correlation between a categorical and continuous variable? [duplicate]

I want to identify/select categorical variables (which could be either object or numeric) that are better related to a single continuous variable than others. My research on the above problem ...
1
vote
0answers
22 views

Detecting seasonality in timestamped events

I have a program that detects events in a large amount of measurement data. When it detects an event, it writes a timestamp. I have thousands of event timestamps. What I wish to do is detect if there ...
1
vote
0answers
24 views

I am getting p value 0 for my population I teven tried zscore for normalization

a1 is the columncolumn a1 g=scipy.stats.mstats.zscore(a1) g stat, p = normaltest(g) print "%.3f, %.9f" %(stat ,p) Result 8.570, 0.00000000 p1=scipy.stats.kstest(g,'norm') p1 KstestResult(...