Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [statistics]

Statistics is a scientific approach to inductive inference and prediction based on probabilistic models of the data. By extension, it covers the design of experiments and surveys to gather data for this purpose.

1
vote
0answers
13 views

Grouping probabilities of full model vs fitting model on subset of variables

Suppose I want to predict some event probability with a set of features. Some of the features may be gender or some other categorical variable. Assume the probabilities are well calibrated. Now, say,...
1
vote
0answers
15 views

t - table and degrees of freedom

I am interested if someone can review this process and give me some tips. (I don't have any data science coworkers or friends to collaborate with...) When the script (.py file) below runs, it will ...
0
votes
0answers
7 views

Extracting disaggregated time series from an aggregate time series

I've got $N = 5000$ individual time series representing hourly electricity demand from $N$ households. I also know whether each house has electric heating or not. Assume there are $N^1$ houses with ...
0
votes
0answers
14 views

What does this residual-fitted value plot show?

I am trying to fit a linear regression model, where my Y and X variables are all continuous. I have applied a vanilla linear regression in R, and I am trying to understand if the model fits all the ...
1
vote
0answers
16 views

Better way to graph percentage of One Categorical Variable by another in R

Looking through the Brfss 2013 dataset (a study that recorded many elements of an observers health habits and outcomes), I wanted to see if there was a correlation between smoking often and being an ...
0
votes
1answer
14 views

Looking for help calculating a probability formula

How do I put this into a calculator or a excel spreadsheet formula. I have never done this math before, but I want to figure it out.
1
vote
0answers
11 views

Metrics for stain normalization

Are there any metrics or methods for assessing stain normalization techniques?
0
votes
0answers
22 views

How to find out the correlation between a categorical and continuous variable? [duplicate]

I want to identify/select categorical variables (which could be either object or numeric) that are better related to a single continuous variable than others. My research on the above problem ...
0
votes
0answers
8 views

Poisson point process application and terminology

I am trying to understand Poisson process that is used in a derivation of maximum likelihood estimation of intrinsic dimension given here https://wiki.math.uwaterloo.ca/statwiki/index.php?title=...
1
vote
0answers
16 views

Detecting seasonality in timestamped events

I have a program that detects events in a large amount of measurement data. When it detects an event, it writes a timestamp. I have thousands of event timestamps. What I wish to do is detect if there ...
1
vote
0answers
14 views

I am getting p value 0 for my population I teven tried zscore for normalization

a1 is the columncolumn a1 g=scipy.stats.mstats.zscore(a1) g stat, p = normaltest(g) print "%.3f, %.9f" %(stat ,p) Result 8.570, 0.00000000 p1=scipy.stats.kstest(g,'norm') p1 KstestResult(...
0
votes
0answers
22 views

Mixed linear Model

I have a dataset with G1 (Grade of first semester)and G2(Grade of second semester)and G3 the final grade. I need to model it using linear mixed model with repeated measure to study the variation ...
0
votes
1answer
24 views

Error in Calculating the z Score for Normality Detection using numpy and scipy

Calculate the z Score for the Normal pdf plot I am not getting the correct values for the z score. For example, the z score for 0.025 comes out as -1.64750894. In the Normal Distribution Table, the ...
7
votes
5answers
2k views

When to use mean vs median

I'm new to data science and stats, so this might seems like a beginner question. I'm working on a dataset where I've user's Twitter followers gain per day. I want to measure the average growth he had ...
0
votes
1answer
11 views

How can I measure the reliability of the specificity of a model with very small train, test, and validation datasets?

Stats newbie here. I have a small dataset of 646 samples that I've trained a reasonably performant model on (~99% test and val accuracy). To complicate things a little bit, the classes are somewhat ...
0
votes
1answer
12 views

What is the first tool to learn start your your data science projects? [closed]

What is the first tool to learn start your your data science projects?
2
votes
1answer
32 views

Fast way of computing covariance matrix of nonstationary kernel in Python

Suppose I have symmetric positive definite covariance function $k:\mathbb{R}\times\mathbb{R}\rightarrow \mathbb{R}$ that is non-stationary (i.e. $k(x,y) \neq g(|x-y|)$ for any function $g$). Is there ...
0
votes
0answers
11 views

How panel data analysis can be used and what are the uses of panel data analysis/regression?

I have a dataset with Cross-section and Time-Series Data. After doing some reading I came accross an area named as "Panel data analysis/regression". However, I am still not clear how and why we use "...
0
votes
0answers
18 views

Big Data Statistical Software for Confidence Intervals/Hypothesis Testing

What are some current approaches for doing statistical analysis (glm, hypothesis testing, confidence intervals, ...) on Big Data (that cannot fit on one machine). I know there are Machine Learning ...
0
votes
1answer
48 views

How can causal inference be used with machine learning?

I am wondering how Causal inference is being used with machine learning and specially where in the data science project ? I have ...
2
votes
0answers
15 views

Multiclass classification problem with more prediction classes than real classes

Can I have a multiclass classification problem with more prediction classes than real classes? For example: I want to predict the channel the user is going to watch. The real classes are "user didn't ...
3
votes
1answer
21 views

Platt's Scaling for multi-label classification?

How does one do platt's scaling for multi-label classification? For example, if the final layer of my DNN is a soft-max activation with 10 classes, then how does platt's scaling work exactly? Do I ...
1
vote
1answer
46 views

Can anyone explain the reasoning behind this post?

I am reading THIS blog post, and I do not understand the logic behind this part: Why is the relu here max(0, Xavg - X) ? And even so, it does not really explain ...
1
vote
0answers
22 views

LSTM - Forecasting usage (real world) [closed]

I see that LSTM is very powerful reconstructing time series it was fed with, but my issue is: => Can LSTM predict future values without requiring real (training/testing) data? My objective is simple:...
2
votes
1answer
29 views

Using classifier's probabilities as independent variables to predict Y

Suppose you have a classification task y~X with (n_samples,m_features). A colleague told me that it is correct to run r different classifiers to predict y based on X and then use the probabilities ...
1
vote
0answers
17 views

How much data would be remain unaffected? [closed]

our data set has missing values. Further examination tells you that they are spread along 1.5 standard deviation from the median with distribution mean = 0 & variance = 5. How much data would ...
1
vote
0answers
39 views

Two-sample K–S test Kolmogorov–Smirnov test in Python

For my evaluation, I have three different time-series data of the following format with different characterstics where the first column is timestamp and the second column is the value. ...
1
vote
0answers
12 views

What is the interpretation of the expectation notation in the GAN formulation?

I'm confused about the expectation notation in the context of GAN loss functions. The GAN loss for the discriminator is binary cross-entropy. ie: is this real or not. real = $D(x)$ (ie: give ...
1
vote
0answers
9 views

Need inputs for Food data set to predict consumer ordering behaviour [closed]

I am looking for food order data set that shows the consumer list and type of food they order regularly. Like : customer_number, gender, day, time, choice_of_food, food_category, quantity(grams), ...
2
votes
0answers
27 views

Measure the “aggregate preference” of points on a 2D plane

Consider some points on a 2D plane, we know corrdinates of all points, how to measure the "aggregate preference" of them. I don't know what is the exact terminology of "aggregate preference". What I ...
2
votes
0answers
15 views

Sub-sampling so that sample statistics match population statistics

I want to investigate the impact of various testing strategies on a product. Let's say chairs. I start with 500 random chairs that I've picked up from garage/yard sales. They come in all shapes and ...
1
vote
4answers
38 views

Which statistical model should I run to get a good model

I'm working on an analysis to analyze projected man-hours vs actual man-hours used for different teams. My data looks like this below. The software I am using is SAS. I've tried a HP linear ...
1
vote
1answer
12 views

the probability distribution of dependent variables

There are three variables, X3 is a function of X1 and X2, ...
6
votes
3answers
82 views

Least Squares optimization

The cost function given as $\hat{\beta} = (Y - \beta X)^T (Y-\beta X)$ is used to evaluate the weights $\beta$. Here $X$ is the data and $Y$ is the output. On taking the derivative, we get the ...
0
votes
0answers
13 views

Quantify similarity between 2 multivariate time series samples

I am looking for methods to analyze the similarity between 2 multivariate samples of a time series data. I have tried and analyzed KS test, euclidean distance and correlation based methods. What are ...
0
votes
1answer
31 views

Intuitive Explanation of R-squared

Here is a nice definition of R-squared that I have found on the internet. R-squared is a statistical measure of how close the data are to the fitted regression line. It is also known as the ...
3
votes
1answer
102 views

How can i find trend time for my articles?

our article is time-based, that means is my article search more in a specific time. as you can see in under chart this article search more in specific period time. if my dataset looks like this(it ...
3
votes
3answers
45 views

How to predict whether or not a customer will renew

I have a dataset of customer contracts that specify a start date and if applicable an end date. Each month a customer is up for renewal. Below is an example of how the data is organized in excel: <...
2
votes
1answer
36 views

How to start building a statistical regression analysis model with multiple categorical/discrete input variables of high dimension in Python

I'm fairly new to data science and ML. I have data of an item going through a release process. I have collected data on various variables such as "product category", "product line", "design country", "...
0
votes
0answers
54 views

Finding unaffected data in a dataset

The data set has missing values. Further examination tells that they are spread along 1.5 standard deviation from the median with distribution mean = 0 & variance = 5. How much data (in percentage)...
1
vote
1answer
17 views

Predicting of the function values

There is a function: $f(t,x,y,z)$, where $t$- time, $x,y,z$- some arguments. The values of $f$ for $t\in [a,b]$ are known (100 samples). What is the most accurate way of predicting the value of $f$ ...
0
votes
0answers
36 views

Data unaffected based on mean,deviation,median

Our data set has missing values. Further examination tells you that they are spread along 1.5 standard deviation from the median with distribution mean = 0 & variance = 5. How much data would ...
1
vote
0answers
7 views

Would it make technical sense to model its two known subelements separately (CLV example)?

Let's assume Customer Lifetime Value (CLV) is defined as average basket x frequency. Option A is to build model predicting CLV ...
0
votes
0answers
26 views

How to solve a classification problem when the independent variables/covariates/feature vectors form a time series?

Cross posted here: https://stats.stackexchange.com/questions/389189/how-to-solve-a-classification-problem-when-the-independent-variables-covariates, but no answer; hence trying here as well! I hope ...
3
votes
1answer
63 views

Is an Arma model equivalent to a 1-layer Recurrent Neural Network without activation function?

Given a time series $f(t)$ to forecast, let us consider an Arma model of the form: $$ f(t) = c + \sum_{i=1}^p a_i f(t-i) + e(t) + \sum_{j=1}^q b_j e(t-j) $$ where $e(t)$ are the forecast errors. On ...
0
votes
1answer
18 views

What is advantage of using the theory of point processes instead of probability distribution functions?

I am interested in applications of point processes in machine learning, so I have been studying the theory of point process. However, I cannot see advantage of the theory of point processes compared ...
1
vote
1answer
30 views

What is the difference between parameters & cooficients in Machine learning?

are both terms interchangeable? I'm kinda new to machine learning field and very confused about these terms in machine learning perspective.
1
vote
1answer
31 views

Test for heteroscedasticity in time series

I want to test heteroscedasticity in time series. The tools in python like: statsmodels.stats.diagnostic.het_breuschpagan require residuals as input obtained by fitting model to data. Since this kind ...
2
votes
1answer
43 views

How to select random data for two different recommender systems?

The business problem: We have two different vendors that offer personalized recommender engines and want to do A/B testing with them. The recommendation will give the user a personalized offer via a ...
1
vote
1answer
13 views

Survey papers on Natural Language Understanding

What are some good survey or review papers on the state of the art of machine learning approaches to grammatical/syntactic structure of natural languages?