Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [statistics]

Statistics is a scientific approach to inductive inference and prediction based on probabilistic models of the data. By extension, it covers the design of experiments and surveys to gather data for this purpose.

0
votes
1answer
13 views

What is the difference between parameters & cooficients in Machine learning?

are both terms interchangeable? I'm kinda new to machine learning field and very confused about these terms in machine learning perspective.
0
votes
1answer
19 views

Test for heteroscedasticity in time series

I want to test heteroscedasticity in time series. The tools in python like: statsmodels.stats.diagnostic.het_breuschpagan require residuals as input obtained by fitting model to data. Since this kind ...
1
vote
1answer
30 views

How to select random data for two different recommender systems?

The business problem: We have two different vendors that offer personalized recommender engines and want to do A/B testing with them. The recommendation will give the user a personalized offer via a ...
0
votes
1answer
9 views

Survey papers on Natural Language Understanding

What are some good survey or review papers on the state of the art of machine learning approaches to grammatical/syntactic structure of natural languages?
2
votes
2answers
48 views

Why do a lot of people use ipython notebook over python file when doing analyzing data? Is it the same in industry?

I have seen that a lot of people write code in ipython notebook when doing statistical analysis on data, apart from easy visualization after each step rather than running the whole code every time on ...
0
votes
2answers
27 views

Statistical analysis vs Data mining

Why the A/B Testing, the Correlation and the Regression are considered methods of Statistical Analysis, while the method of "Association Rules" belong to the Data Mining field ? Which are the ...
0
votes
0answers
8 views

Need Weibull and Gamma Regression Data

I am working on regression with non-normal responses. Now i need a data set(s) in which responses follow Weibull and Gamma distribution, with some reference. kindly data may contain at least 15 ...
0
votes
1answer
12 views

How the original data can be written in the space defined by these M principal components?

Suppose you apply PCA on the data $x_1,...,x_6$ and find that data can be fully described using M principal components $u_1,...,u_M$. How the original data can be written in the space defined by these ...
0
votes
1answer
9 views

Given a 12x12 binary image (only black and white pixels) what is its dimensionality? And how can I define dimensionality of a data space?

Suppose I have a grid 12x12 of pixels that can be only black or white. I can't understand if the dimensionality is 2 or 3. I mean... Is dimension given by 12x12 or 12x12x2 ?
1
vote
0answers
14 views

How to find modes of multivariate time series dataset

I have a dataset of different multivariate time series (with features $x_0, y_0, x_1, y_1$ that are coordinates of starting and ending point of an event within each time series' observation. The data ...
0
votes
1answer
22 views

p-value of binary results

I have a list of ages and then a binary value associated with it (e.g. is the person overweight) I would like to calculate from this the p-value. I have plotted the values and then calculated the ...
0
votes
0answers
3 views

Game Data Analysis (Stats)

Analyzing some game data and wanted some help to know how a specific column affects the game economy dynamics. Game Scenario: • This game data comes from a video game in which players earn items ...
0
votes
2answers
25 views

Predicting missing data. Looking for good data predicting technique

I am analysing data for Countries Trade GDP. Some of the countries have missing GDP value for given a year. However, I have Grand Total for the entire region for that year. Is there a good data ...
0
votes
1answer
12 views

Good t-SNE or bad t-SNE?

I have used tsne to visualise a large dataset and it has produced the following graph. I need help interpreting it, as I have never seen a tsne graph like this before! I am aware that not much ...
0
votes
0answers
18 views

Machine learning over machine learning result

My data is design as a category. This category has groups. Using NLP, the meaning of the algorithm is to find the category. It's a problem of multi-label classification. Using Python and scikit-learn,...
0
votes
1answer
21 views

How can I use a class variable with many possible values in logistic regression?

I am attempting to build a logistic regression model that determines the probability of an outcome based on a set of independent variables. For context, the data is based on a project in which sales ...
3
votes
3answers
79 views

Why 100% accuracy on test data is not good?

I was asked this question in an interview and wasn’t able to give a satisfactory answer not only upto the interviewers' expectations but of my own as well. The question was as above only, he later ...
0
votes
1answer
25 views

feature importance between two data distributions

I have two textual datasets collected from different domains (Twitter and Reddit). I extracted a set of features in the same way from these two datasets, one of these features as an example called ...
0
votes
0answers
14 views

Finding Impact of IV on NPS of responses in A survey

I have a dataset like this. Where the first column is the age of usage of product of a user (in days) and second column is the response of the user to the typical NPS survey question: Using these ...
3
votes
3answers
79 views

Tell me some things a data scientist/statistician needs to do that can't be done in R/Stata/SAS/SPSS [closed]

Why do I need to learn to program in python or Java and learn data structures? I can't think of a single thing statisticians/data scientists need to do that isn't built in as a package in R/stata.
0
votes
0answers
13 views

How to calculate Bias and Standard deviation given the true versus predicted values of ML model?

I am looking to calculate Bias and Standard deviation given the true versus predicted values of ML model. Am I correct in saying: Bias = (average difference between true values and predicted values)/...
0
votes
0answers
10 views

Why do we use Bayesian Error Function in Neural Network when there is less data?

I have read somewhere, not able to recall it, that Bayesian error function is used when there is lack of data to feed into neural networks. But I didn't see any proper reason there. If anyone can ...
0
votes
1answer
24 views

How to get the formulas used by seasonal_decompose for Trend and Seasonality

I'm trying to use decomposition to forecast into the future. From my reading I understand that I can do this by adding a trend formula to a seasonality formula. I know that I can decompose a time ...
-1
votes
0answers
18 views

Titanic Disaster Problem in Kaggle: distribution of numerical feature values across the samples

I'm still a newbie in Data Science and I found the Titanic question and solution here in Kaggle. I've been trying to understand the solution yet I still can't grasp why he used those percentiles for ...
2
votes
1answer
206 views

Select behavior dependant with other factors and its formalization

I'm studying occurence of Behavior11, Behavior12,Behavior2,...
0
votes
0answers
19 views

Generate Decision tree on one multi level categorical feature

I have 100K pairs of categorical observation(feature) and Label , for an example : As you can see, observations are over lapping the labels (different observations can have the same label) and ...
-1
votes
0answers
10 views

Bootstrap method: rule of samples and number of iterations

I am wondering if there is any rule for how many sample and iterations is the best using bootstrapping methods. For example, I have A dataset with 30 data points and B dataset with 35 data points. I ...
1
vote
2answers
75 views

What is the difference between SVM and logistic regression?

While reading the book by Aurelien Geron, I noticed that both logistic regression and SVM predict classes in exactly the same way, so I suspect there must be something that I am missing. In the ...
0
votes
1answer
30 views

Does this correlation make sense?

I'm trying to work on a ML project and I have a dataset and I'm trying to see if there is a correlation between some of the features in my dataset. The dataset contains inspection notes for car parts ...
-1
votes
0answers
18 views

What optimization algorithm is more suitable for timetable rescheduling problem?

I'm working on the project where university course is represented as a to-do list, where: course owner (teacher of the course) can add tasks (containing the URL to the resource needs to learn and two ...
0
votes
0answers
10 views

Code for Point Process Inference

I need to perform inference on a point process, by which I mean I need a statistical model from which I can (at least) estimate likelihoods. The underlying process appears to have a latent structure, ...
4
votes
2answers
107 views

Standard Deviation for Z-scores

I have a set of data that I'm trying to generate a z-score with. I know I need standard deviation as part of my calculations. I am using the formula of: $\sigma = \sqrt{p * n * (1-p)}$ My data is ...
-1
votes
0answers
13 views

Why GP (Gaussian Process) works?

Actually I read through some papers about GP(google the stanford university and oxford university lecture pdf and read) I knew the procedure to do it with detail to choose the kernel and test and the ...
1
vote
0answers
19 views

Forecasting/Predicition for a specific date [closed]

First of all data science is not my experties so apologies if this sounds stupid. I have a data X that is either 1 or 0 and (possibly) dependent on some other variables. I have the values for these ...
1
vote
0answers
8 views

Word frequencies in unbalanced case-control dataset

I have a case-control cohort for which I'm doing analysis of clinical notes. The ratio of cases to controls is 1:4. What I'm looking at is the relative frequency of certain words (e.g. overdose, ...
1
vote
0answers
34 views

Forecasting/predicting techniques for qualitative data?

I have a food alert dataset composed of nominal qualitative variables, such as type of alert, country of origin, action taken, etc. as well as the date on which the alert was recorded. What ...
2
votes
1answer
18 views

How to test the influence of a feature on conversion?

I have a user journey where I have data of the format: userID, did_interact_with_feature(0/1), did_convert(0/1) I want to verify the hypothesis that if a user is engaging with the feature, he's ...
1
vote
0answers
19 views

Is Chi Square goodness parametric or non-parametric?

Does Chi Square goodness of fit require normality assumption? Is it a parameteric or non-parametric test? What it its relation with t-test (parametric test) and u-test (non-parametric test)?
1
vote
1answer
49 views

What the good general regression technqiue for a problem with 50 independent varaibles [closed]

I am a newbie to data science and statistics. I came across this problem, which has 50 independent variables and one dependent variable and trying to identify the good regression technique to start ...
0
votes
1answer
13 views

Wilcoxon W value different from python

I use data from https://en.wikipedia.org/wiki/Wilcoxon_signed-rank_test, W value is 9. But, for the following code W value is 18, what is the reason? ...
0
votes
0answers
9 views

What is the statistical significance of this False-Positive Rate and how will it change with n_samples?

I have a classifier with an FPR of lets say 1:1000 as tested e.g. with 10,000 samples. What is the significance of this FPR and how will it change with n_samples?
1
vote
1answer
33 views

What are the best practices for data formatting?

I used to do some analysis on Excel but my company want to use Python to increase the analysis. I am novice on python and overwhelmed by all of this knowledge I have to learn :'( So for my first ...
0
votes
1answer
40 views

How to Identify p (lag order) for ARIMA Model in Python

here is my auto correlation plot. Generated by the following python code. ...
-1
votes
0answers
13 views

Skew Calculation

I don't have a histogram built from 1D data. Rather, I have irregularly spaced (time,y) data pairs. Interpolating between them isn't valid due to the nature of my data. I'd like to calculate skew, ...
-1
votes
0answers
5 views

Which Keras model, if any, is best for this unusual (K,N)-multinomial predictive task?

Say I have a bunch of images that were classified by $N$ people. Each person classified each image into one of $K$ categories. For each image, the votes for each category are then summed up. As a ...
0
votes
1answer
16 views

Example data source for educaional use

I'm doing project on subject of affinity analysis for my statistical class in college. In order to complete it, I have to acquire sales database with at least 200-300 records, each containing list ...
1
vote
4answers
49 views

Statistics Before Linear Algebra?

I know this is an opinion-based question and will be closed but this is the only place I know that can answer it reasonably and it is a very important matter to me. I am pursuing a machine ...
0
votes
1answer
18 views

Working with few instances of specific target feature over large dataset

I have data over a single, a machine includes different components, all the parts are interacting, the data are tracked for those parts, it tracks power consumption and many other relevant feature ...
0
votes
2answers
33 views

How to tell if the “clusters” I see in my pair plots are statistically significant or occurring by random chance?

I have a data set with one row per subject. Some variables include laboratory parameters for blood chemistry, hematology, etc. I also have some flag variables: any = 1 if the subject experienced an ...
1
vote
0answers
24 views

Books and references for mathematical and intuitive Deep-Learning understanding? [closed]

After doing practical deep learning, I am willing to develop an in-depth intuitive understanding of Deep learning concepts and logic from a mathematical and statistical point of view. What all ...