Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [statistics]

Statistics is a scientific approach to inductive inference and prediction based on probabilistic models of the data. By extension, it covers the design of experiments and surveys to gather data for this purpose.

0
votes
0answers
8 views

Standard Deviation for Z-scores

I have a set of data that I'm trying to generate a z-score with. I know I need standard deviation as part of my calculations. I am using the formula of: $\sigma = \sqrt{p * n * (1-p)}$ My data is ...
0
votes
0answers
5 views

Why GP (Gaussian Process) works?

Actually I read through some papers about GP(google the stanford university and oxford university lecture pdf and read) I knew the procedure to do it with detail to choose the kernel and test and the ...
-1
votes
0answers
16 views

Is there a way to plot 'multi-variable' line graph in R?

Similar to this done in MS ExcelBy 'multi-variable' I mean the columns(variables) of the dataframe on the X-axis and the values they take on the Y-axis and each line representing one entry (or row) in ...
1
vote
0answers
16 views

Forecasting/Predicition for a specific date [on hold]

First of all data science is not my experties so apologies if this sounds stupid. I have a data X that is either 1 or 0 and (possibly) dependent on some other variables. I have the values for these ...
1
vote
0answers
8 views

Word frequencies in unbalanced case-control dataset

I have a case-control cohort for which I'm doing analysis of clinical notes. The ratio of cases to controls is 1:4. What I'm looking at is the relative frequency of certain words (e.g. overdose, ...
1
vote
0answers
25 views

Forecasting/predicting techniques for qualitative data?

I have a food alert dataset composed of nominal qualitative variables, such as type of alert, country of origin, action taken, etc. as well as the date on which the alert was recorded. What ...
2
votes
1answer
13 views

How to test the influence of a feature on conversion?

I have a user journey where I have data of the format: userID, did_interact_with_feature(0/1), did_convert(0/1) I want to verify the hypothesis that if a user is engaging with the feature, he's ...
0
votes
0answers
7 views

How to model channel allocation behavior of the wifi system

I am working on a problem for weeks without progress. Here it is: Inputs are the csv files about activities of many access points (AP), each row has this format [time: mac_address: ...
1
vote
0answers
17 views

Is Chi Square goodness parametric or non-parametric?

Does Chi Square goodness of fit require normality assumption? Is it a parameteric or non-parametric test? What it its relation with t-test (parametric test) and u-test (non-parametric test)?
1
vote
1answer
43 views

What the good general regression technqiue for a problem with 50 independent varaibles [closed]

I am a newbie to data science and statistics. I came across this problem, which has 50 independent variables and one dependent variable and trying to identify the good regression technique to start ...
0
votes
1answer
10 views

Wilcoxon W value different from python

I use data from https://en.wikipedia.org/wiki/Wilcoxon_signed-rank_test, W value is 9. But, for the following code W value is 18, what is the reason? ...
0
votes
0answers
9 views

What is the statistical significance of this False-Positive Rate and how will it change with n_samples?

I have a classifier with an FPR of lets say 1:1000 as tested e.g. with 10,000 samples. What is the significance of this FPR and how will it change with n_samples?
0
votes
1answer
22 views

What are the best practices for data formatting?

I used to do some analysis on Excel but my company want to use Python to increase the analysis. I am novice on python and overwhelmed by all of this knowledge I have to learn :'( So for my first ...
0
votes
1answer
16 views

How to Identify p (lag order) for ARIMA Model in Python

here is my auto correlation plot. Generated by the following python code. ...
0
votes
0answers
8 views

Skew Calculation

I don't have a histogram built from 1D data. Rather, I have irregularly spaced (time,y) data pairs. Interpolating between them isn't valid due to the nature of my data. I'd like to calculate skew, ...
0
votes
0answers
5 views

Which Keras model, if any, is best for this unusual (K,N)-multinomial predictive task?

Say I have a bunch of images that were classified by $N$ people. Each person classified each image into one of $K$ categories. For each image, the votes for each category are then summed up. As a ...
0
votes
1answer
14 views

Example data source for educaional use

I'm doing project on subject of affinity analysis for my statistical class in college. In order to complete it, I have to acquire sales database with at least 200-300 records, each containing list ...
0
votes
0answers
8 views

Loopy normalization constant

My professor and I are conducting some research on belief propgation. Currently, we are attempting to figure out a means to find the proper normalization constant of a distribution that results from ...
1
vote
4answers
42 views

Statistics Before Linear Algebra?

I know this is an opinion-based question and will be closed but this is the only place I know that can answer it reasonably and it is a very important matter to me. I am pursuing a machine ...
0
votes
1answer
11 views

Working with few instances of specific target feature over large dataset

I have data over a single, a machine includes different components, all the parts are interacting, the data are tracked for those parts, it tracks power consumption and many other relevant feature ...
0
votes
2answers
28 views

How to tell if the “clusters” I see in my pair plots are statistically significant or occurring by random chance?

I have a data set with one row per subject. Some variables include laboratory parameters for blood chemistry, hematology, etc. I also have some flag variables: any = 1 if the subject experienced an ...
1
vote
0answers
18 views

Books and references required for mathematical and intuitive Deep-Learning understanding? [closed]

After doing practical deep learning, I am willing to develop an in-depth intuitive understanding of Deep learning concepts and logic from a mathematical and statistical point of view. What all ...
0
votes
0answers
14 views

How to construct an wealth feature for an area, rate or density?

I am trying to model a house price predicting. I have some socioeconomic data according to the postcode area, such as: high income inhabitants, middle income inhabitants, low income inhabitants, ...
0
votes
0answers
13 views

What are the implications of different bin sizes and counts in RFM analysis?

I am building a library for customer segmentation which includes an RFM computation. For skewed datasets with e.g. 60% of frequency values being 1, there would be a disproportionate number of ...
1
vote
0answers
29 views

Semantic segmentation dataset inspection

Is there any tool that can calculate the statistics of semantic segmentation dataset (and possibly the similar types of datasets ...)? So that it shows the number of classes, the total number of ...
0
votes
0answers
20 views

Given a time series of bounded data set, what statistical tool, model or technique should I use to forecast or detect a meaningful change in trend?

I tried my best to come up with a short meaningful title for this question, my apologies if I missed the mark. Here is the concrete problem I'm trying to solve: I am a coach for a large team of ...
0
votes
0answers
24 views

POD, PCA analysis

My toy example is as follows: from some CFD calculations (grid =5x5 cells, each cell is associated with a velocity value) I have extracted 3 snapshots (1 snapshot/case) that represent the velocity of ...
0
votes
0answers
15 views

Possible metrics to judge closeness of predicted curve/points compared to source curve/points?

I have a question. Pardon me if I make any mistakes in posing the question, for I am a beginner in this field. I have a time series dataset of this form : Example : ...
1
vote
2answers
73 views

Multivariate VAR model: ValueError: x already contains a constant

I have already read this question and the associated answer. I have removed any 'all zero' columns, as recommended in the answer. I have 3,169 columns remaining. ...
0
votes
0answers
16 views

Statistics of Model vs. Measured Data

I have 3 [150 x 60,000] arrays. One is measured data, the other 2 are models with inputs based on the metadata for the measured data array. In short, 150 altitudes at 60,000 specific date, time, and ...
0
votes
0answers
18 views

Estimating class prevalence in unlabelled data after predicting labels with a binary classifier

I'm looking to get an estimate of the prevalence of 1's (i.e. the rate of positive labels) in a very large dataset that I have. However, I am hoping to report this percentage as a 95% credible ...
3
votes
2answers
78 views

How can we use Neural Networks for Decision Making intead of Bayesian networks or Desicion Trees?

I am working on Decision Making in Self driving cars and I am wondering how I can use Neural networks (is there any type) ? that can repleace or mimic the bayesian networks or Decision Tree for ...
1
vote
1answer
24 views

Recommender system that connect users with each other , should I go for content based or collaborative filtering?

I am trying to build a system where user come on the platform and he chooses a topic(predefined few topics) and then we connect him with any random online user who chooses the same topic. Then they ...
0
votes
0answers
13 views

Survey results analysis for a social media app - stratified sampling?

I found this question on Glassdoor and was hoping to get some input on it... A user satisfaction survey was conducted for two groups for a social media platform. Assume large sample sizes. Group 1 ...
0
votes
1answer
20 views

What is the quantity sold for a specific fruit & country combination?

What is the algorithm that generates these potential quantities that meet the given criteria? Essentially - there are number of quantities for a fruit and country combination. E.g: ...
0
votes
0answers
8 views

What is the correct way to calculate the entropy of a language model on a data-set of sentences?

I want to fit the parameters of my language model by minimizing the entropy/ maximizing likelihood of my language model on my data-set. However, I am uncertain as how I should go about doing this. ...
0
votes
0answers
20 views

Best practice for outlier removal in Investigating a process deviation

In a controlled process, in which a specific product depicted a deviation in a final product result. The process is time controlled, in which the historical manufacturing experience of the product is ...
0
votes
1answer
63 views

Right Regression Model to use

I am trying to predict reservation count from a dataset with few features. Features are both categorical and continuous. The dependent variable reservations looks like below: My dataset size is ...
1
vote
0answers
99 views

Problem on dataset skewness and deviation [closed]

This is the problem statement : Your data set has missing values and is positively skewed with skewness = 1. Further examination tells you that they are spread along 1.5 standard deviation from the ...
0
votes
1answer
19 views

How to make multiple regression perform better for outliers? (without reducing effect of them)

I have a small dataset(about 60 samples) and I need it to predict well for high target values. There are only a few high values and all models I tried perform poorly for these high values. So I ...
0
votes
2answers
31 views

Replace Values in Vector on Specific Place in R

I want to make $5^{th}$,$10^{th}$,$15^{th}$,$20^{th}$ and $25^{th}$ values of vector an outlier in all xs by using x1 [5]+OT1,...
0
votes
1answer
53 views

A dataset has skewness = 1 with missing data. Standard deviation around median is 1.5. How much data will be unaffected?

There is no other description about the data, if it is univariate, bivariate, etc. neither the type of distribution is given. I recently came across this question, I would like to know how skewness ...
1
vote
0answers
32 views

Improving population weighting

Bit of a noob in this stats world, so apologies in advance for any naiveté. I did a fair bit of stats long ago in college but it's a distant memory, so please assume little knowledge! The dataset I'...
3
votes
1answer
27 views

Should I transform a multiple regression with outliers into ordinal regression?

I have small dataset of about 60 samples that performs poorly in regression. So I wonder how can I transform this task into predicting intervals instead of values. Is it possible to make it perform ...
1
vote
1answer
10 views

Does Box plot with many outliers effects the result?

I am using Haberman's cancer survival dataset https://www.kaggle.com/gilsousa/habermans-survival-data-set to draw a box plot. Here Surv_status is the target variable which has two classes and ...
2
votes
0answers
46 views

Beginner Question Related To Data Science Course [closed]

I am Third Year B.Tech student from 3-tier college of India ,Here is no one fellow or collegous who has somebit knowledge about ML or Data science and I am purely sure that I have strong background of ...
1
vote
0answers
27 views

Perform Pearson's correlation and chi-squared test for feature selection in a dataset with a mixed type of features

I've a dataset of about 200 features and 5000 instances. These features comprise of different data types like percent (string like 4.50%), dollar amount (value between $0 - $1,000,000), discrete ...
0
votes
1answer
120 views

How to read the output of Binary cross entropy?

Suppose for a single training example, the true label is [1 0 0 0 0] while the predictions be [0.1 0.5 0.1 0.1 0.2]. How to calculate its binary cross entropy? And how output value decides whether a ...
1
vote
0answers
17 views

How standardizing and/or log transformation affect prediction result in machine learning models

I recently ran an elastic net model on my data. My predictors are mostly skewed. I found my model perform slightly better when I standardize on log-transformed data than standardizing on original data....
0
votes
0answers
25 views

methods to find causality between 1000 time series

I have an eco-system of 100 applications which each are monitored by let's say 100 metrics publishing every 5 minutes. So my dataset has 10,000 time series. I want to build/learn the dependency graph ...