Questions tagged [statistics]

Statistics is a scientific approach to inductive inference and prediction based on probabilistic models of the data. By extension, it covers the design of experiments and surveys to gather data for this purpose.

Filter by
Sorted by
Tagged with
2
votes
0answers
18 views

difference between normal skewed distribution and skewed distribution

From what I've read normal skewed is a distribution that has all the properties of normal distribution and is skewed: according to this resource:https://www.statisticshowto.datasciencecentral.com/...
1
vote
0answers
8 views

What would be a good threshold number to convert some categories to Not Specified

The current topic is derived from this topic: https://community.rstudio.com/t/39722 (optional to read). I have the following dataset: myds : ...
-3
votes
0answers
21 views

determine the relationship between two nominal variables and two numerical variables [closed]

About statistical variables how to determine the relationship between two nominal variables, and how to determine the relationship between two numerical variables.
0
votes
0answers
20 views

Exploratory Data Analysis for Big Data with continuous and categorical variables (mixed data types)

I have a dataset with 28 variables (6 continuos + 22 categorical). I'm using ...
0
votes
1answer
27 views

why the sigmoid function will be 1 and 0 if we use a fully connected layer that produce a big enough positive(res negative )output

HI I am using a fully connected network that uses sigmoid if we feed a a big enough weights the sigmoid function will finally become 1 or 0 , is there any solution to avoid this ? and will this lead ...
0
votes
1answer
19 views

Traditional Predictive Analytics vs Machine Learning Methods

What is the difference between traditional predictive analytics done using statistics and its tools and, one using machine learning and deep learning? How are we leveraging machine learning and deep ...
1
vote
0answers
13 views

How to graph correctly the Gaussian distribution?

I need to graph the bell curve from the data contained in a .csv file. My problem is at norm_dist_graphic(data): I don't know what data to use in the formule to graph? data.csv: 2 4 1 2 1 3 3 2 1 1 ...
0
votes
0answers
13 views

How to compare two samples taken from the same dataset?

I have data from almost 10 different sensors and was trying to downsample the data. I tried the reservoir algorithm and the uniform sampling and the random sampling techniques. I was wondering how I ...
1
vote
0answers
12 views

How does Markov Chain Monte Carlo work in a neural network from beginning to end?

I am trying to reason through how this works from the beginning of the neural network model to the end from a high level and I want to make sure my understanding is correct, I have probably ...
0
votes
1answer
16 views

Normal distribution and QQ plot

I have data that I plotted with a normal distribution and a QQ plot. I was wondering especially in QQ plot it seems that 95.4% of the data is normally distributed. My question is what does numbers ...
0
votes
1answer
31 views

Populate free space between two dates

I'm new to data science. I'm trying to increase the time-series length for a special calculation. In the original time-series I have 20 weekly reports and I want to increase the amount of occurrences ...
1
vote
2answers
59 views

Fitting a distribution to data

I have a large data set of property values at two points in time. I have calculated the change in value between those two points. I want to assess which distribution most closely matches the variation ...
3
votes
0answers
53 views

bias and variance trade off related question

I am having difficulty to understand the expected squared errors formula in this website: $y=f(x)+e$ true regression line $\hat{y}=\hat{f}(x)$ your estimated regression line $error(x)=\bigg(\...
0
votes
1answer
18 views

Similarity Measure of Simulated Time Series vs Observed time Series

In my work I have an observed Time Series and Simulated ones. I want to compare the Light Curves and check for similarityto find out which simulated curve fits best respectivley which parameters ...
1
vote
1answer
17 views

Causation vs correlation problem in case of fixing other parameters and changing only one

I have a game that players play. D7 retention is defined as percentage of players who installed my game and after exactly 7 days later they played my game. Now I am doing analytics and it shows that ...
1
vote
2answers
26 views

Equation related to Smoothness

If you have a differentiable function $f:\mathbb{R}^d\rightarrow\mathbb{R}$ that is $\beta$-smooth (for all $v$ and all $w$, you have $\|\nabla f(v)-\nabla f(w)\| \leq \beta \|v-w\|$), how can you ...
1
vote
0answers
18 views

Approach for sentiment analysis of Flemish Twitter data (politics)

I have collected about 280.000 tweets posted by Flemish (Dutch) people concerning the previous elections. I used the twitter API and filtered for mentions of know political parties and politicians. ...
3
votes
1answer
54 views

Relation between an underlying function and the underlying probability distribition function of data

I heard and read a lot of times the following statements and got a lot of confusion over time. Statement 1: The goal of machine learning is to get a function from the given data Statement 2: ...
0
votes
0answers
9 views

how to consider sampling when trying to inference exponential parameter

consider this case: There is a price rate for a certain product that changes throw time, The price rate is changed every x minutes. This price get sampled/observed in a non-uniform distribution ...
1
vote
0answers
36 views

Similarity Measure Time Series

In my work I have an observed Time Series and Simulated ones. I want to compare the Light Curves and check for similarityto find out which simulated curve fits best respectivley which parameters ...
0
votes
0answers
13 views

Models to classify data based on different properties

I have a problem that I am trying to solve. I have some time series data, say monthly sales over the past 20 years. I have this information for each customer in a store, there are 10,000 customers (...
0
votes
0answers
27 views

Need Help: Fit simulation data to observed data, with no objectiv function

I want to fit my simulated Lightcurve data to my observed lightcurve. The simulation data is dependent on 32 parameters. Some of those parameters depend on one another. The observation data is not ...
1
vote
0answers
15 views

Identify appropriate variance stabilization transform

I have a heavy tailed data which is heteroscedastic in nature. I need to apply some ML algorithms on this data that are good for normal distribution. So I want to transform the data. I tried Box cox ...
1
vote
1answer
35 views

Comparing models using P-value

I have 2 different ML architectures for a translation task, I evaluate them using BLEU score (higher is better) I've run them 9 times each, yielding the following scores Architecture 1 | 36.52 | 36....
0
votes
0answers
15 views

What type of regression to use when modeling the relationship between a continuous variable and a discrete?

I'd like to create a regression model to find the marginal effect between usage (a rate from 0 - 100%) and tap-distance (1,2,3,4...). I'm working to find how a change in tap-distance to a feature will ...
2
votes
0answers
18 views

How to use zero-inflated negative binomial regression for binary classification task?

I am working on a binary classification problem and I am currently employing XGBoost. The dataset consists of several variables which are count variables. The problem is, these features are highly ...
0
votes
1answer
27 views

How to interpret two continous variables output using GAM?

I am sort of new in R and learning day by day. I really need help with GAM. I have to find out whether association is linear or non-linear by using GAM. The predictor variable is temperature at lag0 ...
0
votes
0answers
20 views

How to get rid off differences between prediction and actuals while predicting or forecasting time series data

I have performed Boxcox transformation on my time series data and processed it through ARIMA modeling. Converted prediction values to the actual. I see significant differences between actual and ...
0
votes
0answers
8 views

What are chain-energies in a CRF?

The CRF from keras_contrib includes the argument use_boundary: Boolean (default True) inidicating if trainable start-end chain energies should be added to model. ...
1
vote
1answer
27 views

Skew and Kurtosis are so similar?

I've been taking the histogram of the optical flow in videos and plotting the kurtosis and skewness of each frame. At the end of the video, I noticed that the skewness and kurtosis follow each other - ...
0
votes
0answers
21 views

t-test with a repeated variable

I want to conduct the t-test on a data I have to check if there is a significant difference in the "fear" emotion between valid and ...
0
votes
3answers
41 views

How to deal with highly skewed (on counts) dependent variables?

I am working on a binary classification problem and the dataset consists of several variables which are count variables. For example, how many times a customer defaulted on a broadband bill payment in ...
0
votes
0answers
11 views

Predict when a customer would buy from previous buying history (Mining Buying patterns)

I'm trying to predict when a customer would buy a product(which month of the year) based on the historical transactions. I have 3 years transactions data and good amount of customers are frequent ones....
0
votes
1answer
12 views

what is the representation/meaning/implication in real life of $P(\text{+})$ in the wiki Drug testing Example about Bayes' theorem

In probability theory and statistics, Bayes' theorem (alternatively Bayes' law or Bayes' rule) describes the probability of an event, based on prior knowledge of conditions that might be related to ...
0
votes
1answer
28 views

Is a new datapoint similar to my reference set?

I have a large reference data set and everyday I get new data points. I would like to know how can I test if those new data points are similar to the reference set? Thanks in advance. Maria
0
votes
1answer
14 views

Data driven project for a self-catering accommodation business

I want to apologise in advance for my ignorance but I'm hitting my head in a wall here as I'm not sure how to proceed with my project and I've got no experience in that field even though I'm doing a ...
0
votes
0answers
20 views

R: Difference in understanding of likelihood and Goodness-of-fit statistics

I am trying to fit normal distribution to my data using fitdist. The log likelihood of fit comes out to be -2788549. Since log likelihood is negative it means likelihood value lies in 0 and 1 which ...
0
votes
0answers
9 views

What statistical test would I use to test that both data sources are matching

Suppose I am only working with two datasets, Data Source 1 with columns Customer ID and ...
0
votes
0answers
36 views

How to detect anomalies (errors and exceptions) in log files?

Is this a good approach? So I'm working on a Root Cause Analysis system which should help find the cause/the root error of failed system builds (packaged in a tarball), through the analysis of log ...
0
votes
0answers
18 views

R: analysis of conditional heteroscedasticity using McLeod Li test

I am using R for McLeod Li test for conditional heteroscedasticity on time series before and after boxcox transformation. The null hypothesis for this test is of no heteroscedasticity. I observed that ...
0
votes
0answers
10 views

R: Error using Shapiro Wilk test for multivariate normality

I want to test if my data is multivariate normal. For this, I am using mvnnormtest package and Shapiro-Wilk multivariate normality test. I am passing the data to function as: ...
0
votes
0answers
8 views

R: Breusch-pagan test heteroscedasticity before and after boxcox transformation

I am trying breusch-pagan on a a feature regressed using multiple variables as: bptest(lm(V1~V2+V3+V4+V5+V6+V7+V8+V9+V10+V11+V12+V13+V14,data=data)) It shows BP ...
1
vote
1answer
16 views

Word2Vec / Doc2Vec - map change of words overtime

I have text data for 10 years and I want to run a Word2Vec model on the text data each year and calculate the distances between words and I would like to compare them over time. Are there any papers ...
1
vote
0answers
27 views

Using neural nets with just one input variable (without response/feature) [closed]

Hi i am using following data from R library ftnonpar: ...
2
votes
1answer
56 views

Variance in statistics vs machine learning

In basic statistics, variance is a measure the variability of the data about its mean. In machine learning, variance is a measure of learning the training data too well/capturing the noise in the data/...
1
vote
1answer
30 views

RMSProp - To a layman?

Can someone explain RMSprop in layman's terms? I have tried reading various resources but they don't talk about the intuition on couple of things: 1) Why is is different from SGD with momentum? 2) ...
0
votes
0answers
7 views

Is it worth graphing a correlation plot between your features and your target?

Is it worth graphing the correlation between the most important features and the target variable after you have done either PCA or L1 Regularization to identify the most important features? I guess ...
0
votes
1answer
51 views

Making Inference from a Correlation Heatmap

I have constructed a Heatmap of my dataset for visualization. I have searched on various sites regarding what can I infer from the heatmap and I am unable to get any clear understanding from them. I ...
0
votes
1answer
28 views

Simple EDA: what to visualize?

I want to visualize a data set for EDA having 8 parameters and 2 class labels. I am confused which parameters to select and on what basis? Also is I want to know whether it would be better to apply ...
0
votes
1answer
36 views

Effect of skewness in data

I am preparing classification model. Many of numeric variables are positives skewed. Should I change a distribution of variables to be more Gaussian?