Questions tagged [distribution]
The distribution tag has no usage guidance.
133
questions
0
votes
0answers
9 views
How to distribute train and test set equally in the matter of low and fast growth cases?
I'm working on a project which we are using python (Random Forest Regression) predicting tumor doubling time. when I plot different test sets I found that my data is not distributed equally into train ...
0
votes
0answers
15 views
How to perform a Monte Carlo simulation with continuous sampling using discrete quantiles?
Assume I have registered the duration of 10 tasks and built the table below with using this data:
Duration
For how many tasks it happened
4 days
5 task
6 days
2 task
8 days
2 task
10 days
1 task
...
0
votes
0answers
9 views
Use A Distribution To Measure Customer Success
I am working at a marketplace startup and one of our success metrics is average number of transactions. To better represent customer success I suggested that we leverage a distribution to measure the ...
0
votes
0answers
19 views
How to generate a positive skewed array?
As the title suggests, I would like to specify the task that I really need help with:
Given three possible distributions of an array:
I know that given a skewed dataset (the left and the right), then ...
0
votes
0answers
30 views
Data Transformation for Machine Learning Regression Task
I am performing a ML regression task, using XGBoost Regressor. I am using financial time series data, namely the Close price of the EUR/USD exchange rate which I will transform into geometric log ...
0
votes
0answers
21 views
A feature is still right-skewed after log scaling. How should it be normalized for machine learning?
I've attached two images below of a heavily right-skewed feature - call it x. I log scaled x, but it is still right-skewed and ...
1
vote
1answer
36 views
Check if distribution per week is the same
I have sales by customer (b2b) and by date. I want to check if the distribution per day inside weeks remains the same from week to week.
Initial dataset
Customer
Date
Sales
Alpha
2019-02-23
527
...
1
vote
0answers
14 views
Criteria for assessing difficulty of a question
I have a list of questions and how many times they have been answered correctly and incorrectly. Based on this, I applied the formula:
...
0
votes
0answers
23 views
Predicting a distribution (instead of a point estimate) with supervised machine learning
I'm new to machine learning, but it seems that supervised learning algorithms that aren't also considered statistical models (random forests, neural networks, etc...) are focused on predicting a point ...
1
vote
0answers
27 views
Testing a Binary Classifier
I have been training a binary multilayer perceptron on a database made out of roughly 3600 0 values, and 4 1 values. Afterwards, I'm testing the MLP on a test set made out of 7 0 values and 7 1 ...
1
vote
0answers
71 views
Estimating the (normal?) distribution of the grayscale values of a region in camera footage
The problem:
Given a small ($25\times25$) region on a $1080$p image extracted from camera footage of a railway system, I am trying to detect whether there is a train present or not. I proceed to take ...
0
votes
1answer
30 views
Non IID variables and SVM Classifier
I am training an SVM model to predict the trend of stock prices (one-day ahead predictions. Classification task). It Had completely slipped from my mind that SVMs assume IID data until I had a ...
1
vote
1answer
78 views
How to make a gaussian distribution in python considering mean. variance. skewness and kurtosis?
np.random.normal(mean,sigma,size) allows to create a gaussian distribution based only on mean and variance. I want to create a distribution based on ...
1
vote
0answers
111 views
Does Double Median Absolute Deviation work for every distribution for the purpose of outlier detection?
I am using Double Median Absolute Deviation for finding outliers in a 1-D data. As mean with standard deviation gets influenced easily by outliers, that's why I chose median based approach. And the ...
0
votes
0answers
10 views
How to implement distributed deep learning on small master-slave architecture through data parallelism approach?
I am a beginner and I would like to deploy the distributed deep learning model followed by Hadoop on a toy example. Like I want to use three Personals computers (PC), one would be work as a parameter ...
1
vote
2answers
168 views
Label distribution over training, validation and test [closed]
I am wondering over whether the number of classes distributed over my training, validation, and test label affects the model.
0
votes
1answer
49 views
derivation for expected value for variance
Hi Im taking a course about probability distribution in datascience and below is derivation of the expected value for the variance
Variance = expected value of the squared difference from mean for ...
0
votes
1answer
23 views
The upper range of a collected dataset is most likely accurate, but the rest may suffer biased omissions: How to call this phenomenon?
Background: In collecting a dataset of a specific unit ordered by a numeric variable, it is possible that the upper 'cloud' of the dataset is correct, while the 'tail' seems inaccurate.
I can thus ...
0
votes
1answer
23 views
Individual models gives quite same distribution on Test set, whereas Ensembling gives better result but very different distribution
I am working on a binary classification problem with unbalanced data (17% for positive class).
The problem is as following:
My three individual models when predicting on the test set (for which I don'...
1
vote
0answers
27 views
Normalizing variables with logarithmic shape
A simple model with two variables [A,B] to train, let's say, a logistic regression or any other classification model:
A: Flat distribution from 0 to 100.
B: A logarithmic distribution from 0
to a few ...
1
vote
1answer
32 views
How to bin a distribution data reported with different frequencies ( salaries ), showing mixed linearity and non-linearity?
I am researching on pay-scales, and wish to receive advise to treat data of salaries.
Objective
My interest is to approximate the salary corresponding to different hierarchical levels in an ...
0
votes
0answers
15 views
Gbm prediction distribution different to training data
I’m doing a regression on a dataset using lightgbm. For the training data the response variable has a non normal distribution which is multimodal. However, the predictions out-of-fold have are ...
2
votes
0answers
31 views
Non-Gaussian like distributions - Classifier of source data fails on target data
I ask you for help on a classification problem (classes are represented by the numbers 0,1 and 2). All features are extracted from time series data (fundamental is sinus shape).
I have a source ...
0
votes
0answers
31 views
Scalling features for competition participants
Hello there and Happy Holidays.
I have a data set with each row representing a race with 6 participants, with each participant having its own column for each feature. The target variable is ...
1
vote
0answers
139 views
Normal vs Uniform Distribution for machine learning
I have a dataset that follows Zipf's law such that the majority of the values are concentrated at one end, with the remaining items containing a very small percentage. Training on the dataset as is ...
1
vote
1answer
31 views
distribution difference between image and text
Once for the task of image captioning I've read that, the features extracted from image and text by deep networks are from two different worlds and got different distribution. My question is how is ...
2
votes
2answers
186 views
A/B Testing (Binomial Distribution vs Random Distribution)
When performing an A/B test for the number of clicks for users viewing (each view is an impression) two variants of an ad, a binomial distribution can be assumed where each variant has a constant ...
1
vote
1answer
67 views
What is a distribution-wise asymmetric measure?
I was trying to understand KL-Divergence, $$D_{KL} \langle P(X) \Vert P(Y) \rangle,$$ and was going through its Wikipedia article. It says the following
In contrast to variation of information, it is ...
1
vote
1answer
83 views
How to find the best fitting parametric distribution for an empirical dataset (stock returns)?
Given some real-valued empirical data (time series), I could convert it to a histogram to have an (non-parametric) empirical distribution of the data, but histograms are blocky and jagged.
Instead, I ...
2
votes
1answer
566 views
Normal distribution and Random Forest
I have big table in dataframe (600k rows) which has y column (the variable I want to predict) and other 4 other columns that are the X.
I have run RF regressor and I got score of 0.87 when I run it ...
1
vote
2answers
54 views
How to generate a random sample and distribute values based in an probability distribution?
I want to generate a random sample based on this probability distribution:
The line is the KDE of the histogram.
My random sample will have n values, the value is ...
0
votes
0answers
19 views
How to find a probability distribution the parameters of which do not impact each other like mean and variance in normal distribution do?
I need to find a probability distribution to fit my data. My data has two important features, duration and activity count. Duration means how long one sequence lasts and activity count means the ...
0
votes
1answer
113 views
How to interpret KDE distribution graph?
I would like to know how to interpret this distribution graph. I have been doing an exercise from the book called 'Python for Finance Cookbook' by Eryk Lewinson. It does not give an in-depth ...
0
votes
1answer
24 views
[under/over]-sampling teaches model the wrong distribution?
TLDR: Will under/oversampling during the training phase teach the model the wrong distribution and adversely affect accuracy?
Let us assume you want to train a classifier to differentiate between ...
0
votes
1answer
66 views
Unable to transform the variable to normal distribution
I have a variable called 'list_price' in my data ,which is not normally distributed.
Here is the screenshot of how the distribution looks like before transformation
Tried transforming it with the ...
0
votes
1answer
35 views
Getting a balanced sample across many variables
Let’s say each element in my population has several attributes.
Let’s call then A, B, C, D, E, F.
Let’s say, for simplicity, each attribute has 10 values (but could be any number between 2 and 30).
...
0
votes
1answer
43 views
Finding the worst affected industry due to COVID in terms of unemployment
My goal is to find the worst affected industries from COVID—19 in terms unemployment.
In terms of the data I will use for this task, I have a time series county-wise unemployment rate data of each ...
1
vote
1answer
24 views
Can the extent of variability within a dataset be reflected through clustering?
As an example: I need to compare the extent of variability amongst houses belonging to 4 different architectural eras - I want to see how different the houses are within each group and then compare ...
1
vote
1answer
38 views
What is the distribution of the goal feature in this dataset
https://www.kaggle.com/kemical/kickstarter-projects
I was checking the distribution of the "goal" feature in this 2016 KickStarter data set. I figure goal would be more normally distributed ...
1
vote
1answer
23 views
A good machine learning approach for distribution of a whole?
So I had done with different classification, regression and clustering approaches for predictions of values etc. I was wondering if there is a machine learning approach for distribution of a whole ...
0
votes
1answer
276 views
Naive Bayes vs Full Bayes model classifiers
I have a hard time to understand when Naive Bayes works better than Full Bayes.
In general, i know that naive bayes does the assumption that features are independent given the class. However, if ...
1
vote
1answer
28 views
Understand how to simulate a statistics [closed]
This solution describes how to simulate statistics to find a confidence interval.
A journalist called 1000 people in town to ask who will they be voting for out of candidates A and B.
The observed ...
1
vote
1answer
26 views
How to draw a sample from data set with respect to a given categorical or numerical variable based on given freely chosen distribution? (Python)
Say I have a data set for some past period. Now new data appears and for a given variable in the data and we find that the distributions have shifted (for example with "age" it would be that suddenly ...
1
vote
0answers
38 views
Professionals appear to interpret sample correlation (e.g. Karl Pearson) as if it represents linear correlation. Is it the correct interpretation? [closed]
I am stressed following the wrong interpretation. What is the correct way of understanding a correlation coefficient.
1
vote
1answer
61 views
1
vote
0answers
13 views
Compare the variances of two categorical distributions in a repeated measure design
I ran two model-building procedures with different parameters on the same sample and obtained the selection of my optimized hyperparameter for each outer fold (each of the analyses had 100 outer folds ...
1
vote
0answers
30 views
Text data distributions comparison
I would like to know what is the best method to compare the different text data distributions. I am working on text classification. I built a model using the old dataset. Now, I would like to know, ...
1
vote
1answer
65 views
How is the E(X) of a Poisson distribution lambda? [closed]
I was recently learning about the Poisson distribution, and was very perplexed about the E(X) equaling lambda. Like how did lambda even come into the picture here, isn't it a symbol of wavelength?
...
2
votes
3answers
1k views
Is there a cost associated with converting Koalas dataframe to Spark dataframe?
I know that pandas works "under the hood" with numpy arrays stored in dictionaries. In contrast, Koalas works with the underlying Spark framework. Does that mean that there is no extra cost associated ...
2
votes
1answer
399 views
How to resample one dataset to conform to the distribution of another dataset?
I have two datasets with 20 features, but with different feature distributions (DS_A and DS_B). How can I sample the DS_A to make its distribution similar to DS_B, with respect to multiple features??
...
