Questions tagged [sampling]
The sampling tag has no usage guidance.
134
questions
0
votes
0answers
17 views
What's the order in applying SMOTE transformation in a pipeline?
Here's the thing, I have an imbalanced data and I was thinking about using SMOTE transformation. However, when doing that using a sklearn pipeline, I get an error because of missing values.
This is my ...
1
vote
1answer
22 views
How should I sample from a mixture distribution?
Let's say we have a mixture distribution, defined by density $f(x)= w_1 p_1(x) + w_2 p_2(x)$, where $w_i$ is a scalar weight. Furthermore, we have efficient methods to evaluate the pdf and cdf/icdf ...
0
votes
0answers
24 views
Difference between test statistic and sample statistic?
Test statistic and sample statistic are used to check statistical significance. How do I understand the procedure in background of reaching a conclusion about an effect-size estimate.
0
votes
0answers
15 views
HR employee attrition modeling - making a balanced sample question
I have dataset 1 (stayers) consisting of 1500 record of HR data demographic data of employees (11 features) who currently are in the company. Dataset 2 (leavers) contains 180 records -same features- ...
1
vote
1answer
24 views
Training a Variational Autoencoder (VAE) for Random Number Generation
I have a complicated 20-dimensional multi-modal distribution and consider training a VAE to learn an approximation of it using 2000 samples. But particularly, with the aim to subsequently generate ...
0
votes
0answers
21 views
How to downsample a dataset with constraints in Python?
I have a dataset that has columns a, b, and others. Both a and ...
1
vote
0answers
12 views
Sequential sampling from Gaussian conditional not working
I'm trying to sequentially sample from a Gaussian Process prior.
The problem is that the samples eventually converge to zero or diverge to infinity.
I'm using the basic conditionals described e.g. ...
1
vote
0answers
22 views
Generating artificial data to extend learning set
I have dataset containing 42 instances(X) and one final Y on which i want to perform LASSO regression.All are continuous and numerical. As the sample size small, I wish to extend it. I am kind of ...
1
vote
0answers
12 views
Adaptive Sampling Strategies for SVM?
I am an Engineer interested in creating a surrogate model of a certain phenomenon in the context of reliability engineering. Essentially my quantity of interest is the Limit state function/stability ...
0
votes
1answer
48 views
Bootstrap clarification for datasets with multiple columns
How do I bootstrap a Dataset/DataFrame with multiple continuous and categorical columns?
For eg: Say I'm trying to bootstrap the colour distribution of M&M's and I have 50 bags (samples) each with ...
0
votes
0answers
14 views
Uniform convergence garantee on sample complexity
I can't understand why the Uniform Convergence guarantees an upper bound and not a lower bound on sample complexity as stated on [1]
Corollary 4.4.
If a class $H$ has the uniform convergence property ...
0
votes
1answer
11 views
Getting a balanced sample across many variables
Let’s say each element in my population has several attributes.
Let’s call then A, B, C, D, E, F.
Let’s say, for simplicity, each attribute has 10 values (but could be any number between 2 and 30).
...
0
votes
1answer
61 views
What is the effect of oversampling on logistic regression?
I am building a model to predict if a customer will use a coupon or not for a given campaign. I am using logistic regression for this model. I took 5 previous campaigns and generally for each campaign ...
1
vote
0answers
25 views
Representation sample size- n [closed]
Need help with identifying a representation sample size 'n'. Let's say I have a very large population- infinite number of participants. I am picking the random sample from this infinite population. I ...
1
vote
0answers
37 views
Chi Square Test Goodness of Fit
I want to use a chi square test but I'm unsure if I'm using it right. The KickStarter website shows the frequency of main categories projects. It is updated once a day. I got a data set of KickStarter ...
0
votes
0answers
19 views
Question regarding strata in Geron's book
In the book "Hands-On Machine Learning with Scikit-Learn and Tensorflow" by Aurélien Géron. There is a regression project explained. My doubt is regarding his example for 'stratified sampling'.
He ...
0
votes
1answer
87 views
What is difference between Standard Normal Distribution and Mean Normalization approaches to feature-scaling?
The tag feature-scaling seems to convey that one of the scaling methods is Standard Normal Distribution. Further, I read an Answer on this site saying that Mean Normalization is a form of feature ...
0
votes
0answers
13 views
Smote oversampling on strings
I have an imbalanced dataset with most transactions being OK and 1% transactions marked as fraud. This dataset has columns with integers (time and transaction amount and timestamp) as well as strings (...
0
votes
0answers
17 views
Time Series Model (n-beats paper) how is sampling / training done?
does any one know what this means?
It is taken from the paper https://openreview.net/pdf?id=r1ecqn4YwB (n-beats time series model).
To update network parameters for one horizon, we sample train ...
1
vote
1answer
16 views
How to draw a sample from data set with respect to a given categorical or numerical variable based on given freely chosen distribution? (Python)
Say I have a data set for some past period. Now new data appears and for a given variable in the data and we find that the distributions have shifted (for example with "age" it would be that suddenly ...
0
votes
1answer
24 views
Is there a name for this form of sampling?
I had an idea to combine oversampling and undersampling in the following way:
Compute the average number of individuals in each class. For classes with a number of individuals greater than this ...
0
votes
0answers
21 views
dealing with imbalanced data for multi-class problem
Based on the experiments I run for a number of times, and the reading I did on imbalanced data for a multiclassification problem such as this paper, resampling techniques like ...
0
votes
1answer
21 views
Dealing with large data: selecting a sample
I'm given a data set to create a model that would predict whether a certain supply chain would be able to deliver the goods without delay or not. I'm doing this in Python.
The data set has 93000 ...
0
votes
0answers
9 views
Estimate machine count for API dependency
I have a SDK, with many APIs, which is used by many apps. Those apps are installed on many machines. I get data containing machine Id, app name and API info . The data is sampled such that 2% of the ...
0
votes
0answers
24 views
Imbalanced Data Classification
This is my code with which I tried to attack my imbalanced dataset from here.
Now I have 2 questions.
When I try to split my data into train/test data and try to run the table() function, I always ...
2
votes
1answer
1k views
SMOTE for regression
I am looking into upsampling an imbalanced dataset for a regression problem (Numerical target variables) in python.
I attached paper and R package that implement SMOTE for regression, can anyone ...
-1
votes
1answer
53 views
Sampling randomly from pd.DataFrame, but ignoring NaN values
I am trying to sample random values from a dataframe where the NaN values should be ignored, without dropping the entire row or column.
My sampling function at the moment looks like this:
...
0
votes
1answer
19 views
Over/Under Sampling for Multi-classification
I'm trying to apply xgboost and random forest for over and under sampling
For imbalanced data:
train shape -> (199991, 23)
However, reverse my expectation. ...
3
votes
0answers
39 views
SuperLearner Cross validation with iid time series
I created a number of ML models in R and I aim at combining them to form an ensemble.
I learned about SuperLearner library which cross validates many models and returns the weight to each model in ...
5
votes
1answer
227 views
Why gradient boosting uses sampling without replacement?
In Random Forest each tree is built selecting a sample with replacement (bootstrap). And I assumed that Gradient Boosting's trees were selected with the same sampling technique. (@BenReiniger ...
1
vote
1answer
36 views
How to resample one dataset to conform to the distribution of another dataset?
I have two datasets with 20 features, but with different feature distributions (DS_A and DS_B). How can I sample the DS_A to make its distribution similar to DS_B, with respect to multiple features??
...
1
vote
1answer
27 views
How to compute modulo of a hash?
Let's say that I have a set of users in my database, that have GUIDs as their IDs. I use xxhash to generate fixed-length hashes for each value, so that I can then ...
2
votes
1answer
21 views
Can sampling like SMOTE/UP/DOWN applied on Validation set?
I am trying to predict classification problem. For that I have used Ranger, Xgboost and naive bayes. My Response class is imbalance . 92:8 ratio. My positive Response is only 8% of whole data.
...
1
vote
1answer
51 views
How to estimate the accuracy on a large dataset?
Given that I have a deep learning model(handover from former colleague).
For some reason, the train/dev set was missing.
In my situation, I want to classify my dataset into 100 categories.
The ...
2
votes
0answers
35 views
Difference between Gibbs sampling and variational Bayes inference
After reading in blogs and books, I came to the conclusion that Gibbs sampling and variation Bayes are methods for estimating or inference of posterior. Below link described but it's difficult to ...
3
votes
0answers
29 views
Stratified selection based on the y response creates a bias in information (Berkson's bias)?
I have a database which has individuals in different months and a target variable which indicates wheter an event happened or not, let's say:
...
2
votes
1answer
38 views
Is there an algorithm for sampling shortest paths?
I have a triangle matrix NxN of distances between vertices (vertex i connected only with vectices j>i) and I'd like to sample path from first to the last and use it as a training sample. Is there an ...
2
votes
1answer
64 views
Using Majority Class to Predict Minority Class
Suppose I want to train a binary model in order to predict the probability of who will buy a personal loan and in the dataset only 5 percent of the examples are people who marked as bought a personal ...
5
votes
1answer
435 views
How are samples selected from training data in Xgboost
In Random Forest, each tree is not fed with the full batch of training data, only a sample.
How does this work for Xgboost? If this sampling happens as well, how does it work for this ML algorithm?
2
votes
1answer
21 views
Gaussian Process for Classification: How to do predictions using MCMC methods
Problem
I was reading about Gaussian Processes for regression in the "Gaussian Processes for Classification" textbook and in a few other online resources. Everywhere I look people seem to avoid ...
0
votes
0answers
91 views
undersampling problem in imblanced dataset ValueError: Unknown label type: 'continuous'
I would like to undersampling the data but I encounter the following error?
...
2
votes
2answers
249 views
How to perform bootstrap validation?
I am working on a binary classification problem.
I ran cross-validation and grid-search on train data. Later I validated the model on my test data as shown below
...
1
vote
2answers
58 views
Bayesian network in Python: both construction and sampling
For a project, I need to create synthetic categorical data containing specific dependencies between the attributes. This can be done by sampling from a pre-defined Bayesian Network. After some ...
3
votes
2answers
53 views
Does Sampling size matters in Multi classification Model
I am working on a multi class classification model where few of the class are with less data compare to other classes.
I used random sampling technique to create a sample from the population keeping ...
1
vote
1answer
131 views
Adjust predicted probability after smote
i have an imbalance data set and I used smote to oversample the minority class and undersample the majority class. now, I want to check the test AUC using predict_proba of the model.
I have two ...
0
votes
0answers
11 views
What are accepted methods for sampling data that will be sampled?
I have a very large (TBs) dataset that I need to sample to a smaller more manageable size. However, I know that the sample will also be sampled to get statistics after it leaves my hands. Are there ...
0
votes
1answer
102 views
Correctly evaluate model with oversampling and cross-validation
I'm dealing with a classic case of dataset with binary imbalanced target (event 3%, non event 97%). My idea is to apply some sort of sampling (over/under, SMOTE etc.) to address the issue.
As I see, ...
0
votes
2answers
946 views
How to find whether a dataset is blanced or imbalanced?
I have few dataset to experiment classification(Multi-class). These datasets are about 400GB. I wanted to know whether the dataset is balanced or imbalanced. How to know that dataset is balance or ...
0
votes
0answers
14 views
How to compare two 3d vectors, which differ by sampling start time
I have some curve, assume it is given by some function f(x(t),y(t)).
This 3d curve is sampled twice:
t=0..99, no noise. This ...
2
votes
0answers
152 views
Training data requirements for NLP models
Are there general guidelines for how much data is required for natural language processing (NLP) classification models? I understand this may depend on the text quality, text length, how accurate the ...
