# Questions tagged [sampling]

The tag has no usage guidance.

137 questions
Filter by
Sorted by
Tagged with
22 views

### What is the most common practice of generating (X,Y) from an arbitrary CDF or PDF?

So I can generate X from arbitrary CDF F(x) by the procedure above. Can it be generalized to two variables? How, exactly? If not, what's the best way to generate <...
72 views

### Bayesian network in Python: both construction and sampling

For a project, I need to create synthetic categorical data containing specific dependencies between the attributes. This can be done by sampling from a pre-defined Bayesian Network. After some ...
17 views

### How to draw a sample from data set with respect to a given categorical or numerical variable based on given freely chosen distribution? (Python)

Say I have a data set for some past period. Now new data appears and for a given variable in the data and we find that the distributions have shifted (for example with "age" it would be that suddenly ...
26 views

### Is there a name for this form of sampling?

I had an idea to combine oversampling and undersampling in the following way: Compute the average number of individuals in each class. For classes with a number of individuals greater than this ...
19 views

### How to generate a random sample and distribute values based in an probability distribution?

I want to generate a random sample based on this probability distribution: The line is the KDE of the histogram. My random sample will have n values, the value is ...
29 views

### Training a Variational Autoencoder (VAE) for Random Number Generation

I have a complicated 20-dimensional multi-modal distribution and consider training a VAE to learn an approximation of it using 2000 samples. But particularly, with the aim to subsequently generate ...
14 views

### Training a Variational Autoencoder (VAE) for Non-Uniform Random Number Generation

I have a complicated 20-dimensional non-uniform distribution and would like generate samples from it. I have considered training a VAE to do so, but my problems are the following: Is my approach ...
11 views

### Positive records : no behavioral data

0 I'm working on a classification model aimed at identifying if behavioral activity within an account (b2b - one account, many contacts) can predict or not an opportunity generation ( a salesperson ...
21 views

### Smote oversampling on strings

I have an imbalanced dataset with most transactions being OK and 1% transactions marked as fraud. This dataset has columns with integers (time and transaction amount and timestamp) as well as strings (...
24k views

### SMOTE and multi class oversampling

I have read that the SMOTE package is implemented for binary classification. In the case of n classes, it creates additional examples for the smallest class. Can I balance all the classes by running ...
37 views

### Dealing with large data: selecting a sample

I'm given a data set to create a model that would predict whether a certain supply chain would be able to deliver the goods without delay or not. I'm doing this in Python. The data set has 93000 ...
24 views

### Sampling trying to keep as much multivariate variance as possible

I was thinking if anyone considered a sampling technique that would try to aim keeping as much of the variance as possible (e.g. as many unique values, or very widely distributed continuous variables)....
687 views

### Oversampling before Cross-Validation, is it a problem?

I have a multi-class classification problem to solve which is highly imbalanced. Obviously I'm doing oversampling, but I'm doing cross-validation with the over-sampled dataset, as a result of which I ...
18 views

### What's the order in applying SMOTE transformation in a pipeline?

Here's the thing, I have an imbalanced data and I was thinking about using SMOTE transformation. However, when doing that using a sklearn pipeline, I get an error because of missing values. This is my ...
24 views

### How should I sample from a mixture distribution?

Let's say we have a mixture distribution, defined by density $f(x)= w_1 p_1(x) + w_2 p_2(x)$, where $w_i$ is a scalar weight. Furthermore, we have efficient methods to evaluate the pdf and cdf/icdf ...
106 views

### Downsample GPS track

I am working with GPS track files (list of X and Y coordinates). I have tracks with a high sampling rate and want to downsample the track for easier handling. The obvious way would be to create a new ...
990 views

### train_test_split ValueError: Input contains NaN

I tried to do a stratified sampling by way of train_test_split in order to save myself some trouble later. So I wrote the following lines: ...
12 views

### Getting a balanced sample across many variables

Letās say each element in my population has several attributes. Letās call then A, B, C, D, E, F. Letās say, for simplicity, each attribute has 10 values (but could be any number between 2 and 30). ...
24 views

### Difference between test statistic and sample statistic?

Test statistic and sample statistic are used to check statistical significance. How do I understand the procedure in background of reaching a conclusion about an effect-size estimate.
15 views

### HR employee attrition modeling - making a balanced sample question

I have dataset 1 (stayers) consisting of 1500 record of HR data demographic data of employees (11 features) who currently are in the company. Dataset 2 (leavers) contains 180 records -same features- ...
21 views

### How to downsample a dataset with constraints in Python?

I have a dataset that has columns a, b, and others. Both a and ...
12 views

### Sequential sampling from Gaussian conditional not working

I'm trying to sequentially sample from a Gaussian Process prior. The problem is that the samples eventually converge to zero or diverge to infinity. I'm using the basic conditionals described e.g. ...
24 views

### Generating artificial data to extend learning set

I have dataset containing 42 instances(X) and one final Y on which i want to perform LASSO regression.All are continuous and numerical. As the sample size small, I wish to extend it. I am kind of ...
13 views

### Adaptive Sampling Strategies for SVM?

I am an Engineer interested in creating a surrogate model of a certain phenomenon in the context of reliability engineering. Essentially my quantity of interest is the Limit state function/stability ...
49 views

### Bootstrap clarification for datasets with multiple columns

How do I bootstrap a Dataset/DataFrame with multiple continuous and categorical columns? For eg: Say I'm trying to bootstrap the colour distribution of M&M's and I have 50 bags (samples) each with ...
37 views

### Chi Square Test Goodness of Fit

I want to use a chi square test but I'm unsure if I'm using it right. The KickStarter website shows the frequency of main categories projects. It is updated once a day. I got a data set of KickStarter ...
14 views

### Uniform convergence garantee on sample complexity

I can't understand why the Uniform Convergence guarantees an upper bound and not a lower bound on sample complexity as stated on [1] Corollary 4.4. If a class $H$ has the uniform convergence property ...
82 views

### Generating a set of different scenarios based on some initial observations

I have a in my hands 3 different time series which model 3 different scenarios (base, downside, upside). Every of this time-series depends on a set of 11 different attributes, which take values for ...
101 views

### Decision trees, categorizacion and oversampling

I want to create a model to predict the propensity to buy a certain product. As my proportion of 1's is very low, I decided to apply oversampling (to get a 10% of 1's and a 90% of 0's). Now, I want to ...
87 views

### What is difference between Standard Normal Distribution and Mean Normalization approaches to feature-scaling?

The tag feature-scaling seems to convey that one of the scaling methods is Standard Normal Distribution. Further, I read an Answer on this site saying that Mean Normalization is a form of feature ...
103 views

### Correctly evaluate model with oversampling and cross-validation

I'm dealing with a classic case of dataset with binary imbalanced target (event 3%, non event 97%). My idea is to apply some sort of sampling (over/under, SMOTE etc.) to address the issue. As I see, ...
72 views

### What is the effect of oversampling on logistic regression?

I am building a model to predict if a customer will use a coupon or not for a given campaign. I am using logistic regression for this model. I took 5 previous campaigns and generally for each campaign ...
25 views

### Representation sample size- n [closed]

Need help with identifying a representation sample size 'n'. Let's say I have a very large population- infinite number of participants. I am picking the random sample from this infinite population. I ...
100 views

### Overfitted model produces similar AUC on test set, so which model do I go with?

I was trying to compare the effect of running GridSearchCV on a dataset which was oversampled prior and oversampled after the training folds are selected. The oversampling approach I used was random ...
692 views

### How to do k-folds in python whilst splitting into 3 sets?

Consider the following data: ...
52 views

### How to estimate the accuracy on a large dataset?

Given that I have a deep learning model(handover from former colleague). For some reason, the train/dev set was missing. In my situation, I want to classify my dataset into 100 categories. The ...
19 views

### Question regarding strata in Geron's book

In the book "Hands-On Machine Learning with Scikit-Learn and Tensorflow" by AurĆ©lien GĆ©ron. There is a regression project explained. My doubt is regarding his example for 'stratified sampling'. He ...
95 views

### Forcing class imbalance to mirror the target data

I'm trying to do binary classification on some data, my source data has a class split of 40% A / 60% B while my target data has a split of 70% A / 30% B. Is it a worthwhile strategy to use SMOTE to ...
22 views

### Time Series Model (n-beats paper) how is sampling / training done?

does any one know what this means? It is taken from the paper https://openreview.net/pdf?id=r1ecqn4YwB (n-beats time series model). To update network parameters for one horizon, we sample train ...
2k views

### SMOTE for regression

I am looking into upsampling an imbalanced dataset for a regression problem (Numerical target variables) in python. I attached paper and R package that implement SMOTE for regression, can anyone ...
21 views

### dealing with imbalanced data for multi-class problem

Based on the experiments I run for a number of times, and the reading I did on imbalanced data for a multiclassification problem such as this paper, resampling techniques like ...
347 views

### Downsampling the dataset to create balanced dataset for neural models

I have a classification dataset with 10k instances and 4 classes and it is unbalanced. 7000 of it belongs to first class, 2000 of it belongs to second 800 of it belongs to third class and remaining ...
9 views

### Estimate machine count for API dependency

I have a SDK, with many APIs, which is used by many apps. Those apps are installed on many machines. I get data containing machine Id, app name and API info . The data is sampled such that 2% of the ...
24 views

### Imbalanced Data Classification

This is my code with which I tried to attack my imbalanced dataset from here. Now I have 2 questions. When I try to split my data into train/test data and try to run the table() function, I always ...
20 views

### Over/Under Sampling for Multi-classification

I'm trying to apply xgboost and random forest for over and under sampling For imbalanced data: train shape -> (199991, 23) However, reverse my expectation. ...
4k views

### K-Fold Cross validation confusion?

I am using K-Fold cross validation to test my trained model, but was amazed that for every K-fold the accuracy is different. For instance, if I use 5 K-fold, every fold has a different accuracy. So, ...
72 views

### Sampling randomly from pd.DataFrame, but ignoring NaN values

I am trying to sample random values from a dataframe where the NaN values should be ignored, without dropping the entire row or column. My sampling function at the moment looks like this: ...
258 views

### Why gradient boosting uses sampling without replacement?

In Random Forest each tree is built selecting a sample with replacement (bootstrap). And I assumed that Gradient Boosting's trees were selected with the same sampling technique. (@BenReiniger ...