Questions tagged [sampling]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0
votes
1answer
22 views

What is the most common practice of generating (X,Y) from an arbitrary CDF or PDF?

So I can generate X from arbitrary CDF F(x) by the procedure above. Can it be generalized to two variables? How, exactly? If not, what's the best way to generate <...
1
vote
2answers
72 views

Bayesian network in Python: both construction and sampling

For a project, I need to create synthetic categorical data containing specific dependencies between the attributes. This can be done by sampling from a pre-defined Bayesian Network. After some ...
1
vote
1answer
17 views

How to draw a sample from data set with respect to a given categorical or numerical variable based on given freely chosen distribution? (Python)

Say I have a data set for some past period. Now new data appears and for a given variable in the data and we find that the distributions have shifted (for example with "age" it would be that suddenly ...
0
votes
1answer
26 views

Is there a name for this form of sampling?

I had an idea to combine oversampling and undersampling in the following way: Compute the average number of individuals in each class. For classes with a number of individuals greater than this ...
1
vote
2answers
19 views

How to generate a random sample and distribute values based in an probability distribution?

I want to generate a random sample based on this probability distribution: The line is the KDE of the histogram. My random sample will have n values, the value is ...
1
vote
1answer
29 views

Training a Variational Autoencoder (VAE) for Random Number Generation

I have a complicated 20-dimensional multi-modal distribution and consider training a VAE to learn an approximation of it using 2000 samples. But particularly, with the aim to subsequently generate ...
0
votes
0answers
14 views

Training a Variational Autoencoder (VAE) for Non-Uniform Random Number Generation

I have a complicated 20-dimensional non-uniform distribution and would like generate samples from it. I have considered training a VAE to do so, but my problems are the following: Is my approach ...
0
votes
0answers
11 views

Positive records : no behavioral data

0 I'm working on a classification model aimed at identifying if behavioral activity within an account (b2b - one account, many contacts) can predict or not an opportunity generation ( a salesperson ...
0
votes
0answers
21 views

Smote oversampling on strings

I have an imbalanced dataset with most transactions being OK and 1% transactions marked as fraud. This dataset has columns with integers (time and transaction amount and timestamp) as well as strings (...
4
votes
3answers
24k views

SMOTE and multi class oversampling

I have read that the SMOTE package is implemented for binary classification. In the case of n classes, it creates additional examples for the smallest class. Can I balance all the classes by running ...
1
vote
2answers
37 views

Dealing with large data: selecting a sample

I'm given a data set to create a model that would predict whether a certain supply chain would be able to deliver the goods without delay or not. I'm doing this in Python. The data set has 93000 ...
1
vote
1answer
24 views

Sampling trying to keep as much multivariate variance as possible

I was thinking if anyone considered a sampling technique that would try to aim keeping as much of the variance as possible (e.g. as many unique values, or very widely distributed continuous variables)....
3
votes
2answers
687 views

Oversampling before Cross-Validation, is it a problem?

I have a multi-class classification problem to solve which is highly imbalanced. Obviously I'm doing oversampling, but I'm doing cross-validation with the over-sampled dataset, as a result of which I ...
0
votes
0answers
18 views

What's the order in applying SMOTE transformation in a pipeline?

Here's the thing, I have an imbalanced data and I was thinking about using SMOTE transformation. However, when doing that using a sklearn pipeline, I get an error because of missing values. This is my ...
1
vote
1answer
24 views

How should I sample from a mixture distribution?

Let's say we have a mixture distribution, defined by density $f(x)= w_1 p_1(x) + w_2 p_2(x)$, where $w_i$ is a scalar weight. Furthermore, we have efficient methods to evaluate the pdf and cdf/icdf ...
1
vote
1answer
106 views

Downsample GPS track

I am working with GPS track files (list of X and Y coordinates). I have tracks with a high sampling rate and want to downsample the track for easier handling. The obvious way would be to create a new ...
0
votes
1answer
990 views

train_test_split ValueError: Input contains NaN

I tried to do a stratified sampling by way of train_test_split in order to save myself some trouble later. So I wrote the following lines: ...
0
votes
1answer
12 views

Getting a balanced sample across many variables

Let’s say each element in my population has several attributes. Let’s call then A, B, C, D, E, F. Let’s say, for simplicity, each attribute has 10 values (but could be any number between 2 and 30). ...
0
votes
0answers
24 views

Difference between test statistic and sample statistic?

Test statistic and sample statistic are used to check statistical significance. How do I understand the procedure in background of reaching a conclusion about an effect-size estimate.
0
votes
0answers
15 views

HR employee attrition modeling - making a balanced sample question

I have dataset 1 (stayers) consisting of 1500 record of HR data demographic data of employees (11 features) who currently are in the company. Dataset 2 (leavers) contains 180 records -same features- ...
0
votes
0answers
21 views
1
vote
0answers
12 views

Sequential sampling from Gaussian conditional not working

I'm trying to sequentially sample from a Gaussian Process prior. The problem is that the samples eventually converge to zero or diverge to infinity. I'm using the basic conditionals described e.g. ...
1
vote
0answers
24 views

Generating artificial data to extend learning set

I have dataset containing 42 instances(X) and one final Y on which i want to perform LASSO regression.All are continuous and numerical. As the sample size small, I wish to extend it. I am kind of ...
1
vote
0answers
13 views

Adaptive Sampling Strategies for SVM?

I am an Engineer interested in creating a surrogate model of a certain phenomenon in the context of reliability engineering. Essentially my quantity of interest is the Limit state function/stability ...
0
votes
1answer
49 views

Bootstrap clarification for datasets with multiple columns

How do I bootstrap a Dataset/DataFrame with multiple continuous and categorical columns? For eg: Say I'm trying to bootstrap the colour distribution of M&M's and I have 50 bags (samples) each with ...
1
vote
0answers
37 views

Chi Square Test Goodness of Fit

I want to use a chi square test but I'm unsure if I'm using it right. The KickStarter website shows the frequency of main categories projects. It is updated once a day. I got a data set of KickStarter ...
0
votes
0answers
14 views

Uniform convergence garantee on sample complexity

I can't understand why the Uniform Convergence guarantees an upper bound and not a lower bound on sample complexity as stated on [1] Corollary 4.4. If a class $H$ has the uniform convergence property ...
1
vote
1answer
82 views

Generating a set of different scenarios based on some initial observations

I have a in my hands 3 different time series which model 3 different scenarios (base, downside, upside). Every of this time-series depends on a set of 11 different attributes, which take values for ...
6
votes
1answer
101 views

Decision trees, categorizacion and oversampling

I want to create a model to predict the propensity to buy a certain product. As my proportion of 1's is very low, I decided to apply oversampling (to get a 10% of 1's and a 90% of 0's). Now, I want to ...
0
votes
1answer
87 views

What is difference between Standard Normal Distribution and Mean Normalization approaches to feature-scaling?

The tag feature-scaling seems to convey that one of the scaling methods is Standard Normal Distribution. Further, I read an Answer on this site saying that Mean Normalization is a form of feature ...
0
votes
1answer
103 views

Correctly evaluate model with oversampling and cross-validation

I'm dealing with a classic case of dataset with binary imbalanced target (event 3%, non event 97%). My idea is to apply some sort of sampling (over/under, SMOTE etc.) to address the issue. As I see, ...
0
votes
1answer
72 views

What is the effect of oversampling on logistic regression?

I am building a model to predict if a customer will use a coupon or not for a given campaign. I am using logistic regression for this model. I took 5 previous campaigns and generally for each campaign ...
1
vote
0answers
25 views

Representation sample size- n [closed]

Need help with identifying a representation sample size 'n'. Let's say I have a very large population- infinite number of participants. I am picking the random sample from this infinite population. I ...
3
votes
2answers
100 views

Overfitted model produces similar AUC on test set, so which model do I go with?

I was trying to compare the effect of running GridSearchCV on a dataset which was oversampled prior and oversampled after the training folds are selected. The oversampling approach I used was random ...
1
vote
1answer
52 views

How to estimate the accuracy on a large dataset?

Given that I have a deep learning model(handover from former colleague). For some reason, the train/dev set was missing. In my situation, I want to classify my dataset into 100 categories. The ...
0
votes
0answers
19 views

Question regarding strata in Geron's book

In the book "Hands-On Machine Learning with Scikit-Learn and Tensorflow" by Aurélien Géron. There is a regression project explained. My doubt is regarding his example for 'stratified sampling'. He ...
0
votes
2answers
95 views

Forcing class imbalance to mirror the target data

I'm trying to do binary classification on some data, my source data has a class split of 40% A / 60% B while my target data has a split of 70% A / 30% B. Is it a worthwhile strategy to use SMOTE to ...
0
votes
0answers
22 views

Time Series Model (n-beats paper) how is sampling / training done?

does any one know what this means? It is taken from the paper https://openreview.net/pdf?id=r1ecqn4YwB (n-beats time series model). To update network parameters for one horizon, we sample train ...
2
votes
1answer
2k views

SMOTE for regression

I am looking into upsampling an imbalanced dataset for a regression problem (Numerical target variables) in python. I attached paper and R package that implement SMOTE for regression, can anyone ...
0
votes
0answers
21 views

dealing with imbalanced data for multi-class problem

Based on the experiments I run for a number of times, and the reading I did on imbalanced data for a multiclassification problem such as this paper, resampling techniques like ...
-1
votes
1answer
347 views

Downsampling the dataset to create balanced dataset for neural models

I have a classification dataset with 10k instances and 4 classes and it is unbalanced. 7000 of it belongs to first class, 2000 of it belongs to second 800 of it belongs to third class and remaining ...
0
votes
0answers
9 views

Estimate machine count for API dependency

I have a SDK, with many APIs, which is used by many apps. Those apps are installed on many machines. I get data containing machine Id, app name and API info . The data is sampled such that 2% of the ...
0
votes
0answers
24 views

Imbalanced Data Classification

This is my code with which I tried to attack my imbalanced dataset from here. Now I have 2 questions. When I try to split my data into train/test data and try to run the table() function, I always ...
0
votes
1answer
20 views

Over/Under Sampling for Multi-classification

I'm trying to apply xgboost and random forest for over and under sampling For imbalanced data: train shape -> (199991, 23) However, reverse my expectation. ...
2
votes
1answer
4k views

K-Fold Cross validation confusion?

I am using K-Fold cross validation to test my trained model, but was amazed that for every K-fold the accuracy is different. For instance, if I use 5 K-fold, every fold has a different accuracy. So, ...
-1
votes
1answer
72 views

Sampling randomly from pd.DataFrame, but ignoring NaN values

I am trying to sample random values from a dataframe where the NaN values should be ignored, without dropping the entire row or column. My sampling function at the moment looks like this: ...
6
votes
1answer
258 views

Why gradient boosting uses sampling without replacement?

In Random Forest each tree is built selecting a sample with replacement (bootstrap). And I assumed that Gradient Boosting's trees were selected with the same sampling technique. (@BenReiniger ...
2
votes
1answer
38 views

Is there an algorithm for sampling shortest paths?

I have a triangle matrix NxN of distances between vertices (vertex i connected only with vectices j>i) and I'd like to sample path from first to the last and use it as a training sample. Is there an ...
1
vote
1answer
36 views

How to resample one dataset to conform to the distribution of another dataset?

I have two datasets with 20 features, but with different feature distributions (DS_A and DS_B). How can I sample the DS_A to make its distribution similar to DS_B, with respect to multiple features?? ...