Questions tagged [sampling]
The sampling tag has no usage guidance.
150
questions
0
votes
1answer
13 views
Subsampling the “right” amout of data to train an ML model
I am training a machine learning model (i.e., a classifier) on a large dataset. I know that I can get the same results using less data (about 30%) but I would like to avoid the trial and error process ...
1
vote
0answers
15 views
Stratification sampling of a json Array [closed]
I have a json array file that i need to create a smaller sample of for testing purposes. A sample of the file looks like:
...
1
vote
1answer
22 views
What is the proper proportion for train and test set for classification system?
I have recently googled the best proportion for training and test set for classifying physiological data between normal and abnormal. Much of the source tells that the proportion should be 70:30 or 80:...
2
votes
2answers
79 views
Undersampling for credit card fraud detection before or after Train/Test Split
I have a credit card dataset with 98% transactions are Non-Fraud and 2% are fraud.
I have been trying to undersample the majotrity class before train and test split and get very good recall and ...
1
vote
0answers
16 views
Sampling n data points from high dimensional data
I have some face images(of a single person), which I ran through an embedding generator to get 128-dimensional embedding.
I have 1000 such embedding (shape of the dataset (1000, 128)). I have a ...
1
vote
1answer
28 views
How to bin a distribution data reported with different frequencies ( salaries ), showing mixed linearity and non-linearity?
I am researching on pay-scales, and wish to receive advise to treat data of salaries.
Objective
My interest is to approximate the salary corresponding to different hierarchical levels in an ...
0
votes
0answers
17 views
Sampling items from a population of subpopulations
I have a population of $n$ items to label and a budget to label only $m$ ($m << n$) of them before training. The population can be partitioned into subpopulations, recursively. In other words, ...
1
vote
1answer
27 views
Recover a integer valued function with *-learning
I have the following problem. From a technical model we have a function $f(n,p)$ approximating its runtime. The function $f$ which maps
$$
f: \mathbb{N} \times \mathbb{P} \to \mathbb{R}_{+}
$$
where $...
0
votes
0answers
20 views
how to know if there is a bias in data collection methods
I am collecting data for machine learning models I want to build for some application.
I started with random sampling (just simply collecting 'recent' data) but I am not getting enough records of ...
1
vote
0answers
25 views
Follow up question regarding Upsampling for Imbalanced Data and the use of ADASYN instead of SMOTE
I have a follow-up question regarding this topic.
I have been working on a project predicting success(1) or failure(0) for organizations by using the Decision Tree and Random Forest algorithms.
My ...
1
vote
1answer
16 views
Is it right to maintain the train distribution in test set for unbalanced data?
If the training set was unbalanced the chances are the model will be biased. But if the data distribution in the test set is the same distribution as the train set, this kind of bias is not going to ...
3
votes
2answers
580 views
Over-sampling: is my model over-fitting?
I would like to ask you some questions on how to consider (good or not) the following results:
...
0
votes
1answer
26 views
How to Present All Categories in All Samples
I have a data contains many categorical columns. When I sampled this data randomly a few times and applied one-hot encoding to categorical columns I noticed that it ended up with datasets with ...
6
votes
2answers
78 views
Is sampling a valid way to reduce complexity?
I'm facing an issue where I have a massive amount of data that I need to cluster. As we know, clustering algorithms can have a very high O complexity, and I'm looking for ways to reduce the time my ...
-1
votes
1answer
33 views
How Should I deal with my imbalanced binary target [closed]
I am trying to model my data with Python and i am having concerns about my binary target variable, because it has 90% cases falling in 0 and 10% of the cases falling in 1. I have tried upsampling my ...
0
votes
1answer
23 views
What is the most common practice of generating (X,Y) from an arbitrary CDF or PDF?
So I can generate X from arbitrary CDF F(x) by the procedure above. Can it be generalized to two variables? How, exactly? If not, what's the best way to generate <...
1
vote
2answers
30 views
How to generate a random sample and distribute values based in an probability distribution?
I want to generate a random sample based on this probability distribution:
The line is the KDE of the histogram.
My random sample will have n values, the value is ...
0
votes
0answers
17 views
Training a Variational Autoencoder (VAE) for Non-Uniform Random Number Generation
I have a complicated 20-dimensional non-uniform distribution and would like generate samples from it. I have considered training a VAE to do so, but my problems are the following:
Is my approach ...
0
votes
0answers
12 views
Positive records : no behavioral data
0
I'm working on a classification model aimed at identifying if behavioral activity within an account (b2b - one account, many contacts) can predict or not an opportunity generation ( a salesperson ...
1
vote
0answers
86 views
What's the order in applying SMOTE transformation in a pipeline?
Here's the thing, I have an imbalanced data and I was thinking about using SMOTE transformation. However, when doing that using a sklearn pipeline, I get an error because of missing values.
This is my ...
1
vote
1answer
32 views
How should I sample from a mixture distribution?
Let's say we have a mixture distribution, defined by density $f(x)= w_1 p_1(x) + w_2 p_2(x)$, where $w_i$ is a scalar weight. Furthermore, we have efficient methods to evaluate the pdf and cdf/icdf ...
0
votes
0answers
26 views
Difference between test statistic and sample statistic?
Test statistic and sample statistic are used to check statistical significance. How do I understand the procedure in background of reaching a conclusion about an effect-size estimate.
0
votes
0answers
17 views
HR employee attrition modeling - making a balanced sample question
I have dataset 1 (stayers) consisting of 1500 record of HR data demographic data of employees (11 features) who currently are in the company. Dataset 2 (leavers) contains 180 records -same features- ...
1
vote
1answer
58 views
Training a Variational Autoencoder (VAE) for Random Number Generation
I have a complicated 20-dimensional multi-modal distribution and consider training a VAE to learn an approximation of it using 2000 samples. But particularly, with the aim to subsequently generate ...
0
votes
0answers
22 views
How to downsample a dataset with constraints in Python?
I have a dataset that has columns a, b, and others. Both a and ...
1
vote
0answers
17 views
Sequential sampling from Gaussian conditional not working
I'm trying to sequentially sample from a Gaussian Process prior.
The problem is that the samples eventually converge to zero or diverge to infinity.
I'm using the basic conditionals described e.g. ...
1
vote
0answers
27 views
Generating artificial data to extend learning set
I have dataset containing 42 instances(X) and one final Y on which i want to perform LASSO regression.All are continuous and numerical. As the sample size small, I wish to extend it. I am kind of ...
1
vote
0answers
13 views
Adaptive Sampling Strategies for SVM?
I am an Engineer interested in creating a surrogate model of a certain phenomenon in the context of reliability engineering. Essentially my quantity of interest is the Limit state function/stability ...
0
votes
1answer
88 views
Bootstrap clarification for datasets with multiple columns
How do I bootstrap a Dataset/DataFrame with multiple continuous and categorical columns?
For eg: Say I'm trying to bootstrap the colour distribution of M&M's and I have 50 bags (samples) each with ...
1
vote
1answer
57 views
Sampling in Text Classification: can the results be considered 'reliable'?
I am testing different models (SVM, Logistic Regression, Naive Bayes, Random Forest) for predicting the class of a spam email. My target is a binary variable.
I am analysing only text, no other fields....
1
vote
0answers
24 views
Uniform convergence garantee on sample complexity
I can't understand why the Uniform Convergence guarantees an upper bound and not a lower bound on sample complexity as stated on [1]
Corollary 4.4.
If a class $H$ has the uniform convergence property ...
0
votes
1answer
16 views
Getting a balanced sample across many variables
Let’s say each element in my population has several attributes.
Let’s call then A, B, C, D, E, F.
Let’s say, for simplicity, each attribute has 10 values (but could be any number between 2 and 30).
...
0
votes
1answer
262 views
What is the effect of oversampling on logistic regression?
I am building a model to predict if a customer will use a coupon or not for a given campaign. I am using logistic regression for this model. I took 5 previous campaigns and generally for each campaign ...
1
vote
0answers
26 views
Representation sample size- n [closed]
Need help with identifying a representation sample size 'n'. Let's say I have a very large population- infinite number of participants. I am picking the random sample from this infinite population. I ...
1
vote
0answers
42 views
Chi Square Test Goodness of Fit
I want to use a chi square test but I'm unsure if I'm using it right. The KickStarter website shows the frequency of main categories projects. It is updated once a day. I got a data set of KickStarter ...
0
votes
0answers
19 views
Question regarding strata in Geron's book
In the book "Hands-On Machine Learning with Scikit-Learn and Tensorflow" by Aurélien Géron. There is a regression project explained. My doubt is regarding his example for 'stratified sampling'.
He ...
0
votes
1answer
96 views
What is difference between Standard Normal Distribution and Mean Normalization approaches to feature-scaling?
The tag feature-scaling seems to convey that one of the scaling methods is Standard Normal Distribution. Further, I read an Answer on this site saying that Mean Normalization is a form of feature ...
0
votes
0answers
52 views
Smote oversampling on strings
I have an imbalanced dataset with most transactions being OK and 1% transactions marked as fraud. This dataset has columns with integers (time and transaction amount and timestamp) as well as strings (...
0
votes
0answers
36 views
Time Series Model (n-beats paper) how is sampling / training done?
does any one know what this means?
It is taken from the paper https://openreview.net/pdf?id=r1ecqn4YwB (n-beats time series model).
To update network parameters for one horizon, we sample train ...
1
vote
1answer
24 views
How to draw a sample from data set with respect to a given categorical or numerical variable based on given freely chosen distribution? (Python)
Say I have a data set for some past period. Now new data appears and for a given variable in the data and we find that the distributions have shifted (for example with "age" it would be that suddenly ...
0
votes
1answer
28 views
Is there a name for this form of sampling?
I had an idea to combine oversampling and undersampling in the following way:
Compute the average number of individuals in each class. For classes with a number of individuals greater than this ...
0
votes
0answers
22 views
dealing with imbalanced data for multi-class problem
Based on the experiments I run for a number of times, and the reading I did on imbalanced data for a multiclassification problem such as this paper, resampling techniques like ...
1
vote
2answers
45 views
Dealing with large data: selecting a sample
I'm given a data set to create a model that would predict whether a certain supply chain would be able to deliver the goods without delay or not. I'm doing this in Python.
The data set has 93000 ...
2
votes
1answer
4k views
SMOTE for regression
I am looking into upsampling an imbalanced dataset for a regression problem (Numerical target variables) in python.
I attached paper and R package that implement SMOTE for regression, can anyone ...
-1
votes
1answer
289 views
Sampling randomly from pd.DataFrame, but ignoring NaN values
I am trying to sample random values from a dataframe where the NaN values should be ignored, without dropping the entire row or column.
My sampling function at the moment looks like this:
...
0
votes
1answer
23 views
Over/Under Sampling for Multi-classification
I'm trying to apply xgboost and random forest for over and under sampling
For imbalanced data:
train shape -> (199991, 23)
However, reverse my expectation. ...
3
votes
0answers
63 views
SuperLearner Cross validation with iid time series
I created a number of ML models in R and I aim at combining them to form an ensemble.
I learned about SuperLearner library which cross validates many models and returns the weight to each model in ...
6
votes
1answer
399 views
Why gradient boosting uses sampling without replacement?
In Random Forest each tree is built selecting a sample with replacement (bootstrap). And I assumed that Gradient Boosting's trees were selected with the same sampling technique. (@BenReiniger ...
1
vote
1answer
150 views
How to resample one dataset to conform to the distribution of another dataset?
I have two datasets with 20 features, but with different feature distributions (DS_A and DS_B). How can I sample the DS_A to make its distribution similar to DS_B, with respect to multiple features??
...
1
vote
1answer
80 views
How to compute modulo of a hash?
Let's say that I have a set of users in my database, that have GUIDs as their IDs. I use xxhash to generate fixed-length hashes for each value, so that I can then ...
