Questions tagged [sampling]
The sampling tag has no usage guidance.
168
questions
2
votes
3answers
60 views
What could go wrong if I sample before classification?
I have a million entries in a table that I can use to train a binary classifier. Only 30 thousand of them are positive. Is there anything fundamentally wrong with selecting around 30 thousand negative ...
0
votes
0answers
9 views
how to convert an array of non regularly interleaved coordinates to a matrix of weights using interpolation to obtain uniform sampling
I have an array of coordinates each one with an associated timestamp. Something like:
...
0
votes
0answers
13 views
Train Test Split for Imbalance Data set for credit card transaction data set
I am currently working on a credit card transaction datasets for fraud detection, and I am unsure how to go about splitting the data.
Transactions are time related data, do I split them like how you ...
0
votes
0answers
19 views
Negative samples generation for Q&A model training
I'm pondering over what's the best time to generate negative samples for a Q&A dataset for training a sentence-bert model for a semantic similarity task.
I have a dataset in the form of pairs:
...
0
votes
1answer
24 views
Sampling a data based on average and variance of another data
I have a set of textual datasets that have the following average and variance tokens lengths:
...
1
vote
0answers
31 views
under sampling the dataset of multi-label classifiction
I have a multi-label dataset, whose label distribution looks something like this, with label on x-axis and number of rows it occurs in the dataset in y-axis.
...
0
votes
0answers
25 views
Is it always appropriate to use SMOTE in an imbalanced multiclass dataset?
Is it good practice to always use SMOTE and random undersampling in an imbalanced multiclass dataset or are there exceptions? In context, I am using a traditional machine learning model (SVC) for ...
0
votes
0answers
30 views
Compare standard deviations in different samples?
I have some data which you can group based on different variables.
I know how to test if they have significantly different means. But what the deviation inside the samples?
0
votes
0answers
17 views
Chi - Square test for Validating Sampled Data
I have a large dataset (stored in a dataframe) that needs to be sampled, so I have performed sampling on it (sampled data also stored in a dataframe) and now wish to check if the sample data is ...
0
votes
0answers
15 views
What is the best way to create input data samples using in XGBoost for predicting number of next days that customer will come back to store
I'm building the tree-based model like a XGBoost to solve the problem about customer purchase cycle.
And I think, I will build 2 models which one is predicting the customer will come back to store in ...
2
votes
1answer
41 views
How create a representative small subset from a huge dataset, for local development?
I have a time series problem and the dataset I'm using is rather huge. Around 100GB. For local development I'm trying to subset this into a very small batch around 50MB, just to make sure unit tests ...
1
vote
0answers
21 views
Stratified sampling - use of proxy variable
For splitting of the data into train/test/val I use stratified sampling. Is it appropriate to define strata using information extracted from the dataset? E.g. use machine-learning to model proxy ...
1
vote
0answers
10 views
Data preparation for password attrition study
Our company sells contracts to businesses to gain access to various reports they might be interested in. A contract lasts one year and consists of a service tier (good, better, best) and a number of ...
1
vote
1answer
23 views
Sampling Technique for mixed data type
I am looking for a very specific sampling technique which pertains to a very large dataset with mixed data type i.e, I have categorical as well as continous variables and want to have a sample that ...
0
votes
0answers
23 views
repeated train/test splitting and assessing performance variability
I have a question related to performance variability and how to assess different methods.
I want to compare the result of 5 different classifiers on the same dataset (let's say 20 newsgroup dataset).
...
0
votes
0answers
14 views
VAE generating same results during test time
I have trained a VAE to generate a style transferred sentence, from a negative sentence to a positive sentence. The underlying concept of VAE tells us that the sampling is done randomly, to which Mean ...
2
votes
1answer
26 views
Discrepency in 25th percentile of standard deviation calculation
For each of the following sample sizes [3,5,7,9], I Need to calculate the 25th percentile of the standard deviation for the values sampled. And the total number of trials will be 10.
DataFrame
...
0
votes
0answers
32 views
Is it right method to remove instances that are hard to predict before train test split?
In a binary classification problem,
I have a slightly unbalanced medical dataset with class distribution:
0:5600, 1:1500
0 without a problem and 1 with a problem.
I tried many pipelines, automls, and ...
1
vote
1answer
29 views
Multiclass data redistribution
I want to redistribute the data in classes according to new proportions and wonder what is the optimal way to do it.
For example I have
...
2
votes
1answer
82 views
Negative sampling for graph representation learning
I was watching a lecture about graph representation learning (from here) and got a little bit confusing about how they define the negative samping procedure.
In the presentation J. Leskovec briefly ...
1
vote
1answer
38 views
Is there a relationship between learning rate and training set size?
I have a large dataset to use for training a Neural Network model. However, I don't have enough resources to do a proper hyperparameters tuning on the whole dataset. Therefore, my idea is to tune the ...
1
vote
1answer
32 views
Reducing (Variance) | the gap between my weights
I have ML ready samples. And each sample has a weight.
The weights distribute between [0-1]
My problem arise because there are a lot of samples which are ...
0
votes
0answers
18 views
LSTM long sequences reduction algorithm
suppose I have a large multi-variate TimeSeries dataset with very long sequences (~50k)
I want to reduce each sequence size to constant size for LSTM training.
I thought about splitting each sequence ...
0
votes
1answer
105 views
Controlling the sampling from Variational AutoEncoder (VAE)
Suppose a Variational Autoencoder (VAE) is trained with mnist data. To sample, one draws from normal distribution. My question is: suppose I am interested in generating only 1s and no other digits. ...
0
votes
1answer
57 views
Subsampling the “right” amout of data to train an ML model
I am training a machine learning model (i.e., a classifier) on a large dataset. I know that I can get the same results using less data (about 30%) but I would like to avoid the trial and error process ...
1
vote
0answers
21 views
Stratification sampling of a json Array [closed]
I have a json array file that i need to create a smaller sample of for testing purposes. A sample of the file looks like:
...
1
vote
1answer
33 views
What is the proper proportion for train and test set for classification system?
I have recently googled the best proportion for training and test set for classifying physiological data between normal and abnormal. Much of the source tells that the proportion should be 70:30 or 80:...
2
votes
2answers
127 views
Undersampling for credit card fraud detection before or after Train/Test Split
I have a credit card dataset with 98% transactions are Non-Fraud and 2% are fraud.
I have been trying to undersample the majotrity class before train and test split and get very good recall and ...
1
vote
2answers
25 views
Sampling n data points from high dimensional data
I have some face images(of a single person), which I ran through an embedding generator to get 128-dimensional embedding.
I have 1000 such embedding (shape of the dataset (1000, 128)). I have a ...
1
vote
1answer
32 views
How to bin a distribution data reported with different frequencies ( salaries ), showing mixed linearity and non-linearity?
I am researching on pay-scales, and wish to receive advise to treat data of salaries.
Objective
My interest is to approximate the salary corresponding to different hierarchical levels in an ...
0
votes
0answers
17 views
Sampling items from a population of subpopulations
I have a population of $n$ items to label and a budget to label only $m$ ($m << n$) of them before training. The population can be partitioned into subpopulations, recursively. In other words, ...
1
vote
1answer
35 views
Recover a integer valued function with *-learning
I have the following problem. From a technical model we have a function $f(n,p)$ approximating its runtime. The function $f$ which maps
$$
f: \mathbb{N} \times \mathbb{P} \to \mathbb{R}_{+}
$$
where $...
0
votes
0answers
25 views
how to know if there is a bias in data collection methods
I am collecting data for machine learning models I want to build for some application.
I started with random sampling (just simply collecting 'recent' data) but I am not getting enough records of ...
1
vote
0answers
36 views
Follow up question regarding Upsampling for Imbalanced Data and the use of ADASYN instead of SMOTE
I have a follow-up question regarding this topic.
I have been working on a project predicting success(1) or failure(0) for organizations by using the Decision Tree and Random Forest algorithms.
My ...
1
vote
1answer
29 views
Is it right to maintain the train distribution in test set for unbalanced data?
If the training set was unbalanced the chances are the model will be biased. But if the data distribution in the test set is the same distribution as the train set, this kind of bias is not going to ...
3
votes
2answers
585 views
Over-sampling: is my model over-fitting?
I would like to ask you some questions on how to consider (good or not) the following results:
...
0
votes
1answer
26 views
How to Present All Categories in All Samples
I have a data contains many categorical columns. When I sampled this data randomly a few times and applied one-hot encoding to categorical columns I noticed that it ended up with datasets with ...
6
votes
2answers
88 views
Is sampling a valid way to reduce complexity?
I'm facing an issue where I have a massive amount of data that I need to cluster. As we know, clustering algorithms can have a very high O complexity, and I'm looking for ways to reduce the time my ...
-1
votes
1answer
39 views
How Should I deal with my imbalanced binary target [closed]
I am trying to model my data with Python and i am having concerns about my binary target variable, because it has 90% cases falling in 0 and 10% of the cases falling in 1. I have tried upsampling my ...
0
votes
1answer
25 views
What is the most common practice of generating (X,Y) from an arbitrary CDF or PDF?
So I can generate X from arbitrary CDF F(x) by the procedure above. Can it be generalized to two variables? How, exactly? If not, what's the best way to generate <...
1
vote
2answers
59 views
How to generate a random sample and distribute values based in an probability distribution?
I want to generate a random sample based on this probability distribution:
The line is the KDE of the histogram.
My random sample will have n values, the value is ...
1
vote
0answers
194 views
What's the order in applying SMOTE transformation in a pipeline?
Here's the thing, I have an imbalanced data and I was thinking about using SMOTE transformation. However, when doing that using a sklearn pipeline, I get an error because of missing values.
This is my ...
1
vote
1answer
49 views
How should I sample from a mixture distribution?
Let's say we have a mixture distribution, defined by density $f(x)= w_1 p_1(x) + w_2 p_2(x)$, where $w_i$ is a scalar weight. Furthermore, we have efficient methods to evaluate the pdf and cdf/icdf ...
0
votes
0answers
32 views
Difference between test statistic and sample statistic?
Test statistic and sample statistic are used to check statistical significance. How do I understand the procedure in background of reaching a conclusion about an effect-size estimate.
0
votes
0answers
23 views
HR employee attrition modeling - making a balanced sample question
I have dataset 1 (stayers) consisting of 1500 record of HR data demographic data of employees (11 features) who currently are in the company. Dataset 2 (leavers) contains 180 records -same features- ...
1
vote
1answer
114 views
Training a Variational Autoencoder (VAE) for Random Number Generation
I have a complicated 20-dimensional multi-modal distribution and consider training a VAE to learn an approximation of it using 2000 samples. But particularly, with the aim to subsequently generate ...
1
vote
0answers
17 views
Sequential sampling from Gaussian conditional not working
I'm trying to sequentially sample from a Gaussian Process prior.
The problem is that the samples eventually converge to zero or diverge to infinity.
I'm using the basic conditionals described e.g. ...
1
vote
0answers
28 views
Generating artificial data to extend learning set
I have dataset containing 42 instances(X) and one final Y on which i want to perform LASSO regression.All are continuous and numerical. As the sample size small, I wish to extend it. I am kind of ...
1
vote
0answers
13 views
Adaptive Sampling Strategies for SVM?
I am an Engineer interested in creating a surrogate model of a certain phenomenon in the context of reliability engineering. Essentially my quantity of interest is the Limit state function/stability ...
0
votes
1answer
255 views
Bootstrap clarification for datasets with multiple columns
How do I bootstrap a Dataset/DataFrame with multiple continuous and categorical columns?
For eg: Say I'm trying to bootstrap the colour distribution of M&M's and I have 50 bags (samples) each with ...
