Last call to make your voice heard! Our 2022 Developer Survey closes in less than a week. Take survey.

Questions tagged [sampling]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
6 views

Optimally sample from multiple distributions

I have two datasets both of the form from the table below. I am interested in downselecting from dataset A by sampling from the distribution of values from dataset B. However, I want to consider both ...
user avatar
0 votes
0 answers
17 views

Slice NumPy arrays differently along axes (without looping)

I am trying to analyze a temporal signal sampled by a 2D sensor. Effectively, this means integrating the signal values for each sensor pixel (array row/column coordinate) at the times each pixel is ...
user avatar
0 votes
0 answers
11 views

How to improve L2 loss for generative autoencoder

I am working with a modified generative autoencoder and having issues getting the L2 sufficiently low. I think problem is that because my data is over a very large range and is standardized to values ...
user avatar
  • 13
0 votes
0 answers
7 views

Difference between Jackknife vs bootsrap vs cross validation

I have doubts about the differences between these three methods and I would like to clarify the following: Main differences Advantages of one over the other Context of use of each method etc... If ...
user avatar
  • 320
0 votes
0 answers
7 views

the mean and standard deviation aren't the same as those of the input data i provided after sampling

I have a log-normal mean and a standard deviation. after i converted them to the underlying normal distribution's parameters mu and sigma, I sampled from the log-normal distribution however when i ...
user avatar
0 votes
0 answers
17 views

under sample to get specific number of samples per class using tomek links of imblearn

I have a dataset with classes in my target column distributed like shown below. ...
user avatar
0 votes
0 answers
12 views

How to use Splitting for startifying in sklearn for multiple files

I have csv data file for binary classification. I divided it into 5 multiple files and tried to apply the stratification technique so the class label has the same proportion for all the files. but I ...
user avatar
  • 175
0 votes
0 answers
13 views

KDE Sampling with negative density and/or class-specific weighting

I have a dataset which contains two overlapping distributions/classes of points. I have been trying to sample from just one of these distributions/classes using the scikit learn Kernel Density class, ...
user avatar
  • 13
0 votes
0 answers
19 views

Scikit Learn Random Forest Classifier Hyperparameter Min Target Sample Size

From reading the docs on Scikit Learn, I haven't been able to find an answer, but does anyone know if there is a way to specify to always include a specific number out of the max sample size of ...
user avatar
0 votes
0 answers
32 views

Generating new sample with same distribution

I have a timeseries data for 1 week. The data contains readings from a device for certain hours of the day. There are about 8-10 readings per day at different timestamps. The timestamps recorded for ...
user avatar
  • 1
1 vote
0 answers
17 views

Interrupted Time Series with Unevenly Distributed Samples

I'm working on causal inference using Interrupted Time Series Design. I have multiple samples per day and am selecting my analysis bandwidth based on pre-treatment RMSE on leave-on-out cross ...
user avatar
0 votes
1 answer
18 views

How can I distribute samples optimally to fit a model?

I'm trying to fit a model to a low number (~5-10) of data points. I might be able to suggest the optimal distribution of the data points beforehand, knowing a bit about the data and the model I ...
user avatar
0 votes
1 answer
26 views

Recommended number of features for regression problem

In the following link the answer recommends a feauture amount of N/3 for regression (or it is quoted). Where N corresponds to the sample size: How many features to sample using Random Forests Is there ...
user avatar
0 votes
0 answers
37 views

Sampling from Discrete Space using Latin Hypercube

I wanted to use Discrete Latin Hypercube to sample 20 samples from a space created by an array that has 4096 values. So my desired output would be something like the following: ...
user avatar
0 votes
0 answers
25 views

How to detect data drift for a small size serving dataset with respect to a large training dataset?

The training data contains almost 6 million records and the serving/inference data that I get in batch contains about 400 records. Before my model does any predictions on the inference data I need to ...
user avatar
2 votes
3 answers
88 views

What could go wrong if I sample before classification?

I have a million entries in a table that I can use to train a binary classifier. Only 30 thousand of them are positive. Is there anything fundamentally wrong with selecting around 30 thousand negative ...
user avatar
  • 186
0 votes
0 answers
12 views

how to convert an array of non regularly interleaved coordinates to a matrix of weights using interpolation to obtain uniform sampling

I have an array of coordinates each one with an associated timestamp. Something like: ...
user avatar
  • 101
0 votes
0 answers
25 views

Train Test Split for Imbalance Data set for credit card transaction data set

I am currently working on a credit card transaction datasets for fraud detection, and I am unsure how to go about splitting the data. Transactions are time related data, do I split them like how you ...
user avatar
0 votes
0 answers
57 views

Negative samples generation for Q&A model training

I'm pondering over what's the best time to generate negative samples for a Q&A dataset for training a sentence-bert model for a semantic similarity task. I have a dataset in the form of pairs: ...
user avatar
0 votes
1 answer
24 views

Sampling a data based on average and variance of another data

I have a set of textual datasets that have the following average and variance tokens lengths: ...
user avatar
  • 240
0 votes
0 answers
84 views

under sampling the dataset of multi-label classifiction

I have a multi-label dataset, whose label distribution looks something like this, with label on x-axis and number of rows it occurs in the dataset in y-axis. ...
user avatar
0 votes
0 answers
73 views

Is it always appropriate to use SMOTE in an imbalanced multiclass dataset?

Is it good practice to always use SMOTE and random undersampling in an imbalanced multiclass dataset or are there exceptions? In context, I am using a traditional machine learning model (SVC) for ...
user avatar
0 votes
0 answers
30 views

Compare standard deviations in different samples?

I have some data which you can group based on different variables. I know how to test if they have significantly different means. But what the deviation inside the samples?
user avatar
0 votes
0 answers
19 views

Chi - Square test for Validating Sampled Data

I have a large dataset (stored in a dataframe) that needs to be sampled, so I have performed sampling on it (sampled data also stored in a dataframe) and now wish to check if the sample data is ...
user avatar
0 votes
0 answers
21 views

What is the best way to create input data samples using in XGBoost for predicting number of next days that customer will come back to store

I'm building the tree-based model like a XGBoost to solve the problem about customer purchase cycle. And I think, I will build 2 models which one is predicting the customer will come back to store in ...
user avatar
  • 43
2 votes
1 answer
50 views

How create a representative small subset from a huge dataset, for local development?

​ I have a time series problem and the dataset I'm using is rather huge. Around 100GB. For local development I'm trying to subset this into a very small batch around 50MB, just to make sure unit tests ...
user avatar
1 vote
0 answers
21 views

Stratified sampling - use of proxy variable

For splitting of the data into train/test/val I use stratified sampling. Is it appropriate to define strata using information extracted from the dataset? E.g. use machine-learning to model proxy ...
user avatar
0 votes
0 answers
11 views

Data preparation for password attrition study

Our company sells contracts to businesses to gain access to various reports they might be interested in. A contract lasts one year and consists of a service tier (good, better, best) and a number of ...
user avatar
1 vote
1 answer
25 views

Sampling Technique for mixed data type

I am looking for a very specific sampling technique which pertains to a very large dataset with mixed data type i.e, I have categorical as well as continous variables and want to have a sample that ...
user avatar
0 votes
0 answers
43 views

repeated train/test splitting and assessing performance variability

I have a question related to performance variability and how to assess different methods. I want to compare the result of 5 different classifiers on the same dataset (let's say 20 newsgroup dataset). ...
user avatar
  • 321
0 votes
0 answers
43 views

VAE generating same results during test time

I have trained a VAE to generate a style transferred sentence, from a negative sentence to a positive sentence. The underlying concept of VAE tells us that the sampling is done randomly, to which Mean ...
user avatar
2 votes
1 answer
27 views

Discrepency in 25th percentile of standard deviation calculation

For each of the following sample sizes [3,5,7,9], I Need to calculate the 25th percentile of the standard deviation for the values sampled. And the total number of trials will be 10. DataFrame ...
user avatar
0 votes
0 answers
38 views

Is it right method to remove instances that are hard to predict before train test split?

In a binary classification problem, I have a slightly unbalanced medical dataset with class distribution: 0:5600, 1:1500 0 without a problem and 1 with a problem. I tried many pipelines, automls, and ...
user avatar
  • 3
1 vote
1 answer
31 views

Multiclass data redistribution

I want to redistribute the data in classes according to new proportions and wonder what is the optimal way to do it. For example I have ...
user avatar
2 votes
1 answer
157 views

Negative sampling for graph representation learning

I was watching a lecture about graph representation learning (from here) and got a little bit confusing about how they define the negative samping procedure. In the presentation J. Leskovec briefly ...
user avatar
1 vote
1 answer
99 views

Is there a relationship between learning rate and training set size?

I have a large dataset to use for training a Neural Network model. However, I don't have enough resources to do a proper hyperparameters tuning on the whole dataset. Therefore, my idea is to tune the ...
user avatar
  • 95
1 vote
1 answer
37 views

Reducing (Variance) | the gap between my weights

I have ML ready samples. And each sample has a weight. The weights distribute between [0-1] My problem arise because there are a lot of samples which are ...
user avatar
0 votes
1 answer
282 views

Controlling the sampling from Variational AutoEncoder (VAE)

Suppose a Variational Autoencoder (VAE) is trained with mnist data. To sample, one draws from normal distribution. My question is: suppose I am interested in generating only 1s and no other digits. ...
user avatar
  • 103
0 votes
1 answer
113 views

Subsampling the “right” amout of data to train an ML model

I am training a machine learning model (i.e., a classifier) on a large dataset. I know that I can get the same results using less data (about 30%) but I would like to avoid the trial and error process ...
user avatar
  • 1
1 vote
0 answers
28 views

Stratification sampling of a json Array [closed]

I have a json array file that i need to create a smaller sample of for testing purposes. A sample of the file looks like: ...
user avatar
1 vote
1 answer
50 views

What is the proper proportion for train and test set for classification system?

I have recently googled the best proportion for training and test set for classifying physiological data between normal and abnormal. Much of the source tells that the proportion should be 70:30 or 80:...
user avatar
  • 21
2 votes
2 answers
174 views

Undersampling for credit card fraud detection before or after Train/Test Split

I have a credit card dataset with 98% transactions are Non-Fraud and 2% are fraud. I have been trying to undersample the majotrity class before train and test split and get very good recall and ...
user avatar
2 votes
3 answers
71 views

Sampling n data points from high dimensional data

I have some face images(of a single person), which I ran through an embedding generator to get 128-dimensional embedding. I have 1000 such embedding (shape of the dataset (1000, 128)). I have a ...
user avatar
1 vote
1 answer
35 views

How to bin a distribution data reported with different frequencies ( salaries ), showing mixed linearity and non-linearity?

I am researching on pay-scales, and wish to receive advise to treat data of salaries. Objective My interest is to approximate the salary corresponding to different hierarchical levels in an ...
user avatar
0 votes
0 answers
19 views

Sampling items from a population of subpopulations

I have a population of $n$ items to label and a budget to label only $m$ ($m << n$) of them before training. The population can be partitioned into subpopulations, recursively. In other words, ...
user avatar
  • 635
1 vote
2 answers
47 views

Recover a integer valued function with *-learning

I have the following problem. From a technical model we have a function $f(n,p)$ approximating its runtime. The function $f$ which maps $$ f: \mathbb{N} \times \mathbb{P} \to \mathbb{R}_{+} $$ where $...
user avatar
0 votes
0 answers
27 views

how to know if there is a bias in data collection methods

I am collecting data for machine learning models I want to build for some application. I started with random sampling (just simply collecting 'recent' data) but I am not getting enough records of ...
user avatar
1 vote
0 answers
54 views

Follow up question regarding Upsampling for Imbalanced Data and the use of ADASYN instead of SMOTE

I have a follow-up question regarding this topic. I have been working on a project predicting success(1) or failure(0) for organizations by using the Decision Tree and Random Forest algorithms. My ...
user avatar
1 vote
1 answer
76 views

Is it right to maintain the train distribution in test set for unbalanced data?

If the training set was unbalanced the chances are the model will be biased. But if the data distribution in the test set is the same distribution as the train set, this kind of bias is not going to ...
user avatar
3 votes
2 answers
591 views

Over-sampling: is my model over-fitting?

I would like to ask you some questions on how to consider (good or not) the following results: ...
user avatar
  • 295