Questions tagged [sampling]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
2
votes
3answers
60 views

What could go wrong if I sample before classification?

I have a million entries in a table that I can use to train a binary classifier. Only 30 thousand of them are positive. Is there anything fundamentally wrong with selecting around 30 thousand negative ...
0
votes
0answers
9 views

how to convert an array of non regularly interleaved coordinates to a matrix of weights using interpolation to obtain uniform sampling

I have an array of coordinates each one with an associated timestamp. Something like: ...
0
votes
0answers
13 views

Train Test Split for Imbalance Data set for credit card transaction data set

I am currently working on a credit card transaction datasets for fraud detection, and I am unsure how to go about splitting the data. Transactions are time related data, do I split them like how you ...
0
votes
0answers
19 views

Negative samples generation for Q&A model training

I'm pondering over what's the best time to generate negative samples for a Q&A dataset for training a sentence-bert model for a semantic similarity task. I have a dataset in the form of pairs: ...
0
votes
1answer
24 views

Sampling a data based on average and variance of another data

I have a set of textual datasets that have the following average and variance tokens lengths: ...
1
vote
0answers
31 views

under sampling the dataset of multi-label classifiction

I have a multi-label dataset, whose label distribution looks something like this, with label on x-axis and number of rows it occurs in the dataset in y-axis. ...
0
votes
0answers
25 views

Is it always appropriate to use SMOTE in an imbalanced multiclass dataset?

Is it good practice to always use SMOTE and random undersampling in an imbalanced multiclass dataset or are there exceptions? In context, I am using a traditional machine learning model (SVC) for ...
0
votes
0answers
30 views

Compare standard deviations in different samples?

I have some data which you can group based on different variables. I know how to test if they have significantly different means. But what the deviation inside the samples?
0
votes
0answers
17 views

Chi - Square test for Validating Sampled Data

I have a large dataset (stored in a dataframe) that needs to be sampled, so I have performed sampling on it (sampled data also stored in a dataframe) and now wish to check if the sample data is ...
0
votes
0answers
15 views

What is the best way to create input data samples using in XGBoost for predicting number of next days that customer will come back to store

I'm building the tree-based model like a XGBoost to solve the problem about customer purchase cycle. And I think, I will build 2 models which one is predicting the customer will come back to store in ...
2
votes
1answer
41 views

How create a representative small subset from a huge dataset, for local development?

​ I have a time series problem and the dataset I'm using is rather huge. Around 100GB. For local development I'm trying to subset this into a very small batch around 50MB, just to make sure unit tests ...
1
vote
0answers
21 views

Stratified sampling - use of proxy variable

For splitting of the data into train/test/val I use stratified sampling. Is it appropriate to define strata using information extracted from the dataset? E.g. use machine-learning to model proxy ...
1
vote
0answers
10 views

Data preparation for password attrition study

Our company sells contracts to businesses to gain access to various reports they might be interested in. A contract lasts one year and consists of a service tier (good, better, best) and a number of ...
1
vote
1answer
23 views

Sampling Technique for mixed data type

I am looking for a very specific sampling technique which pertains to a very large dataset with mixed data type i.e, I have categorical as well as continous variables and want to have a sample that ...
0
votes
0answers
23 views

repeated train/test splitting and assessing performance variability

I have a question related to performance variability and how to assess different methods. I want to compare the result of 5 different classifiers on the same dataset (let's say 20 newsgroup dataset). ...
0
votes
0answers
14 views

VAE generating same results during test time

I have trained a VAE to generate a style transferred sentence, from a negative sentence to a positive sentence. The underlying concept of VAE tells us that the sampling is done randomly, to which Mean ...
2
votes
1answer
26 views

Discrepency in 25th percentile of standard deviation calculation

For each of the following sample sizes [3,5,7,9], I Need to calculate the 25th percentile of the standard deviation for the values sampled. And the total number of trials will be 10. DataFrame ...
0
votes
0answers
32 views

Is it right method to remove instances that are hard to predict before train test split?

In a binary classification problem, I have a slightly unbalanced medical dataset with class distribution: 0:5600, 1:1500 0 without a problem and 1 with a problem. I tried many pipelines, automls, and ...
1
vote
1answer
29 views

Multiclass data redistribution

I want to redistribute the data in classes according to new proportions and wonder what is the optimal way to do it. For example I have ...
2
votes
1answer
82 views

Negative sampling for graph representation learning

I was watching a lecture about graph representation learning (from here) and got a little bit confusing about how they define the negative samping procedure. In the presentation J. Leskovec briefly ...
1
vote
1answer
38 views

Is there a relationship between learning rate and training set size?

I have a large dataset to use for training a Neural Network model. However, I don't have enough resources to do a proper hyperparameters tuning on the whole dataset. Therefore, my idea is to tune the ...
1
vote
1answer
32 views

Reducing (Variance) | the gap between my weights

I have ML ready samples. And each sample has a weight. The weights distribute between [0-1] My problem arise because there are a lot of samples which are ...
0
votes
0answers
18 views

LSTM long sequences reduction algorithm

suppose I have a large multi-variate TimeSeries dataset with very long sequences (~50k) I want to reduce each sequence size to constant size for LSTM training. I thought about splitting each sequence ...
0
votes
1answer
105 views

Controlling the sampling from Variational AutoEncoder (VAE)

Suppose a Variational Autoencoder (VAE) is trained with mnist data. To sample, one draws from normal distribution. My question is: suppose I am interested in generating only 1s and no other digits. ...
0
votes
1answer
57 views

Subsampling the “right” amout of data to train an ML model

I am training a machine learning model (i.e., a classifier) on a large dataset. I know that I can get the same results using less data (about 30%) but I would like to avoid the trial and error process ...
1
vote
0answers
21 views

Stratification sampling of a json Array [closed]

I have a json array file that i need to create a smaller sample of for testing purposes. A sample of the file looks like: ...
1
vote
1answer
33 views

What is the proper proportion for train and test set for classification system?

I have recently googled the best proportion for training and test set for classifying physiological data between normal and abnormal. Much of the source tells that the proportion should be 70:30 or 80:...
2
votes
2answers
127 views

Undersampling for credit card fraud detection before or after Train/Test Split

I have a credit card dataset with 98% transactions are Non-Fraud and 2% are fraud. I have been trying to undersample the majotrity class before train and test split and get very good recall and ...
1
vote
2answers
25 views

Sampling n data points from high dimensional data

I have some face images(of a single person), which I ran through an embedding generator to get 128-dimensional embedding. I have 1000 such embedding (shape of the dataset (1000, 128)). I have a ...
1
vote
1answer
32 views

How to bin a distribution data reported with different frequencies ( salaries ), showing mixed linearity and non-linearity?

I am researching on pay-scales, and wish to receive advise to treat data of salaries. Objective My interest is to approximate the salary corresponding to different hierarchical levels in an ...
0
votes
0answers
17 views

Sampling items from a population of subpopulations

I have a population of $n$ items to label and a budget to label only $m$ ($m << n$) of them before training. The population can be partitioned into subpopulations, recursively. In other words, ...
1
vote
1answer
35 views

Recover a integer valued function with *-learning

I have the following problem. From a technical model we have a function $f(n,p)$ approximating its runtime. The function $f$ which maps $$ f: \mathbb{N} \times \mathbb{P} \to \mathbb{R}_{+} $$ where $...
0
votes
0answers
25 views

how to know if there is a bias in data collection methods

I am collecting data for machine learning models I want to build for some application. I started with random sampling (just simply collecting 'recent' data) but I am not getting enough records of ...
1
vote
0answers
36 views

Follow up question regarding Upsampling for Imbalanced Data and the use of ADASYN instead of SMOTE

I have a follow-up question regarding this topic. I have been working on a project predicting success(1) or failure(0) for organizations by using the Decision Tree and Random Forest algorithms. My ...
1
vote
1answer
29 views

Is it right to maintain the train distribution in test set for unbalanced data?

If the training set was unbalanced the chances are the model will be biased. But if the data distribution in the test set is the same distribution as the train set, this kind of bias is not going to ...
3
votes
2answers
585 views

Over-sampling: is my model over-fitting?

I would like to ask you some questions on how to consider (good or not) the following results: ...
0
votes
1answer
26 views

How to Present All Categories in All Samples

I have a data contains many categorical columns. When I sampled this data randomly a few times and applied one-hot encoding to categorical columns I noticed that it ended up with datasets with ...
6
votes
2answers
88 views

Is sampling a valid way to reduce complexity?

I'm facing an issue where I have a massive amount of data that I need to cluster. As we know, clustering algorithms can have a very high O complexity, and I'm looking for ways to reduce the time my ...
-1
votes
1answer
39 views

How Should I deal with my imbalanced binary target [closed]

I am trying to model my data with Python and i am having concerns about my binary target variable, because it has 90% cases falling in 0 and 10% of the cases falling in 1. I have tried upsampling my ...
0
votes
1answer
25 views

What is the most common practice of generating (X,Y) from an arbitrary CDF or PDF?

So I can generate X from arbitrary CDF F(x) by the procedure above. Can it be generalized to two variables? How, exactly? If not, what's the best way to generate <...
1
vote
2answers
59 views

How to generate a random sample and distribute values based in an probability distribution?

I want to generate a random sample based on this probability distribution: The line is the KDE of the histogram. My random sample will have n values, the value is ...
1
vote
0answers
194 views

What's the order in applying SMOTE transformation in a pipeline?

Here's the thing, I have an imbalanced data and I was thinking about using SMOTE transformation. However, when doing that using a sklearn pipeline, I get an error because of missing values. This is my ...
1
vote
1answer
49 views

How should I sample from a mixture distribution?

Let's say we have a mixture distribution, defined by density $f(x)= w_1 p_1(x) + w_2 p_2(x)$, where $w_i$ is a scalar weight. Furthermore, we have efficient methods to evaluate the pdf and cdf/icdf ...
0
votes
0answers
32 views

Difference between test statistic and sample statistic?

Test statistic and sample statistic are used to check statistical significance. How do I understand the procedure in background of reaching a conclusion about an effect-size estimate.
0
votes
0answers
23 views

HR employee attrition modeling - making a balanced sample question

I have dataset 1 (stayers) consisting of 1500 record of HR data demographic data of employees (11 features) who currently are in the company. Dataset 2 (leavers) contains 180 records -same features- ...
1
vote
1answer
114 views

Training a Variational Autoencoder (VAE) for Random Number Generation

I have a complicated 20-dimensional multi-modal distribution and consider training a VAE to learn an approximation of it using 2000 samples. But particularly, with the aim to subsequently generate ...
1
vote
0answers
17 views

Sequential sampling from Gaussian conditional not working

I'm trying to sequentially sample from a Gaussian Process prior. The problem is that the samples eventually converge to zero or diverge to infinity. I'm using the basic conditionals described e.g. ...
1
vote
0answers
28 views

Generating artificial data to extend learning set

I have dataset containing 42 instances(X) and one final Y on which i want to perform LASSO regression.All are continuous and numerical. As the sample size small, I wish to extend it. I am kind of ...
1
vote
0answers
13 views

Adaptive Sampling Strategies for SVM?

I am an Engineer interested in creating a surrogate model of a certain phenomenon in the context of reliability engineering. Essentially my quantity of interest is the Limit state function/stability ...
0
votes
1answer
255 views

Bootstrap clarification for datasets with multiple columns

How do I bootstrap a Dataset/DataFrame with multiple continuous and categorical columns? For eg: Say I'm trying to bootstrap the colour distribution of M&M's and I have 50 bags (samples) each with ...