Questions tagged [sampling]
The sampling tag has no usage guidance.
124
questions
-1
votes
0answers
10 views
How to manually collect rectangular training data samples from images? [closed]
I want to collect training samples from images.
That can mean different things depending on the context. I think of the simplest case, which should be most commonly required. Because it is so common, ...
0
votes
0answers
24 views
Chi Square Test Goodness of Fit
I want to use a chi square test but I'm unsure if I'm using it right. The KickStarter website shows the frequency of main categories projects. It is updated once a day. I got a data set of KickStarter ...
0
votes
0answers
19 views
Question regarding strata in Geron's book
In the book "Hands-On Machine Learning with Scikit-Learn and Tensorflow" by Aurélien Géron. There is a regression project explained. My doubt is regarding his example for 'stratified sampling'.
He ...
0
votes
0answers
15 views
Sample size calculation for AB testing
I have sales data for 500 stores and lets say I want to run an AB test on a sample of test and control stores with the objective of getting statistically significant increase in sales(lift). My ...
0
votes
0answers
7 views
Smote oversampling on strings
I have an imbalanced dataset with most transactions being OK and 1% transactions marked as fraud. This dataset has columns with integers (time and transaction amount and timestamp) as well as strings (...
0
votes
0answers
14 views
Time Series Model (n-beats paper) how is sampling / training done?
does any one know what this means?
It is taken from the paper https://openreview.net/pdf?id=r1ecqn4YwB (n-beats time series model).
To update network parameters for one horizon, we sample train ...
1
vote
1answer
16 views
How to draw a sample from data set with respect to a given categorical or numerical variable based on given freely chosen distribution? (Python)
Say I have a data set for some past period. Now new data appears and for a given variable in the data and we find that the distributions have shifted (for example with "age" it would be that suddenly ...
0
votes
1answer
24 views
Is there a name for this form of sampling?
I had an idea to combine oversampling and undersampling in the following way:
Compute the average number of individuals in each class. For classes with a number of individuals greater than this ...
0
votes
0answers
19 views
dealing with imbalanced data for multi-class problem
Based on the experiments I run for a number of times, and the reading I did on imbalanced data for a multiclassification problem such as this paper, resampling techniques like ...
0
votes
1answer
20 views
Dealing with large data: selecting a sample
I'm given a data set to create a model that would predict whether a certain supply chain would be able to deliver the goods without delay or not. I'm doing this in Python.
The data set has 93000 ...
0
votes
0answers
9 views
Estimate machine count for API dependency
I have a SDK, with many APIs, which is used by many apps. Those apps are installed on many machines. I get data containing machine Id, app name and API info . The data is sampled such that 2% of the ...
0
votes
0answers
23 views
Imbalanced Data Classification
This is my code with which I tried to attack my imbalanced dataset from here.
Now I have 2 questions.
When I try to split my data into train/test data and try to run the table() function, I always ...
1
vote
1answer
673 views
SMOTE for regression
I am looking into upsampling an imbalanced dataset for a regression problem (Numerical target variables) in python.
I attached paper and R package that implement SMOTE for regression, can anyone ...
-1
votes
1answer
41 views
Sampling randomly from pd.DataFrame, but ignoring NaN values
I am trying to sample random values from a dataframe where the NaN values should be ignored, without dropping the entire row or column.
My sampling function at the moment looks like this:
...
0
votes
1answer
19 views
Over/Under Sampling for Multi-classification
I'm trying to apply xgboost and random forest for over and under sampling
For imbalanced data:
train shape -> (199991, 23)
However, reverse my expectation. ...
2
votes
0answers
27 views
SuperLearner Cross validation with iid time series
I created a number of ML models in R and I aim at combining them to form an ensemble.
I learned about SuperLearner library which cross validates many models and returns the weight to each model in ...
5
votes
1answer
170 views
Why gradient boosting uses sampling without replacement?
In Random Forest each tree is built selecting a sample with replacement (bootstrap). And I assumed that Gradient Boosting's trees were selected with the same sampling technique. (@BenReiniger ...
1
vote
1answer
22 views
How to resample one dataset to conform to the distribution of another dataset?
I have two datasets with 20 features, but with different feature distributions (DS_A and DS_B). How can I sample the DS_A to make its distribution similar to DS_B, with respect to multiple features??
...
1
vote
1answer
24 views
How to compute modulo of a hash?
Let's say that I have a set of users in my database, that have GUIDs as their IDs. I use xxhash to generate fixed-length hashes for each value, so that I can then ...
2
votes
1answer
20 views
Can sampling like SMOTE/UP/DOWN applied on Validation set?
I am trying to predict classification problem. For that I have used Ranger, Xgboost and naive bayes. My Response class is imbalance . 92:8 ratio. My positive Response is only 8% of whole data.
...
1
vote
1answer
44 views
How to estimate the accuracy on a large dataset?
Given that I have a deep learning model(handover from former colleague).
For some reason, the train/dev set was missing.
In my situation, I want to classify my dataset into 100 categories.
The ...
2
votes
0answers
29 views
Difference between Gibbs sampling and variational Bayes inference
After reading in blogs and books, I came to the conclusion that Gibbs sampling and variation Bayes are methods for estimating or inference of posterior. Below link described but it's difficult to ...
3
votes
0answers
29 views
Stratified selection based on the y response creates a bias in information (Berkson's bias)?
I have a database which has individuals in different months and a target variable which indicates wheter an event happened or not, let's say:
...
2
votes
1answer
34 views
Is there an algorithm for sampling shortest paths?
I have a triangle matrix NxN of distances between vertices (vertex i connected only with vectices j>i) and I'd like to sample path from first to the last and use it as a training sample. Is there an ...
2
votes
1answer
50 views
Using Majority Class to Predict Minority Class
Suppose I want to train a binary model in order to predict the probability of who will buy a personal loan and in the dataset only 5 percent of the examples are people who marked as bought a personal ...
5
votes
1answer
315 views
How are samples selected from training data in Xgboost
In Random Forest, each tree is not fed with the full batch of training data, only a sample.
How does this work for Xgboost? If this sampling happens as well, how does it work for this ML algorithm?
2
votes
1answer
21 views
Gaussian Process for Classification: How to do predictions using MCMC methods
Problem
I was reading about Gaussian Processes for regression in the "Gaussian Processes for Classification" textbook and in a few other online resources. Everywhere I look people seem to avoid ...
0
votes
0answers
73 views
undersampling problem in imblanced dataset ValueError: Unknown label type: 'continuous'
I would like to undersampling the data but I encounter the following error?
...
2
votes
2answers
180 views
How to perform bootstrap validation?
I am working on a binary classification problem.
I ran cross-validation and grid-search on train data. Later I validated the model on my test data as shown below
...
1
vote
2answers
48 views
Bayesian network in Python: both construction and sampling
For a project, I need to create synthetic categorical data containing specific dependencies between the attributes. This can be done by sampling from a pre-defined Bayesian Network. After some ...
3
votes
2answers
39 views
Does Sampling size matters in Multi classification Model
I am working on a multi class classification model where few of the class are with less data compare to other classes.
I used random sampling technique to create a sample from the population keeping ...
1
vote
1answer
100 views
Adjust predicted probability after smote
i have an imbalance data set and I used smote to oversample the minority class and undersample the majority class. now, I want to check the test AUC using predict_proba of the model.
I have two ...
0
votes
0answers
11 views
What are accepted methods for sampling data that will be sampled?
I have a very large (TBs) dataset that I need to sample to a smaller more manageable size. However, I know that the sample will also be sampled to get statistics after it leaves my hands. Are there ...
0
votes
1answer
91 views
Correctly evaluate model with oversampling and cross-validation
I'm dealing with a classic case of dataset with binary imbalanced target (event 3%, non event 97%). My idea is to apply some sort of sampling (over/under, SMOTE etc.) to address the issue.
As I see, ...
0
votes
2answers
622 views
How to find whether a dataset is blanced or imbalanced?
I have few dataset to experiment classification(Multi-class). These datasets are about 400GB. I wanted to know whether the dataset is balanced or imbalanced. How to know that dataset is balance or ...
0
votes
0answers
13 views
How to compare two 3d vectors, which differ by sampling start time
I have some curve, assume it is given by some function f(x(t),y(t)).
This 3d curve is sampled twice:
t=0..99, no noise. This ...
2
votes
0answers
115 views
Training data requirements for NLP models
Are there general guidelines for how much data is required for natural language processing (NLP) classification models? I understand this may depend on the text quality, text length, how accurate the ...
2
votes
3answers
106 views
Highly Imbalanced dataset fro classes more than 200
I have a text dataset where I need to train a classifier to classify the titles into categories. The dataset shape is more than 575000. There are 256 target classes here. The problem is the dataset is ...
4
votes
3answers
276 views
Why did sampling boost the performance of my model?
I have an imbalanced dataset with 88 positive samples and 128575 negative samples. I was reluctant to over/undersample the data since it's a biological dataset and I didn't want to introduce synthetic ...
0
votes
0answers
16 views
How to compare two samples taken from the same dataset?
I have data from almost 10 different sensors and was trying to downsample the data. I tried the reservoir algorithm and the uniform sampling and the random sampling techniques. I was wondering how I ...
1
vote
1answer
17 views
Sampling trying to keep as much multivariate variance as possible
I was thinking if anyone considered a sampling technique that would try to aim keeping as much of the variance as possible (e.g. as many unique values, or very widely distributed continuous variables)....
0
votes
0answers
21 views
Sampling user IDs from a large data set for retention analysis
Let's say I have a data set with user identities and their respective orders, of the form:
...
0
votes
1answer
699 views
train_test_split ValueError: Input contains NaN
I tried to do a stratified sampling by way of train_test_split in order to save myself some trouble later. So I wrote the following lines:
...
0
votes
0answers
30 views
Can we have a sampled sigmoid instead of softmax?
Thh solution proposed here: is for softmax negative sampling. How do we do a sigmoid negative sampling? I couldnt find a corresponding 'tf.nn.sampled_sigmoid_loss' function.
0
votes
1answer
103 views
Random forest with zero precision for unbalanced test data
Apologies if this is a basic question.
I have a very unbalanced dataset in which the records are labelled by one of two classes, class1 (negative class) and class2 (positive class):
class 1: 1.5 ...
1
vote
0answers
31 views
Which machine learning methods can be used to address MonteCarlo sampling problems?
I would like to pick the brains of machine learning experts here to point me to a branch of machine learning technique that is best suited for my context. I would like to estimate the average value $\...
1
vote
0answers
58 views
Variable Importance changes with oversampling
I am currently using Xgboost for a binary classification problem with highly imbalanced data in R.
I have used oversampling to train the model. This worked well, now however it comes to measuring ...
0
votes
2answers
58 views
Forcing class imbalance to mirror the target data
I'm trying to do binary classification on some data, my source data has a class split of 40% A / 60% B while my target data has a split of 70% A / 30% B.
Is it a worthwhile strategy to use SMOTE to ...
3
votes
2answers
216 views
Is it OK to use the testing sample to compare algorithms?
I'm working on a little project where my dataset have 6k lines and around 300 features, with a simple binary outcome.
Since I'm still learning ML, I want to try all the algorithms I can manage to ...
3
votes
1answer
1k views
SVM SMOTE fit_resample() function runs forever with no result
Problem
fit_resample(X,y)
is taking too long to complete execution for 2million rows.
Dataset specifications
I have a labeled dataset about network features, ...
