Questions tagged [sampling]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
-1
votes
0answers
10 views

How to manually collect rectangular training data samples from images? [closed]

I want to collect training samples from images. That can mean different things depending on the context. I think of the simplest case, which should be most commonly required. Because it is so common, ...
0
votes
0answers
24 views

Chi Square Test Goodness of Fit

I want to use a chi square test but I'm unsure if I'm using it right. The KickStarter website shows the frequency of main categories projects. It is updated once a day. I got a data set of KickStarter ...
0
votes
0answers
19 views

Question regarding strata in Geron's book

In the book "Hands-On Machine Learning with Scikit-Learn and Tensorflow" by Aurélien Géron. There is a regression project explained. My doubt is regarding his example for 'stratified sampling'. He ...
0
votes
0answers
15 views

Sample size calculation for AB testing

I have sales data for 500 stores and lets say I want to run an AB test on a sample of test and control stores with the objective of getting statistically significant increase in sales(lift). My ...
0
votes
0answers
7 views

Smote oversampling on strings

I have an imbalanced dataset with most transactions being OK and 1% transactions marked as fraud. This dataset has columns with integers (time and transaction amount and timestamp) as well as strings (...
0
votes
0answers
14 views

Time Series Model (n-beats paper) how is sampling / training done?

does any one know what this means? It is taken from the paper https://openreview.net/pdf?id=r1ecqn4YwB (n-beats time series model). To update network parameters for one horizon, we sample train ...
1
vote
1answer
16 views

How to draw a sample from data set with respect to a given categorical or numerical variable based on given freely chosen distribution? (Python)

Say I have a data set for some past period. Now new data appears and for a given variable in the data and we find that the distributions have shifted (for example with "age" it would be that suddenly ...
0
votes
1answer
24 views

Is there a name for this form of sampling?

I had an idea to combine oversampling and undersampling in the following way: Compute the average number of individuals in each class. For classes with a number of individuals greater than this ...
0
votes
0answers
19 views

dealing with imbalanced data for multi-class problem

Based on the experiments I run for a number of times, and the reading I did on imbalanced data for a multiclassification problem such as this paper, resampling techniques like ...
0
votes
1answer
20 views

Dealing with large data: selecting a sample

I'm given a data set to create a model that would predict whether a certain supply chain would be able to deliver the goods without delay or not. I'm doing this in Python. The data set has 93000 ...
0
votes
0answers
9 views

Estimate machine count for API dependency

I have a SDK, with many APIs, which is used by many apps. Those apps are installed on many machines. I get data containing machine Id, app name and API info . The data is sampled such that 2% of the ...
0
votes
0answers
23 views

Imbalanced Data Classification

This is my code with which I tried to attack my imbalanced dataset from here. Now I have 2 questions. When I try to split my data into train/test data and try to run the table() function, I always ...
1
vote
1answer
673 views

SMOTE for regression

I am looking into upsampling an imbalanced dataset for a regression problem (Numerical target variables) in python. I attached paper and R package that implement SMOTE for regression, can anyone ...
-1
votes
1answer
41 views

Sampling randomly from pd.DataFrame, but ignoring NaN values

I am trying to sample random values from a dataframe where the NaN values should be ignored, without dropping the entire row or column. My sampling function at the moment looks like this: ...
0
votes
1answer
19 views

Over/Under Sampling for Multi-classification

I'm trying to apply xgboost and random forest for over and under sampling For imbalanced data: train shape -> (199991, 23) However, reverse my expectation. ...
2
votes
0answers
27 views

SuperLearner Cross validation with iid time series

I created a number of ML models in R and I aim at combining them to form an ensemble. I learned about SuperLearner library which cross validates many models and returns the weight to each model in ...
5
votes
1answer
170 views

Why gradient boosting uses sampling without replacement?

In Random Forest each tree is built selecting a sample with replacement (bootstrap). And I assumed that Gradient Boosting's trees were selected with the same sampling technique. (@BenReiniger ...
1
vote
1answer
22 views

How to resample one dataset to conform to the distribution of another dataset?

I have two datasets with 20 features, but with different feature distributions (DS_A and DS_B). How can I sample the DS_A to make its distribution similar to DS_B, with respect to multiple features?? ...
1
vote
1answer
24 views

How to compute modulo of a hash?

Let's say that I have a set of users in my database, that have GUIDs as their IDs. I use xxhash to generate fixed-length hashes for each value, so that I can then ...
2
votes
1answer
20 views

Can sampling like SMOTE/UP/DOWN applied on Validation set?

I am trying to predict classification problem. For that I have used Ranger, Xgboost and naive bayes. My Response class is imbalance . 92:8 ratio. My positive Response is only 8% of whole data. ...
1
vote
1answer
44 views

How to estimate the accuracy on a large dataset?

Given that I have a deep learning model(handover from former colleague). For some reason, the train/dev set was missing. In my situation, I want to classify my dataset into 100 categories. The ...
2
votes
0answers
29 views

Difference between Gibbs sampling and variational Bayes inference

After reading in blogs and books, I came to the conclusion that Gibbs sampling and variation Bayes are methods for estimating or inference of posterior. Below link described but it's difficult to ...
3
votes
0answers
29 views

Stratified selection based on the y response creates a bias in information (Berkson's bias)?

I have a database which has individuals in different months and a target variable which indicates wheter an event happened or not, let's say: ...
2
votes
1answer
34 views

Is there an algorithm for sampling shortest paths?

I have a triangle matrix NxN of distances between vertices (vertex i connected only with vectices j>i) and I'd like to sample path from first to the last and use it as a training sample. Is there an ...
2
votes
1answer
50 views

Using Majority Class to Predict Minority Class

Suppose I want to train a binary model in order to predict the probability of who will buy a personal loan and in the dataset only 5 percent of the examples are people who marked as bought a personal ...
5
votes
1answer
315 views

How are samples selected from training data in Xgboost

In Random Forest, each tree is not fed with the full batch of training data, only a sample. How does this work for Xgboost? If this sampling happens as well, how does it work for this ML algorithm?
2
votes
1answer
21 views

Gaussian Process for Classification: How to do predictions using MCMC methods

Problem I was reading about Gaussian Processes for regression in the "Gaussian Processes for Classification" textbook and in a few other online resources. Everywhere I look people seem to avoid ...
0
votes
0answers
73 views

undersampling problem in imblanced dataset ValueError: Unknown label type: 'continuous'

I would like to undersampling the data but I encounter the following error? ...
2
votes
2answers
180 views

How to perform bootstrap validation?

I am working on a binary classification problem. I ran cross-validation and grid-search on train data. Later I validated the model on my test data as shown below ...
1
vote
2answers
48 views

Bayesian network in Python: both construction and sampling

For a project, I need to create synthetic categorical data containing specific dependencies between the attributes. This can be done by sampling from a pre-defined Bayesian Network. After some ...
3
votes
2answers
39 views

Does Sampling size matters in Multi classification Model

I am working on a multi class classification model where few of the class are with less data compare to other classes. I used random sampling technique to create a sample from the population keeping ...
1
vote
1answer
100 views

Adjust predicted probability after smote

i have an imbalance data set and I used smote to oversample the minority class and undersample the majority class. now, I want to check the test AUC using predict_proba of the model. I have two ...
0
votes
0answers
11 views

What are accepted methods for sampling data that will be sampled?

I have a very large (TBs) dataset that I need to sample to a smaller more manageable size. However, I know that the sample will also be sampled to get statistics after it leaves my hands. Are there ...
0
votes
1answer
91 views

Correctly evaluate model with oversampling and cross-validation

I'm dealing with a classic case of dataset with binary imbalanced target (event 3%, non event 97%). My idea is to apply some sort of sampling (over/under, SMOTE etc.) to address the issue. As I see, ...
0
votes
2answers
622 views

How to find whether a dataset is blanced or imbalanced?

I have few dataset to experiment classification(Multi-class). These datasets are about 400GB. I wanted to know whether the dataset is balanced or imbalanced. How to know that dataset is balance or ...
0
votes
0answers
13 views

How to compare two 3d vectors, which differ by sampling start time

I have some curve, assume it is given by some function f(x(t),y(t)). This 3d curve is sampled twice: t=0..99, no noise. This ...
2
votes
0answers
115 views

Training data requirements for NLP models

Are there general guidelines for how much data is required for natural language processing (NLP) classification models? I understand this may depend on the text quality, text length, how accurate the ...
2
votes
3answers
106 views

Highly Imbalanced dataset fro classes more than 200

I have a text dataset where I need to train a classifier to classify the titles into categories. The dataset shape is more than 575000. There are 256 target classes here. The problem is the dataset is ...
4
votes
3answers
276 views

Why did sampling boost the performance of my model?

I have an imbalanced dataset with 88 positive samples and 128575 negative samples. I was reluctant to over/undersample the data since it's a biological dataset and I didn't want to introduce synthetic ...
0
votes
0answers
16 views

How to compare two samples taken from the same dataset?

I have data from almost 10 different sensors and was trying to downsample the data. I tried the reservoir algorithm and the uniform sampling and the random sampling techniques. I was wondering how I ...
1
vote
1answer
17 views

Sampling trying to keep as much multivariate variance as possible

I was thinking if anyone considered a sampling technique that would try to aim keeping as much of the variance as possible (e.g. as many unique values, or very widely distributed continuous variables)....
0
votes
0answers
21 views

Sampling user IDs from a large data set for retention analysis

Let's say I have a data set with user identities and their respective orders, of the form: ...
0
votes
1answer
699 views

train_test_split ValueError: Input contains NaN

I tried to do a stratified sampling by way of train_test_split in order to save myself some trouble later. So I wrote the following lines: ...
0
votes
0answers
30 views

Can we have a sampled sigmoid instead of softmax?

Thh solution proposed here: is for softmax negative sampling. How do we do a sigmoid negative sampling? I couldnt find a corresponding 'tf.nn.sampled_sigmoid_loss' function.
0
votes
1answer
103 views

Random forest with zero precision for unbalanced test data

Apologies if this is a basic question. I have a very unbalanced dataset in which the records are labelled by one of two classes, class1 (negative class) and class2 (positive class): class 1: 1.5 ...
1
vote
0answers
31 views

Which machine learning methods can be used to address MonteCarlo sampling problems?

I would like to pick the brains of machine learning experts here to point me to a branch of machine learning technique that is best suited for my context. I would like to estimate the average value $\...
1
vote
0answers
58 views

Variable Importance changes with oversampling

I am currently using Xgboost for a binary classification problem with highly imbalanced data in R. I have used oversampling to train the model. This worked well, now however it comes to measuring ...
0
votes
2answers
58 views

Forcing class imbalance to mirror the target data

I'm trying to do binary classification on some data, my source data has a class split of 40% A / 60% B while my target data has a split of 70% A / 30% B. Is it a worthwhile strategy to use SMOTE to ...
3
votes
2answers
216 views

Is it OK to use the testing sample to compare algorithms?

I'm working on a little project where my dataset have 6k lines and around 300 features, with a simple binary outcome. Since I'm still learning ML, I want to try all the algorithms I can manage to ...
3
votes
1answer
1k views

SVM SMOTE fit_resample() function runs forever with no result

Problem fit_resample(X,y) is taking too long to complete execution for 2million rows. Dataset specifications I have a labeled dataset about network features, ...