Questions tagged [sampling]
The sampling tag has no usage guidance.
155
questions
2
votes
1answer
23 views
Discrepency in 25th percentile of standard deviation calculation
For each of the following sample sizes [3,5,7,9], I Need to calculate the 25th percentile of the standard deviation for the values sampled. And the total number of trials will be 10.
DataFrame
...
0
votes
0answers
23 views
Is it right method to remove instances that are hard to predict before train test split?
In a binary classification problem,
I have a slightly unbalanced medical dataset with class distribution:
0:5600, 1:1500
0 without a problem and 1 with a problem.
I tried many pipelines, automls, and ...
1
vote
1answer
28 views
Multiclass data redistribution
I want to redistribute the data in classes according to new proportions and wonder what is the optimal way to do it.
For example I have
...
2
votes
1answer
43 views
Negative sampling for graph representation learning
I was watching a lecture about graph representation learning (from here) and got a little bit confusing about how they define the negative samping procedure.
In the presentation J. Leskovec briefly ...
1
vote
1answer
31 views
Is there a relationship between learning rate and training set size?
I have a large dataset to use for training a Neural Network model. However, I don't have enough resources to do a proper hyperparameters tuning on the whole dataset. Therefore, my idea is to tune the ...
1
vote
1answer
29 views
Reducing (Variance) | the gap between my weights
I have ML ready samples. And each sample has a weight.
The weights distribute between [0-1]
My problem arise because there are a lot of samples which are ...
0
votes
0answers
17 views
LSTM long sequences reduction algorithm
suppose I have a large multi-variate TimeSeries dataset with very long sequences (~50k)
I want to reduce each sequence size to constant size for LSTM training.
I thought about splitting each sequence ...
0
votes
1answer
55 views
Controlling the sampling from Variational AutoEncoder (VAE)
Suppose a Variational Autoencoder (VAE) is trained with mnist data. To sample, one draws from normal distribution. My question is: suppose I am interested in generating only 1s and no other digits. ...
0
votes
1answer
34 views
Subsampling the “right” amout of data to train an ML model
I am training a machine learning model (i.e., a classifier) on a large dataset. I know that I can get the same results using less data (about 30%) but I would like to avoid the trial and error process ...
1
vote
0answers
19 views
Stratification sampling of a json Array [closed]
I have a json array file that i need to create a smaller sample of for testing purposes. A sample of the file looks like:
...
1
vote
1answer
28 views
What is the proper proportion for train and test set for classification system?
I have recently googled the best proportion for training and test set for classifying physiological data between normal and abnormal. Much of the source tells that the proportion should be 70:30 or 80:...
2
votes
2answers
103 views
Undersampling for credit card fraud detection before or after Train/Test Split
I have a credit card dataset with 98% transactions are Non-Fraud and 2% are fraud.
I have been trying to undersample the majotrity class before train and test split and get very good recall and ...
1
vote
2answers
23 views
Sampling n data points from high dimensional data
I have some face images(of a single person), which I ran through an embedding generator to get 128-dimensional embedding.
I have 1000 such embedding (shape of the dataset (1000, 128)). I have a ...
1
vote
1answer
30 views
How to bin a distribution data reported with different frequencies ( salaries ), showing mixed linearity and non-linearity?
I am researching on pay-scales, and wish to receive advise to treat data of salaries.
Objective
My interest is to approximate the salary corresponding to different hierarchical levels in an ...
0
votes
0answers
17 views
Sampling items from a population of subpopulations
I have a population of $n$ items to label and a budget to label only $m$ ($m << n$) of them before training. The population can be partitioned into subpopulations, recursively. In other words, ...
1
vote
1answer
30 views
Recover a integer valued function with *-learning
I have the following problem. From a technical model we have a function $f(n,p)$ approximating its runtime. The function $f$ which maps
$$
f: \mathbb{N} \times \mathbb{P} \to \mathbb{R}_{+}
$$
where $...
0
votes
0answers
23 views
how to know if there is a bias in data collection methods
I am collecting data for machine learning models I want to build for some application.
I started with random sampling (just simply collecting 'recent' data) but I am not getting enough records of ...
1
vote
0answers
34 views
Follow up question regarding Upsampling for Imbalanced Data and the use of ADASYN instead of SMOTE
I have a follow-up question regarding this topic.
I have been working on a project predicting success(1) or failure(0) for organizations by using the Decision Tree and Random Forest algorithms.
My ...
1
vote
1answer
20 views
Is it right to maintain the train distribution in test set for unbalanced data?
If the training set was unbalanced the chances are the model will be biased. But if the data distribution in the test set is the same distribution as the train set, this kind of bias is not going to ...
3
votes
2answers
584 views
Over-sampling: is my model over-fitting?
I would like to ask you some questions on how to consider (good or not) the following results:
...
0
votes
1answer
26 views
How to Present All Categories in All Samples
I have a data contains many categorical columns. When I sampled this data randomly a few times and applied one-hot encoding to categorical columns I noticed that it ended up with datasets with ...
6
votes
2answers
83 views
Is sampling a valid way to reduce complexity?
I'm facing an issue where I have a massive amount of data that I need to cluster. As we know, clustering algorithms can have a very high O complexity, and I'm looking for ways to reduce the time my ...
-1
votes
1answer
33 views
How Should I deal with my imbalanced binary target [closed]
I am trying to model my data with Python and i am having concerns about my binary target variable, because it has 90% cases falling in 0 and 10% of the cases falling in 1. I have tried upsampling my ...
0
votes
1answer
24 views
What is the most common practice of generating (X,Y) from an arbitrary CDF or PDF?
So I can generate X from arbitrary CDF F(x) by the procedure above. Can it be generalized to two variables? How, exactly? If not, what's the best way to generate <...
1
vote
2answers
46 views
How to generate a random sample and distribute values based in an probability distribution?
I want to generate a random sample based on this probability distribution:
The line is the KDE of the histogram.
My random sample will have n values, the value is ...
0
votes
0answers
18 views
Training a Variational Autoencoder (VAE) for Non-Uniform Random Number Generation
I have a complicated 20-dimensional non-uniform distribution and would like generate samples from it. I have considered training a VAE to do so, but my problems are the following:
Is my approach ...
0
votes
0answers
12 views
Positive records : no behavioral data
0
I'm working on a classification model aimed at identifying if behavioral activity within an account (b2b - one account, many contacts) can predict or not an opportunity generation ( a salesperson ...
1
vote
0answers
138 views
What's the order in applying SMOTE transformation in a pipeline?
Here's the thing, I have an imbalanced data and I was thinking about using SMOTE transformation. However, when doing that using a sklearn pipeline, I get an error because of missing values.
This is my ...
1
vote
1answer
41 views
How should I sample from a mixture distribution?
Let's say we have a mixture distribution, defined by density $f(x)= w_1 p_1(x) + w_2 p_2(x)$, where $w_i$ is a scalar weight. Furthermore, we have efficient methods to evaluate the pdf and cdf/icdf ...
0
votes
0answers
29 views
Difference between test statistic and sample statistic?
Test statistic and sample statistic are used to check statistical significance. How do I understand the procedure in background of reaching a conclusion about an effect-size estimate.
0
votes
0answers
18 views
HR employee attrition modeling - making a balanced sample question
I have dataset 1 (stayers) consisting of 1500 record of HR data demographic data of employees (11 features) who currently are in the company. Dataset 2 (leavers) contains 180 records -same features- ...
1
vote
1answer
93 views
Training a Variational Autoencoder (VAE) for Random Number Generation
I have a complicated 20-dimensional multi-modal distribution and consider training a VAE to learn an approximation of it using 2000 samples. But particularly, with the aim to subsequently generate ...
0
votes
0answers
22 views
How to downsample a dataset with constraints in Python?
I have a dataset that has columns a, b, and others. Both a and ...
1
vote
0answers
17 views
Sequential sampling from Gaussian conditional not working
I'm trying to sequentially sample from a Gaussian Process prior.
The problem is that the samples eventually converge to zero or diverge to infinity.
I'm using the basic conditionals described e.g. ...
1
vote
0answers
28 views
Generating artificial data to extend learning set
I have dataset containing 42 instances(X) and one final Y on which i want to perform LASSO regression.All are continuous and numerical. As the sample size small, I wish to extend it. I am kind of ...
1
vote
0answers
13 views
Adaptive Sampling Strategies for SVM?
I am an Engineer interested in creating a surrogate model of a certain phenomenon in the context of reliability engineering. Essentially my quantity of interest is the Limit state function/stability ...
0
votes
1answer
144 views
Bootstrap clarification for datasets with multiple columns
How do I bootstrap a Dataset/DataFrame with multiple continuous and categorical columns?
For eg: Say I'm trying to bootstrap the colour distribution of M&M's and I have 50 bags (samples) each with ...
1
vote
1answer
69 views
Sampling in Text Classification: can the results be considered 'reliable'?
I am testing different models (SVM, Logistic Regression, Naive Bayes, Random Forest) for predicting the class of a spam email. My target is a binary variable.
I am analysing only text, no other fields....
1
vote
0answers
29 views
Uniform convergence garantee on sample complexity
I can't understand why the Uniform Convergence guarantees an upper bound and not a lower bound on sample complexity as stated on [1]
Corollary 4.4.
If a class $H$ has the uniform convergence property ...
0
votes
1answer
25 views
Getting a balanced sample across many variables
Let’s say each element in my population has several attributes.
Let’s call then A, B, C, D, E, F.
Let’s say, for simplicity, each attribute has 10 values (but could be any number between 2 and 30).
...
0
votes
1answer
391 views
What is the effect of oversampling on logistic regression?
I am building a model to predict if a customer will use a coupon or not for a given campaign. I am using logistic regression for this model. I took 5 previous campaigns and generally for each campaign ...
1
vote
0answers
27 views
Representation sample size- n [closed]
Need help with identifying a representation sample size 'n'. Let's say I have a very large population- infinite number of participants. I am picking the random sample from this infinite population. I ...
1
vote
0answers
45 views
Chi Square Test Goodness of Fit
I want to use a chi square test but I'm unsure if I'm using it right. The KickStarter website shows the frequency of main categories projects. It is updated once a day. I got a data set of KickStarter ...
0
votes
1answer
104 views
What is difference between Standard Normal Distribution and Mean Normalization approaches to feature-scaling?
The tag feature-scaling seems to convey that one of the scaling methods is Standard Normal Distribution. Further, I read an Answer on this site saying that Mean Normalization is a form of feature ...
1
vote
1answer
25 views
How to draw a sample from data set with respect to a given categorical or numerical variable based on given freely chosen distribution? (Python)
Say I have a data set for some past period. Now new data appears and for a given variable in the data and we find that the distributions have shifted (for example with "age" it would be that suddenly ...
0
votes
1answer
31 views
Is there a name for this form of sampling?
I had an idea to combine oversampling and undersampling in the following way:
Compute the average number of individuals in each class. For classes with a number of individuals greater than this ...
1
vote
2answers
49 views
Dealing with large data: selecting a sample
I'm given a data set to create a model that would predict whether a certain supply chain would be able to deliver the goods without delay or not. I'm doing this in Python.
The data set has 93000 ...
2
votes
1answer
5k views
SMOTE for regression
I am looking into upsampling an imbalanced dataset for a regression problem (Numerical target variables) in python.
I attached paper and R package that implement SMOTE for regression, can anyone ...
-1
votes
1answer
429 views
Sampling randomly from pd.DataFrame, but ignoring NaN values
I am trying to sample random values from a dataframe where the NaN values should be ignored, without dropping the entire row or column.
My sampling function at the moment looks like this:
...
0
votes
1answer
27 views
Over/Under Sampling for Multi-classification
I'm trying to apply xgboost and random forest for over and under sampling
For imbalanced data:
train shape -> (199991, 23)
However, reverse my expectation. ...
