I am collecting data for machine learning models I want to build for some application.

I started with random sampling (just simply collecting 'recent' data) but I am not getting enough records of interest, which I want to use as my target. The process is currently automated.

I don't want to use SMOTE unless it works really well with text(NLP) data.

I also don't really want to tackle this as an anomally detection either.

Idealy, I want to build a look-alike model to give "authenticity score".

I could manually search for targets I am lacking or set up another collecting method, which may be more efficient than the random sampling.

What I am afraid of is that this might introduce some biases in my data.

How can I know if there is a bias?

My data consists of text, categorical, and continuous variables.

If all the samples are mixed and shuffled nicely, would that be alright unless training data is from one sampling method and testing data is from another?

  • $\begingroup$ Is the issue that you’re winding up with unbalanced classes in a classification problem? $\endgroup$ – Dave Dec 31 '20 at 20:53
  • $\begingroup$ yes that is my main issue cuz currently less han 1% of the collected samples are my target $\endgroup$ – Hiro Nakagame Dec 31 '20 at 20:54
  • $\begingroup$ Look up Stephan Kolassa on Cross Validated, stats.stackexchange.com. He has a post on why unbalanced classes aren’t a problem at all! $\endgroup$ – Dave Dec 31 '20 at 21:07
  • $\begingroup$ artificialy oversampling the minority class (if this is the case) IS going to introduce bias. In simple terms unbiased sampling means representative sampling in the sense that it reflects the underlying distribution. Since this is unknown, random sampling is the best approach in general as long as it is indeed diversified and random $\endgroup$ – Nikos M. Jan 1 at 17:36
  • $\begingroup$ Theoretically, I understand that it would introduce bias. If I call random samples A and artificially targeted samples B, I expect the target variable to have different distributions.But if A and B have very similar distributions for other continuous/categorical features, would there be any issues/concerns because of the sampling methonds? $\endgroup$ – Hiro Nakagame Jan 2 at 6:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.