0
$\begingroup$

I have a dataset that is bigger than I need it to be. In fact, bigger than my hardware can handle. So I'm trying to lower the number of samples. And I'm not sure what is the right approach to do so. Here are some other facts:

  • I'll be applying some clustering algorithm afterward
  • The outliers are the important data points in the dataset (there could be 1 outlier per each 100k records)
  • There are 13 columns in this dataset each with a different distribution

I understand that if I wanted to keep the current distribution of the data, I should resample uniformly. But the fact is that the clustering algorithm does not benefit from having many close (almost identical) data points. So I would rather resample in a way that the resulting dataset is relatively uniform. But I'm not sure if this is a valid approach and if it is, how to do it (for instance in Python with Pandas).

$\endgroup$
9
  • $\begingroup$ I don't understand what is the goal, is it outlier detection or clustering? Also it depends a lot on how the system is going to be used: the distribution for clustering or for outlier detection should be the same as the one it receives "in production". $\endgroup$
    – Erwan
    Jun 24 at 1:03
  • $\begingroup$ Thx, you are right but I'm not there yet to consider the design for the production system. I'm just trying to take the first steps and understand the data. The outliers are actually real data points. It's just that they're way different from the rest of the dataset (at least that's my expectation) and there are too few of them. So, I need to make sure that by resampling, I'm not losing them. BTW, the reason why I assumed resampling with different distributions could still work is cuz I'm planning to use DBSCAN. I believe it doesn't care about the number of data points but only the closest ones $\endgroup$
    – Mehran
    Jun 24 at 1:26
  • $\begingroup$ Be careful: DBScan uses density, and density depends on frequency. In general changing the distribution might cause the outliers not to be outliers anymore. For example let's say the data is made of the following values with their frequencies: 1x100, 2x50, 3x10, 7x1, 8x1, 9x1. From this distribution it's clear that 7, 8 and 9 are outliers. However if we remove duplicate values then there's only 1, 2, 3, 7, 8, 9, each value only once. Now 7, 8, 9 are not outliers anymore. $\endgroup$
    – Erwan
    Jun 24 at 1:55
  • $\begingroup$ Maybe I'm wrong but the way I see a flaw in your assumption is that DBSCAN relies on distance and locality and not density (I'm absolutely new to it but that's the way I see it). This means, in your latter example, depending on the DBSCAN's parameters (e.g. eps=0.5, min_samples=1), 1-9 could each be a separate cluster. Just like the earlier example. Thus the number of data points does not matter. In my use case, the term outlier is used loosely. They are simply data points with less frequency. Also, I can identify outliers by just eyeballing (once clustered). It's just too high-dimensional. $\endgroup$
    – Mehran
    Jun 24 at 2:09
  • $\begingroup$ I'm not very knowledgeable about DBScan myself but the first letter D stands for Density: Density-based spatial clustering of applications with noise. But if you're sure that the outliers are clearly distinguishable only by their features, then you're right that the distribution shouldn't matter. You might be a bit optimistic about the results of the clustering though: with a lot of data it rarely gives a very clear picture especially at the detailed level of specific points. $\endgroup$
    – Erwan
    Jun 24 at 2:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.