Let's say I have 1-D data following a Gaussian distribution. I want to extract from this database the meaningful information, that is the information that lies far away from the mean.

One way to do that is to apply a function (e.g. $x \to |x-\mu|$ where $\mu$ is the mean of the distribution) and sort the values I get. But I want to keep track of how meaningful the data is as I am collecting it all along this process (as I do the bottom-up approach : I gather the samples one by one and create another (smaller) database). Is there an intuitive measure of this concept ?

I thought about comparing the histograms between the initial database and the one I create as I gather the samples. For example, the Wasserstein metric gives us a way to compare two probability distributions. An intuitive effect of the extraction I am making would be that the distance between the histograms does not decrease by much at first, as I only fill the error that does not contribute much to the distance. Is that a good approach ?

Next level : If my data is made of groups of samples (still, the entire database follows a Gaussian distribution), what would be the intuitive way to select groups that have meaningful information ?

I though about giving each group a weight that is the average of the variance of each sample that is in the group, and then sort the groups with the weights they have. But does this really keep track of the useful information ?

Next Next level : What about a n-D data following a (multivariate) Gaussian distribution ?

This is kinda theoric. If you have any questions feel free. Any article on the matter is also appreciated.

  • $\begingroup$ I'm not sure I grasp the idea: does the process need to be fully incremental, or can you start by collecting a small sample in order to correctly estimate the mean and std dev? And you want to collect the points far from the mean, but how far or how much in proportion of the data? Fair warning: I'm curious about the problem but I'm not sure I can help ;) $\endgroup$ – Erwan Jul 29 '19 at 23:02
  • $\begingroup$ I have access to the whole initial database, thus I know the mean and the standard deviation. I'd say I want to have a uniform distribution in the end (I want to know every information the Gaussian distribution gives me in the same manner). As I have little information far away from the mean, and a lot of information around the mean, I want to focus on what I called the meaningful information. For the Next level, I know that I will have samples in each group that will be close to the mean, so I do focus on the samples that lie far away from the mean. $\endgroup$ – Flewer47 Jul 30 '19 at 15:59
  • $\begingroup$ I gave it a try but as I suspected I'm out of my depth, so I gave up :D. I'd suggest you ask this on stats.stackexchange.com, for theoretical questions like this you're much more likely to get a good answer there. Don't forget to mention what you said in the comment, I think it really makes the question clearer. $\endgroup$ – Erwan Jul 30 '19 at 21:11
  • $\begingroup$ Thanks for your time ! $\endgroup$ – Flewer47 Jul 31 '19 at 16:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.