I have a very long dataset and I'm trying to reduce it by grouping the data in periods of 24 hours. In this way, there will be a single data point that represents that day, but they must yield the same (or a good approximation) a and b coefficients when I fit a line with y = ax + b. I thought about taking the mean of each group, or the median, or perhaps the centroid, but I'm not sure which metric would be appropriate.

Another reason to reduce my dataset is to plot it faster, as of today it takes much longer than I can afford.

To summarize: as in the image below I have the grey points and need to group them and find the orange squares that would provide the same line when I perform a linear regression.

enter image description here

  • $\begingroup$ Means would probably be fine, although they wouldn't give the same result. Slightly related: en.wikipedia.org/wiki/Law_of_total_variance. Or you could do a regression on a small random sample of all points from your dataset, that should give a good approximation of the coefficients unless your distribution is heavy tailed. $\endgroup$
    – Valentas
    Oct 7 at 15:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.