Metric to combine average with number of accurance

Question

I was analysing a dataset which has got two main columns which I can name like:

category:String amount: Int

So I wanted to know the average of amount for each of the category. So I did something like:

df.groupBy("category").agg(avg("amount") as average)

The problem is for categories I don't have many observations, sometimes just one or two. But for other categories I have around 5000.

So I would like to know if there is any metrics where I can reflect that the average may be more or less accurate depending on the amount of observations used for the average.

Answer 1

The metric you are looking for is the standard error. From Wikipedia:

the standard error equals the standard deviation divided by the square root of the sample size.

So as the sample size increases the standard error decreases. You can also see this for an explanation with some code in R.