I was analysing a dataset which has got two main columns which I can name like:
category:String amount: Int
So I wanted to know the average of amount for each of the category. So I did something like:
df.groupBy("category").agg(avg("amount") as average)
The problem is for categories I don't have many observations, sometimes just one or two. But for other categories I have around 5000.
So I would like to know if there is any metrics where I can reflect that the average may be more or less accurate depending on the amount of observations used for the average.