I think "balanced" can have different meanings in different contexts, but typically a "balanced" dataset is one in which the class labels are uniformly distributed. Of course, an "imbalanced" dataset is one in which the distribution of class labels is not uniform.
I'm not aware of discussions about balance metrics in the literature. One straightforward option is to measure the KL-divergence between the actual distribution of the class labels and a uniform distribution over the class labels. A KL-divergence of zero would indicate a perfectly balanced dataset. The KL-divergence will grow to infinity as the actual distribution differs from the perfectly-balanced distribution. This metric would allow you to compare the degree of imbalance between two datasets.
Unfortunately, KL-divergence is pretty fragile. It's technically undefined when one of the classes has zero examples. In practice you could treat such datasets as if they're infinitely imbalanced. This is what scipy.stats.entropy does.
However, this treatment has one undesirable side-effect. Any dataset in which there are zero examples of a class gets the same "imbalance score" (infinity). This doesn't match our intuition about imbalance. Suppose we have two datasets with examples from the classes "dog", "cat", and "horse". Let's say dataset A has 19 dogs, 1 cat, and 0 horses. Dataset B has 10 dogs, 10 cats, and 0 horses.
Ideally our metric would say that Dataset A is more imbalanced than Dataset B, but KL-divergence will give the same result for both datasets.
We can correct this problem by adding a tiny epsilon to each of the class distributions before calculating KL-divergence. This both solves the problem of being undefined when a class has zero examples, and it also corrects for the above scenario where two different datasets get the same imbalance score.
