I have a dataset with 5000 observations with class 0 and 300 observations with class 1. I would like to run some statistical analysis, for example on the average length of strings, the number of words, ... I was calculating the average. Do you think it makes sense to calculate and compare these values with a similar difference In classes?
The answer to this question really depends on the insights you are after with these statistics.
Generally speaking however, it is wise to investigate per class in such a situation. Taking your examples this would entail comparing average string lengths for both classes. This will help you understand how both classes are (dis)similar given the dataset.