Types of averages when analyzing sentences

Question

I have a list of words and their frequencies in a text corpus. So there are words like "a", "what", "some" that have really high frequencies, and other like "neurodegenerative" that are less popular.

I want to analyze sentences by assigning to each word its score and then determine if one sentence is more "technical", or more specific to a domain than others. For example:

"I have a dog and a cat." vs. "Mitochondria is the powerhouse of the cell."

I was thinking of just calculating the average of these frequencies, but sometimes I have a sentence like:

"Migraine is a serious headache.", with average 640, and

"Typical examples of continuous functions which are not holomorphic are complex conjugation and taking the real part.", with average 600, because of the many short, very common words.

Is there any better way of evaluating such sentences to give a more realistic score, or average, that would indicate how "niche" they are?

Answer 1

It might depend on what you will then use the scores for. For instance, should one long sentence score higher than two shorter sentences even if all three sentences have the same density of technical words? If so, adding rather than averaging the scores? Or adding, then doing an adjustment for sentence length.

The other way to get the more technical words to have more weight, when you take the mean, is to raise their score to a power. The power becomes a hyperparameter you can tune for, but simply squaring scores would be enough to test the idea.

I'll also mention https://en.wikipedia.org/wiki/Tf%E2%80%93idf, in case you were not aware of it; the see also section might also bring up some ideas.