2
$\begingroup$

I am measuring the "proximity" of certain words (proper nouns) within a certain text (Silmarillion) by determining the occurrences of these words within the text and, via binning, creating a discrete distribution of them.

I then compare these distributions (% of occurrences per bin) with the Hellinger distance, and I run Mathematica's "FindClusters" function on the resulting distance matrix. The output shows me how the words are clustered within the text. I leave it to Mathematica to determine the number of clusters. So far, so good.

Clearly, my chosen bin width impacts the distributions which I attempt to cluster, so I was wondering if there is any measure for the discriminating power of a clustering; the maximum information content between "all elements in one cluster" and "each element is its own cluster."

I would then run a loop with increasing bin width to find the clustering telling me "the most" about the structure of the text.

A link to a source would actually suffice - I wasn't able to find anything, probably because I am not sure of the appropriate search terms.

Improve this question
$\endgroup$
2
  • $\begingroup$ Welcome to DataScienceSE. I'm not very knowledgeable about this but this could be the same evaluation problem as the one in topic modeling, so hopefully looking at a few papers in this area should give you an idea how to evaluate. Also I don't fully understand your method, do you use the words in the close context of the proper noun to represent them as a distribution? $\endgroup$
    – Erwan
    Mar 17, 2022 at 2:44
  • $\begingroup$ Thanks for the hint, Erwan. I have extracted all proper nouns and am trying to cluster them according to THEIR proximity to each other in the text. Since the Silmarillion is more a "History" text than a story, many of the places and characters come and go. $\endgroup$
    – Matthias
    Mar 17, 2022 at 11:55

1 Answer 1

Reset to default
0
$\begingroup$

Okay, I appear to have found some stuff:

I have implemented the Dunn index and am currently running the clusters for a sequence of bin width, taking note of the index for subsequent visual inspection of the results.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.