I have a data set of about 1000 observations like so:
Project Description A B C D Sorter Assessment .4 0 .2 0 Renewal - Maintenance 0 0 0 0 Rebuild .5 0 0 0 Sorter Assembly 0 .4 0 .1
I would like to hierarchically cluster the observations based on the words used in Project Description, but only do so if they explain the variation among the A-D variables.
- If a project description includes 'Assessment' and that's correlated with variation in variables A-D, the word 'Assessment' would be included as a branch of the hierarchical tree.
- It may be that out of all the project descriptions that contain the word 'assessment', there is variation in variables A-D which is correlated with other words in the project description like 'sorter' or 'renewal', in which case those would become sub-branches.
How might I go about doing this? I am new to machine-learning, so if you have any resources to suggest on this topic or examples in R, that'd be extremely helpful. Thanks.
Edit: Perhaps a reproducible example might be more helpful. Here is a link to a data set containing 480,000 Rotten Tomato Critic Reviews. Is it possible to cluster the words used, only including words that are useful in predicting whether or not the critic rated the movie as rotten/fresh?