Most clustering examples on the net are unsupervised learning. There is a given vectorization into a 2D space and the algorithm discovers clusters.
However, what if the input data that I want it to train for is desired clustering? So that the training data would be similar to classification, but in an open ended result set.
An example would be this list of entries
- Susan walks
- Susan is sitting.
- Hear Susan sing.
- David walks
- Robert sits
The verbs (intents) are finite, Walk, Sing, Sit. But in addition to classification of intent, I also need to cluster the intents to identify the entities. This is an open ended set.
If I have 100 000 records of data on this form with identified verb and entity.
Susan walks | Walk | Susan
Susan is sitting | Sit | Susan
Hear Susan sing | Sing | Susan
David walks | Walk | David
Robert sits | Sit | Robert
The language has been simplified in this example. The point is that there may be variations on phrasing, but it is pretty simple english. The input data will have a lot of different names.
I would like an engine that would be good at classifying intent (easily solved), but it should also group together the intents that are likely to refer to the same person. So that the runtime can encounter "Alexander" and will be able to cluster his datapoints even though the name wasn't in the training data.
Is this poissible? What sort of algorithm is this?
And what would the training data look like?
Susan walks | Walk | PersonA
Susan is sitting | Sit | PersonA
Hear Susan sing | Sing | PersonA
David walks | Walk | PersonB
Robert sits | Sit | PersonC
I want the algorithm to identify the correct clusters. I don't want the names "Susan" or "David" in the last column. Nor do I want "PersonA" there, because it would be just as correct to call Susan "PersonB". But I want some way of indicating desirable clustering without naming them.
Most clustering examples I have looked at have already solved the problem of vectorization. But I want the computer to decide the correct vectorization to use that will produce the desirable clustering.
I took a look at LUIS that does a separation between intent and entities. But it seemed that the "entity" was just another classification within a predefined set. Not that you could use the data to discover entity instances.