Using human created small groups to identify entirely new small groups

Question

I'm relatively new to data science, but an old hat at analytics. I'm looking for some direction on a project that I'm wanting to work on.

I'm working with discrete objects, that when small changes were made to them would get brand-new IDs. Various aspects about them would remain the same, and a field such as a name field would very often be very closely related, but not always. So, if we use people as an example, when I change Bob Smith's age from 30 to 31, an entirely new record is created for bob, with an entirely new ID, but all of the other data is closely related, or exactly the same. At the time of its creation, it wasn't important to keep track of if these were all the same Bob Smith or not, so the records are otherwise not related. Assume each record has perhaps 5-7 attributes to work with.

The most frequent case is that the object gets created once, and never has any modifications that cause a new record to be created. On the other extreme a single object may get say, 7 or 8 modifications that create new records, but not hundreds, possibly not even tens.

It has since become important to establish which of these records belong together, and for everything new created after a certain date, these associations were maintained... manually. By humans.

I've considered using something similar to a string distance function, but instead of characters making up a string, using attribute values making up a record to identify the 'distance' between records.

I'm curious though if the more recent records, where the association between these objects is tracked could be used as some kind of training set to train a model to create groups out of past data.

Obviously, my fictitious system doesn't make any sense, it's analogous to the real problem, and I'm trying to simplify it for consumption here.

I'm not even sure how to ask the question to begin searching for an answer or a method.

Input appreciated!

Answer 1

Welcome to the Data Science World.

Note: I am not 100% sure about your project and details are not known. So I will only point you in some directions.

As I understand your question, your problem is to group "nearby" objects without a formal definition of "nearby". But you have some cases where humans annotated which objects belong together. This basically leads to two tasks:

1. Define a meaningful distance metric
2. Group nearby objects using above metric

Finding a metric

Here, I would like to point you to metric learning, which does exactly this. If you are looking for a quick start, scikit-learn provides some methods and a tutorial.

Depending on your data and the type of anntotations, you might need to perform some preprocessing, e.g. some embeddings and maybe look for some different methods than the ones in scikit-learn, but I hope I could point you into a direction.

Grouping objects

The first thing that comes to ones mind whould be to use some clustering, but clustering algorithms are mainly designed to give a small number of clusters. But you are looking for a huge number of clusters, so you might have to search a bit. Hierachical clustering might be a way to start with.

Or you look for related segmentation algorithms (mostly known from the image domain). You could build a graph of nearby objects and try some graph cut algorithms.

I hope this could point your search into some promissing directions.