I have 2 ddbb with around 60,000 samples each. Both have the same features (same column names) that represent particular things with text or categories (turned into numbers). Each sample in a ddbb is assumed to refer to a different particular thing. But there are some objects that are represented in both ddbb, yet with somewhat different values in the same-name column (like different open descriptions, or classified as another category).
The aim is to train a machine learning model that recognizes when the two “descriptions” refer to the same thing.
We have manually recognized a few thousand "duplicated" cases that we have labelled correspondingly. This label is the target variable to learn. Yet, the supervised classification examples I have seen, work with a single data frame and/or predict something about a single row of features.
How does it work when we are not trying to predict what a sample represents, but the relation between 2 samples (specifically now if they refer to the same object or not)?
I do not even know how to feed two dataframes into scikit-learn or even auto scikit-learn (or similars) so that it can handle the task of recognizing if a sample from one represents the same thing than another sample in the other one...Maybe I should concatenate them so that there is just one df, but then it would have to compare all with all the rest...Or does it not make a big difference?
Any idea or hint about how to proceed here or how to frame the problem better?