1
$\begingroup$

I would like to train a scalable model that has as an input row of a database and has an output of either 1 or 0 depending upon whether it has seen this entry of the database before or not. The purpose of this is to then be able to check against a similar database and determine which entries are the same in the first database, without ever having to share the database. I can't hash the entries and hash-match or any other techniques like that.

That is, let's say two organizations are both using a person database with just floats and with columns = ["name" : float, "age": float, "net_worth": float]. Now, imagine that I am the first group, and I train a model that will overfit on my data, 'memorizing' it in some way. Then, what I want to be able to do is send that model to the other organization, who could apply it to each element of their table. This will then tell the other organization which entries of their dataset are also shared by me.

I understand that this is a unique (if not downright weird) way to try and solve this problem, but this is the approach that I'm trying to take. Does anyone have any solutions they can think of?

$\endgroup$
  • $\begingroup$ Technically by sending the model to the other organization you would be sending them a lot of information about the first database. Hashing the entries seems a much safer way, why you can't use it? $\endgroup$ – Erwan Sep 16 '19 at 13:50
  • $\begingroup$ Legal requirements about data sharing in Australia, it's a workaround $\endgroup$ – Milan Leonard Sep 18 '19 at 2:23
  • $\begingroup$ Are you sure this is a legal workaround then? A model may contain a lot of personal information, usually readable in clear. A "maximally overfit" model would be exactly the original training data itself. $\endgroup$ – Erwan Sep 18 '19 at 8:15
1
$\begingroup$

I'm not sure if a machine learning approach is a good choice for such a task. I am tempted to say no. The reason is mainly, that ML usually is used for the discovery of patterns or in other words of similarities while they are designed to generalize so they still recognize a pattern even if it looks different due to some randomness (noise etc.).

In your case however it sounds more like you want to discover exact matches.

You also have to consider, that you cannot store your data in an arbitrarily compact form without reducing information. So even if you would find a suitable ML model for your purpose, you cannot expect it to consume much less space as the part of the database for which you want to find matches in the other database.

$\endgroup$
  • $\begingroup$ Yeah, I know that it is a very weird solution, and I'm not trying to compress my information really. Just trying to make a workaround for some rules within the intelligence community $\endgroup$ – Milan Leonard Sep 18 '19 at 2:26
0
$\begingroup$

You could train a binary classifier since that's all you care about (seen or not). I would recommend xgboost binary.

$\endgroup$
  • $\begingroup$ The problem with this approach as I understood it is to train a classifier, I have to provide it data is has seen (which is fine) as well as data that it hasn't seen, which is paradoxical. Like, if I can feed it a training example that it hasn't seen to classify that it hasn't seen it, then it's seen it. Like it seems a bit like a paradox $\endgroup$ – Milan Leonard Sep 18 '19 at 2:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.