I've two different datasets A(x1, x2, x3...xn) and B(y1, y2, y3...yn). Each instance in A is linked with an instance in B i.e. only one unique pair exists.

x1 -> y1
x1 -> y2
.     .
x2 -> y3
x3 -> y3
x4 -> y4
x5 -> y5
predict(xnew  -> ynew)?


Also, instances can be duplicate in A as well as B. If I get a new instance in A and a new instance in B, I want to predict whether they both will be linked or not. Please suggest a way to do this. Thanks in advance.

• If I understood correctly there cannot be any solution to your problem, since, each pair is unique without any additional attribute information. There are link prediction algorithms but they reside on the fact of common links. In your case, every 2 nodes are linked (e.x. a1 -> b2) without any other possible linkage, leaving no room for link inference between the new nodes and the already nodes since there are no connections. Each A node is on it's own and B as well. If that's not the case please provide more information to your question. Commented Oct 10, 2019 at 17:00
• @Grzegorz pls look at the edit Commented May 15, 2020 at 16:33

Make X as feature
Y as Label
Build a model on it with 80% example.
Predict Y for 20% and check if it is very close to Y_actual.
If you are getting a good score. Then you are good.

Predict Y for new X
If Y_pred is close to Y_actual. You may assume both are linked.

• tried this already, does not seem to be good approach for real valued numeric data. Commented May 17, 2020 at 13:52

From the top of my head I think you could build something similar to an autoencoder.

$$X$$ as input, $$Y$$ as label: $$Y' = f_\theta(X) \approx Y$$ as label, $$f_\theta$$ is a Neural Network, $$\theta$$ the weights and your loss $$\mathcal{L} = d(Y,Y')$$, where $$d$$ is some convex distance measure like Mean-Squared-Error or Binary-Crossentropy (if you scale your output and labels between $$(0,1)$$).

If you have new data $$\{\hat{X},\hat{Y} \}$$, feed $$\hat{X}$$ to your network $$f_\theta$$ and look for the sample in $$\hat{Y}$$ which has the minimal distance $$d(\hat{Y},\hat{Y'})$$

• How is this different from training conventional ML models like SVM, Decision trees? Because this approach is failing for me. Commented May 17, 2020 at 19:03

This seems to me now to be a typical case of a bipartite graph. In the real world, this can be viewed with customers and products; where each customer is linked to a product that they bought, but node customers or product nodes cannot be linked to each other.

I do not know the SOTA for that, but a useful tool to check with Python is NetworkX. Perhaps, you could create a bipartite graph from your data and use one of their link prediction algorithms.

• seems like a promising solution! Will try it definitely. Thanks! Commented May 17, 2020 at 19:01