1
$\begingroup$

I'm using PytorchGeometric to train a graph convolutional network for regression over nodes problem (the graph models physical phenomena in the network of sensors; the network of sensors is actually the network of measurements distributed across the power grid (powers, currents, voltages), and the goal of the GNN is to predict some unmeasured variables in the graph.). In the training dataset there graphs with different topologies (i.e. different edge_index tensors), and each of which has input and label tensors, which consist of float values for each node in the graph.

The training curves look good, the loss curve is converging to a small value and there are no exploding nor vanishing gradients.

There are 1000 different graph topologies in the training set and around 2000 training samples. So, when the trained model is tested on graphs whose topology occurs 2 or 3 times in the training set, the results are great, almost the same as the test sample labels for each node (the input values of nodes are different, only the topology is already seen). When the trained model is tested on graphs whose topology occurs one in the training set, the results are slightly worse.

But when the model is tested on the unseen (but similar) graph topology, the results are completely wrong. All of the graphs in the training and test set are generated synthetically, using the same random process, so the topologies come from the same graph distribution.

Since the graph models physical phenomena in the network of sensors, I would expect that the GNN should be able to learn how sensor information impacts the neighboring variables, even for the unseen graphs.

I've tried going deeper into the graph and adding the convolutional layers. I used the convolutional layers: https://github.com/rusty1s/pytorch_geometric/blob/master/torch_geometric/nn/conv/gcn_conv.py#L188

Did someone have a similar problem? Are there some GNN models that are better at generalizing on unseen graph topologies?

Cheers!

New contributor
sesli is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
$\endgroup$
0
0
$\begingroup$

your problem is very interesting.

Is it possible to know which are your hyper parameters?

It seems that there is an overfitting or something similar.

Nevertheless, here are some ideas that could help:

1- The size of the GNN might be too big and it cannot learn new configurations.

2- Maybe use a drop out to erase some neural weights and allow more generalisation.

3- I don't know if GNN can use batches to learn on subgraphs (with a specific size limit) instead of complete graphs. I'm currently working with similar issues but in a different field.

Please let me know if you have more information to suggest more ideas.

New contributor
nico59128 is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
$\endgroup$
2
  • $\begingroup$ Hi, thanks for the suggestions! I'm not sure if there is any overfitting, since the gradient norms have expected values, but anyhow, here are the hyperparameters (obtained by hyperparm. optimization): - learning rate: 0.0004 (but also experimented with weight decay) - batch size: 4 (so the tensor corresponding to the 4 graphs are concatenated) - 4 graph convolutional layers, each having the output size of 32 and with ReLU activations in between - Adam optimizer - I've actually tried the dropout with the dropout probability of 0.2, but the overall performance deteriorated. $\endgroup$ – sesli yesterday
  • $\begingroup$ As for the learning on subgraphs, I think that in general, it might be possible and easy to implement. But in this power systems case, every sensor in the network can have an influence on the node values that are being learned (using the reading from all the sensors in the graphs some values related to all nodes in the graph are estimated). $\endgroup$ – sesli yesterday
0
$\begingroup$

My unconventional answer to your question is that you have stumbled upon something that is usual in machine learning even if it is not usually recognised as such.

The fact is that models learn from the data they see and model those data and capture their structure as best as they can (sometimes they capture more structure than desirable or even existent, in other words they capture noise).

They cannot learn from unseen data. If it happens that unseen data are simply variations of seen data, then all is (usually) well. Else performance on unseen data on average is not better than random. This is especially true as the models (and the data) get more complex (as are graph networks).

A possible remedy is to avoid overfitting your networks so they may generalise better, but this means the training (and validation) performance will necessarily drop. Another remedy is to feed those unseen synthetically generated graphs into your network during training, although I don't think this will eventually address the issue. A third possible remedy is to alter the architecture of the network (eg add more layers, ..).

$\endgroup$
2
  • $\begingroup$ Thanks for the comment! Well my biggest concern is that unseen data are variations of seen data. All those graphs have the same number of nodes and similar number of edges, the number of combinations of graph topologies is large, so so many of then can't be covered with the training set. $\endgroup$ – sesli yesterday
  • $\begingroup$ Maybe they are not so similar as you might think. For example topology might change and this can impact the results. Another approach is to alter the architecture of the networkm eg add more layers etc.. $\endgroup$ – Nikos M. 20 hours ago

Your Answer

sesli is a new contributor. Be nice, and check out our Code of Conduct.

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.