3
$\begingroup$

As you know, in a federated learning setting, clients train local versions of the joint global model with their Non i.i.d data and each submit an update to the global model which would be aggregated into the next joint global model.

The normalization which happens by Batch Normalization layers during the training phase is based on the local batch statistics. My question is, How should one aggregate these local statistics (batch normalization parameters) for the global model so they represent the global statistics of all the data? I am talking about beta, alpha, moving mean and variance for each batch normalization layer. Should we treat them like weight and biases of fully connected (or Conv) layers and simply average them?

$\endgroup$

1 Answer 1

4
$\begingroup$

One approach would be to simply average everything, as proposed in the FedAvg preprint.

Some very recent preprints suggest only relaying the learned parameters back to the central server and keeping local batch normalization (BN) statistics separate, as proposed by the SiloBN preprint.

The authors of this paper claim that:

Keeping BN statistics local permits the federated training of a model robust to the heterogeneity of the different centers, as the local statistics ensure that the intermediate activations are centered to a similar value across centers.

To paraphrase them; they distinguish BN statistics as encodings for local domain information whereas the learned parameters are to be domain-invariant. I believe their aggregation method for the relayed information is simply averaging.

Here's their first figure for reference: enter image description here

$\endgroup$
1
  • 1
    $\begingroup$ Thanks Benji, I read the SiloBN paper. Helped a lot. $\endgroup$
    – O'ara
    Commented Sep 3, 2020 at 17:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.