How to train a layer multiple times in a pass for variable amount of input features?

Question

I'm training an LSTM on multi-feature time-series location data. How can I design a network structure that can take one primary location's features, plus an arbitrary number $n$ of extra "neighbor" locations' features alongside the primary location's features? (All locations have the same data features for the same dates)

I have an idea but I do not know how exactly to implement it properly, nor if it is even the correct approach.

For example, assume the dataset has columns like: date, A, B, C, D, location_name, latitude, longitude. The numerical features would be A, B, C, D. In my data they have all been normalized by series in the whole dataset (i.e. all A for all locations have a common mean and std. dev., but B's mean and std. dev. is different than A's).

• The primary LSTM would take $3$ features in this example: the primary location's A, B, C features.
• The secondary LSTM would take $4$ features: another location's A, B, C, distance features (at the same dates as the primary sequence), where distance is generated and represents how far this location is from the primary location. This intends to let the LSTM account for distance in the relationship while training.
• The goal is that the network predicts the latest D value for the primary location.

On the forward pass, this hypothetical network would:

1. feed the $3$ features of the primary location through the primary LSTM, and produce an output;
2. feed the $4$ features each of $n$ neighbor locations through the secondary LSTM, one pass for each location, producing $n$ total outputs from the secondary LSTM;
3. combine all $n+1$ intermediate LSTM outputs through a third layer (ex. Dense?) that will produce $1$ output: the latest D value for the primary location.

and this is where my idea is stuck. First, I do not know what this third layer can be. I would be looking for a layer that can transform an arbitrary number of inputs into one output, and also be able to backpropagate. I know I could perform a weighted average of all the values along the feature axis, but I can't backpropagate an average (to my knowledge). Second, I do not know if the secondary LSTM needs to take a different loss than the primary LSTM during backpropagation.

The appeal to me of having two LSTMs and one other layer is that backpropagation should be a nice constant time with respect to $n$ neighbors ($3+4n$ total features).

What should the third layer be? Should I only have one LSTM, combine the outputs with an average weighted by location distance of each output?

Or am I approaching this problem wrong? I have never worked with a network that can handle a variable number of features. I am partially aware of CNNs but I only know of them as handling non-time-dependent data.