I want to perform a regression with a Neural Network using (environmental) spatiotemporal data. They share a target variable and have the same dimensions (latitude, longitude and time) but they are expressed in different styles/data shapes and show different features that I want to use as predictors.

My assumption: it would improve the training if the networks "knows" that the dimensions are shared as the time and location are important for training.

My question is: how can I train a Neural Network with these separate inputs, while making sure the network "knows" that the dimensions belong together? Here are the two data shapes:

  • Dataset 1 has the data on a regular grid/matrix that has the size "latitude x longitude x time x features", which means that the dimension information (latitude, longitude, time) is contained in the position within the grid. They are very similar to images used in e.g. image classification that have different features/channels (where the equivalent to my features are the RGB-channels), only that there's an additional time dimension. I'd like to keep this data structure so I'd be able to use techniques from image classification, like working with 2D/3D convolutional neural networks.
  • Dataset 2 is a very sparse dataset with individual locations over time (point measurements), which is most efficiently displayed as a csv-style table, where the features and also the latitude, longitude and time dimensions are shown as individual columns. I'm reluctant to turn these sparse datapoints into a similar grid as the first dataset, as it would result in 99.9% NaNs and I'm not sure how the network would handle that, even with data imputation.

I'm aware that you can't "label" features for a neural network so that it "knows" where they belong together and that the process of scaling/normalization will further anonymize things. Is there any kind of trick I can employ to better connect the two separate inputs given that they share the same dimensions? And at the same time make clear that the features are not the same and it is just additional information/predictors? Is there a way I could approach this? Or is it simply not possible and I'd have to turn either of the two into the other data type?

So far the closest solution I've come across was to build a neural network with two separate inputs following this example (which is using tensorflow/keras in Python). Unless I misunderstand it, it feeds in a image/table batch per timestep, which seems to be a way to get around my problem if the only dimension were time, but I'm not sure how to account for longitude/latitude. There are also a few more posts like here but they just explain that you can use multiple inputs - they don't talk about how different inputs could work together/influence each other.


1 Answer 1


If I understand properly you have to sets of input features $F_1$ and $F_2$: the first is image-like, and the second is sparse being probably represented as a vector. I assume $F_1$ to have shape $(A,B,C)$ and $F_2$ to be $(D,)$.

If so, you can try various network designs:

  • Stack $F_2$ to $F_1$: use a dense layer to expand $F_2$ to match the shape of $F_1$, then concatenate along last (channel) axis, and go for a CNN. In this way you combine the two feature sets at the the input level.
f2 = dense(units=A * B * D)(f2)
f2 = reshape((A, B, D))(f2)
f = concatenate(axis=-1)([f1, f2])  # f is now (A, B, C + D)
# ...
  • Combine in the middle: Alternatively, you can apply a CNN on $F_1$ and flattening the output feature maps to a vector, which is then combine with the second features as follows:
f1 = CNN(f1)
f1 = flatten(f1)  # reshaped to (K,)
f2 = dense(units=K)(f2)  # expand to (K,)

# combine:
f = f1 + f2
# or
f = f1 * f2
# or even
f = concatenate()([f1, f2])

# final layer

To get a more broad understanding on how some features can influence another read this article.

  • $\begingroup$ Sorry for the late reply and thanks, this is really helpful. What confuses me is how reshaping and concatenating affects the shared information (for the positions). By which I mean: F2 is essentially a list of point measurements of F1, they cover some (but not nearly all) positions within the image-like F1 (i.e. longitude and latitude, or you could alternatively say a pixel at position x and y within the image) and this location is stored as an additional feature in a column/vector of F2, along with the measured variable that the images don't have. $\endgroup$ Jun 20, 2023 at 10:40
  • $\begingroup$ So if I understand correctly reshaping and concatenating works for combining individual sets of different inputs (e.g. images and vectors) that belong together (like the image and label of a puppy or an image and the style to paint in the link you provided), but as I understand it wouldn't work in my case $\endgroup$ Jun 20, 2023 at 10:43
  • $\begingroup$ @Tobitobitobi Actually, operations like concatenation, addition or multiplication allow different sets of features to interact. But before performing such operation, you need to have matching shape and so a linear (or dense) layer transforms the auxiliary feature set to the correct dimensionality and then a reshape to the correct shape. After doing so you can concatenate for example. Once you have the concatenated features you'll add the hidden layers, to make them learn from the "combined" sets of features. This generally works for any pairs of features, you only need to try it. $\endgroup$ Jun 21, 2023 at 17:43
  • $\begingroup$ I see - sorry, I guess it's difficult to get my head around how the features can interact, hence the initial doubt. I will try that, thanks! $\endgroup$ Jun 23, 2023 at 10:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.