I'm looking for an approach to classify a similar dataset to the exposed next. Let's say we have an image with some elements inside it (imagine a large building footprint with several structures). What I want to classify are those individuals but taking into account the others. The main issue here is that some individuals are common between them, but having the whole context any can give some insight. Each element is very simple, just some colored lines over a black background, so if the footprint has n elements I extract n 256x256 images to be classified.
The approach tested so far is the individual classification of each image with a classic CNN (namely CNN1). Then, in a next step I take each feature map of the last fully connected layer of CNN1 and compose them in order to create the individual context of the target individual, so the output would be
stack(feature_map_target, max_pool(feature_map_1, feature_map_2,..., feature_map_n-1)), with
feature_map_i the footprint indviduals discarding the targeted one. Lastly, this context is the input of a new CNN2 with a couple of fully connected layers which performs a classification at its end.
For now I'm not succeeding (around 65% of accuracy), because there are some individuals very similar between them and I'm loosing the context trained in CNN2 when classifying. Any light here?