I'm using an RNN consisting of GRU cells to compare two bounding box trajectories and determine whether they belong to the same agent or not. In other words, I am only interested in a single final probability score at the final time step.
What I'm unsure about is how to formulate the loss function in this case. I see two options:
1) force the network to output the correct label at every time step i.e. if I am providing a positive training sample whose output should be 1, then my loss function would be a vector of ones subtracted by the network's output at each time step
2) only check the output at the final time step and use only that in my loss function.
Intuitively, the second option makes more sense, but I'm sure there are other factors that come into play too.