If your objects move slow enough you can just find the nearest object on the other frame. In your case the objects move too fast. If you significantly increase the frame rate you will be able to see a one pixel movement. Then you just find the closest object on the previous frame and this identify the same object. If you want to have a movement on the scale you showed you should track the object across multiple frames until you reach the need distance. And you will not only have a movement vector, but also a full path from one point to the other.
If you cannot increase the frame rate the task becomes complicated even for humans. I had trouble identifying the same objects. It means that the task will be really hard for an algorithm. You can still try the nearest neighbor, but it will have much worse results than with a high frame rate. Any additional information to differentiate the objects will significantly improve matching, like color and shape.
After nearest neighbor you can also run an algorithm to minimize a loss function. Where your loss is a distance between the matches and a penalty for non matched objects and a huge penalty for same matching. You minimize by something like evolutionary algorithm, or empirical hill descent. The variables are the matched object number from the other frame for each object on this frame.