How to detect changes between when the changes between 2 frames of videos that are significant enough to be counted for video classification? Thinking of the problem as analogous to edge detection in an image where the change between 2 adjacent pixel is threshold-ed to detect an edge.
Most video formats should already mark such frames marked as "Key Frames" (Frames 1 in this example).
It is possible to get identifiers of such frames (or split video around such frame). For example, mkvmerge tool.