Is there algorithmic difference between analyzing video and an image, say for example,if I want object recognition? Or do I just have to analyze every frame of the the video just as an image?

Example, detecting an object in a single image is easy compared to video becuase time dimension is added to video. In addition, on video, in every frame, the object is most probably moving, which makes the frames in motion... So how can u handle the time factor and the "in motion" part in the video. Those are the problem I imagine in video, it would be nice if u add ur own thoughts on it. Thanks

  • $\begingroup$ The most likely answer is yes, though if can you clarify what you mean as "just as an image" and "object recognition" ( segmentation + classification ? ) the differences will become clearer. $\endgroup$ Aug 12, 2015 at 15:15
  • $\begingroup$ Like counting people in a cam video and counting people in image. Is there anything thing I should consider for video? $\endgroup$ Aug 12, 2015 at 15:25
  • $\begingroup$ For video you have a time component, so methods that involve tracking as a component would become relevant. You might develop a model of human movement which would help increase accuracy with video sequences. Depending on frame rate and object velocity, deformation or blurring of the object in the video frame may be an issue or not. $\endgroup$ Aug 12, 2015 at 16:07

1 Answer 1


This is a huge topic, so I will just give you a high-level overview and some pointers to more info.

Yes, there are definitely ways to work with video that are different from working with individual still images.

At the simplest level, it could be running an object detector such as HoG (or a sliding-window convnet) on each frame, and then some means of assigning nearby detections in adjacent frames to be the same object, and discarding detections which do not seem to have continuity in time. Many of the algorithms in this field seem to look at a single frame as a building block to looking at the whole sequence, where perhaps the data from adjacent frames is combined, aggregated, and/or used to disambiguate the current frame.

Another approach is to first estimate object motion between frames (using optical flow, phase correlation, pyramid block matching or another method) and then treat areas of multiple frames that are collocated after accounting for motion as being the same object. This is very powerful but limited by the accuracy of the motion estimation.

In newer research, there is a back-and-forth between finding where things are (detection) and how things move (tracking), where each task can help the other one, for example (Kalal 2010) or (Andriluka 2008), to the point that the two parts of the algorithm are no longer separable. Kalal's TLD algorithm is a recently famous version of this.

There are also algorithms which work directly in the spatiotemporal (or sometimes just temporal) domain. An example of purely temporal would be detecting vehicles by the periodicity of variation of their wheel spokes.

Some of the frequently studied model problems are:

  • People, vehicles or other objects detection and tracking throughout a video sequence. For example, using the CalTech Pedestrians dataset or other standard benchmarks.

  • Hand gesture recognition, either for games, user interface, or sometimes sign language recognition. Often using depth data, for example Kinect video.

  • Activity recognition, for example walking vs standing. Due to the nature of this problem, it is much more common to see pure spatiotemporal algorithms here, for example (Sadanand and Corso 2012).

  • Simultaneous location and mapping (SLAM). Often found in robotics, this problem is basically to build a 3D model of the environment from video of a single camera (or stereo or depth video) which is moving around; the environment is often assumed to be static. This is often done by running some kind of feature detector such as SIFT or SURF, matching up features in successive frames, and then building point clouds from the relative 3D motion implied by features.

Here are some more references:

Shah, Mubarak, and Ramesh Jain, eds. Motion-based recognition. Vol. 9. Springer Science & Business Media, 2013.

Turk, Matthew. "Gesture recognition." Computer Vision: A Reference Guide (2014): 346-349.

Rosenfeld, Azriel, David Doermann, and Daniel DeMenthon, eds. Video mining. Vol. 6. Springer Science & Business Media, 2013.

Kalal, Zdenek, Krystian Mikolajczyk, and Jiri Matas. "Tracking-learning-detection." Pattern Analysis and Machine Intelligence, IEEE Transactions on 34.7 (2012): 1409-1422.

M. Andriluka, S. Roth, B. Schiele. People-Tracking-by-Detection and People-Detection-by-Tracking. Computer Vision and Pattern Recognition (CVPR) 2008

Sadanand, Sreemanananth, and Jason J. Corso. "Action bank: A high-level representation of activity in video." Computer Vision and Pattern Recognition (CVPR), 2012

  • $\begingroup$ Thanks, just what i wanted, nice intuition....i'll see all the references. $\endgroup$ Aug 17, 2015 at 11:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.