In 2016 some people at MIT's CSAIL group "made an important new breakthrough in predictive vision, developing an algorithm that can anticipate interactions more accurately than ever before." They wrote an article, Teaching machines to predict the future.
They made a great video which shows their results. They trained an algorithm on YouTube videos and TV shows to predict when two individuals will shake hands, hug, kiss, or slap five.
The researchers, Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba, published a paper at CVPR 2016 entitled Anticipating Visual Representations with Unlabeled Video.
Chia-Wen Cheng has a repository of an LSTM-based model implemented in TensorFlow.
For a more up to date list of papers on predicting actions or activities see this list.