Generative adversarial networks for video prediction with action control

Zhihang Hu, Jason T.L. Wang

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Scopus citations


The ability of predicting future frames in video sequences, known as video prediction, is an appealing yet challenging task in computer vision. This task requires an in-depth representation of video sequences and a deep understanding of real-word causal rules. Existing approaches for tackling the video prediction problem can be classified into two categories: deterministic and stochastic methods. Deterministic methods lack the ability of generating possible future frames and often yield blurry predictions. On the other hand, although current stochastic approaches can predict possible future frames, their models lack the ability of action control in the sense that they cannot generate the desired future frames conditioned on a specific action. In this paper, we propose new generative adversarial networks (GANs) for stochastic video prediction. Our framework, called VPGAN, employs an adversarial inference model and a cycle-consistency loss function to empower the framework to obtain more accurate predictions. In addition, we incorporate a conformal mapping network structure into VPGAN to enable action control for generating desirable future frames. In this way, VPGAN is able to produce fake videos of an object moving along a specific direction. Experimental results show that a combination of VPGAN and pre-trained image segmentation models outperforms existing stochastic video prediction methods.

Original languageEnglish (US)
Title of host publicationArtificial Intelligence IJCAI 2019 International Workshops - Revised Selected Best Papers
EditorsAmal El Fallah Seghrouchni, David Sarne
Number of pages19
ISBN (Print)9783030561499
StatePublished - 2020
Event28th International Joint Conference on Artificial Intelligence, IJCAI 2019 - Macau, China
Duration: Aug 10 2019Aug 12 2019

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume12158 LNAI
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349


Conference28th International Joint Conference on Artificial Intelligence, IJCAI 2019

All Science Journal Classification (ASJC) codes

  • Theoretical Computer Science
  • General Computer Science


  • Cycle-consistency
  • Deep learning
  • Video prediction


Dive into the research topics of 'Generative adversarial networks for video prediction with action control'. Together they form a unique fingerprint.

Cite this