Dual-view Snapshot Compressive Imaging via Optical Flow Aided Recurrent Neural Network

Ruiying Lu, Bo Chen, Guanliang Liu, Ziheng Cheng, Mu Qiao, Xin Yuan

Research output: Contribution to journalArticlepeer-review

Abstract

Dual-view snapshot compressive imaging (SCI) aims to capture videos from two field-of-views (FoVs) using a 2D sensor (detector) in a single snapshot, achieving joint FoV and temporal compressive sensing, and thus enjoying the advantages of low-bandwidth, low-power and low-cost. However, it is challenging for existing model-based decoding algorithms to reconstruct each individual scene, which usually require exhaustive parameter tuning with extremely long running time for large scale data. In this paper, we propose an optical flow-aided recurrent neural network for dual video SCI systems, which provides high-quality decoding in seconds. Firstly, we develop a diversity amplification method to enlarge the differences between scenes of two FoVs, and design a deep convolutional neural network with dual branches to separate different scenes from the single measurement. Secondly, we integrate the bidirectional optical flow extracted from adjacent frames with the recurrent neural network to jointly reconstruct each video in a sequential manner. Extensive results on both simulation and real data demonstrate the superior performance of our proposed model in short inference time. The code and data are available at https://github.com/RuiyingLu/OFaNet-for-Dual-view-SCI.

Original languageEnglish (US)
Pages (from-to)3279-3298
Number of pages20
JournalInternational Journal of Computer Vision
Volume129
Issue number12
DOIs
StatePublished - Dec 2021
Externally publishedYes

All Science Journal Classification (ASJC) codes

  • Software
  • Computer Vision and Pattern Recognition
  • Artificial Intelligence

Keywords

  • Compressive imaging
  • Compressive sensing
  • Convolutional neural network
  • Deep learning
  • Field of view
  • Optical flow
  • Recurrent neural network
  • Snapshot

Fingerprint

Dive into the research topics of 'Dual-view Snapshot Compressive Imaging via Optical Flow Aided Recurrent Neural Network'. Together they form a unique fingerprint.

Cite this