ESSOP: Efficient and scalable stochastic outer product architecture for deep learning

Vinay Joshi, Geethan Karunaratne, Manuel Le Gallo, Irem Boybat, Christophe Piveteau, Abu Sebastian, Bipin Rajendran, Evangelos Eleftheriou

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Deep neural networks (DNNs) have surpassed human-level accuracy in a variety of cognitive tasks but at the cost of significant memory/time requirements in DNN training. This limits their deployment in energy and memory limited applications that require real-time learning. Matrix-vector multiplications (MVM) and vector-vector outer product (VVOP) are the two most expensive operations associated with training of DNNs. Strategies to improve the efficiency of MVM computation in hardware have been demonstrated with minimal impact on training accuracy. However, the VVOP computation remains a relatively less explored bottleneck even with the aforementioned strategies. Stochastic computing (SC) has been proposed to improve the efficiency of VVOP computation but on relatively shallow networks with bounded activation functions and floating-point (FP) scaling of activation gradients. In this paper, we propose ESSOP, an efficient and scalable stochastic outer product architecture based on the SC paradigm. We introduce efficient techniques to generalize SC for weight update computation in DNNs with the unbounded activation functions (e.g., ReLU), required by many state-of-the-art networks. Our architecture reduces the computational cost by re-using random numbers and replacing certain FP multiplication operations by bit shift scaling. We show that the ResNet-32 network with 33 convolution layers and a fully-connected layer can be trained with ESSOP on the CIFAR-10 dataset to achieve baseline comparable accuracy. Hardware design of ESSOP at 14 nm technology node shows that, compared to a highly pipelined FP16 multiplier design, ESSOP is 82.2 % and 93.7 % better in energy and area efficiency respectively for outer product computation.

Original languageEnglish (US)
Title of host publication2020 IEEE International Symposium on Circuits and Systems, ISCAS 2020 - Proceedings
PublisherInstitute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)9781728133201
StatePublished - 2020
Event52nd IEEE International Symposium on Circuits and Systems, ISCAS 2020 - Virtual, Online
Duration: Oct 10 2020Oct 21 2020

Publication series

NameProceedings - IEEE International Symposium on Circuits and Systems
Volume2020-October
ISSN (Print)0271-4310

Conference

Conference52nd IEEE International Symposium on Circuits and Systems, ISCAS 2020
CityVirtual, Online
Period10/10/2010/21/20

All Science Journal Classification (ASJC) codes

  • Electrical and Electronic Engineering

Fingerprint

Dive into the research topics of 'ESSOP: Efficient and scalable stochastic outer product architecture for deep learning'. Together they form a unique fingerprint.

Cite this