Hardware Acceleration for Postdecision State Reinforcement Learning in IoT Systems

Jianchi Sun, Nikhilesh Sharma, Jacob Chakareski, Nicholas Mastronarde, Yingjie Lao

Research output: Contribution to journalArticlepeer-review

2 Scopus citations

Abstract

Reinforcement learning (RL) is increasingly being used to optimize resource-constrained wireless Internet of Things (IoT) devices. However, existing RL algorithms that are lightweight enough to be implemented on these devices, such as $Q$ -learning, converge too slowly to effectively adapt to the experienced information source and channel dynamics, while deep RL algorithms are too complex to be implemented on these devices. By integrating basic models of the IoT system into the learning process, the so-called postdecision state (PDS)-based RL can achieve faster convergence speeds than these alternative approaches at lower complexity than deep RL; however, its complexity may still hinder the real-time and energy-efficient operations on IoT devices. In this article, we develop efficient hardware accelerators for PDS-based RL. We first develop an arithmetic hardware acceleration architecture and then propose a stochastic computing (SC)-based reconfigurable hardware architecture. By using simple bitwise computations enabled by SC, we eliminate costly multiplications involved in PDS learning, which simultaneously reduces the hardware area and power consumption. We show that the computational efficiency can be further improved by using extremely short stochastic representations without sacrificing learning performance. We demonstrate our proposed approach on a simulated wireless IoT sensor that must transmit delay-sensitive data over a fading channel while minimizing its energy consumption. Our experimental results show that our arithmetic accelerator is $5.3\times $ faster than $Q$ -learning and $2.6\times $ faster than a baseline hardware architecture, while the proposed SC-based architecture further reduces the critical path of the arithmetic accelerator by 87.9%.

Original languageEnglish (US)
Pages (from-to)9889-9903
Number of pages15
JournalIEEE Internet of Things Journal
Volume9
Issue number12
DOIs
StatePublished - Jun 15 2022

All Science Journal Classification (ASJC) codes

  • Signal Processing
  • Information Systems
  • Hardware and Architecture
  • Computer Science Applications
  • Computer Networks and Communications

Keywords

  • Action evaluation
  • hardware acceleration
  • Internet of Things (IoT) systems
  • latency sensitive resource-constrained online operation
  • postdecision state learning
  • reinforcement learning
  • stochastic computing (SC)
  • wireless communication

Fingerprint

Dive into the research topics of 'Hardware Acceleration for Postdecision State Reinforcement Learning in IoT Systems'. Together they form a unique fingerprint.

Cite this