Large Scale String Analytics in Arkouda

Zhihui Du, Oliver Alvarado Rodriguez, David A. Bader

Research output: Chapter in Book/Report/Conference proceedingConference contribution


Large scale data sets from the web, social networks, and bioinformatics are widely available and can often be rep-resented by strings and suffix arrays are highly efficient data structures enabling string analysis. But, our personal devices and corresponding exploratory data analysis (EDA) tools cannot handle big data sets beyond the local memory. Arkouda is a framework under early development that brings together the productivity of Python at the user side with the high-performance of Chapel at the server-side. In this paper, an efficient suffix array data structure design and integration method are given first. A suffix array algorithm library integration method instead of one single suffix algorithm is presented to enable runtime performance optimization in Arkouda since different suffix array algorithms may have very different practical performances for strings in various applications. A parallel suffix array construction algorithm framework is given to further exploit hierarchical parallelism on multiple locales in Chapel. A corresponding benchmark is developed to evaluate the feasibility of the provided suffix array integration method and measure the end-To-end performance. Experimental results show that the proposed solution can provide data scientists an easy and efficient method to build suffix arrays with high performance in Python. All our codes are open source and available from GitHub (

Original languageEnglish (US)
Title of host publication2021 IEEE High Performance Extreme Computing Conference, HPEC 2021
PublisherInstitute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)9781665423694
StatePublished - 2021
Event2021 IEEE High Performance Extreme Computing Conference, HPEC 2021 - Virtual, Online, United States
Duration: Sep 20 2021Sep 24 2021

Publication series

Name2021 IEEE High Performance Extreme Computing Conference, HPEC 2021


Conference2021 IEEE High Performance Extreme Computing Conference, HPEC 2021
Country/TerritoryUnited States
CityVirtual, Online

All Science Journal Classification (ASJC) codes

  • Modeling and Simulation
  • Artificial Intelligence
  • Computer Networks and Communications
  • Computer Science Applications
  • Hardware and Architecture
  • Computational Mathematics


  • Arkouda
  • exploratory data analysis
  • large scale string sets
  • suffix array construction algorithm


Dive into the research topics of 'Large Scale String Analytics in Arkouda'. Together they form a unique fingerprint.

Cite this