Spark-based large-scale matrix inversion for big data processing

Yang Liang, Jun Liu, Cheng Fang, Nirwan Ansari

Research output: Chapter in Book/Report/Conference proceedingConference contribution

3 Scopus citations

Abstract

Matrix inversion is a fundamental operation to solve linear equations for many computational applications. However, it is a challenging task to invert large-scale matrices of extremely high order (several thousands), which are common in most of web-scale systems like social networks and recommendation systems. In this paper, we present a LU decomposition based block-recursive algorithm for large-scale matrix inversion, and its well-designed implementation with optimized data structure, reduction of space complexity and effective matrix multiplication on the Spark parallel computing platform. The experimental evaluation results show that the proposed algorithm is efficient to invert large-scale matrices on a cluster composed of commodity servers and scalable to invert even larger matrices. The proposed algorithm and implementation will be a solid base to build a high-performance linear algebra library on Spark for big data processing.

Original languageEnglish (US)
Title of host publication2016 IEEE Conference on Computer Communications Workshops, INFOCOM WKSHPS 2016
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages718-723
Number of pages6
ISBN (Electronic)9781467399555
DOIs
StatePublished - Sep 6 2016
Event35th IEEE Conference on Computer Communications Workshops, INFOCOM WKSHPS 2016 - San Francisco, United States
Duration: Apr 10 2016Apr 14 2016

Publication series

NameProceedings - IEEE INFOCOM
Volume2016-September
ISSN (Print)0743-166X

Other

Other35th IEEE Conference on Computer Communications Workshops, INFOCOM WKSHPS 2016
CountryUnited States
CitySan Francisco
Period4/10/164/14/16

All Science Journal Classification (ASJC) codes

  • Computer Science(all)
  • Electrical and Electronic Engineering

Keywords

  • LU decomposition
  • Spark
  • distributed computing
  • linear algebra
  • matrix inversion
  • parallel algorithm

Fingerprint Dive into the research topics of 'Spark-based large-scale matrix inversion for big data processing'. Together they form a unique fingerprint.

Cite this