Active workflow system for Near Real-Time extreme-scale science

Yanwei Zhang, Qing Liu, Scott Klasky, Matthew Wolf, Karsten Schwan, Greg Eisenhauer, Jong Choi, Norbert Podhorszki

Research output: Contribution to conferencePaperpeer-review

3 Scopus citations


In recent years, streaming-based data processing has been gaining substantial traction for dealing with overwhelming data generated by real-time applications, from both enterprise sources and scientific computing. In this work, however, we look at an emerging class of scientific data with Near Real-Time (NRT) requirement, in which data is typically generated in a bursty fashion with the near real-time constraints being applied primarily between bursts, rather than within a stream. A key challenge for this types of data sources is that the processing time per data element is not uniform, and not always feasible to predict. Given the observations on the increasing unpredictability of compute load and system dynamics, this work looks to adapt streamingbased approach to the context of this new class of large experiments and simulations that have complex run-time control and analysis issues. In particular, we deploy a novel two-tier scheme for handling the increasing unpredictability of runtime behaviors: Instead of relying on determining what and where to run the scientific workflows beforehand or partial dynamically, the decision will also be adaptively enhanced online according to system runtime status. This is enabled by embedding workflow along with data streams. Specifically, we break data outputs generated from experiments or simulations into multiple self-describing "chunks", which we call active data objects. As such, if there is a transient hotspot observed, a data object with unfinished workflow pipeline can break its previous schedule and search for a least loaded location to continue the execution. Our preliminary experiment results based on synthetic workloads demonstrate the proposed active workflow system as a very promising solution by outperforming the state-of-the-art semi-dynamic workflow schedulers with an improved workflow completion time, as well as a good scalability.

Original languageEnglish (US)
Number of pages9
StatePublished - 2014
Externally publishedYes
Event2014 1st Workshop on Parallel Programming for Analytics Applications, PPAA 2014 - Orlando, FL, United States
Duration: Feb 16 2014Feb 16 2014


Other2014 1st Workshop on Parallel Programming for Analytics Applications, PPAA 2014
Country/TerritoryUnited States
CityOrlando, FL

All Science Journal Classification (ASJC) codes

  • Computer Science Applications
  • Software


  • Distributed workflow scheduler
  • Load balancing
  • Near real-time science
  • Scientific workflow system
  • Stream processing
  • System dynamics


Dive into the research topics of 'Active workflow system for Near Real-Time extreme-scale science'. Together they form a unique fingerprint.

Cite this