Improving throughput and reliability of distributed scientific workflows for streaming data processing

Yi Gu; Qishi Wu; Xin Liu; Dantong Yu

doi:10.1109/HPCC.2011.52

Improving throughput and reliability of distributed scientific workflows for streaming data processing

Yi Gu, Qishi Wu, Xin Liu, Dantong Yu

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

6 Scopus citations

Abstract

With the advent of next-generation scientific applications, the workflow-based computing technology has become an indispensable research method for managing and streamlining large-scale distributed data processing. This paper investigates a problem of mapping distributed workflows for streaming data processing in faulty networks where nodes and links are subject to probabilistic failures. We formulate this problem as a bi-objective optimization problem in terms of both throughput and reliability, and propose a decentralized layer-oriented method to achieve high throughput for smooth data flow while satisfying a prespecified overall failure rate bound for a guaranteed level of reliability. The superiority of the proposed mapping solution is illustrated by both extensive simulation-based performance comparisons with existing algorithms and experimental results from a real-life scientific workflow deployed in wide-area networks.

Original language	English (US)
Title of host publication	Proc.- 2011 IEEE International Conference on HPCC 2011 - 2011 IEEE International Workshop on FTDCS 2011 - Workshops of the 2011 Int. Conf. on UIC 2011- Workshops of the 2011 Int. Conf. ATC 2011
Pages	347-354
Number of pages	8
DOIs	https://doi.org/10.1109/HPCC.2011.52
State	Published - 2011
Externally published	Yes
Event	13th IEEE International Workshop on FTDCS 2011, the 8th International Conference on ATC 2011, the 8th International Conference on UIC 2011 and the 13th IEEE International Conference on HPCC 2011 - Banff, AB, Canada Duration: Sep 2 2011 → Sep 4 2011

Publication series

Name	Proc.- 2011 IEEE International Conference on HPCC 2011 - 2011 IEEE International Workshop on FTDCS 2011 -Workshops of the 2011 Int. Conf. on UIC 2011- Workshops of the 2011 Int. Conf. ATC 2011

Other

Other	13th IEEE International Workshop on FTDCS 2011, the 8th International Conference on ATC 2011, the 8th International Conference on UIC 2011 and the 13th IEEE International Conference on HPCC 2011
Country/Territory	Canada
City	Banff, AB
Period	9/2/11 → 9/4/11

All Science Journal Classification (ASJC) codes

Artificial Intelligence
Computer Networks and Communications

Keywords

Reliability
distributed computing
fault tolerance
frame rate
workflow mapping

Access to Document

10.1109/HPCC.2011.52

Cite this

Gu, Y., Wu, Q., Liu, X., & Yu, D. (2011). Improving throughput and reliability of distributed scientific workflows for streaming data processing. In Proc.- 2011 IEEE International Conference on HPCC 2011 - 2011 IEEE International Workshop on FTDCS 2011 - Workshops of the 2011 Int. Conf. on UIC 2011- Workshops of the 2011 Int. Conf. ATC 2011 (pp. 347-354). Article 6063011 (Proc.- 2011 IEEE International Conference on HPCC 2011 - 2011 IEEE International Workshop on FTDCS 2011 -Workshops of the 2011 Int. Conf. on UIC 2011- Workshops of the 2011 Int. Conf. ATC 2011). https://doi.org/10.1109/HPCC.2011.52

Gu, Yi ; Wu, Qishi ; Liu, Xin et al. / Improving throughput and reliability of distributed scientific workflows for streaming data processing. Proc.- 2011 IEEE International Conference on HPCC 2011 - 2011 IEEE International Workshop on FTDCS 2011 - Workshops of the 2011 Int. Conf. on UIC 2011- Workshops of the 2011 Int. Conf. ATC 2011. 2011. pp. 347-354 (Proc.- 2011 IEEE International Conference on HPCC 2011 - 2011 IEEE International Workshop on FTDCS 2011 -Workshops of the 2011 Int. Conf. on UIC 2011- Workshops of the 2011 Int. Conf. ATC 2011).

@inproceedings{6b35b3d04e6c4c06ac106a7b3977bf62,

title = "Improving throughput and reliability of distributed scientific workflows for streaming data processing",

abstract = "With the advent of next-generation scientific applications, the workflow-based computing technology has become an indispensable research method for managing and streamlining large-scale distributed data processing. This paper investigates a problem of mapping distributed workflows for streaming data processing in faulty networks where nodes and links are subject to probabilistic failures. We formulate this problem as a bi-objective optimization problem in terms of both throughput and reliability, and propose a decentralized layer-oriented method to achieve high throughput for smooth data flow while satisfying a prespecified overall failure rate bound for a guaranteed level of reliability. The superiority of the proposed mapping solution is illustrated by both extensive simulation-based performance comparisons with existing algorithms and experimental results from a real-life scientific workflow deployed in wide-area networks.",

keywords = "Reliability, distributed computing, fault tolerance, frame rate, workflow mapping",

author = "Yi Gu and Qishi Wu and Xin Liu and Dantong Yu",

year = "2011",

doi = "10.1109/HPCC.2011.52",

language = "English (US)",

isbn = "9780769545387",

series = "Proc.- 2011 IEEE International Conference on HPCC 2011 - 2011 IEEE International Workshop on FTDCS 2011 -Workshops of the 2011 Int. Conf. on UIC 2011- Workshops of the 2011 Int. Conf. ATC 2011",

pages = "347--354",

booktitle = "Proc.- 2011 IEEE International Conference on HPCC 2011 - 2011 IEEE International Workshop on FTDCS 2011 - Workshops of the 2011 Int. Conf. on UIC 2011- Workshops of the 2011 Int. Conf. ATC 2011",

note = "13th IEEE International Workshop on FTDCS 2011, the 8th International Conference on ATC 2011, the 8th International Conference on UIC 2011 and the 13th IEEE International Conference on HPCC 2011 ; Conference date: 02-09-2011 Through 04-09-2011",

}

Gu, Y, Wu, Q, Liu, X & Yu, D 2011, Improving throughput and reliability of distributed scientific workflows for streaming data processing. in Proc.- 2011 IEEE International Conference on HPCC 2011 - 2011 IEEE International Workshop on FTDCS 2011 - Workshops of the 2011 Int. Conf. on UIC 2011- Workshops of the 2011 Int. Conf. ATC 2011., 6063011, Proc.- 2011 IEEE International Conference on HPCC 2011 - 2011 IEEE International Workshop on FTDCS 2011 -Workshops of the 2011 Int. Conf. on UIC 2011- Workshops of the 2011 Int. Conf. ATC 2011, pp. 347-354, 13th IEEE International Workshop on FTDCS 2011, the 8th International Conference on ATC 2011, the 8th International Conference on UIC 2011 and the 13th IEEE International Conference on HPCC 2011, Banff, AB, Canada, 9/2/11. https://doi.org/10.1109/HPCC.2011.52

Improving throughput and reliability of distributed scientific workflows for streaming data processing. / Gu, Yi; Wu, Qishi; Liu, Xin et al.
Proc.- 2011 IEEE International Conference on HPCC 2011 - 2011 IEEE International Workshop on FTDCS 2011 - Workshops of the 2011 Int. Conf. on UIC 2011- Workshops of the 2011 Int. Conf. ATC 2011. 2011. p. 347-354 6063011 (Proc.- 2011 IEEE International Conference on HPCC 2011 - 2011 IEEE International Workshop on FTDCS 2011 -Workshops of the 2011 Int. Conf. on UIC 2011- Workshops of the 2011 Int. Conf. ATC 2011).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Improving throughput and reliability of distributed scientific workflows for streaming data processing

AU - Gu, Yi

AU - Wu, Qishi

AU - Liu, Xin

AU - Yu, Dantong

PY - 2011

Y1 - 2011

N2 - With the advent of next-generation scientific applications, the workflow-based computing technology has become an indispensable research method for managing and streamlining large-scale distributed data processing. This paper investigates a problem of mapping distributed workflows for streaming data processing in faulty networks where nodes and links are subject to probabilistic failures. We formulate this problem as a bi-objective optimization problem in terms of both throughput and reliability, and propose a decentralized layer-oriented method to achieve high throughput for smooth data flow while satisfying a prespecified overall failure rate bound for a guaranteed level of reliability. The superiority of the proposed mapping solution is illustrated by both extensive simulation-based performance comparisons with existing algorithms and experimental results from a real-life scientific workflow deployed in wide-area networks.

AB - With the advent of next-generation scientific applications, the workflow-based computing technology has become an indispensable research method for managing and streamlining large-scale distributed data processing. This paper investigates a problem of mapping distributed workflows for streaming data processing in faulty networks where nodes and links are subject to probabilistic failures. We formulate this problem as a bi-objective optimization problem in terms of both throughput and reliability, and propose a decentralized layer-oriented method to achieve high throughput for smooth data flow while satisfying a prespecified overall failure rate bound for a guaranteed level of reliability. The superiority of the proposed mapping solution is illustrated by both extensive simulation-based performance comparisons with existing algorithms and experimental results from a real-life scientific workflow deployed in wide-area networks.

KW - Reliability

KW - distributed computing

KW - fault tolerance

KW - frame rate

KW - workflow mapping

UR - http://www.scopus.com/inward/record.url?scp=81555209726&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=81555209726&partnerID=8YFLogxK

U2 - 10.1109/HPCC.2011.52

DO - 10.1109/HPCC.2011.52

M3 - Conference contribution

AN - SCOPUS:81555209726

SN - 9780769545387

T3 - Proc.- 2011 IEEE International Conference on HPCC 2011 - 2011 IEEE International Workshop on FTDCS 2011 -Workshops of the 2011 Int. Conf. on UIC 2011- Workshops of the 2011 Int. Conf. ATC 2011

SP - 347

EP - 354

BT - Proc.- 2011 IEEE International Conference on HPCC 2011 - 2011 IEEE International Workshop on FTDCS 2011 - Workshops of the 2011 Int. Conf. on UIC 2011- Workshops of the 2011 Int. Conf. ATC 2011

T2 - 13th IEEE International Workshop on FTDCS 2011, the 8th International Conference on ATC 2011, the 8th International Conference on UIC 2011 and the 13th IEEE International Conference on HPCC 2011

Y2 - 2 September 2011 through 4 September 2011

ER -

Gu Y, Wu Q, Liu X, Yu D. Improving throughput and reliability of distributed scientific workflows for streaming data processing. In Proc.- 2011 IEEE International Conference on HPCC 2011 - 2011 IEEE International Workshop on FTDCS 2011 - Workshops of the 2011 Int. Conf. on UIC 2011- Workshops of the 2011 Int. Conf. ATC 2011. 2011. p. 347-354. 6063011. (Proc.- 2011 IEEE International Conference on HPCC 2011 - 2011 IEEE International Workshop on FTDCS 2011 -Workshops of the 2011 Int. Conf. on UIC 2011- Workshops of the 2011 Int. Conf. ATC 2011). doi: 10.1109/HPCC.2011.52

Improving throughput and reliability of distributed scientific workflows for streaming data processing

Abstract

Publication series

Other

All Science Journal Classification (ASJC) codes

Keywords

Access to Document

Other files and links

Fingerprint

Cite this