Feature Overcorrelation in Deep Graph Neural Networks: A New Perspective

Wei Jin; Xiaorui Liu; Yao Ma; Charu Aggarwal; Jiliang Tang

doi:10.1145/3534678.3539445

Feature Overcorrelation in Deep Graph Neural Networks: A New Perspective

Wei Jin, Xiaorui Liu, Yao Ma, Charu Aggarwal, Jiliang Tang

Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

18 Scopus citations

Abstract

Recent years have witnessed remarkable success achieved by graph neural networks (GNNs) in many real-world applications such as recommendation and drug discovery. Despite the success, oversmoothing has been identified as one of the key issues which limit the performance of deep GNNs. It indicates that the learned node representations are highly indistinguishable due to the stacked aggregators. In this paper, we propose a new perspective to look at the performance degradation of deep GNNs, i.e., feature overcorrelation. Through empirical and theoretical study on this matter, we demonstrate the existence of feature overcorrelation in deeper GNNs and reveal potential reasons leading to this issue. To reduce the feature correlation, we propose a general framework DeCorr which can encourage GNNs to encode less redundant information. Extensive experiments have demonstrated that DeCorr can help enable deeper GNNs and is complementary to existing techniques tackling the oversmoothing issue.

Original language	English (US)
Title of host publication	KDD 2022 - Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining
Publisher	Association for Computing Machinery
Pages	709-719
Number of pages	11
ISBN (Electronic)	9781450393850
DOIs	https://doi.org/10.1145/3534678.3539445
State	Published - Aug 14 2022
Event	28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2022 - Washington, United States Duration: Aug 14 2022 → Aug 18 2022

Publication series

Name	Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

Conference

Conference	28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2022
Country/Territory	United States
City	Washington
Period	8/14/22 → 8/18/22

All Science Journal Classification (ASJC) codes

Software
Information Systems

Keywords

deep models
graph neural networks
semi-supervised learning

Access to Document

10.1145/3534678.3539445

Cite this

Jin, W., Liu, X., Ma, Y., Aggarwal, C., & Tang, J. (2022). Feature Overcorrelation in Deep Graph Neural Networks: A New Perspective. In KDD 2022 - Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (pp. 709-719). (Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining). Association for Computing Machinery. https://doi.org/10.1145/3534678.3539445

@inproceedings{7dabb4b5efdc41039b2bbcc32fe13642,

title = "Feature Overcorrelation in Deep Graph Neural Networks: A New Perspective",

abstract = "Recent years have witnessed remarkable success achieved by graph neural networks (GNNs) in many real-world applications such as recommendation and drug discovery. Despite the success, oversmoothing has been identified as one of the key issues which limit the performance of deep GNNs. It indicates that the learned node representations are highly indistinguishable due to the stacked aggregators. In this paper, we propose a new perspective to look at the performance degradation of deep GNNs, i.e., feature overcorrelation. Through empirical and theoretical study on this matter, we demonstrate the existence of feature overcorrelation in deeper GNNs and reveal potential reasons leading to this issue. To reduce the feature correlation, we propose a general framework DeCorr which can encourage GNNs to encode less redundant information. Extensive experiments have demonstrated that DeCorr can help enable deeper GNNs and is complementary to existing techniques tackling the oversmoothing issue.",

keywords = "deep models, graph neural networks, semi-supervised learning",

author = "Wei Jin and Xiaorui Liu and Yao Ma and Charu Aggarwal and Jiliang Tang",

note = "Publisher Copyright: {\textcopyright} 2022 ACM.; 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2022 ; Conference date: 14-08-2022 Through 18-08-2022",

year = "2022",

month = aug,

day = "14",

doi = "10.1145/3534678.3539445",

language = "English (US)",

series = "Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining",

publisher = "Association for Computing Machinery",

pages = "709--719",

booktitle = "KDD 2022 - Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining",

}

Jin, W, Liu, X, Ma, Y, Aggarwal, C & Tang, J 2022, Feature Overcorrelation in Deep Graph Neural Networks: A New Perspective. in KDD 2022 - Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Association for Computing Machinery, pp. 709-719, 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2022, Washington, United States, 8/14/22. https://doi.org/10.1145/3534678.3539445

Feature Overcorrelation in Deep Graph Neural Networks: A New Perspective. / Jin, Wei; Liu, Xiaorui; Ma, Yao et al.
KDD 2022 - Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. Association for Computing Machinery, 2022. p. 709-719 (Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Feature Overcorrelation in Deep Graph Neural Networks

T2 - 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2022

AU - Jin, Wei

AU - Liu, Xiaorui

AU - Ma, Yao

AU - Aggarwal, Charu

AU - Tang, Jiliang

PY - 2022/8/14

Y1 - 2022/8/14

N2 - Recent years have witnessed remarkable success achieved by graph neural networks (GNNs) in many real-world applications such as recommendation and drug discovery. Despite the success, oversmoothing has been identified as one of the key issues which limit the performance of deep GNNs. It indicates that the learned node representations are highly indistinguishable due to the stacked aggregators. In this paper, we propose a new perspective to look at the performance degradation of deep GNNs, i.e., feature overcorrelation. Through empirical and theoretical study on this matter, we demonstrate the existence of feature overcorrelation in deeper GNNs and reveal potential reasons leading to this issue. To reduce the feature correlation, we propose a general framework DeCorr which can encourage GNNs to encode less redundant information. Extensive experiments have demonstrated that DeCorr can help enable deeper GNNs and is complementary to existing techniques tackling the oversmoothing issue.

AB - Recent years have witnessed remarkable success achieved by graph neural networks (GNNs) in many real-world applications such as recommendation and drug discovery. Despite the success, oversmoothing has been identified as one of the key issues which limit the performance of deep GNNs. It indicates that the learned node representations are highly indistinguishable due to the stacked aggregators. In this paper, we propose a new perspective to look at the performance degradation of deep GNNs, i.e., feature overcorrelation. Through empirical and theoretical study on this matter, we demonstrate the existence of feature overcorrelation in deeper GNNs and reveal potential reasons leading to this issue. To reduce the feature correlation, we propose a general framework DeCorr which can encourage GNNs to encode less redundant information. Extensive experiments have demonstrated that DeCorr can help enable deeper GNNs and is complementary to existing techniques tackling the oversmoothing issue.

KW - deep models

KW - graph neural networks

KW - semi-supervised learning

UR - http://www.scopus.com/inward/record.url?scp=85132171748&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85132171748&partnerID=8YFLogxK

U2 - 10.1145/3534678.3539445

DO - 10.1145/3534678.3539445

M3 - Conference contribution

AN - SCOPUS:85132171748

T3 - Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

SP - 709

EP - 719

BT - KDD 2022 - Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

PB - Association for Computing Machinery

Y2 - 14 August 2022 through 18 August 2022

ER -

Jin W, Liu X, Ma Y, Aggarwal C, Tang J. Feature Overcorrelation in Deep Graph Neural Networks: A New Perspective. In KDD 2022 - Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. Association for Computing Machinery. 2022. p. 709-719. (Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining). doi: 10.1145/3534678.3539445

Feature Overcorrelation in Deep Graph Neural Networks: A New Perspective

Abstract

Publication series

Conference

All Science Journal Classification (ASJC) codes

Keywords

Access to Document

Other files and links

Fingerprint

Cite this