Deep neural networks with knowledge instillation

Fan Yang, Ninghao Liu, Mengnan Du, Kaixiong Zhou, Shuiwang Ji, Xia Hu

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Scopus citations


Deep neural network (DNN) has become an effective computational tool because of its superior performance in practice. However, the generalization of DNN still largely depends on the training data, no matter in quantity or quality. In this paper, we propose a knowledge instillation framework, named NeuKI, for feed-forward DNN, aiming to enhance learning performance with the aid of knowledge. This task is particularly challenging due to the complicated nature of knowledge and numerous variants of DNN architectures. To bridge the gap, we construct a separate knowledge-DNN faithfully encoding the instilled knowledge for joint training. The core idea is to regularize the training of target-DNN with the constructed knowledge-DNN, so that the instilled knowledge can guide the model training. The proposed NeuKI is demonstrated to be applicable to both knowledge rules and constraints, where rules are encoded by structure and constraints are handled by loss. Experiments are conducted on several real-world datasets from different domains, and the results demonstrate the effectiveness of NeuKI in improving learning performance, as well as relevant data efficiency and model interpretability.

Original languageEnglish (US)
Title of host publicationProceedings of the 2020 SIAM International Conference on Data Mining, SDM 2020
EditorsCarlotta Demeniconi, Nitesh Chawla
PublisherSociety for Industrial and Applied Mathematics Publications
Number of pages9
ISBN (Electronic)9781611976236
StatePublished - 2020
Externally publishedYes
Event2020 SIAM International Conference on Data Mining, SDM 2020 - Cincinnati, United States
Duration: May 7 2020May 9 2020

Publication series

NameProceedings of the 2020 SIAM International Conference on Data Mining, SDM 2020


Conference2020 SIAM International Conference on Data Mining, SDM 2020
Country/TerritoryUnited States

All Science Journal Classification (ASJC) codes

  • Computer Science Applications
  • Software


Dive into the research topics of 'Deep neural networks with knowledge instillation'. Together they form a unique fingerprint.

Cite this