Generalized anomaly detection model for windows-based malicious program behavior

Xin Tang, Constantine N. Manikopoulos, Sotirios G. Ziavras

Research output: Contribution to journalArticlepeer-review

2 Scopus citations


In this paper we demonstrate that it is possible in general to detect Windows-based malicious program behavior. Since S. Forrest et al. used the N-grams method to classify system call trace data, dynamic learning has become a promising research area. However, most research works have been done in the UNIX environment and have limited scope. In Forrest's original model, "Self" is defined based on a normal process whereas "Non-Self" corresponds to one or two malicious processes. We extend this technique into the Windows environment. In our model, "Self" is defined to represent the general pattern of hundreds ofWindows program behaviors; "Non-Self" is defined to represent all program behaviors that fall out of norm. Because of the difficulty in collecting program behavior, insufficient research results are available. We collected around 1000 system call traces of various normal and malicious programs in the Windows OS. A normal profile was built using a Hidden Markov Model (HMM). The evaluation was based on the entire trace. Our classification results are promising.

Original languageEnglish (US)
Pages (from-to)428-435
Number of pages8
JournalInternational Journal of Network Security
Issue number3
StatePublished - 2008

All Science Journal Classification (ASJC) codes

  • Computer Networks and Communications


  • Intrusion detection
  • Malicious codes
  • Markov process
  • Program behavior


Dive into the research topics of 'Generalized anomaly detection model for windows-based malicious program behavior'. Together they form a unique fingerprint.

Cite this