System Call-Based Detection of Malicious Processes

Raymond Canzanese, Spiros Mancoridis, Moshe Kam

Research output: Chapter in Book/Report/Conference proceedingConference contribution

35 Scopus citations

Abstract

System call analysis is a behavioral malware detection technique that is popular due to its promising detection results and ease of implementation. This study describes a system that uses system call analysis to detect malware that evade traditional defenses. The system monitors executing processes to identify compromised hosts in production environments. Experimental results compare the effectiveness of multiple feature extraction strategies and detectors based on their detection accuracy at low false positive rates. Logistic regression and support vector machines consistently outperform log-likelihood ratio and signature detectors as processing and detection methods. A feature selection study indicates that a relatively small set of system call 3-grams provide detection accuracy comparable to that of more complex models. A case study indicates that the detection system performs well against a variety of malware samples, benign workloads, and host configurations.

Original languageEnglish (US)
Title of host publicationProceedings - 2015 IEEE International Conference on Software Quality, Reliability and Security, QRS 2015
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages119-124
Number of pages6
ISBN (Electronic)9781467379892
DOIs
StatePublished - Sep 21 2015
EventIEEE International Conference on Software Quality, Reliability and Security, QRS 2015 - Vancouver, Canada
Duration: Aug 3 2015Aug 5 2015

Publication series

NameProceedings - 2015 IEEE International Conference on Software Quality, Reliability and Security, QRS 2015

Other

OtherIEEE International Conference on Software Quality, Reliability and Security, QRS 2015
Country/TerritoryCanada
CityVancouver
Period8/3/158/5/15

All Science Journal Classification (ASJC) codes

  • Computer Networks and Communications
  • Software

Keywords

  • detection
  • machine learning
  • malware
  • system call analysis

Fingerprint

Dive into the research topics of 'System Call-Based Detection of Malicious Processes'. Together they form a unique fingerprint.

Cite this