Combating against attacks on encrypted protocols

Attacks against encrypted protocols are becoming increasingly popular. They pose a serious challenge to the conventional Intrusion Detection Systems (IDSs) which heavily rely on inspecting the network packet fields and are consequently unable to monitor encrypted sessions. IDSs can be broadly categorized into two types: signature-based and anomaly-based IDSs. The signature-based IDSs rely on previous attack signatures but are often ineffective against new attacks. On the other hand, anomaly-based detection systems depend on detecting the change in the protocol behavior caused by an attack. The latter can be employed to detect novel attacks, and therefore are often preferred over their signature-based counterpart. In this paper, we envision an anomaly-based IDS which can detect attacks against popular encrypted protocols, such as SSH and SSL. The proposed system creates a normal behavior profile and uses nonparametric Cusum algorithm to detect deviation from the normal profile. Upon detecting an anomaly, the proposed mechanism generates an alert, sets a delay to the protocol response, and traces back the attacker. The effectiveness of the proposed detection scheme is verified via simulations.