Distributed early worm detection based on payload histograms

Epidemic worms has become a social problem owing to their potency in paralyzing the Internet, thus affecting our way of life. Recent researches have pointed out that epidemic worms can propagate similar payloads rapidly. It was shown that it is possible to evaluate similarities between these payloads in terms of a 256-dimensional vector based on histograms of the appearance frequencies of 256 character codes. This observation has also been confirmed by our earlier works. However, this method, if applied to flows from only one network, which means a network managed by an independent organization, is prone to a high rate of false positives in cases such as when normal emails are sent through a mailing list. To overcome this problem, we propose a new scheme which checks for any similarity between flows detected at several IDSs in a distributed environment. The proposed scheme is based on the fact that normal payloads propagating from different networks are different, whereas in the case of epidemic worms payloads even propagated through different networks but generated by the same worm exhibit similarity. We have demonstrated the effectiveness of the proposed scheme through extensive experiments using real network traffic that contains worms.