UC Riverside

The current fight between security experts and malware authors is an arms race. In this race, malware authors devise new attacks and exploits new vulnerabilities while the experts can only deflect the attacks and patch up the vulnerability after damage has been inflicted. Defending against miscreants is a difficult task precisely because experts do not know what attacks may come in the future. The ultimate goal of our work is to utilizing graph-mining and Machine Learning techniques to (a) develop profiles of user and malware behaviors and (b) detect anomalies and identify malicious actors. In this dissertation, we present three pieces of work that are aimed toward achieve that goal. The first is a graph-based approach designed to leverage P2P bots' behaviors to detect them when they lay dormant in the local network and wait for instructions from the botmasters. The second is a probabilistic algorithm based on the Stochastic Block Model that is designed to infer the group structure of users from their web browsing behaviors and leverage the group structure to detect when users in the network visit malicious websites. The third is an in-depth study of users' exposure to web-based malware from the point of view of the malicious websites and the users themselves, where we explore the methods with which web-based malware spread and investigate their characteristics and temporal behaviors.