The number of mobile applications (i.e., apps) is rapidly growing, as the mobile computing becomes an integral part of the modern user experience. Malicious apps have infiltrated open marketplaces for mobile platforms. These malicious apps can exfiltrate user's private data, abuse of system resources, or disrupting regular services. Despite the recent advances on mobile security, the problem of detecting vulnerable and malicious mobile apps with high detection accuracy remains an open problem.

In this thesis, we address the problem of Android security by presenting a new quantitative program analysis framework for security vetting of Android apps. We first introduce a highly accurate proactive detection solution for detecting individual malicious apps. Our approach enforces benign property as opposed of chasing malware signatures, and uses one complex feature rather than multi-feature as in the existing malware detection methods. In particular, we statically extract a data-flow feature on how user inputs trigger sensitive critical operations, a property referred to as the user-trigger dependence. This feature is extracted through nontrivial Android-specific static program analysis, which can be used in various quantitative analytical methods. Our evaluation on thousands of malicious apps and free popular apps gives a detection accuracy (2% false negative rate and false positive rate) that is better than, or at least competitive against, the state-of-the-art. Furthermore, our method discovers new malicious apps available in the Google Play store that have not been previously detected by anti-virus scanning tools.

Second, we present a new app collusion detection approach and algorithms to analyze pairs or groups of communicating apps. App collusion is a new technique utilized by the attackers to evade standard detection. It is a new threat where two or more apps, appearing benign, communicate to perform malicious task. Most of the existing solutions assume the attack model of a stand-alone malicious app, and hence cannot detect app collusion. We first demonstrate experimental evidence on the technical challenges associated with detecting app collusion. Then, we address these challenges by introducing a scalable and an in-depth cross-app static flow analysis approach to identify the risk level associated with communicating apps. Our approach statically analyzes the sensitivity and the context of each inter-app communication with low analysis complexity, and defines fine-grained security policies for the inter-app communication risk detection. Our evaluation results on thousands of free popular apps indicate that our technique is effective. It generates four times fewer false positives compared to the state-of-the-art collusion-detection solution, enhancing the detection capability. The advantages of our inter-app communication analysis approach are the analysis scalability with low complexity, and the substantially improved detection accuracy compared to the state-of-the-art solution. These types of proactive defenses solutions allow defenders to stay proactive when defending against constantly evolving malware threats.