Jelesnianski, Christopher Stanislaw2023-01-102023-01-102023-01-09vt_gsexam:35944http://hdl.handle.net/10919/113108Over the years, many defense techniques have been proposed by the security community. Even so, few have been adopted by the general public and deployed in production. This limited defense deployment and weak security has serious consequences, as large scale cyber-attacks are now a common occurrence in society. One major obstacle that stands in the way is practicality, the quality of being designed for actual use or having usefulness or convenience. For example, an exploit mitigation design may be considered not practical to deploy if it imposes high performance overhead, despite offering excellent and robust security guarantees. This is because achieving hallmarks of practical design, such as minimizing adverse side-effects like performance degradation or memory monopolization, is difficult in practice, especially when trying to provide a high level of security for users. Secure and practical exploit mitigation design must successfully navigate several challenges. To illustrate, modern-day attacks, especially code re-use attacks, understand that rudimentary defenses such as Data Execution Prevention (DEP) and Address Space Layout Randomization (ASLR) will be deployed moving forward. These attacks have therefore evolved and diversified their angles of attack to become capable of leveraging a multitude of different code components. Accordingly, the security community has uncovered these threats and maintained progress in providing possible resolutions with new exploit mitigation designs. More specifically though, defenses have had to correspondingly extend their capabilities to protect more aspects of code, leading to defense techniques becoming increasingly complex. Trouble then arises as supporting such fine-grained defenses brings inherent disadvantages such as significant hardware resource utilization that could be otherwise used for useful work. This complexity has made performance, security, and scalability all competing ideals in practical system design. At the same time, other recent efforts have implemented mechanisms with negligible performance impact, but do so at the risk of weaker security guarantees. This dissertation first formalizes the challenges in modern exploit mitigation design. To illustrate these challenges, this dissertation presents a survey from the perspective of both attacker and defender to provide an overview of this current security landscape. This includes defining an informal taxonomy of exploit mitigation strategies, explaining prominent attack vectors that are faced by security experts today, and identifying and defining code components that are generally abused by code re-use. This dissertation then presents two practical design solutions. Both defense system designs uphold goals of achieving realistic performance, providing strong security guarantees, being robust for modern application code-bases, and being able to scale across the system at large. The first practical exploit mitigation design this dissertation presents is MARDU. MARDU is a novel re-randomization approach that utilizes on-demand randomization and the concept of code trampolines to support sharing of code transparently system-wide. To the best of my knowledge, MARDU is the first presented re-randomization technique capable of runtime code sharing for re-randomized code system-wide. Moreover, MARDU is one of the very few re-randomization mechanisms capable of performing seamless live thread migration to newly randomized code without pausing application execution. This dissertation describes the full design, implementation, and evaluation of MARDU to demonstrate its merits and show that careful design can uphold all practical design goals. For instance, scalability is a major challenge for randomization strategies, especially because traditional OS design expects code to be placed in known locations so that it can be reached by multiple processes, while randomization is purposefully trying to achieve the opposite, being completely unpredictable. This clash in expectations between system and defense design breaks a few very important assumptions for an application's runtime environment. This forces most randomization mechanisms to abandon the hope of upholding memory deduplication. MARDU resolves this challenge by applying trampolines to securely reach functions protected under secure memory. Even with this new calling convention in place, MARDU shows re-randomization degradation can be significantly reduced without sacrificing randomization entropy. Moreover, MARDU shows it is capable of defeating prominent code re-use variants with this practical design. This dissertation then presents its second practical exploit mitigation solution, BASTION. BASTION is a fine-grained system call filtering mechanism aimed at significantly strengthening the security surrounding system calls. Like MARDU, BASTION upholds the principles of this dissertation and was implemented with practicality in mind. BASTION's design is based on empirical observation of what a legitimate system call invocation consists of. BASTION introduces System Call Integrity to enforce the correct and intended use of system calls within a program. In order to enforce this novel security policy, BASTION proposes three new specialized contexts for the effective enforcement of legitimate system call usage. Namely, these contexts enforce that: system calls are only invoked with the correct calling convention, system calls are reached through legitimate control-flow paths, and all system call arguments are free from attacker corruption. By enforcing System Call Integrity with the previously mentioned contexts, this dissertation adds further evidence that context-sensitive defense strategies are superior to context-insensitive ones. BASTION is able to prevent over 32 real-world and synthesized exploits in its security evaluation and incurs negligible performance overhead (0.60%-2.01%). BASTION demonstrates that narrow and specialized exploit mitigation designs can be effective in more than one front, to the point that BASTION not only revents code re-use, but is capable of defending against any attack class that requires the utilization of system calls.ETDenCreative Commons Attribution 4.0 InternationalExploit MitigationPractical DesignSystem SoftwareRandomizationRuntime DefenseReturn-Oriented ProgrammingCode Re-use AttacksSystem CallsSystem Call SpecializationDissertation