Improving Operating System Security, Reliability, and Performance through Intra-Unikernel Isolation, Asynchronous Out-of-kernel IPC, and Advanced System Servers
| dc.contributor.author | Sung, Mincheol | en |
| dc.contributor.committeechair | Ravindran, Binoy | en |
| dc.contributor.committeemember | Wang, Haining | en |
| dc.contributor.committeemember | Nikolaev, Ruslan | en |
| dc.contributor.committeemember | Hicks, Matthew | en |
| dc.contributor.committeemember | Zeng, Haibo | en |
| dc.contributor.department | Electrical and Computer Engineering | en |
| dc.date.accessioned | 2023-03-29T08:00:13Z | en |
| dc.date.available | 2023-03-29T08:00:13Z | en |
| dc.date.issued | 2023-03-28 | en |
| dc.description.abstract | Computer systems are vulnerable to security exploits, and the security of the operating system (OS) is crucial as it is often a trusted entity that applications rely on. Traditional OSs have a monolithic design where all components are executed in a single privilege layer, but this design is increasingly inadequate as OS code sizes have become larger and expose a large attack surface. Microkernel OSs and multiserver OSs improve security and reliability through isolation, but they come at a performance cost due to crossing privilege layers through IPCs, system calls, and mode switches. Library OSs, on the other hand, implement kernel components as libraries which avoids crossing privilege layers in performance-critical paths and thereby improves performance. Unikernels are a specialized form of library OSs that consist of a single application compiled with the necessary kernel components, and execute in a single address space, usually atop a hypervisor for strong isolation. Unikernels have recently gained popularity in various application domains due to their better performance and security. Although unikernels offer strong isolation between each instance due to virtualization, there is no isolation within a unikernel. Since the model eliminates the traditional separation between kernel and user parts of the address space, the subversion of a kernel or application component will result in the subversion of the entire unikernel. Thus, a unikernel must be viewed as a single unit of trust, reducing security. The dissertation's first contribution is intra-unikernel isolation: we use Intel's Memory Protection Keys (MPK) primitive to provide per-thread permission control over groups of virtual memory pages within a unikernel's single address space, allowing different areas of the address space to be isolated from each other. We implement our mechanisms in RustyHermit, a unikernel written in Rust. Our evaluations show that the mechanisms have low overhead and retain unikernel's low system call latency property: 0.6% slowdown on applications including memory/compute intensive benchmarks as well as micro-benchmarks. Multiserver OS, a type of microkernel OS, has high parallelism potential due to its inherent compartmentalization. However, the model suffers from inferior performance. This is due to inter-process communication (IPC) client-server crossings that require context switches for single-core systems, which are more expensive than traditional system calls; on multi-core systems (now ubiquitous), they have poor resource utilization. The dissertation's second contribution is Aoki, a new approach to IPC design for microkernel OSs. Aoki incorporates non-blocking concurrency techniques to eliminate in-kernel blocking synchronization which causes performance challenges for state-of-the-art microkernels. Aoki's non-blocking (i.e., lock-free and wait-free) IPC design not only improves performance and scalability, but also enhances reliability by preventing thread starvation. In a multiserver OS setting, the design also enables the reconnection of stateful servers after failure without loss of IPC states. Aoki solves two problems that have plagued previous microkernel IPC designs: reducing excessive transitions between user and kernel modes and enabling efficient recovery from failures. We implement Aoki in the state-of-the-art seL4 microkernel. Results from our experiments show that Aoki outperforms the baseline seL4 in both fastpath IPC and cross-core IPC, with improvements of 2.4x and 20x, respectively. The Aoki IPC design enables the design of system servers for multiserver OSs with higher performance and reliability. The dissertation's third and final contribution is the design of a fault-tolerant storage server and a copy-free file system server. We build both servers using NetBSD OS's rumprun unikernel, which provides robust isolation through hardware virtualization, and is capable of handling a wide range of storage devices including NVMe. Both servers communicate with client applications using Aoki's IPC design, which yields scalable IPC. In the case of the storage server, the IPC also enables the server to transparently recover from server failures and reconnect to client applications, with no loss of IPC state and no significant overhead. In the copy-free file system server's design, applications grant the server direct memory access to file I/O data buffers for high performance. The performance problems solved in the server designs have challenged all prior multiserver/microkernel OSs. Our evaluations show that both servers have a performance comparable to Linux and the rumprun baseline. | en |
| dc.description.abstractgeneral | Computer security is extremely important, especially when it comes to the operating system (OS) – the foundation upon which all applications execute. Traditional OSs adopt a monolithic design in which all of their components execute at a single privilege level (for achieving high performance). However, this design degrades security as the vulnerability of a single component can be exploited to compromise the entire system. The problem is exacerbated when the OS codebase becomes large, as is the current trend. To overcome this security challenge, researchers have developed alternative OS models such as microkernels, multiserver OSs, library OSs, and recently, unikernels. The unikernel model has recently gained popularity in application domains such as cloud computing, the internet of things (IoT), and high-performance computing due to its improved security and performance. In this model, a single application is compiled together with its necessary OS components to produce a single, small executable image. Unikernels execute atop a hypervisor, a software layer that provides strong isolation between unikernels, usually by leveraging special hardware instructions. Both ideas improve security. The dissertation's first contribution improves the security of unikernels by enabling isolation within a unikernel. This allows different components of a unikernel (e.g., safe code, unsafe code, kernel code, user code) to be isolated from each other. Thus, the vulnerability of a single component cannot be exploited to compromise the entire system. We used Intel's Memory Protection Keys (MPK), a hardware feature of Intel CPUs, to achieve this isolation. Our implementation of the technique and experimental evaluations revealed that the technique has low overhead and high performance. The dissertation's second contribution improves the performance of multiserver OSs. This OS model has excellent potential for parallelization, but its performance is hindered by slow communication between applications and OS subsystems (which are programmed as clients and servers, respectively). We develop Aoki, an Inter-Process Communication (IPC) technique that enables faster and more reliable communication between clients and servers in multiserver OSs. Our implementation of Aoki in the state-of-the-art seL4 microkernel and evaluations reveal that the technique improves IPC latency over seL4's by as much as two orders of magnitude. The dissertation's third and final contribution is the design of two servers for multiserver OSs: a storage server and a file system server. The servers are built as unikernels running atop the Xen hypervisor and are powered by Aoki's IPC mechanism for communication between the servers and applications. The storage server is designed to recover its state after a failure with no loss of data and little overhead, and the file system server is designed to communicate with applications with little overhead. Our evaluations show that both servers achieve their design goals: they have comparable performance to that of state-of-the-art high-performance OSes such as Linux. | en |
| dc.description.degree | Doctor of Philosophy | en |
| dc.format.medium | ETD | en |
| dc.identifier.other | vt_gsexam:36640 | en |
| dc.identifier.uri | http://hdl.handle.net/10919/114217 | en |
| dc.language.iso | en | en |
| dc.publisher | Virginia Tech | en |
| dc.rights | In Copyright | en |
| dc.rights.uri | http://rightsstatements.org/vocab/InC/1.0/ | en |
| dc.subject | Operating Systems | en |
| dc.subject | Microkernels | en |
| dc.subject | Unikernels | en |
| dc.subject | IPC | en |
| dc.subject | Security | en |
| dc.subject | Failure Recovery | en |
| dc.title | Improving Operating System Security, Reliability, and Performance through Intra-Unikernel Isolation, Asynchronous Out-of-kernel IPC, and Advanced System Servers | en |
| dc.type | Dissertation | en |
| thesis.degree.discipline | Computer Engineering | en |
| thesis.degree.grantor | Virginia Polytechnic Institute and State University | en |
| thesis.degree.level | doctoral | en |
| thesis.degree.name | Doctor of Philosophy | en |
Files
Original bundle
1 - 1 of 1