Mitigating Privilege Escalation

A Thesis Submitted to the Faculty

in partial fulfillment of the requirements for the degree of

Doctor of Philosophy

by

Scott Brookes

Thayer School of Engineering Dartmouth College

Hanover, New Hampshire

May 2018

Examining Committee:

Chairman Stephen Taylor, Ph.D.

Member Sergey Bratus, Ph.D.

Member George Cybenko, Ph.D.

Member Steve Chapin, Ph.D.

F. Jon Kull, Ph.D. Dean of Graduate and Advanced Studies

ProQuest Number:

All rights reserved

INFORMATION TO ALL USERS The quality of this reproduction is dependent upon the quality of the copy submitted.

In the unlikely event that the author did not send a complete manuscript and there are missing pages, these will be noted. Also, if material had to be removed,

a note will indicate the deletion.

ProQuest

Published by ProQuest LLC ( ). Copyright of the Dissertation is held by the Author.

All rights reserved. This work is protected against unauthorized copying under Title 17, United States Code

Microform Edition © ProQuest LLC.

ProQuest LLC. 789 East Eisenhower Parkway

P.O. Box 1346 Ann Arbor, MI 48106 – 1346

10822683

10822683

2018

Abstract

One particularly difficult challenge in the computer security landscape is preventing

privilege escalation. This type of attack happens when an actor is granted access

to some piece of hardware with limited permissions but manages to circumvent the

security policies meant to contain them. Although a simple bug in the operating

system, or even in user libraries, can be sufficient to enable this type of attack,

such a vulnerability is also relatively easy to fix. Privilege escalation mechanisms

represent a more challenging security risk because they are methods by which generic

vulnerabilities (such as a buffer overflow) can be leveraged to escalate privilege.

This thesis describes a collection of operating system hardening techniques de-

signed to mitigate the risks of common privilege escalation mechanisms. This includes

non-deterministic loading techniques to randomize code, leveraging the virtualization

features of modern hardware to protect operating system code, and a novel operat-

ing system design paradigm. A proof-of-concept prototype was developed for each

of these techniques using the Bear research microkernel. The code for all techniques

described in this thesis is available at https://github.com/SCSLaboratory/BearOS.

Each of the techniques described in this thesis is evaluated in terms of the addi-

tional security it offers alongside the performance cost of the technique. The security

analysis of each technique attempts to describe (and quantify where possible) the

types of privilege escalation mechanisms that the technique interrupts. Meanwhile,

macro- and micro-benchmarks that are compatible with the Bear microkernel illus-

ii

trate the practicality of each of these techniques for deployment on real-world systems.

Synthesizing four different security mechanisms that each address unique types

of privilege escalation threats, the thesis provides a glimpse of a hardened operating

system. Contrary to the standard practice of “patching” the status quo in response to

each new threat, it attempts to visualize a next-generation operating system design

that brings together the best features of non-determinism, virtualization, and hard-

ware resource utilization in order to present a more secure computing system that

can still meet the ever-increasing performance requirements of modern computing

applications.

iii

Contents

Abstract ii

Contents iv

List of Tables viii

List of Figures ix

List of Code Snippets xi

List of Algorithms xii

1 Introduction 1

1.1 Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

1.2 Metrics of Success . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

1.3 Publications and Contributions . . . . . . . . . . . . . . . . . . . . . 8

1.4 Thesis Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

2 Background and State of the Art 13

2.1 Virtual Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

2.1.1 Case Study: Recursive Paging . . . . . . . . . . . . . . . . . . 15

2.1.2 The Virtual Address Space . . . . . . . . . . . . . . . . . . . . 17

2.2 Message Passing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

iv

2.3 Scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

3 Related Work 24

3.1 Privilege Escalation Mitigation . . . . . . . . . . . . . . . . . . . . . 25

3.1.1 Techniques based on Virtualization . . . . . . . . . . . . . . . 29

3.1.2 Other Techniques . . . . . . . . . . . . . . . . . . . . . . . . . 35

3.1.3 Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

3.1.4 Privilege Escalation Mitigation: Conclusion . . . . . . . . . . 42

3.2 Execute-Only Memory . . . . . . . . . . . . . . . . . . . . . . . . . . 42

3.3 Code Diversification . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

3.4 Asymmetrical Multiprocessing . . . . . . . . . . . . . . . . . . . . . . 46

3.5 Operating System Design Paradigm Shifts . . . . . . . . . . . . . . . 48

3.5.1 Microkernels . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

3.5.2 ExoKernel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

3.5.3 Unikernels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

4 ExOShim 51

4.1 ExOShim: Background . . . . . . . . . . . . . . . . . . . . . . . . . . 52

4.1.1 The Kernel and Virtual Memory . . . . . . . . . . . . . . . . 52

4.1.2 Virtualization and the EPT . . . . . . . . . . . . . . . . . . . 53

4.2 ExOShim: Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

4.2.1 Assumptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

4.2.2 ExOShim . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

4.3 ExOShim: Implementation . . . . . . . . . . . . . . . . . . . . . . . . 58

4.3.1 Providing ExOShim a Context . . . . . . . . . . . . . . . . . . 58

4.3.2 Building the EPT . . . . . . . . . . . . . . . . . . . . . . . . . 60

4.3.3 Starting Virtualization . . . . . . . . . . . . . . . . . . . . . . 65

4.4 ExOShim: Evaluation and Analysis . . . . . . . . . . . . . . . . . . . 69

v

4.4.1 Prototype Complexity . . . . . . . . . . . . . . . . . . . . . . 69

4.4.2 Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

4.4.3 Security Implications . . . . . . . . . . . . . . . . . . . . . . . 71

4.5 ExOShim: Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . 72

4.5.1 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

4.5.2 Final Thoughts . . . . . . . . . . . . . . . . . . . . . . . . . . 73

5 Diversification: KPLT 75

5.1 KPLT: Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

5.1.1 Using the Virtual Memory Abstraction . . . . . . . . . . . . . 76

5.1.2 The Contents of a KPLT . . . . . . . . . . . . . . . . . . . . . 78

5.2 KPLT: Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . 79

5.2.1 KPLT Creation at Load-Time . . . . . . . . . . . . . . . . . . 80

5.2.2 Run-time KPLT Maintenance and Manipulation . . . . . . . . 82

5.3 KPLT: Evaluation and Analysis . . . . . . . . . . . . . . . . . . . . . 86

5.3.1 Security Implications . . . . . . . . . . . . . . . . . . . . . . . 86

5.3.2 Increase in Diversity . . . . . . . . . . . . . . . . . . . . . . . 87

5.3.3 Performance Cost . . . . . . . . . . . . . . . . . . . . . . . . . 88

5.3.4 Remaining Work and Challenges . . . . . . . . . . . . . . . . 88

5.4 KPLT: Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88

6 Diversification: Überdiversity 89

6.1 Diversification Algorithms . . . . . . . . . . . . . . . . . . . . . . . . 91

6.1.1 Run-time Complexity and Termination Analysis . . . . . . . . 94

6.1.2 Proof of Uniformly Distributed Variants . . . . . . . . . . . . 97

6.2 Überdiversity: Implementation . . . . . . . . . . . . . . . . . . . . . . 100

6.2.1 ELF & the Diversity Loader . . . . . . . . . . . . . . . . . . . 100

6.2.2 The Virtual Memory Abstraction . . . . . . . . . . . . . . . . 101

vi

6.2.3 Diversifying the Entire Software Stack . . . . . . . . . . . . . 103

6.3 Überdiversity: Evaluation and Analysis . . . . . . . . . . . . . . . . . 104

6.3.1 Quantification of Diversity Achieved . . . . . . . . . . . . . . 104

6.3.2 Performance Cost . . . . . . . . . . . . . . . . . . . . . . . . . 110

6.3.3 Security Implications . . . . . . . . . . . . . . . . . . . . . . . 114

6.4 Überdiversity: Conclusion . . . . . . . . . . . . . . . . . . . . . . . . 116

7 KUCS 118

7.1 KUCS: Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121

7.1.1 Separating Kernel and User Cores . . . . . . . . . . . . . . . . 123

7.1.2 Fine-Grained Virtualization . . . . . . . . . . . . . . . . . . . 126

7.1.3 Increasing Performance . . . . . . . . . . . . . . . . . . . . . . 128

7.2 KUCS: Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . 130

7.2.1 KUCSBear . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130

7.2.2 Interrupts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132

7.2.3 Virtual Memory . . . . . . . . . . . . . . . . . . . . . . . . . . 141

7.2.4 Message Passing . . . . . . . . . . . . . . . . . . . . . . . . . 142

7.2.5 Scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151

7.3 KUCS: Evaluation and Analysis . . . . . . . . . . . . . . . . . . . . . 157

7.3.1 Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157

7.3.2 Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159

7.3.3 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168

7.4 KUCS: Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177

8 Conclusion 180

Bibliography 183

vii

List of Tables

3.1 Summary of Privilege Escalation Mitigation Methods . . . . . . . . . 40

3.2 Characteristics of Privilege Escalation Mitigation Methods . . . . . . 41

3.3 Existing Execute-Only Memory Techniques . . . . . . . . . . . . . . . 44

3.4 Existing Code Diversification Techniques . . . . . . . . . . . . . . . . 45

4.1 Processor Benchmarks for ExOShim . . . . . . . . . . . . . . . . . . . 70

4.2 ExOShim’s Bear Test Suite Performance . . . . . . . . . . . . . . . . 72

6.1 Fork/Exec Test with Überdiversity . . . . . . . . . . . . . . . . . . . 110

6.2 Processor Benchmarks with Überdiversity . . . . . . . . . . . . . . . 112

7.1 System Call Mechanism Performance . . . . . . . . . . . . . . . . . . 164

7.2 Processor Benchmarks for KUCSBear . . . . . . . . . . . . . . . . . . 165

7.3 System Benchmarking for KUCSBear . . . . . . . . . . . . . . . . . . 167

viii

List of Figures

1.1 Threats and Mitigations Explored in this Thesis . . . . . . . . . . . . 3

2.1 x86 Virtual Memory Translation . . . . . . . . . . . . . . . . . . . . . 14

2.2 Recursive and Conventional Paging Structure Addressing . . . . . . . 16

2.3 Traditional Operating System Design . . . . . . . . . . . . . . . . . . 21

3.1 Return-to-User Privilege Escalation Attack . . . . . . . . . . . . . . . 28

3.2 SecVisor’s Protection Against ret2usr Attacks . . . . . . . . . . . . . 31

3.3 kGuard’s Protection Against ret2usr Attacks . . . . . . . . . . . . . . 36

4.1 Example Memory Disclosure Vulnerability . . . . . . . . . . . . . . . 52

4.2 Overview of ExOShim Protections . . . . . . . . . . . . . . . . . . . . 55

4.3 ExOShim Benchmark Suite Performance . . . . . . . . . . . . . . . . 71

5.1 Function Calls with KPLT . . . . . . . . . . . . . . . . . . . . . . . . 77

5.2 Assembly KPLT Entry . . . . . . . . . . . . . . . . . . . . . . . . . . 78

6.1 A modified FYS Algorithm . . . . . . . . . . . . . . . . . . . . . . . . 99

6.2 Cache Performance Benchmarks with Überdiversity . . . . . . . . . . 111

6.3 Worst-Case Memory Overhead with Überdiversity . . . . . . . . . . . 113

7.1 KUCS Design Overview . . . . . . . . . . . . . . . . . . . . . . . . . 122

7.2 Asynchronous System Calls . . . . . . . . . . . . . . . . . . . . . . . 129

ix

7.3 Kernel Mappings to a KUCS Remote Process . . . . . . . . . . . . . 143

7.4 Initializing the Message Passing Ring Buffer . . . . . . . . . . . . . . 148

7.5 KUCS System Call Mechanisms . . . . . . . . . . . . . . . . . . . . . 149

7.6 System Call Mechanism Performance . . . . . . . . . . . . . . . . . . 163

7.7 System Benchmarking for KUCSBear . . . . . . . . . . . . . . . . . . 167

x

List of Code Snippets

2.1 State-of-the-Art Message Sending from User-Space . . . . . . . . . . . 19

2.2 User-space “ping” System Call . . . . . . . . . . . . . . . . . . . . . . 20

4.1 Building ExOShim’s EPT . . . . . . . . . . . . . . . . . . . . . . . . 61

5.1 Creating a KPLT Entry . . . . . . . . . . . . . . . . . . . . . . . . . 79

7.1 Interrupt Handling Control Flow . . . . . . . . . . . . . . . . . . . . 133

7.2 Naive Remote Memory Mapping . . . . . . . . . . . . . . . . . . . . . 141

7.3 Message Passing with the Ring Buffer from User-space . . . . . . . . 145

7.4 Kernel’s Initialization for Message Passing Ring Buffer . . . . . . . . 146

7.5 Message Passing with the Ring Buffer from the Kernel . . . . . . . . 147

xi

List of Algorithms

4.1 ExOShim’s Initialization Routine . . . . . . . . . . . . . . . . . . . . . 59

6.1 Greedy Diversity Loader Algorithm . . . . . . . . . . . . . . . . . . . . 91

6.2 Scaling Diversity Loader Algorithm . . . . . . . . . . . . . . . . . . . . 92

6.3 Generic Diversity Loader Algorithm . . . . . . . . . . . . . . . . . . . 94

6.4 Fisher-Yates Shuffle Algorithm . . . . . . . . . . . . . . . . . . . . . . 98

xii

Chapter 1

Introduction

Problem Current operating system designs enable multiple privilege escalation

mechanisms by which an attacker can leverage simple bugs to compromise the system.

Hypothesis Modern multicore hardware can significantly mitigate the risk of priv-

ilege escalation with acceptable performance cost.

Isolation is a cornerstone of security in modern computer systems. As the com-

plexities associated with even the most benign instances of computation skyrocket

and processors are called on to handle hundreds or thousands of such instances si-

multaneously, isolating instances from one another is the only practical way to protect

them.

One of the most fundamental forms of isolation is separating the user or appli-

cation from the kernel or operating system. As a trusted actor that manages all

applications on the system, the kernel has the direct ability to compromise every

application running on a system. If a malicious actor manages to compromise the

kernel, they can easily compromise all applications on the system. Consequently,

isolation is most commonly enforced by hardware. In particular, system memory can

be marked for the user or for the supervisor (kernel) and the processor can operate

1

in different modes, corresponding to user and kernel.

Unfortunately, hardware protections are not 100% successful. The ability of a

malicious user to gain read, write, or execution privileges on memory reserved ex-

clusively for the kernel is known as privilege escalation. Quantifying the impact of

this issue is difficult. Although as of early 2018 only about 5% of reported Common

Vulnerabilities and Exposures (CVEs) since 1999 are listed with type “Gain Privi-

leges,” privilege escalation attacks may also be classified as “Gain information” (9%)

or “Code Execution” (29%) [2]. Furthermore, these statistics only discuss vulnera-

bilities. There are also privilege escalation mechanisms that use seemingly unrelated

vulnerabilities to achieve the escalation of privileges. Using a mechanism such as

return-to-user [93] which uses a compromised return address to force code execution

into user-space without changing the CPU privilege mode, a simple buffer overflow

(15% of all reported vulnerabilities) located in kernel code could give the user kernel

privilege.

The Meltdown vulnerability [109] provides a clear example of how devastating

privilege escalation can be. This vulnerability enables a user-space process to read

arbitrary memory, circumventing paging permissions. This would be listed as a single

vulnerability in the statistics referenced previously, but the protected memory of

almost any operating system running on any one of millions of affected processors

produced over almost a decade could be read by a simple user program.

In an age where many of the world’s secrets are entrusted to computers and the

operating systems that run them, privilege escalation constitutes a major threat. This

thesis explores a collection of significant privilege escalation mechanisms and presents

novel techniques for mitigating them. While they are applicable to operating systems

in general, each of the research efforts described in this thesis was prototyped on a

research microkernel called Bear [123]. The capabilities described here, as well as

the Bear microkernel itself, are all available as open source software under the MIT

2

Privilege Escalation

ExOShim

Execute

Read

Memory Disclosure

Meltdown

Kernel Code Injection

Diversi�cation

KUCS

ret2usr

Kernel ROP

Threat Solution

Figure 1.1: The left side of this figure shows a breakdown of the vulnerabilities and mechanisms examined in this thesis. The right side pairs the techniques explored in this thesis to the privilege escalation mechanisms they address.

software license at https://github.com/SCSLaboratory/BearOS.

1.1 Approach

This thesis presents novel mitigation techniques for five different privilege escalation

mechanisms. The mechanisms and their associated mitigation techniques are listed

in Figure 1.1.

Privilege Escalation Methods

Memory Disclosure Often, “memory disclosure” refers to a vulnerability, rather

than a mechanism. In a memory disclosure, information leaks from a privileged source

to an unprivileged receiver. This is a common issue in monolithic operating systems

that load drivers, potentially buggy pieces of software written by third parties, into

the operating system with privilege. The memory disclosure mitigation explored

in this thesis, ExOShim, enforces a security policy that identifies and prohibits any

3

memory disclosure, regardless of the mechanism used to accomplish such a disclosure.

Meltdown First appearing in the public eye in January 2018, Meltdown [109] is

a vulnerability found in most Intel x86 processor implementations and some other

processors, including ARM and PowerPC. Meltdown takes advantage of speculative

execution and cache timing attacks to recover the results of computations that cause

a fault before such a fault is actually delivered. In other words, a user process can

issue reads to kernel memory and then retrieve the data of the read before the fault

is delivered. This is a demonstration of the dangers of leaving such a vital security

property – the separation of kernel- and user-space memory – to a complex hard-

ware implementation that is all but invisible to the programmer. The Kernel- and

User-space Core-based Separation (KUCS) operating system design provides stronger

separation between the user- and kernel-space such that the hardware vulnerability

allowing Meltdown can no longer impact the system.

Kernel ROP Return-oriented-programming (ROP) is a process by which a care-

fully crafted stack uses return instructions to force control flow through a series of

snippets of existing code (called “gadgets”) to preform computation. Even small pro-

grams have enough gadgets to perform arbitrary execution with a carefully controlled

stack, so any program as big and complex as an operating system can be subjected

to complex ROP attacks. This is an example of how mechanism differs from vul-

nerability ; any simple kernel stack overflow (the vulnerability) can enable a kernel

ROP attack (the mechanism). Software randomization techniques are the standard

technique for mitigating ROP as they make it more difficult to discover the location

of the gadgets of interest and subsequently build the stack necessary to preform the

desired computation. This thesis explores the widespread use of non-determinism on

many existing systems and offers novel techniques and analysis that compliment this

rich field of research.

4

Kernel Code Injection Perhaps the most naive method of escalating to kernel

privilege is to inject malicious instructions directly into the kernel code. One example

of how this may be done is with a kernel routine that copies data from user memory

into kernel memory. If the attacker can force execution to jump to this previously

copied memory, they can execute code that they control with all the privileges of

the kernel. While most kernels use Write xor Execute (W⊕X) security policies, this

is an example of a defensive strategy that targets a vulnerability; not necessarily

the mechanism itself. On the contrary, KUCS enables security policies that ensure

that no code absent from the kernel at boot-time can ever be executed with kernel

privilege.

ret2usr If the attacker can control a single return address in the kernel, they can

force the kernel to jump to user-controlled memory without passing through the

normal context-switching routine required to change the mode of the processor into

user-mode. After its discovery, chip manufacturers began to implement hardware fea-

tures such as Intel’s Supervisor Mode Execution Protection (SMEP) in order to cause

a system fault if user-marked memory is executed without the CPU in user mode.

Despite this hardware patch, which may not be used by corrupted, old, or buggy ker-

nel implementations, this mechanism demonstrates the dangers of weak separation

between kernel- and user-space on modern hardware. As previously discussed, Melt-

down is another attack exploiting this weak separation. KUCS enforces a stronger

separation that is not vulnerable to ret2usr type attacks.

Privilege Escalation Mitigation Techniques

ExOShim is a memory disclosure mitigation that uses the virtualization features

of modern processors. In particular, the extended page tables (EPT) included with

Intel’s VT-x virtualization extensions provide the capability to mark memory pages as

5

execute-only, a feature not available in the kernel paging system. ExOShim provides

a shim layer that enables virtualization and uses these fine-grained permission bits

in the EPT to mark the kernel code as execute-only. Furthermore, it keeps the

operating system as the most basic trusted software entity. Rather than implementing

a trusted hypervisor (which becomes a large attack surface in its own right) it uses

the virtualization features to enforce a security policy that the kernel initializes. In

fact, ExOShim manages to offer this protection in a way that cannot be hijacked,

modified, or disabled after initialization.

Diversification Although there have been decades of research into software ran-

domization techniques, this thesis extends this body of knowledge with two new tech-

niques: the Kernel Procedure Linkage Table (KPLT) and Überdiversity. The KPLT

treats the kernel, which is conventionally mapped into every process, as a shared ob-

ject. It provides a method for randomizing kernel mappings on a per-process basis, so

that each process uses unique virtual addresses to access shared kernel routines. The

second technique, Überdiversity, presents a prototype that explores just how far load-

Related Tags