Zookeeper

Table of Contents

Part I. ZooKeeper Concepts and Basics
1. Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
The ZooKeeper Mission 4
How the World Survived without ZooKeeper 6
What ZooKeeper Doesn’t Do 6
The Apache Project 7
Building Distributed Systems with ZooKeeper 7
Example: Master-Worker Application 9
Master Failures 10
Worker Failures 10
Communication Failures 11
Summary of Tasks 12
Why Is Distributed Coordination Hard? 12
ZooKeeper Is a Success, with Caveats 14
2. Getting to Grips with ZooKeeper. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
ZooKeeper Basics 17
API Overview 18
Different Modes for Znodes 19
Watches and Notifications 20
Versions 23
ZooKeeper Architecture 23
ZooKeeper Quorums 24
Sessions 25
Getting Started with ZooKeeper 26
First ZooKeeper Session 27
iii
States and the Lifetime of a Session 30
ZooKeeper with Quorums 31
Implementing a Primitive: Locks with ZooKeeper 35
Implementation of a Master-Worker Example 35
The Master Role 36
Workers, Tasks, and Assignments 38
The Worker Role 39
The Client Role 40
Takeaway Messages 42
Part II. Programming with ZooKeeper
3. Getting Started with the ZooKeeper API. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
Setting the ZooKeeper CLASSPATH 45
Creating a ZooKeeper Session 45
Implementing a Watcher 47
Running the Watcher Example 49
Getting Mastership 51
Getting Mastership Asynchronously 56
Setting Up Metadata 59
Registering Workers 60
Queuing Tasks 64
The Admin Client 65
Takeaway Messages 68
4. Dealing with State Change. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
One-Time Triggers 70
Wait, Can I Miss Events with One-Time Triggers? 70
Getting More Concrete: How to Set Watches 71
A Common Pattern 72
The Master-Worker Example 73
Mastership Changes 73
Master Waits for Changes to the List of Workers 77
Master Waits for New Tasks to Assign 79
Worker Waits for New Task Assignments 82
Client Waits for Task Execution Result 85
An Alternative Way: Multiop 87
Watches as a Replacement for Explicit Cache Management 90
Ordering Guarantees 91
Order of Writes 91
Order of Reads 91
iv | Table of Contents
Order of Notifications 92
The Herd Effect and the Scalability of Watches 93
Takeaway Messages 94
5. Dealing with Failure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
Recoverable Failures 99
The Exists Watch and the Disconnected Event 102
Unrecoverable Failures 103
Leader Election and External Resources 104
Takeaway Messages 108
6. ZooKeeper Caveat Emptor. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
Using ACLs 109
Built-in Authentication Schemes 110
SASL and Kerberos 113
Adding New Schemes 113
Session Recovery 113
Version Is Reset When Znode Is Re-Created 114
The sync Call 114
Ordering Guarantees 116
Order in the Presence of Connection Loss 116
Order with the Synchronous API and Multiple Threads 117
Order When Mixing Synchronous and Asynchronous Calls 118
Data and Child Limits 118
Embedding the ZooKeeper Server 118
Takeaway Messages 119
7. The C Client. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
Setting Up the Development Environment 121
Starting a Session 122
Bootstrapping the Master 124
Taking Leadership 130
Assigning Tasks 132
Single-Threaded versus Multithreaded Clients 136
Takeaway Messages 138
8. Curator: A High-Level API for ZooKeeper. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
The Curator Client 139
Fluent API 140
Listeners 141
State Changes in Curator 143
A Couple of Edge Cases 144
Table of Contents | v
Recipes 144
Leader Latch 144
Leader Selector 146
Children Cache 149
Takeaway Messages 151
Part III. Administering ZooKeeper
9. ZooKeeper Internals. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
Requests, Transactions, and Identifiers 156
Leader Elections 157
Zab: Broadcasting State Updates 161
Observers 166
The Skeleton of a Server 167
Standalone Servers 167
Leader Servers 168
Follower and Observer Servers 169
Local Storage 170
Logs and Disk Use 170
Snapshots 172
Servers and Sessions 173
Servers and Watches 174
Clients 175
Serialization 175
Takeaway Messages 176
10. Running ZooKeeper. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
Configuring a ZooKeeper Server 178
Basic Configuration 179
Storage Configuration 179
Network Configuration 181
Cluster Configuration 183
Authentication and Authorization Options 186
Unsafe Options 186
Logging 188
Dedicating Resources 189
Configuring a ZooKeeper Ensemble 190
The Majority Rules 190
Configurable Quorums 191
Observers 193
Reconfiguration 193
vi | Table of Contents
Managing Client Connect Strings 197
Quotas 200
Multitenancy 201
File System Layout and Formats 202
Transaction Logs 202
Snapshots 203
Epoch Files 204
Using Stored ZooKeeper Data 205
Four-Letter Words 205
Monitoring with JMX 207
Connecting Remotely 213
Tools 214
Takeaway Messages 214
Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215