...
- The coordinator decides that a server is dead and initiates recovery. I will use the term "condemned" to Let's refer to such a master as condemned.
- The condemned server isn't actually dead (perhaps it was temporarily partitioned, or was unable to respond to "ping" requests because of a temporary heavy load, etc.). ILet'll s use the term "zombie" to refer to this such a master; note that not all condemned masters are zombies but, by definition, all zombies are condemned.
- After recovery, a recovery master begins accepting requests for some of the condemned masterzombie's data, including writes.
- Meanwhile, the zombie continues to serve requests, causing clients to see stale data.
...
- At the beginning of recovery the coordinator contacts every server in the cluster. This is already necessary for the proper operation of recovery; it means that once recovery has begun every server in the cluster knows that a master has been condemned.
- If the condemned master is not really dead and then it will continue to send PING requests to other servers every 10ms.
- When another server receives a PING from a condemned server it responds with a special "Dude, you're a zombie!" response. If a server receives such a response it places itself in a limbo state, which means it will not service any requests from clients until it has contacted the coordinator. This is analogous to expiration of a lease.
- If a limbo server contacts the coordinator, the coordinator can respond in one of two ways:
- It can tell the condemned master that it no longer owns any tablets.
- It can pardon the condemned master, tell it to continue normal operation, and abort any recovery that was in progress for the master. This is analogous to renewing the master's lease.
- If a server fails to receive a response from a PING request, it must contact the coordinator as described above so that the coordinator can decide whether the PING target should be recovered. But, in addition, whenever a PING times out a server must place itself into limbo state until it has contacted the coordinator. This is necessary to handle partitions where a zombie loses communication with the rest of the cluster.
- Finally, if a server receives a PING request when it is in limbo state it indicates that fact in its response, and the server sending the PING request must put itself in limbo and contact the coordinator. This ensures that, in the event of a partition, information about the partition spreads rapidly throughout a disconnected group of servers (see below for an example).
...