Child pages
  • Failures
Skip to end of metadata
Go to start of metadata

Failure Brainstorm

Failure Modes

  • Loss of machine
    • What does this mean? When we label a misbehaving machine as toast?
    • Multiple machine types: client, master, backup (separate service colocated w/ master), coordinator
  • Loss of rack
    • Common failure scenario - must be able to handle gracefully
    • Implies we need # of racks >= # of copies of replicated data for durability
  • Network Partitions
    • Should not lose sanity when these occur.

Failure Symptoms

  • RPC Timeout
    • Client->Server or Server->Server request doesn't yield quick response
  • RPC Error
    • Client or Server node detects corrupt RPC, corrupt payload or server explicitly indicates error
  • Assertion failure
    • Internal software consistency violated; likely abort(3)'s
  • Error During Recovery
    • Many potentialities: backup sources die, reconstitution destinations die (e.g if distributed index reconstruction), reconstituting master dies, data missing/invalid/unsalvagable, etc.
  • Full power loss
    • All lights go out.
  • Partial power loss/Network partition
    • Part(s) of datacenter either goes down, or appears to, while other part(s) stay up.
  • Missing heartbeat
    • Controller or master nodes expected a regular liveness ping, but didn't get one.
  • Cluster coordinator dies/becomes unreachable
  • Client asks wrong server for object
    • Likely response: Not my data, not my problem.
  • Backup I/O Error
    • Backup disk/ssd/etc gives I/O error. Common enough to handle specially or just SIITH?
  • No labels