Linearizable RPC & TX progress

TODO for SOSP Camera Ready

• 1 M client benchmark / cstlee

• Full-mix for TPC-C benchmark / seojin

• Measure 1-level deeper for throughput (factor analysis?)
• Clarify the term linearizability: make proposals. / collin
• Talk about ultimate client / John??
• Clarify exp setup for TPC-C benchmark.
• Comparision with tell from VLDB.
• Extra paragraph lease

Tentative schedule for both linearizability and TX.

Nov 3: Today

Nov 14: TX design draft

Nov 21: Linearizability implementation

Jan 27: TX Implementation

Feb 27: 1st draft & Measurements

Mar 28: Final

Transaction Components (Assumes 4 + 1 days / wk)

Client Driven Transaction

• Prep Rpc and Handling - Seojin / DONE!
• Client driven transactions "task" - Collin / DONE!
• Transaction object & API - Collin /  DONE!
• Multiple objects per prepare RPC - DONE!

Transaction Cleanup Coordinator (Recovery Coordinator)

• Timer and Rpc for initiating recovery protocol. - Seojin / Half DONE.
• Request Abort Rpc and Handling - Seojin / 2 day
• Transaction Cleanup "task" - Collin / DONE!
• Transaction Cleanup crash recovery - Collin / DONE!

Common Components (Mostly in Data Master)

• Decision Rpc - seojin / DONE!
• lock table - Collin / DONE!
• Special GC rules for transaction entires (& lease expiration handling) - Seojin / 3 day
• Transaction log records & recovery - Seojin / DONE!

Benchmark

Micro-benchmarks

• Throughput and latency for TX varying # of objects and size of objects / READ vs. WRITE / # of servers. No contention.
• Varied contention
• (Conditional) Write latency for varied size of objects.
• Crash recovery for client / server?
• Scalability of leases?

Macro-benchmark

• Run TPC-C and compare with other systems (eg. H-store)
• YCSB?
• Ask Ankita and SYang

Current Client Side TimeTrace:

(+  21.5 ns): sendPrepareRpc find first object
(+  41.6 ns): sendPrepareRpc objectFinder.lookup
(+ 186.1 ns): sendPrepareRpc prepareRpcs.emplace_back
(+  72.3 ns): sendPrepareRpc objectFinder.lookup and service locator comparison
(+  43.6 ns): appendOp append data to rpc
(+  94.1 ns): sendPrepareRpc break loop -- objectFinder.lookup and service locator comparison
(+  13.3 ns): send call
(+  14.7 ns): send get ackId
(+ 456.2 ns): send session->sendRequest
(+  14.7 ns): sendPrepareRpc send return

(+  34.1 ns): processPrepareRpcs check for ready Rpcs
(+ 100.6 ns): processPrepareRpcs prepareRpcs.erase
=====================================================
(+1092.8 ns): total

Linearizability Components

• increment handling
• Coordinator RPC GetLeaseInfo w/ batching

Coordinator

• Cluster Clock and Time support - Collin
• Lease manager - Collin

Master

• Backup's log reading during recovery - Collin (6days)
• Recovery from log <logReplay> (6 days)
• Handling of Linearizable RPC in MasterService - Seojin (3 days)
• Durable logging - Seojin (3 days)
• Log cleaning for rpc log entry - Collin (4 days)
• In-memory client table (UnackedRpcResult) - Seojin

Client

• Linearizable RPC handling. (using RpcTracker and ClusterClock) - Seojin (3 days)
• Client lease - Collin (4 days)
• In-memory RPC table (RpcTracker) - Seojin