In a recent run of Jonathan's test case, a server crashed you the beginning of the test with the following log message:
1452986298.788889565 Logger.cc:800 in assertionError ERROR: Assertion `rpcs[rpcId % len].id == rpcId' failed at /home/ouster/remote/ramcloud/src/UnackedRpcResults.cc:669 in void RAMCloud::UnackedRpcResults::Client::updateResult(uint64_t, void*)
I was able to extract the following information from the stack trace in the log:
This was invoked from UnackedRpcResults::recordCompletion, line 381,
which was invoked from UnackedRpcHandle::~UnackedRpcHandle, line 99,
which was invoked from the destructor for "std::vector<UnackedRpcHandle> rpcHandles", declared in MasterService::txPrepare, line 2790
I have only seen this problem happen once.