We're updating the issue view to help you get more done. 

Repeated transactions on same object exceeds memory capacity

Description

After executing a few hundred or a few thousand simple read/write transactions against a single object of size 1KB in RAMCloud (reading the value, writing the exact value back, and committing the transaction), the master server reports that it is out of memory indefinitely. Presumably the server should run the log cleaner to free up space, but this doesn't appear to be happening.

The code for this test case can be found here (TxMemoryTestCase), along with all the instructions necessary to setup a cluster and run the test case:
https://github.com/ellitron/RAMCloudClientTests

The client reports the following:

{{1483833005.315541431 MemoryTestCase.cc:110 in main NOTICE[1]: Executing transaction 805

1483833005.315569671 MemoryTestCase.cc:110 in main NOTICE[1]: Executing transaction 806

1483833005.315597629 MemoryTestCase.cc:110 in main NOTICE[1]: Executing transaction 807

1483833005.315679432 RpcWrapper.cc:220 in isReady NOTICE[1]: Server basic+infud:host=rc01,lid=32,qpn=1067 returned STATUS_RETRY from TX_PREPARE request: Log is out of space! Transaction abort-vote wasn't logged.
1483833010.316025864 RpcWrapper.cc:220 in isReady NOTICE[1]: (3284 duplicates of this message were skipped) Server basic+infud:host=rc01,lid=32,qpn=1067 returned STATUS_RETRY from TX_PREPARE request: Log is out of space! Transaction abort-vote wasn't logged.
1483833010.316025864 ClientLeaseAgent.cc:54 in getLease NOTICE[1]: (696 duplicates of this message were skipped) Blocked waiting for lease to renew. 1483833015.316193006 RpcWrapper.cc:220 in isReady NOTICE[1]: (3258 duplicates of this message were skipped) Server basic+infud:host=rc01,lid=32,qpn=1067 returned STATUS_RETRY from TX_PREPARE request: Log is out of space! Transaction abort-vote wasn't logged.
1483833020.317365024 RpcWrapper.cc:220 in isReady NOTICE[1]: (3253 duplicates of this message were skipped) Server basic+infud:host=rc01,lid=32,qpn=1067 returned STATUS_RETRY from TX_PREPARE request: Log is out of space! Transaction abort-vote wasn't logged.
1483833025.318351744 RpcWrapper.cc:220 in isReady NOTICE[1]: (3265 duplicates of this message were skipped) Server basic+infud:host=rc01,lid=32,qpn=1067 returned STATUS_RETRY from TX_PREPARE request: Log is out of space! Transaction abort-vote wasn't logged.
...}}

The master server reports the following:

{{1483833005.315657915 AbstractLog.cc:308 in hasSpaceFor WARNING[6]: Memory capacity exceeded; must delete objects before any more new objects can be created
1483833010.316031530 AbstractLog.cc:308 in hasSpaceFor WARNING[6]: (6569 duplicates of this message were skipped) Memory capacity exceeded; must delete objects before any more new objects can be created
1483833015.316200668 AbstractLog.cc:308 in hasSpaceFor WARNING[6]: (6517 duplicates of this message were skipped) Memory capacity exceeded; must delete objects before any more new objects can be created
1483833020.317375052 AbstractLog.cc:308 in hasSpaceFor WARNING[6]: (6507 duplicates of this message were skipped) Memory capacity exceeded; must delete objects before any more new objects can be created
1483833025.318365799 AbstractLog.cc:308 in hasSpaceFor WARNING[6]: (6531 duplicates of this message were skipped) Memory capacity exceeded; must delete objects before any more new objects can be created
1483833030.319230306 AbstractLog.cc:308 in hasSpaceFor WARNING[6]: (4388 duplicates of this message were skipped) Memory capacity exceeded; must delete objects before any more new objects can be created
1483833035.320536727 AbstractLog.cc:308 in hasSpaceFor WARNING[6]: (3279 duplicates of this message were skipped) Memory capacity exceeded; must delete objects before any more new objects can be created}}

Logs are attached.

Environment

None

Status

Assignee

Unassigned

Reporter

Jonathan Ellithorpe

Labels

None

Priority

Blocker