RAMCloud servers do not currently manage their threads very efficiently. This page documents some of the problems with the current approach, and also serves as a collection point for design notes on a new "nanoscheduler" that might solve the problems.
Problems
- The degree of concurrency within a server is configured when it starts, for example by specifying the number of worker threads for each service. However, it's not clear how to set these configuration options. The "right" configuration depends on how many cores the machine has, and it could vary dynamically depending on workload.
- If all of the workers for server are busy, the dispatch thread queues any new requests. However, the turnaround time for dispatching requests from the queue is very slow. For example, in the YCSB "A" workload, it can take 3-5 microseconds between when a worker thread finishes its current request and when it receives a new request to start processing.
- In the YCSB "A" workload at 90% memory utilization, having 3 worker threads for the master service runs slower than having only 1 (12.4 kops/sec/server vs, 16.1 kops/sec/server).
- Housekeeping tasks such as the log cleaner compete for threads with the workers. It would be better to reduce the number of workers temporarily while the cleaners running so that the operating system does not have to deschedule any of RAMCloud's threads.
- There is no mechanism for the operating system to tell RAMCloud how many threads it's going to schedule, or for RAMCloud to pick which threads those are. A new OS API for this could be very useful (but, need to review Scheduler Activations to see if perhaps that already solved this problem).