Waking up worker threads

Owned by John Ousterhout
Last updated: Jul 05, 2011Version comment
6 min read

In the current version of RAMCloud worker threads pull while waiting for new work, but put themselves to sleep if 100 microseconds elapse and no new requests have arrived. Mohammad Alizadeh and Masato Yasuda discovered that it can be expensive to wake them up again once they have been put to sleep. Here is an e-mail describing their measurements, in which they space out requests by different amounts of time and measure the round-trip latency for short reads. By increasing the polling period the overheads for waking up the worker thread dropped again.

We see that with 100ms polling, there still is a slight increase in latency as we 
go to 10 and 20ms inter-read period, but it's less than 2us on average. When we use a 200ms 
period, we again see a latency around 65us. I've verified that the same behavior holds
with DEBUG=yes; only the numbers are somewhat higher.

We'll use a 100ms polling period for the rest of our experiments. Any issues 
we should be worried about with this?

Mohammad

##########################################################
# DEBUG=no, pollMicros=100us

--- 0
Mean Latency for 100B reads: 5.34008 us -- 100000 samples
Min Latency for 100B reads: 4.359 us
Max Latency for 100B reads: 75.695 us
25th percentile Latency for 100B reads: 4.518 us
50th percentile Latency for 100B reads: 5.756 us
75th percentile Latency for 100B reads: 5.905 us
90th percentile Latency for 100B reads: 6.389 us
99th percentile Latency for 100B reads: 7.655 us

--- 50us
Mean Latency for 100B reads: 5.41838 us -- 100000 samples
Min Latency for 100B reads: 4.377 us
Max Latency for 100B reads: 106.523 us
25th percentile Latency for 100B reads: 4.572 us
50th percentile Latency for 100B reads: 5.773 us
75th percentile Latency for 100B reads: 5.936 us
90th percentile Latency for 100B reads: 6.377 us
99th percentile Latency for 100B reads: 7.804 us

--- 100us
Mean Latency for 100B reads: 5.72131 us -- 100000 samples
Min Latency for 100B reads: 4.364 us
Max Latency for 100B reads: 99.841 us
25th percentile Latency for 100B reads: 4.59 us
50th percentile Latency for 100B reads: 5.784 us
75th percentile Latency for 100B reads: 6.094 us
90th percentile Latency for 100B reads: 8.22 us
99th percentile Latency for 100B reads: 10.444 us

--- 200us
Mean Latency for 100B reads: 8.27036 us -- 100000 samples
Min Latency for 100B reads: 4.896 us
Max Latency for 100B reads: 114.699 us
25th percentile Latency for 100B reads: 7.371 us
50th percentile Latency for 100B reads: 8.287 us
75th percentile Latency for 100B reads: 8.855 us
90th percentile Latency for 100B reads: 9.365 us
99th percentile Latency for 100B reads: 11.066 us

--- 300us
Mean Latency for 100B reads: 9.15849 us -- 100000 samples
Min Latency for 100B reads: 4.736 us
Max Latency for 100B reads: 117.281 us
25th percentile Latency for 100B reads: 7.46 us
50th percentile Latency for 100B reads: 8.421 us
75th percentile Latency for 100B reads: 8.895 us
90th percentile Latency for 100B reads: 9.589 us
99th percentile Latency for 100B reads: 51.104 us

--- 400us
Mean Latency for 100B reads: 22.234 us -- 100000 samples
Min Latency for 100B reads: 6.928 us
Max Latency for 100B reads: 111.516 us
25th percentile Latency for 100B reads: 7.583 us
50th percentile Latency for 100B reads: 8.901 us
75th percentile Latency for 100B reads: 49.068 us
90th percentile Latency for 100B reads: 52.687 us
99th percentile Latency for 100B reads: 70.758 us

--- 500us
Mean Latency for 100B reads: 39.1592 us -- 100000 samples
Min Latency for 100B reads: 6.853 us
Max Latency for 100B reads: 155.294 us
25th percentile Latency for 100B reads: 8.969 us
50th percentile Latency for 100B reads: 49.707 us
75th percentile Latency for 100B reads: 64.993 us
90th percentile Latency for 100B reads: 69.404 us
99th percentile Latency for 100B reads: 73.52 us

--- 1ms
Mean Latency for 100B reads: 58.2559 us -- 100000 samples
Min Latency for 100B reads: 6.602 us
Max Latency for 100B reads: 175.089 us
25th percentile Latency for 100B reads: 52.57 us
50th percentile Latency for 100B reads: 66.168 us
75th percentile Latency for 100B reads: 69.771 us
90th percentile Latency for 100B reads: 71.802 us
99th percentile Latency for 100B reads: 81.252 us

--- 10ms
Mean Latency for 100B reads: 65.2651 us -- 10000 samples
Min Latency for 100B reads: 11.972 us
Max Latency for 100B reads: 149.737 us
25th percentile Latency for 100B reads: 66.765 us
50th percentile Latency for 100B reads: 69.395 us
75th percentile Latency for 100B reads: 71.258 us
90th percentile Latency for 100B reads: 73.159 us
99th percentile Latency for 100B reads: 81.85 us

--- 20ms
Mean Latency for 100B reads: 63.4412 us -- 10000 samples
Min Latency for 100B reads: 12.531 us
Max Latency for 100B reads: 174.477 us
25th percentile Latency for 100B reads: 54.566 us
50th percentile Latency for 100B reads: 68.907 us
75th percentile Latency for 100B reads: 71.214 us
90th percentile Latency for 100B reads: 73.165 us
99th percentile Latency for 100B reads: 83.254 us

######################################################
# DEBUG=no, pollMicros=100ms

--- 0
Mean Latency for 100B reads: 5.362 us -- 100000 samples
Min Latency for 100B reads: 4.367 us
Max Latency for 100B reads: 105.105 us
25th percentile Latency for 100B reads: 4.542 us
50th percentile Latency for 100B reads: 5.783 us
75th percentile Latency for 100B reads: 5.928 us
90th percentile Latency for 100B reads: 6.384 us
99th percentile Latency for 100B reads: 7.662 us

--- 50us
Mean Latency for 100B reads: 5.43884 us -- 100000 samples
Min Latency for 100B reads: 4.366 us
Max Latency for 100B reads: 103.341 us
25th percentile Latency for 100B reads: 4.554 us
50th percentile Latency for 100B reads: 5.792 us
75th percentile Latency for 100B reads: 5.939 us
90th percentile Latency for 100B reads: 6.438 us
99th percentile Latency for 100B reads: 7.757 us

--- 100us
Mean Latency for 100B reads: 5.49459 us -- 100000 samples
Min Latency for 100B reads: 4.375 us
Max Latency for 100B reads: 98.99 us
25th percentile Latency for 100B reads: 4.571 us
50th percentile Latency for 100B reads: 5.794 us
75th percentile Latency for 100B reads: 6.127 us
90th percentile Latency for 100B reads: 6.48 us
99th percentile Latency for 100B reads: 7.93 us

--- 200us
Mean Latency for 100B reads: 5.49273 us -- 100000 samples
Min Latency for 100B reads: 4.384 us
Max Latency for 100B reads: 112.433 us
25th percentile Latency for 100B reads: 4.57 us
50th percentile Latency for 100B reads: 5.792 us
75th percentile Latency for 100B reads: 6.029 us
90th percentile Latency for 100B reads: 6.495 us
99th percentile Latency for 100B reads: 7.897 us

--- 300us
Mean Latency for 100B reads: 5.55089 us -- 100000 samples
Min Latency for 100B reads: 4.431 us
Max Latency for 100B reads: 110.76 us
25th percentile Latency for 100B reads: 4.724 us
50th percentile Latency for 100B reads: 5.838 us
75th percentile Latency for 100B reads: 6.12 us
90th percentile Latency for 100B reads: 6.492 us
99th percentile Latency for 100B reads: 7.905 us

--- 400us
Mean Latency for 100B reads: 5.67296 us -- 100000 samples
Min Latency for 100B reads: 4.392 us
Max Latency for 100B reads: 134.91 us
25th percentile Latency for 100B reads: 4.804 us
50th percentile Latency for 100B reads: 5.891 us
75th percentile Latency for 100B reads: 6.232 us
90th percentile Latency for 100B reads: 6.665 us
99th percentile Latency for 100B reads: 8.186 us

--- 500us
Mean Latency for 100B reads: 5.83287 us -- 100000 samples
Min Latency for 100B reads: 4.67 us
Max Latency for 100B reads: 110.55 us
25th percentile Latency for 100B reads: 5.039 us
50th percentile Latency for 100B reads: 6.011 us
75th percentile Latency for 100B reads: 6.389 us
90th percentile Latency for 100B reads: 6.921 us
99th percentile Latency for 100B reads: 8.69 us

--- 1ms
Mean Latency for 100B reads: 5.95246 us -- 100000 samples
Min Latency for 100B reads: 4.858 us
Max Latency for 100B reads: 104.654 us
25th percentile Latency for 100B reads: 5.051 us
50th percentile Latency for 100B reads: 6.305 us
75th percentile Latency for 100B reads: 6.506 us
90th percentile Latency for 100B reads: 6.906 us
99th percentile Latency for 100B reads: 8.412 us

--- 10ms
Mean Latency for 100B reads: 6.78109 us -- 10000 samples
Min Latency for 100B reads: 5.631 us
Max Latency for 100B reads: 75.12 us
25th percentile Latency for 100B reads: 5.837 us
50th percentile Latency for 100B reads: 7.063 us
75th percentile Latency for 100B reads: 7.29 us
90th percentile Latency for 100B reads: 7.995 us
99th percentile Latency for 100B reads: 9.691 us

--- 20ms
Mean Latency for 100B reads: 7.06089 us -- 10000 samples
Min Latency for 100B reads: 5.757 us
Max Latency for 100B reads: 75.19 us
25th percentile Latency for 100B reads: 6.097 us
50th percentile Latency for 100B reads: 7.242 us
75th percentile Latency for 100B reads: 7.603 us
90th percentile Latency for 100B reads: 8.382 us
99th percentile Latency for 100B reads: 10 us

--- 200ms
Mean Latency for 100B reads: 68.6457 us -- 1000 samples
Min Latency for 100B reads: 28.151 us
Max Latency for 100B reads: 149.218 us
25th percentile Latency for 100B reads: 57.599 us
50th percentile Latency for 100B reads: 72.539 us
75th percentile Latency for 100B reads: 75.953 us
90th percentile Latency for 100B reads: 81.025 us
99th percentile Latency for 100B reads: 108.116 us