Q1: Can we have just one thread pool instead?
The currently running Arachne user thread yields
The main loop polls to check if it needs to call sched_yield() to relinquish the core
Q1: Should Arachne know about the blocked user thread?
hq6: There does not seem to be much use in Arachne being aware of the blocked user thread, since it is blocked due to a kernel operation, and there's not much Arachne could do with that information.
Q2: How do we notice the blocking syscall finishes if we have disabled the interrupts to achieve full core isolation?
hq6: It may make sense to keep device interrupts, and disable only timer interrupts.
Q3: Should we consider the possibility of async. syscall mechanism (e.g., FlexSC, Linux Syslets, etc.)?
hq6: Time permitting, we should experiment with the FlexSC idea of piling up syscalls on a remote core and measure application-level throughput and latency under load. The hidden costs of context switching between the user and kernel that FlexSC aims to reduce still exists in our scheme above.
Q4: What if we have too many threads blocking in the syscalls at the same time? Should we try to cap the size of the thread pool?
hq6: This seems like a self-limiting problem, in the sense that if the core is spending a large proportion of time in the kernel and handling interrupts due to IO completions, then it will not have much time to accept many new user threads to issue further syscalls.
Q5: Is the scheduling class going to be notified when threads are blocked on syscalls and woken?
Yes, the hook functions `dequeue_task` and `task_woken` of the `sched_class` interface (defined at http://lxr.free-electrons.com/source/kernel/sched/sched.h#L1193) will be called respectively.
Scheduling policies currently available in Linux 4.7 are defined at: http://lxr.free-electrons.com/source/include/uapi/linux/sched.h?v=4.7#L35.
The scheduling policy used by a particular thread is indicated by the `policy` field in `task_struct`: http://lxr.free-electrons.com/source/include/linux/sched.h?v=4.7#L1495.
This is a simplistic Arachne main loop implementation that achieves the functionality mentioned above.
void Arachne::MainLoop() { while (1) { while (!relinquishThisCore() && !yieldToUnblockedSibling()) { // Pick the next Arachne user thread to run // and switch to its context; this function // returns when the Arachne user thread // decides to yield the control schedule_next_thread(); } // If this thread have been downgraded and // lost its binding to the core, just exit the // the main loop and terminate gracefully if (!boundToThisCore()) return; sched_yield(); } } |
(DYNAMIC) FULL CORE ISOLATION
Changes to the load balancer:
Move non-CPU-bound interrupts to non-isolated cores
Reduce CPU-bound interrupts as much as possible
Turn off the scheduler tick entirely
TO BE CONTINUED...