Issue 5847059: Rewrite the scheduler, for simplicity, and better error handling.
DescriptionRewrite the scheduler, for simplicity, and better error handling.
During a recent code audit, a number of issues with the relation hook scheduler
were discovered.
Say we get notified of three membership events from the same
child watch, two joins and a depart. if the first join hook
errors, we need to keep both it and the subsequent membership
events around.
Previously on error we'd execute the next few hooks,
regardless of the error, which is problematic. The error handler will stop
the scheduler, but we'll still end up processing the events since the
executioner doesn't check if its running till its done processing a clock.
On error we need to stop processing immediately, we also need
to keep all events for the around, including the failed
one, to properly support retry (albeit still subject to reduction via
additional events.)
On a resolved --retry, we should continue the event processing stream where
we left off (ie start execution against the failed hook.)
The branch includes expanded unit test coverage of various failure scenarios,
and concurrent activity scenarios, as well as two new implementations of the
scheduler (one buried in the history which attempts to fix the previous impl).
The latest schduler implementation, does away with the notion of clocks and
instead uses a simple list (wrapped in a deferred queue)
as the primary data structure. The additional member_versions, context_members
data structures are used only at ingest/put time, the consumer relies soley on
the queue. This simplifies the scheduler implementation, and avoided some
issues with the previous implementation that had to bookeep across multiple data
structures in the face of concurrent activity.
The alias expansion of join to modified was also removed from the lifecycle
and placed directly in the scheduler. This makes it more reliable in the face of an
error of either join/modified hook, and subjects the additional synthetic event
to reduction.
https://code.launchpad.net/~hazmat/juju/scheduler-peek-list/+merge/98104
(do not edit description out of merge proposal)
Patch Set 1 #
Total comments: 14
MessagesTotal messages: 3
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
