Threaded object update and EvaluationContext

Made objects update happening from multiple threads. It is a task-based scheduling system which uses current dependency graph for spawning new tasks. This means threading happens on object level, but the system is flexible enough for higher granularity.

Technical details:

- Uses task scheduler which was recently commited to trunk (that one which Brecht ported from Cycles).
- Added two utility functions to dependency graph:
  * DAG_threaded_update_begin, which is called to initialize threaded objects update. It will also schedule root DAG node to the queue, hence starting evaluation process.

    Initialization will fill in DAG nodes valency (valency is how much dependencies which are not updated yet current node does have). This valency is used by task threads for faster detecting which nodes might be scheduled.

  * DAG_threaded_update_handle_node_updated which is called from task thread function when node was fully handled.

    This function decreases valency of node childs an schedules childs with zero valency.

  As it might have become clear, task thread receives DAG nodes and decides which callback to call for it. Currently only BKE_object_handle_update is called for object DAG nodes.

  In the future it'll call node->callback() from Ali's new dependency graph.

- This required adding paradigma of derivedRender, which is used by modifiers when preparing scene for rendering.

  This is mainly needed to stop using get_object_dm() from modifiers' apply callback. Such a call was only a workaround for dependency graph glitch when rendering scene with, say, boolean modifiers before displaying this scene.

  This derivedReder is stored in Object and being created by database_init_objects and it happens in threads (the same way as it's done for scene update).

  Unfortunately, this bumps memory usage, but could not see how to actually solve it atm. using Locked interface from GSoC branch will help here,

- Added paradigma of EvaluaitonContext. Currently it's more like just a replacement for G.is_rendering which fails in some circumstances.

  Future idea of this context is to also store all the local data needed for objects evaluation such as local time, copied objects (by Copy-on-Write system).

  So now there're two types of EvaluationContext:
  * Context used for viewport updated and owned by Main. in the future this context might be easily moved to Window or Screen to allow per-window/per-screen local time.

  * Context used by render engines to evaluate objects for render purposes. Render engine is an owner of this context.

  This context is passed to all object update routines.

There still might be some TODOs in the patch, but no actual stoppers i think. And it might be some left-overs from unfinished work in the branch. Also we might want to have "locked interface rendering" perhaps?

[brechtvl, 2013-10-29 16:27:25]

Some initial comments from first pass reading code, still need to dive into
scene.c, depsgraph.c and convertblender.c.

https://codereview.appspot.com/15340044/diff/1/b/source/blender/blenkernel/BK...
File b/source/blender/blenkernel/BKE_depsgraph.h (right):

https://codereview.appspot.com/15340044/diff/1/b/source/blender/blenkernel/BK...
b/source/blender/blenkernel/BKE_depsgraph.h:121: +void DAG_threaded_init(void);
Maybe leave out the "threaded" part of these function names?

https://codereview.appspot.com/15340044/diff/1/b/source/blender/blenkernel/BK...
File b/source/blender/blenkernel/BKE_scene.h (right):

https://codereview.appspot.com/15340044/diff/1/b/source/blender/blenkernel/BK...
b/source/blender/blenkernel/BKE_scene.h:117: +/* TODO(sergey): Find a better
place for this. */
The dependency graph header file perhaps?

https://codereview.appspot.com/15340044/diff/1/b/source/blender/blenkernel/BK...
b/source/blender/blenkernel/BKE_scene.h:123: +void
BKE_scene_update_tagged_ex(struct EvaluationContext *evaluation_context, struct
Main *bmain, struct Scene *scene, bool use_threads);
When do you not want this to be threaded? Are there cases where this has thread
safety issues, or is it more done because you haven't tested using threaded
updates everywhere?

https://codereview.appspot.com/15340044/diff/1/b/source/blender/blenkernel/de...
File b/source/blender/blenkernel/depsgraph_private.h (right):

https://codereview.appspot.com/15340044/diff/1/b/source/blender/blenkernel/de...
b/source/blender/blenkernel/depsgraph_private.h:97: +	uint32_t valency;  /*
valency of the node is a number of parents which are not updated yet
This could be named better I think, valency is a pretty generic name and doesn't
really say anything about things being updated. Maybe "num_pending_parents".

https://codereview.appspot.com/15340044/diff/1/b/source/blender/blenkernel/in...
File b/source/blender/blenkernel/intern/constraint.c (right):

https://codereview.appspot.com/15340044/diff/1/b/source/blender/blenkernel/in...
b/source/blender/blenkernel/intern/constraint.c:3355: +		 *               render
flag for cnstraints. before this use viewport dm as it
typo: cnstraints (same further in the file)

https://codereview.appspot.com/15340044/diff/1/b/source/blender/editors/rende...
File b/source/blender/editors/render/render_internal.c (right):

https://codereview.appspot.com/15340044/diff/1/b/source/blender/editors/rende...
b/source/blender/editors/render/render_internal.c:52: +#include "BKE_curve.h"
Adding these includes seems kind of unnecessary for just a whitespace change? :)

https://codereview.appspot.com/15340044/diff/1/b/source/blender/editors/scree...
File b/source/blender/editors/screen/screen_edit.c (right):

https://codereview.appspot.com/15340044/diff/1/b/source/blender/editors/scree...
b/source/blender/editors/screen/screen_edit.c:1441:
+		BKE_scene_set_background(bmain, sc->scene);
It makes sense to do this update here, I would say the main todo in this case is
that it would be better if we could detect if anything actually changes that
requires a depsgraph rebuild, but probably not so important to solve now.

https://codereview.appspot.com/15340044/diff/1/b/source/blender/makesrna/inte...
File b/source/blender/makesrna/intern/rna_scene_api.c (right):

https://codereview.appspot.com/15340044/diff/1/b/source/blender/makesrna/inte...
b/source/blender/makesrna/intern/rna_scene_api.c:60: +	/* TODO(sergey): Use
context from G.main? */
This one is a bit complicated, for cycles (and other external render engines)
motion blur we need to be able to do this update for render, and not just for
the viewport. Some possible solutions, one is to provide an option to do this
for render, or we could add a function to RenderEngine that should be used
instead of this one and uses the evaluation context owned by the render, or we
expose the concept of an evaluation context to the API somehow.

I think we aren't quite ready yet to expose this to the API too much, so perhaps
this needs to be postponed a bit until more of the depsgraph work is getting
implemented and not part of this patch.

[brechtvl, 2013-10-29 17:05:47]

This file crashes when rendering with the patch:
lib/tests/particles_and_hair/particle_dupligroups.blend

https://codereview.appspot.com/15340044/diff/1/b/source/blender/render/intern...
File b/source/blender/render/intern/source/convertblender.c (right):

https://codereview.appspot.com/15340044/diff/1/b/source/blender/render/intern...
b/source/blender/render/intern/source/convertblender.c:5098: +static void
create_derivedRender_add_task(void *node, void *user_data);
The whole system in this file seems to be missing for external engines like
Cycles? I'm not sure what we decided here, to postpone this derivedRender or
solve it, but in current state I don't think we can merge it because it breaks
Cycles render in many cases (for example motion blur).

[sergey.vfx, 2013-12-11 10:35:48]

Let's move to http://developer.blender.org/D94