On-surface brush

This patch adds an optional brush display mode that visualizes the intersection between the brush's sphere of influence and the mesh. The code is based off Jason Wilkins' GSoC branch.

The most significant change from the original code is that it does not require any copying or preservation of the GL depth buffer; instead the PBVH is used to partially redraw the necessary parts of the depth buffer. This could be a bit slower than making a copy of the depth buffer, but I think that relies too much on graphics cards/drivers behaving properly.

Unchanged, however, is a requirement on the stencil buffer. Currently the patch unconditionally adds the stencil buffer to the GL-initialization code for all platforms in GHOST. (I've only tested X11 here, the Mac and Win code I just copied over.) I'm not sure if this is considered safe, or if we should have a compile-time option, or a command-line startup option, or better run-time checking (i.e. if drawable request fails, create a new request without the stencil buffer bit.) Or some combination of these.

[psy-fi, 2012-01-19 12:55:07]

Did a first review on the OpenGL code with some comments. Looks OK to me. About
the stencil buffer, I don't think it will be problematic since it's an old
feature (GL 1.0), but it never hurts to be careful. I think most colour formats
with 24-bit depth support stencil too to make a nice 32-bit alignment in video
memory. So I expect we should be safe there.

http://codereview.appspot.com/5554059/diff/1/source/blender/editors/sculpt_pa...
File source/blender/editors/sculpt_paint/paint_cursor.c (right):

http://codereview.appspot.com/5554059/diff/1/source/blender/editors/sculpt_pa...
source/blender/editors/sculpt_paint/paint_cursor.c:547: 
The explanation is more or less as follows:

1) Draw the outer radius sphere and write 3 int the stencil buffer where the
sphere fails the depth test (this will be fragments occluded by the painted
mesh. GL_REPLACE in the second argument of glStencilOp means replace the stencil
value when the depth test fails). The stencil test is disabled here(GL_ALWAYS in
glStenciFunc means it always passes)

http://codereview.appspot.com/5554059/diff/1/source/blender/editors/sculpt_pa...
source/blender/editors/sculpt_paint/paint_cursor.c:551: 
2) If we have an inner radius, draw a second sphere, decrease the value by one
each time the sphere fails the depth test. What happens is that if just the back
faces of the inner sphere are occluded in that pixel we have 2 in the stencil
buffer, but if both sides of the sphere are occluded, we have 1.

This call to glStencilFunc is extraneous as I think GL_DECR always decreases by
one

http://codereview.appspot.com/5554059/diff/1/source/blender/editors/sculpt_pa...
source/blender/editors/sculpt_paint/paint_cursor.c:561: 
3) The step that remains is to obviously draw the intersection itself. Now we
use the stencil test to only draw on places where stencil equals 1 or 3. These
are places where the mesh is inside the outer sphere and outside the inner
sphere. On places where it is inside the inner sphere the stencil is zero so we
the stencil will fail there.

http://codereview.appspot.com/5554059/diff/1/source/blender/editors/sculpt_pa...
source/blender/editors/sculpt_paint/paint_cursor.c:567: glStencilOp(GL_KEEP,
GL_KEEP, GL_KEEP);
Again, extraneous glStencilOp call.

[psy-fi, 2012-01-19 13:04:31]

Small correction, also an extra observation, this method may fail if the brush
sphere is partially occluded by another part of the mesh. What will happen is
that the occluding mesh surface will make the depth test fail on the outer
sphere. I don't know how brush influence works but I think this will not be the
desired result. I will have to test it later.

http://codereview.appspot.com/5554059/diff/1/source/blender/editors/sculpt_pa...
File source/blender/editors/sculpt_paint/paint_cursor.c (right):

http://codereview.appspot.com/5554059/diff/1/source/blender/editors/sculpt_pa...
source/blender/editors/sculpt_paint/paint_cursor.c:561: 
I meant "On places where it is inside the inner sphere the stencil is two so the
stencil will fail there."

[nicholasbishop, 2012-02-13 19:21:04]

Fix texture overlay and update comment

[nicholasbishop, 2012-02-13 19:30:10]

On 2012/02/13 19:21:04, nicholasbishop wrote:
> Fix texture overlay and update comment

A couple updates:
* Thanks for the explanation of the stencil stuff psy-fi, I updated the comment.

* The texture overlay option now gets stenciled when wrap mode is on. Since
brush texture sampling is screen-space, there's no need to map the texture to
the object to draw it correctly. As with the existing texture overlay, this will
not draw the texture if the mapping mode is 3D (rather than Fixed or Tiled.) 3D
mode would require evaluating face vertices individually and drawing with e.g.
glColorMaterial. I'd consider this a separate feature though, prefer to wait on
implementing that.

A couple of additional notes:
* The existing overlay code draws textures in the "tiled" mapping mode across
the entire view, rather than just inside the brush circle. When stenciling is
on, it now gets clipped to the brush sphere. Both behaviors make some sense, but
maybe we should choose one and stick to it.
* The existing overlay code draw the texture alpha flipped, so transparent areas
are darkest. This is a bit counter-intuitive to me, especially since the radial
control (FKEY) does not invert. Again, might be nice to choose one.

[nicholasbishop, 2012-02-13 21:36:08]

Sergey noted on IRC that the on-surface brush can look bad with the layer brush.
This is due to the height of the layer brush not being limited to the brush
size. We could "fix" by adding such a limit, but not sure if in other cases a
very tall layer brush is useful.

The grab brush has a similar issue, once the grab "spike" becomes sufficiently
long to exit the brush sphere, the outer border of the sphere gets drawn over
the spike.

[Jason.A.Wilkins, 2012-02-16 21:54:32]

I had z-fighting show up on some platforms when using this approach to draw the
"wrap" mode overlay (where the texture map is shown on the surface).

I would not be afraid of the z-buffer approach since it is a pretty fundamental
part of current graphics APIs and shouldn't be broken.  

However I do not see anything wrong with removing that requirement if z-fighting
is not a problem.