Fix for #36226, Select Linked works not in touch with Prefs.

3 parts:

1) To allow distinguishing uninitialized (not set) properties in the keymap items, a few changes to the RNA struct comparison function are needed: Instead of allowing only strict/non-strict comparison of 2 properties A and B in a struct, this now has 3 modes:
* STRICT: compare only the actual property values (same as 'strict' before)
* UNSET_MATCH_ANY: if either A or B is unset, consider them a match (same as non-strict before)
* UNSET_MATCH_NONE: if one property is set and the other not, consider them a mismatch.

The new UNSET_MATCH_NONE mode is useful for keymaps, because it allows keeping user-defined property values in the keymap even if they match the default property value (see wm_keymap_diff function in wm_keymap.c)

2) Fix for the RNA_property_is_idprop function: This was checking the "magic" flag, which is really only useful internally in RNA. Use the PROP_IDPROPERTY flag instead to identify id properties which can be unset (i.e. delete the underlying id property storage). The function is not used anywhere sofar, so no risk of functionality breaking.

3) A new operator is added for unsetting ID properties in the RMB context menu, next to the "Reset" operator. This only works on ID properties and deletes the ID property storage, so that the default value is used.

In the user preferences for keymaps the properties are shown in an inactive layout to indicate that the default value is used (which some operators such as the "select linked" op from the report use to trigger automatic behavior). When the user sets a property it gets set and stays that way until explicitly "unset" using the new operator.

[ideasman42, 2013-09-19 09:15:37]

Found one issue I'm hesitant about changing and some minor issues, otherwise
LGTM code-wise.

On the usability side would be good to have other devs check before committing
into trunk since it changes how keymap editing works.

https://codereview.appspot.com/13244048/diff/1/source/blender/editors/interfa...
File source/blender/editors/interface/interface_ops.c (right):

https://codereview.appspot.com/13244048/diff/1/source/blender/editors/interfa...
source/blender/editors/interface/interface_ops.c:508: static int
unset_property_button_poll(bContext *C)
This poll() function is quite intensive, it loops over button area blocks and
searches for the button.

We try to keep poll functions fairly minimal - I think the poll function could
be left out or just check ED_operator_regionactive or so. The menu item can be
left out if unset isnt possible.

https://codereview.appspot.com/13244048/diff/1/source/blender/editors/interfa...
source/blender/editors/interface/interface_ops.c:520: static int
unset_property_button_exec(bContext *C, wmOperator *UNUSED(op))
very similar to reset_default_button_exec(). could de-duplicate here.

https://codereview.appspot.com/13244048/diff/1/source/blender/editors/interfa...
File source/blender/editors/interface/interface_templates.c (right):

https://codereview.appspot.com/13244048/diff/1/source/blender/editors/interfa...
source/blender/editors/interface/interface_templates.c:3374: /* add property */
would add /* XXX, check on a better way to denote unset-state */

https://codereview.appspot.com/13244048/diff/1/source/blender/makesrna/intern...
File source/blender/makesrna/intern/rna_access.c (right):

https://codereview.appspot.com/13244048/diff/1/source/blender/makesrna/intern...
source/blender/makesrna/intern/rna_access.c:4968: return (prop->flag &
PROP_IDPROPERTY) != 0;
Not sure about this commit, could have bad side-effects (since relase is coming
up).
In the case where you're using this, you can use
RNA_struct_idprops_check(ptr->type) instead of checking the prop and modifying
this function.

[lukas.toenne1, 2013-09-19 10:26:31]

https://codereview.appspot.com/13244048/diff/1/source/blender/editors/interfa...
File source/blender/editors/interface/interface_ops.c (right):

https://codereview.appspot.com/13244048/diff/1/source/blender/editors/interfa...
source/blender/editors/interface/interface_ops.c:508: static int
unset_property_button_poll(bContext *C)
On 2013/09/19 09:15:37, ideasman42 wrote:
> This poll() function is quite intensive, it loops over button area blocks and
> searches for the button.
> 
> We try to keep poll functions fairly minimal - I think the poll function could
> be left out or just check ED_operator_regionactive or so. The menu item can be
> left out if unset isnt possible.

Done.

https://codereview.appspot.com/13244048/diff/1/source/blender/editors/interfa...
source/blender/editors/interface/interface_ops.c:520: static int
unset_property_button_exec(bContext *C, wmOperator *UNUSED(op))
On 2013/09/19 09:15:37, ideasman42 wrote:
> very similar to reset_default_button_exec(). could de-duplicate here.

True. Added operator_button_property_finish function for doing update calls and
return value in the success case.

https://codereview.appspot.com/13244048/diff/1/source/blender/editors/interfa...
File source/blender/editors/interface/interface_templates.c (right):

https://codereview.appspot.com/13244048/diff/1/source/blender/editors/interfa...
source/blender/editors/interface/interface_templates.c:3374: /* add property */
On 2013/09/19 09:15:37, ideasman42 wrote:
> would add /* XXX, check on a better way to denote unset-state */

Will try different UI concepts, leaving as it is here for now.

https://codereview.appspot.com/13244048/diff/1/source/blender/makesrna/intern...
File source/blender/makesrna/intern/rna_access.c (right):

https://codereview.appspot.com/13244048/diff/1/source/blender/makesrna/intern...
source/blender/makesrna/intern/rna_access.c:4968: return (prop->flag &
PROP_IDPROPERTY) != 0;
On 2013/09/19 09:15:37, ideasman42 wrote:
> Not sure about this commit, could have bad side-effects (since relase is
coming
> up).
> In the case where you're using this, you can use
> RNA_struct_idprops_check(ptr->type) instead of checking the prop and modifying
> this function.

This function does not seem to be used anywhere yet, so would not risk breakage.
It also doesn't work as expected, which is why i changed it in the first place.

RNA_struct_idprops_check also doesn't tell whether a property uses ID props,
just whether the struct generally supports them.

Then again, the PROP_IDPROPERTY is checked in RNA_property_unset anyway, so not
strictly necessary to check in execute (in poll it would be useful but this was
removed as you suggested). All it does it prevent unnecessary updates when
unsetting non-ID properties, which has no effect.

Removing this change for the time being.

[lukas.toenne1, 2013-09-19 10:27:47]

Fixed some issues pointed out by ideasman42

[lukas.toenne1, 2013-09-19 10:30:51]

Minor compiler error fix

[lukas.toenne1, 2013-09-19 14:21:02]

UI improvements

[lukas.toenne1, 2013-09-19 14:24:13]

Made some UI improvements:
* Buttons get a new flag for drawing unset properties. This can be activated by
passing an according flag to uiItem functions. It is only used in the keymap
user prefs, no need to show this info in other places so far.

* Tooltips get an additional *UNSET* string attached for unset properties. This
also happens only in the keymap user prefs.