Fix for #1128: java.lang.String should be mapped to PyUnicode, not PyString

This has the potential to break a lot of things, but it is the right thing to do. I hope we are on time to introduce it on 2.5.

I've fixed all the regressions found by running the test suite, fixing the APIs that really need to return str objects. That was done explicitly declaring the return value as PyString on the Java site. On the Python side, I had to introduce StringUtil.asPyString as a helper to force the String -> PyString conversion (Another option is to do java_string.decode('iso8859-1'), but seems rather expensive)

[Leo Soto M., 2008-09-13 23:58:02]

same patch, ignoring whitespace changes

[Nicholas Riley, 2008-09-14 00:22:45]

comments

http://codereview.appspot.com/5655/diff/206/225
File jython/Lib/javapath.py (right):

http://codereview.appspot.com/5655/diff/206/225#newcode103
Line 103: return asPyString(File(sys.getPath(path)).exists())
This is a bool, doesn't need asPyString()

http://codereview.appspot.com/5655/diff/206/225#newcode108
Line 108: return asPyString(File(path).isAbsolute())
This is a bool, doesn't need asPyString()

http://codereview.appspot.com/5655/diff/206/225#newcode113
Line 113: return asPyString(File(sys.getPath(path)).isFile())
This is a bool, doesn't need asPyString()

http://codereview.appspot.com/5655/diff/206/225#newcode118
Line 118: return asPyString(File(sys.getPath(path)).isDirectory())
This is a bool, doesn't need asPyString()

http://codereview.appspot.com/5655/diff/206/227
File jython/Lib/os.py (right):

http://codereview.appspot.com/5655/diff/206/227#newcode113
Line 113: return asPyString(File(getcwd()))
This is called from Java code (jna-posix), not from Python code; it's fine if it
returns a Java string.

http://codereview.appspot.com/5655/diff/206/227#newcode630
Line 630: return StringUtil.asPyString(StringUtil.fromBytes(buf))
don't need the "StringUtil."

http://codereview.appspot.com/5655/diff/206/228
File jython/Lib/test/test_exceptions.py (right):

http://codereview.appspot.com/5655/diff/206/228#newcode342
Line 342: from test import test_support
looks unrelated, check in separately

http://codereview.appspot.com/5655/diff/206/224
File jython/src/com/ziclix/python/sql/handler/MySQLDataHandler.java (right):

http://codereview.appspot.com/5655/diff/206/224#newcode57
Line 57: // XXX: Only works with ASCII data!
tabs? looks like indentation is messed up here

http://codereview.appspot.com/5655/diff/206/208
File jython/src/org/python/core/PyStringMap.java (right):

http://codereview.appspot.com/5655/diff/206/208#newcode144
Line 144: buf.append(new PyString((String)key).__repr__().toString());
This seems a really complicated way to say:

"'" + key + "'"

http://codereview.appspot.com/5655/diff/206/209
File jython/src/org/python/core/PySystemState.java (right):

http://codereview.appspot.com/5655/diff/206/209#newcode105
Line 105: public static String byteorder = "big";
This needs fixing too.

http://codereview.appspot.com/5655/diff/206/207
File jython/src/org/python/core/adapter/ClassicPyObjectAdapter.java (right):

http://codereview.appspot.com/5655/diff/206/207#newcode63
Line 63: return new PyUnicode((String) o);
tabs?

http://codereview.appspot.com/5655/diff/206/210
File jython/src/org/python/core/util/StringUtil.java (right):

http://codereview.appspot.com/5655/diff/206/210#newcode89
Line 89: return new PyString(string);
perhaps toPyString (parallel with toBytes above)?

http://codereview.appspot.com/5655/diff/206/217
File jython/src/org/python/modules/time/Time.java (right):

http://codereview.appspot.com/5655/diff/206/217#newcode654
Line 654: return new PyString(s);
This can produce Unicode.  Consider:

LANG=es_ES.UTF-8 jythont -c "import time; print time.strftime('%A')"
s?bado

[Leo Soto M., 2008-09-21 16:57:17]

http://codereview.appspot.com/5655/diff/206/224
File jython/src/com/ziclix/python/sql/handler/MySQLDataHandler.java (right):

http://codereview.appspot.com/5655/diff/206/224#newcode57
Line 57: // XXX: Only works with ASCII data!
On 2008/09/14 00:22:45, Nicholas Riley wrote:
> tabs? looks like indentation is messed up here

Sure. I think I've fixed it, but when using diff -w whitespace changes are lost.

http://codereview.appspot.com/5655/diff/206/208
File jython/src/org/python/core/PyStringMap.java (right):

http://codereview.appspot.com/5655/diff/206/208#newcode144
Line 144: buf.append(new PyString((String)key).__repr__().toString());
On 2008/09/14 00:22:45, Nicholas Riley wrote:
> This seems a really complicated way to say:
> 
> "'" + key + "'"

Unless key.contains("'"). We could replicate PyString#__repr__ logic here, but I
think this is better.

http://codereview.appspot.com/5655/diff/206/210
File jython/src/org/python/core/util/StringUtil.java (right):

http://codereview.appspot.com/5655/diff/206/210#newcode89
Line 89: return new PyString(string);
On 2008/09/14 00:22:45, Nicholas Riley wrote:
> perhaps toPyString (parallel with toBytes above)?

To me, toPyString sounds more like a transformation, which is not the case here,
where we are just wrapping the String. Also note that asPyString parallels
PyString#asString.

http://codereview.appspot.com/5655/diff/206/217
File jython/src/org/python/modules/time/Time.java (right):

http://codereview.appspot.com/5655/diff/206/217#newcode654
Line 654: return new PyString(s);
On 2008/09/14 00:22:45, Nicholas Riley wrote:
> This can produce Unicode.  Consider:
> 
> LANG=es_ES.UTF-8 jythont -c "import time; print time.strftime('%A')"
> s?bado

So, strftime should start to return unicode in all cases?

That would break many doctests. Also this would break code that mixes the return
of strftime with other strings and then uses that value on places where a str is
expected (not an unicode). os.environ is one of such places. 

So, how about returning new PyUnicode(s).encode('utf-8')?

[BTW, What does CPython in your example? On Linux it doesn't seem to localize
the info, so I can't reproduce it :( ]