code review 5718045: misc/vim: restore fileencodings.

misc/vim: restore fileencodings.
Currently, ftdetect/gofiletype.vim set fileencodings to open the file as
utf-8 encoding event if the file does not contain multibyte characters.
But fileencodings is global option.

$ vim foo.txt
:set fileencodings
utf-8,ucs-bom,cp932

$ vim foo.go
:set fileencodings
utf-8

This change restore fileencodings before opening the file.
Also added specify fileformats=unix.

[mattn, 2012-03-02 02:54:31]

Hello golang-dev@googlecode.com (cc: golang-dev@googlegroups.com),

I'd like you to review this change to
http://go.googlecode.com/hg/

[dsymonds, 2012-03-02 06:45:39]

Would it be better to replace "fileencoding=utf-8" with
"encoding=utf-8" (and force fileencoding to be empty)? Go source files
are defined to be UTF-8.


Dave.

[mattn, 2012-03-02 07:51:53]

> On Friday, March 2, 2012 3:45:39 PM UTC+9, David Symonds wrote:
>  Would it be better to replace "fileencoding=utf-8" with
>  "encoding=utf-8" (and force fileencoding to be empty)? Go source files
>  are defined to be UTF-8.

No. Many japanese uses set encoding=cp932 yet. And they treat files which 
is cp932/utf-8 encoded both.
Of couse if go source file contain utf-8 characters which can't convert 
into cp932, vim show "conversion error".
However, they users double byte character sets. Because utf-8's glyph is 
not same as cp932's one.
And, if changing encoding to utf-8, some vim plugins which is running then 
may break behavior.

[dsymonds, 2012-03-02 07:57:27]

On Fri, Mar 2, 2012 at 6:51 PM, mattn <mattn.jp@gmail.com> wrote:

> No. Many japanese uses set encoding=cp932 yet. And they treat files which is
> cp932/utf-8 encoded both.
> Of couse if go source file contain utf-8 characters which can't convert into
> cp932, vim show "conversion error".

cp932 is not a valid representation of Go source code, which is
defined by the specification to be UTF-8.
  http://golang.org/doc/go_spec.html#Source_code_representation

[mattn, 2012-03-02 08:35:02]

> cp932 is not a valid representation of Go source code, which is
>  defined by the specification to be UTF-8.
>    http://golang.org/doc/go_spec.html#Source_code_representation 

Yes, I know. When setting like below,

set encoding=cp932
set fileencodings=utf-8

vim try to open the file as utf-8 encoding. Thus, it can read/write utf-8 
file.

On Friday, March 2, 2012 4:57:27 PM UTC+9, David Symonds wrote:
>
> On Fri, Mar 2, 2012 at 6:51 PM, mattn <mattn.jp@gmail.com> wrote:
>
> > No. Many japanese uses set encoding=cp932 yet. And they treat files 
> which is
> > cp932/utf-8 encoded both.
> > Of couse if go source file contain utf-8 characters which can't convert 
> into
> > cp932, vim show "conversion error".
>
> cp932 is not a valid representation of Go source code, which is
> defined by the specification to be UTF-8.
>   http://golang.org/doc/go_spec.html#Source_code_representation
>
>

[mattn, 2012-03-08 05:51:42]

ping

[dsymonds, 2012-03-08 05:57:55]

Sorry, I haven't forgotten about this. I've just been away. I'll study
this closer tomorrow.

[dsymonds, 2012-03-09 13:09:00]

I'm still not comfortable with this yet. Something that is bugging me
is that this seems more complex than it should be. Can you point at
any other file type plugins that do this kind of machination?


Dave.

[mattn, 2012-03-13 08:42:41]

> I'm still not comfortable with this yet. Something that is bugging me
> is that this seems more complex than it should be. Can you point at
> any other file type plugins that do this kind of machination?

No, Go is particular. For example, java endorse to write the code as utf-8 
encoding. But it's not "MUST". Go enforce to write the code as utf-8.

I (or CJK people) often write the code(C/C++/ruby/perl/python etc) which is 
not utf-8 encoding with vim. They always set options like following.

:set encoding=cp932 " or another DBCS encoding.
:set fileencodings=utf-8,cp932 " and more encodings to guess.

Because, utf-8 is not upper compatible of DBCS. Andalso, we open file both 
DBCS/UTF-8 at a time.
Someone may use terminal emurator which don't support utf-8. AFAIK, who use 
encoding=utf-8 on windows isn't many. I know some japanese use non-utf8 
locale on unix yet. But gofiletype.vim change global option 'encoding'. 
it's rudely.

Thanks.

[dsymonds, 2012-03-13 12:10:00]

On Tue, Mar 13, 2012 at 7:42 PM, mattn <mattn.jp@gmail.com> wrote:

> Someone may use terminal emurator which don't support utf-8. AFAIK, who use
> encoding=utf-8 on windows isn't many. I know some japanese use non-utf8
> locale on unix yet. But gofiletype.vim change global option 'encoding'. it's
> rudely.

Okay, this makes sense to me.

Is there a way to change the encoding for a particular buffer? If so,
we should definitely do that. If not, we'll have to proceed with your
change.


Dave.

[mattn, 2012-03-13 16:21:32]

> Is there a way to change the encoding for a particular buffer? If so,
> we should definitely do that. If not, we'll have to proceed with your
> change.

vim don't have buffer local encoding option.

[dsymonds, 2012-03-13 22:06:49]

On Mar 14, 2012 3:20 AM, "mattn" <mattn.jp@gmail.com> wrote:

> > Is there a way to change the encoding for a particular buffer? If so,
> > we should definitely do that. If not, we'll have to proceed with your
> > change.
>
> vim don't have buffer local encoding option.

Yeah, didn't think so. Okay, I'll review this today.

[dsymonds, 2012-03-14 04:36:41]

One other thing to try: ++enc allows overriding fileencoding on a per-file
basis. Is there a way you can hook that in instead? That would be a lot less
invasive.

If that doesn't work, please add the comment I mention below, and I'll LGTM this
CL.

http://codereview.appspot.com/5718045/diff/3/misc/vim/ftdetect/gofiletype.vim
File misc/vim/ftdetect/gofiletype.vim (right):

http://codereview.appspot.com/5718045/diff/3/misc/vim/ftdetect/gofiletype.vim...
misc/vim/ftdetect/gofiletype.vim:1: let s:current_fileformats = ''
Add a comment at the top here.

" We take care to preserve the user's fileencodings and fileformats,
" because those settings are global (not buffer local), yet we want
" to override them for loading Go files, which are defined to be UTF-8.

[mattn, 2012-03-14 06:28:01]

> One other thing to try: ++enc allows overriding fileencoding on a per-file
basis. Is there a way you can hook that in instead? That would be a lot less
invasive.

":e ++enc=utf-8 foo.go" is meaning that set fileencodings to just only utf-8
before opening foo.go. vimscript can't know what value specified in '++enc'. We
can know fileencodings value with hook BufReadPre. But script can't know whether
the value is specified in vimrc or ++enc. Andalso, fileencoding(not
fileencodings) have two meanings. The first is 'current file encoding'. When
change this vaule, buffer will be saved as specified encoding. The second is
'default encodings when open new file'. Thus if set fileencoding=utf-8 in
BufReadPre, the value will be utf-8 for new file, but if open exist file, it
make different from real fileencoding . i.e. We shouldn't use fileencoding
option also for this trick.
To solve this problem,

* Reading exist file:

1. Store current fileencodings/fileformats
2. Set just only utf-8/unix for fileencodings/fileformats before reading file.
3. Restore original value for fileencodings/fileformats.

* Open new buffer:
1. set utf8/unix for fileencoding/fileformat.
 This don't effect modified flag.

http://codereview.appspot.com/5718045/diff/3/misc/vim/ftdetect/gofiletype.vim
File misc/vim/ftdetect/gofiletype.vim (right):

http://codereview.appspot.com/5718045/diff/3/misc/vim/ftdetect/gofiletype.vim...
misc/vim/ftdetect/gofiletype.vim:1: let s:current_fileformats = ''
On 2012/03/14 04:36:42, dsymonds wrote:
> Add a comment at the top here.
> 
> " We take care to preserve the user's fileencodings and fileformats,
> " because those settings are global (not buffer local), yet we want
> " to override them for loading Go files, which are defined to be UTF-8.

Done.

[dsymonds, 2012-03-14 06:29:19]

LGTM

[dsymonds, 2012-03-14 07:43:09]

*** Submitted as http://code.google.com/p/go/source/detail?r=ca61f2a46162 ***

misc/vim: restore fileencodings.
Currently, ftdetect/gofiletype.vim set fileencodings to open the file as
utf-8 encoding event if the file does not contain multibyte characters.
But fileencodings is global option.

$ vim foo.txt
:set fileencodings
utf-8,ucs-bom,cp932

$ vim foo.go
:set fileencodings
utf-8

This change restore fileencodings before opening the file.
Also added specify fileformats=unix.

R=golang-dev, dsymonds
CC=golang-dev
http://codereview.appspot.com/5718045

Committer: David Symonds <dsymonds@golang.org>