[google/4.6] Add xfails for arm-grtev2-linux-gnueabi

[dougkwan, 2012-03-12 18:58:43]

Hi Diego

	This patch adds arm-grtev2-linux-gnueabi.xfail for our 4.6 branch
	so that we can track regressions.  This just established the test
	baseline.  The failures need to be investigated.

-Doug

2012-03-12   Doug Kwan  <dougkwan@google.com>

	* contrib/testsuite-management/arm-grtev2-linux-gnueabi.xfail:
	New file.

Index: contrib/testsuite-management/arm-grtev2-linux-gnueabi.xfail
===================================================================
--- contrib/testsuite-management/arm-grtev2-linux-gnueabi.xfail	(revision 0)
+++ contrib/testsuite-management/arm-grtev2-linux-gnueabi.xfail	(revision 0)
@@ -0,0 +1,74 @@
+# Failures in ./gcc/testsuite/gcc/gcc.sum:
+# *** gcc:
+FAIL: gcc.dg/automversn_1.c (test for excess errors)
+UNRESOLVED: gcc.dg/automversn_1.c compilation failed to produce executable
+UNRESOLVED: gcc.dg/automversn_1.c scan-tree-dump auto_clone
"foo.autoclone.original"
+UNRESOLVED: gcc.dg/automversn_1.c scan-tree-dump auto_clone "foo.autoclone.0"
+FAIL: gcc.dg/builtin-apply2.c execution test
+FAIL: gcc.dg/tls/pr42894.c (test for excess errors)
+FAIL: gcc.dg/torture/stackalign/builtin-apply-2.c  -O0  execution test
+FAIL: gcc.dg/torture/stackalign/builtin-apply-2.c  -O0  execution test
+FAIL: gcc.dg/torture/stackalign/builtin-apply-2.c  -O1  execution test
+FAIL: gcc.dg/torture/stackalign/builtin-apply-2.c  -Os  execution test
+FAIL: gcc.dg/torture/tls/tls-test.c  -O0  execution test
+FAIL: gcc.dg/torture/tls/tls-test.c  -O1  execution test
+FAIL: gcc.dg/torture/tls/tls-test.c  -O2  execution test
+FAIL: gcc.dg/torture/tls/tls-test.c  -O3 -fomit-frame-pointer  execution test
+FAIL: gcc.dg/torture/tls/tls-test.c  -O3 -g  execution test
+FAIL: gcc.dg/torture/tls/tls-test.c  -Os  execution test
+FAIL: gcc.dg/torture/tls/tls-test.c  -O2 -flto -flto-partition=none  execution
test
+FAIL: gcc.dg/torture/tls/tls-test.c  -O2 -flto  execution test
+FAIL: gcc.dg/tree-prof/switch-case-1.c scan-rtl-dump-times expand "Succ
edge[^\n]*count:2000" 1
+FAIL: gcc.dg/vect/vect-multitypes-11.c scan-tree-dump-times vect "vectorized 1
loops" 1
+FAIL: gcc.dg/vect/vect-multitypes-12.c scan-tree-dump-times vect "vectorized 1
loops" 1
+FAIL: gcc.dg/vect/vect-reduc-dot-s16b.c scan-tree-dump-times vect "vectorized 1
loops" 0
+FAIL: gcc.dg/vect/vect-reduc-pattern-1a.c scan-tree-dump-times vect "vectorized
1 loops" 0
+FAIL: gcc.dg/vect/vect-reduc-pattern-1b.c scan-tree-dump-times vect "vectorized
1 loops" 0
+FAIL: gcc.dg/vect/vect-reduc-pattern-1c.c scan-tree-dump-times vect "vectorized
1 loops" 0
+FAIL: gcc.dg/vect/vect-reduc-pattern-2a.c scan-tree-dump-times vect "vectorized
1 loops" 0
+FAIL: gcc.dg/vect/vect-reduc-pattern-2b.c scan-tree-dump-times vect "vectorized
1 loops" 0
+XPASS: gcc.dg/vect/slp-reduc-3.c scan-tree-dump-times vect "vectorizing stmts
using SLP" 1
+FAIL: gcc.dg/vect/wrapv-vect-reduc-pattern-2c.c scan-tree-dump-times vect
"vectorized 1 loops" 0
+XPASS: gcc.dg/vect/no-scevccp-outer-16.c scan-tree-dump-times vect "OUTER LOOP
VECTORIZED." 1
+XPASS: gcc.dg/vect/no-scevccp-outer-17.c scan-tree-dump-times vect "OUTER LOOP
VECTORIZED." 1
+XPASS: gcc.dg/vect/no-scevccp-outer-19.c scan-tree-dump-times vect "OUTER LOOP
VECTORIZED." 1
+XPASS: gcc.dg/vect/no-scevccp-outer-21.c scan-tree-dump-times vect "OUTER LOOP
VECTORIZED." 1
+FAIL: gcc.target/arm/pr42575.c scan-assembler-not mov
+FAIL: gcc.target/arm/thumb-ltu.c (test for excess errors)
+UNRESOLVED: gcc.target/arm/thumb-ltu.c scan-assembler-not uxtb
+
+# Failures in ./gcc/testsuite/gfortran/gfortran.sum:
+# *** gfortran:
+FAIL: gfortran.dg/pr45636.f90  -O  scan-tree-dump-times forwprop2 "memset" 0
+FAIL: gfortran.dg/vect/fast-math-pr38968.f90 execution test
+FAIL: gfortran.dg/vect/fast-math-pr38968.f90 scan-tree-dump vect "vectorized 1
loops"
+
+# Failures in ./gcc/testsuite/g++/g++.sum:
+# *** g++:
+FAIL: g++.dg/abi/forced.C execution test
+FAIL: g++.dg/abi/local1.C execution test
+FAIL: g++.dg/thread-ann/thread_annot_lock-82.C  (test for warnings, line 47)
+XPASS: g++.dg/uninit-pred-3_b.C (test for excess errors)
+FAIL: g++.dg/tree-prof/mversn13.C execution,    -g  -fprofile-generate
+UNRESOLVED: g++.dg/tree-prof/mversn13.C compilation,  -g  -fprofile-use
+UNRESOLVED: g++.dg/tree-prof/mversn13.C execution,    -g  -fprofile-use
+FAIL: g++.dg/tree-prof/mversn13.C execution,    -O0  -fprofile-generate
+UNRESOLVED: g++.dg/tree-prof/mversn13.C compilation,  -O0  -fprofile-use
+UNRESOLVED: g++.dg/tree-prof/mversn13.C execution,    -O0  -fprofile-use
+FAIL: g++.dg/tree-prof/mversn13.C execution,    -O1  -fprofile-generate
+UNRESOLVED: g++.dg/tree-prof/mversn13.C compilation,  -O1  -fprofile-use
+UNRESOLVED: g++.dg/tree-prof/mversn13.C execution,    -O1  -fprofile-use
+FAIL: g++.dg/tree-prof/mversn13.C execution,    -O2  -fprofile-generate
+UNRESOLVED: g++.dg/tree-prof/mversn13.C compilation,  -O2  -fprofile-use
+UNRESOLVED: g++.dg/tree-prof/mversn13.C execution,    -O2  -fprofile-use
+FAIL: g++.dg/tree-prof/mversn13.C execution,    -O3  -fprofile-generate
+UNRESOLVED: g++.dg/tree-prof/mversn13.C compilation,  -O3  -fprofile-use
+UNRESOLVED: g++.dg/tree-prof/mversn13.C execution,    -O3  -fprofile-use
+FAIL: g++.dg/tree-prof/mversn13.C execution,    -O3 -g  -fprofile-generate
+UNRESOLVED: g++.dg/tree-prof/mversn13.C compilation,  -O3 -g  -fprofile-use
+UNRESOLVED: g++.dg/tree-prof/mversn13.C execution,    -O3 -g  -fprofile-use
+FAIL: g++.dg/tree-prof/mversn13.C execution,    -Os  -fprofile-generate
+UNRESOLVED: g++.dg/tree-prof/mversn13.C compilation,  -Os  -fprofile-use
+UNRESOLVED: g++.dg/tree-prof/mversn13.C execution,    -Os  -fprofile-use
+FAIL: g++.dg/vect/pr36648.cc scan-tree-dump-times vect "vectorized 1 loops" 1
+FAIL: g++.dg/vect/pr36648.cc scan-tree-dump-times vect "vectorizing stmts using
SLP" 1

--
This patch is available for review at http://codereview.appspot.com/5794063

[Diego Novillo, 2012-03-12 18:59:34]

On 12/03/12 14:58 , Doug Kwan wrote:

> 2012-03-12   Doug Kwan<dougkwan@google.com>
>
> 	* contrib/testsuite-management/arm-grtev2-linux-gnueabi.xfail:
> 	New file.

OK.

Diego.

[mikestump_comcast.net, 2012-03-12 19:28:01]

On Mar 12, 2012, at 11:59 AM, Diego Novillo wrote:
> On 12/03/12 14:58 , Doug Kwan wrote:
> 
>> 2012-03-12   Doug Kwan<dougkwan@google.com>
>> 
>> 	* contrib/testsuite-management/arm-grtev2-linux-gnueabi.xfail:
>> 	New file.
> 
> OK.

Hum, kinda makes be wish we had all the xfail files for the release branches and
trunk for all primary and secondary platforms and that make check did something
more intelligent.  :-)  One advantage, people doing day to day work, would
rarely have to do two make check runs, as they could do just one and do the
analysis against the checked in file.  They would only have to do two, if the
file isn't up-to-date.

So, I guess the question is, is there a down side to doing that?  I can imagine
a

	if [ -r "$srcdir/contrib/testsuite-management/$target.xfail ]; then
		$srcdir/contrib/testsuite-management/validate_failures.py bla bla
	fi

added into make check somewhere.

[Diego Novillo, 2012-03-12 19:37:33]

On 12/03/12 15:27 , Mike Stump wrote:

> So, I guess the question is, is there a down side to doing that?  I can
imagine a
>
> 	if [ -r "$srcdir/contrib/testsuite-management/$target.xfail ]; then
> 		$srcdir/contrib/testsuite-management/validate_failures.py bla bla
> 	fi

Yeah.  I had something like this in mind originally, but never followed 
through (we use the validator from within our own build harness).

Ideally, though, we would not even need this hack.  'make check' should 
return 0 when every test is nominal.  Period.

Our own guidelines are the culprit here: '... , this means comparing 
post-patch test results to pre-patch results by testing twice or 
comparing with recent posts to the gcc-testresults list.' 
(http://gcc.gnu.org/contribute.html#testing).

I have argued before that we should change this, but I am yet to do 
anything concrete about it.


Diego.

[mikestump_comcast.net, 2012-03-12 21:06:38]

On Mar 12, 2012, at 12:37 PM, Diego Novillo wrote:
> Ideally, though, we would not even need this hack.  'make check' should return
0 when every test is nominal.  Period.

Yeah, that pig has yet to achieve lift-off.  :-)

[ramana.radhakrishnan_linaro.org, 2012-03-12 22:57:52]

On 12 March 2012 18:58, Doug Kwan <dougkwan@google.com> wrote:
> Hi Diego
>
>        This patch adds arm-grtev2-linux-gnueabi.xfail for our 4.6 branch
>        so that we can track regressions.  This just established the test
>        baseline.  The failures need to be investigated.

Just out of curiosity, were these when you ran them cross on qemu or
when you ran these native. It's probably worth noting that as well.
There are times when you'll see differences in test results especially
on recent trunk ( atomic tests depend on the version of gdb installed
, cross testing on qemu pretty much means threaded tests are well
let's say flaky) .

This is probably something that ought to be recorded along with the
environment in which the tests were run to ease comparison.

This failure here suggests that you are runing on qemu.

+FAIL: gfortran.dg/vect/fast-math-pr38968.f90 execution test

I've noticed that this is something that times out depending on the
orientation of the sun , moon and earth and the performance of qemu on
your machine but on a native device runs just fine.

regards,
Ramana

[dougkwan, 2012-03-12 23:04:35]

Hi Ramana,

   We know the limit of the QEMU and already noticed failure due to
the simulator.  Like I said, this is used as the baseline.  We are
going to look at the failures carefully to categorize them.  We
noticed that some tests fail randomly on QEMU, these are marked as
flaky and the validator script ignores those.

Thanks

-Doug

On Mon, Mar 12, 2012 at 3:57 PM, Ramana Radhakrishnan
<ramana.radhakrishnan@linaro.org> wrote:
> On 12 March 2012 18:58, Doug Kwan <dougkwan@google.com> wrote:
>> Hi Diego
>>
>>        This patch adds arm-grtev2-linux-gnueabi.xfail for our 4.6 branch
>>        so that we can track regressions.  This just established the test
>>        baseline.  The failures need to be investigated.
>
> Just out of curiosity, were these when you ran them cross on qemu or
> when you ran these native. It's probably worth noting that as well.
> There are times when you'll see differences in test results especially
> on recent trunk ( atomic tests depend on the version of gdb installed
> , cross testing on qemu pretty much means threaded tests are well
> let's say flaky) .
>
> This is probably something that ought to be recorded along with the
> environment in which the tests were run to ease comparison.
>
> This failure here suggests that you are runing on qemu.
>
> +FAIL: gfortran.dg/vect/fast-math-pr38968.f90 execution test
>
> I've noticed that this is something that times out depending on the
> orientation of the sun , moon and earth and the performance of qemu on
> your machine but on a native device runs just fine.
>
> regards,
> Ramana