[pph] Clean up PPH tests

[Lawrence Crowl, 2011-06-05 22:46:33]

The purpose of this patch is to make PPH testing clean.

Some tests have assembly comparison failures between regular compiles and PPH
compiles.  When this miscompare is exected, we now mark the test with a
comment "pph asm xdiff".  The driver lib/dg-pph.exp looks for this comment to
report either FAIL or XFAIL.

Test p1mean.cc enters infinite recursion.  Instead of disabling the test with
a syntax error, we now run it under a dg-timeout.

Test c1return-5.h was trying to grep assembly, but since we produce pre-parsed
headers, not assembly, the test was in error.  We have moved the assembly
check to c1return-5.cc where it belongs.

All PPH tests expected to have PPH compilation failures are marked with
// { dg-xfail-if "comment" { "*-*-*" } { "-fpph-map=pph.map" } }
The regular compiles do not have the -fpph-map option, and hence are not
expected to fail.  One test has a comment INFINITE; one test has a comment
BOGUS; and the rest have a comment ICE.

The driver lib/dg-pph.exp test control logic now stops a test when any
compilation fails to produce assembly output.  If the test file contains the
above directive, that compilation failure is expected and reported as such.
Otherwise, it is reported as an unexpected failure.

PPH is wrongly producing ICEs and errors.  We mark applicable messages with
dg-bogus.  Because the line/column information is often wrong, we mark the
with a line of "0", which avoids line comparisons.  We also prune output as
needed.

All tests are now either PASS or XFAIL, with the exception of unavoidable
bogus XPASS messages for "test for bogus messages" and "test for excess
errors".  These are unavoidable because they cannot be conditionalized based
on compiler options, and they do not appear in regular compiles.

If you run the output of

    make check-c++ RUNTESTFLAGS=pph.exp

through

    sed -e '
        /^XPASS: .*test for bogus messages/     d
        /^XPASS: .*test for excess errors/      d

        /^#/            p
        /^ERROR: /      p
        /^XFAIL: /      p
        /^XPASS: /      p
        /^FAIL: /       p

        d
    '

you should get no individual tests listed.


Index: gcc/testsuite/ChangeLog.pph

2011-06-05  Lawrence Crowl  <crowl@google.com>

	* lib/dg-pph.exp (dg-pph-hdr): Add blank lines to logfile.
	Format if statements and calls consistently.
	(dg-pph-neg): Remove extraneous assembly files. Add blank
	lines to logfile. Format if statements and calls consistently.
	(dg-pph-pos): Stop test when a compilation fails. Recognize
	expected failing compilations. Recognize expected assembly
	miscompares. Parenthesize failure notes. Remove extraneous
	assembly files. Add blank lines to logfile. Format if
	statements and calls consistently.
	* g++.dg/pph/c1simple.cc: Add comment "pph asm xdiff"
	indicating expected assembly difference failure. Add line
	spacing to test.
	* g++.dg/pph/c1struct.cc: Likewise.
	* g++.dg/pph/c1variables.cc: Likewise.
	* g++.dg/pph/c2builtin1.cc: Likewise.
	* g++.dg/pph/x1globalref.cc: Likewise.
	* g++.dg/pph/x1hardlookup.cc: Likewise.
	* g++.dg/pph/x1struct1.cc: Likewise.
	* g++.dg/pph/x1struct2.cc: Likewise.
	* g++.dg/pph/x2nontrivinit.cc: Likewise.
	* c120060625-1.h: Add dg-xfail-if for -fpph-map. Add bogus
	message for ICE. Prune output. Add line spacing to text.
	* g++.dg/pph/c120060625-1.cc: Likewise.
	* g++.dg/pph/c1eabi1.h: Likewise.
	* g++.dg/pph/c1eabi1.cc: Likewise.
	* g++.dg/pph/x1autometh.cc: Likewise.
	* g++.dg/pph/x1functions.cc: Likewise.
	* g++.dg/pph/x1special.cc: Likewise.
	* g++.dg/pph/x1template.cc: Likewise.
	* g++.dg/pph/x1tmplclass.cc: Likewise.
	* g++.dg/pph/x1tmplfunc.cc: Likewise.
	* g++.dg/pph/x1typerefs.cc: Likewise.
	* g++.dg/pph/x1variables.cc: Likewise.
	* g++.dg/pph/p1mean.cc: Change syntax error preventing infinite
	recursion to a dg-timeout. Add dg-xfail-if for -fpph-map.
	* g++.dg/pph/c1return-5.h: Remove dg-final as we do not generate
	assembly on compiling headers. Change dg directive location.
	* g++.dg/pph/c1return-5.cc: Add dg-final as we _do_ generate
	assembly on compiling sources. Copy up dg-options from
	c1return-5.h as well.


Index: gcc/testsuite/lib/dg-pph.exp
===================================================================
--- gcc/testsuite/lib/dg-pph.exp	(revision 174608)
+++ gcc/testsuite/lib/dg-pph.exp	(working copy)
@@ -24,13 +24,14 @@ proc dg-pph-hdr { subdir test options ma
     global runtests dg-do-what-default
 
     # If we're only testing specific files and this isn't one of them, skip it.
-    if ![runtest_file_p $runtests $test] {
+    if { ![runtest_file_p $runtests $test] } {
 	return
     }
 
     set nshort "$subdir/[file tail $test]"
     set bname "[file rootname [file tail $nshort]]"
-    verbose "Testing $nshort, $options" 1
+    verbose -log "\nTesting $nshort, $options"
+
     set dg-do-what-default preparse
     dg-test -keep-output $test "$options $mapflag -I." ""
 
@@ -41,16 +42,19 @@ proc dg-pph-neg { subdir test options ma
     global runtests dg-do-what-default
 
     # If we're only testing specific files and this isn't one of them, skip it.
-    if ![runtest_file_p $runtests $test] {
+    if { ![runtest_file_p $runtests $test] } {
 	return
     }
 
     set nshort "$subdir/[file tail $test]"
     set bname "[file rootname [file tail $nshort]]"
-    verbose "Testing $nshort, $options" 1
+    verbose -log "\nTesting $nshort, $options"
+
     set dg-do-what-default compile
     dg-test -keep-output $test "$options $mapflag -I." ""
-    file_on_host delete "$bname.s"
+    if { ![file_on_host exists "$bname.s"] } {
+	file_on_host delete "$bname.s"
+    }
 }
 
 proc dg-pph-pos { subdir test options mapflag suffix } {
@@ -58,64 +62,72 @@ proc dg-pph-pos { subdir test options ma
     global runtests dg-do-what-default
 
     # If we're only testing specific files and this isn't one of them, skip it.
-    if ![runtest_file_p $runtests $test] {
+    if { ![runtest_file_p $runtests $test] } {
 	return
     }
 
     set nshort "$subdir/[file tail $test]"
     set bname "[file rootname [file tail $nshort]]"
-    verbose "Testing $nshort, $options" 1
-    set dg-do-what-default compile
+    verbose -log "\nTesting $nshort, $options"
 
-    set have_errs [llength [grep $test "{\[ \t\]\+dg-error\[ \t\]\+.*\[
\t\]\+}"]]
     # Compile the file the first time for a base case.
+    set dg-do-what-default compile
     dg-test -keep-output $test "$options -I." ""
 
-    if { $have_errs } {
-	verbose -log "regular compilation failed"
-	fail "$nshort $options, regular compilation failed"
-	return
-    }
-
-    if { ! [ file_on_host exists "$bname.s" ] } {
-	verbose -log "regular assembly file '$bname.s' missing"
-	fail "$nshort $options, regular assembly missing"
+    # Quit if it did not compile successfully.
+    if { ![file_on_host exists "$bname.s"] } {
+	# All regular compiles should pass.
+	fail "$nshort $options (regular assembly missing)"
 	return
     }
 
     # Rename the .s file into .s-pph to compare it after the second build.
     remote_upload host "$bname.s" "$bname.s-pph"
     remote_download host "$bname.s-pph"
+    file_on_host delete "$bname.s"
+
+    verbose -log ""
 
     # Compile a second time using the pph files.
     dg-test -keep-output $test "$options $mapflag -I." ""
 
-    if { $have_errs } {
-	verbose -log "PPH compilation failed"
-	fail "$nshort $options, PPH compilation failed"
-	return
-    }
-
-    if { ! [ file_on_host exists "$bname.s" ] } {
-	verbose -log "PPH assembly file '$bname.s' missing"
-	fail "$nshort $options, PPH assembly missing"
+    # Quit if it did not compile successfully.
+    if { ![file_on_host exists "$bname.s"] } {
+	# Expect assembly to be missing when the compile is an expected fail.
+	if { [llength [grep $test "dg-xfail-if.*-fpph-map"]] } {
+	    xfail "$nshort $options (pph assembly missing)"
+	} else {
+	    fail "$nshort $options (pph assembly missing)"
+	}
 	return
     }
 
     # Rename the .s file into .s+pph to compare it.
     remote_upload host "$bname.s" "$bname.s+pph"
     remote_download host "$bname.s+pph"
+    file_on_host delete "$bname.s"
+
+    verbose -log ""
 
     # Compare the two assembly files.  They should be identical.
-    set tmp [ diff "$bname.s-pph" "$bname.s+pph" ]
-    if { $tmp == 0 } {
-	verbose -log "assembly file '$bname.s-pph', '$bname.s+pph' comparison error"
-	fail "$nshort $options assembly comparison"
-    } elseif { $tmp == 1 } {
-	pass "$nshort $options assembly comparison"
+    set adiff [diff "$bname.s-pph" "$bname.s+pph"]
+    # The sources mark when they expect the comparison to differ.
+    set xdiff [llength [grep $test "pph asm xdiff"]]
+    if { $adiff == 0 } {
+	fail "$nshort $options comparison failure"
+    } elseif { $adiff == 1 } {
+	if { $xdiff } {
+	    xpass "$nshort $options (assembly identical)"
+	} else {
+	    pass "$nshort $options (assembly identical)"
+	}
 	file_on_host delete "$bname.s-pph"
 	file_on_host delete "$bname.s+pph"
     } else {
-	fail "$nshort $options assembly comparison"
+	if { $xdiff } {
+	    xfail "$nshort $options (assembly mismatch)"
+	} else {
+	    fail "$nshort $options (assembly mismatch)"
+	}
     }
 }
Index: gcc/testsuite/g++.dg/pph/x1globalref.cc
===================================================================
--- gcc/testsuite/g++.dg/pph/x1globalref.cc	(revision 174608)
+++ gcc/testsuite/g++.dg/pph/x1globalref.cc	(working copy)
@@ -1,4 +1,7 @@
+// pph asm xdiff
+
 #include "x1globalref.h"
+
 type x = 2;
 type y = x;
 type f() { return n+m+a[3]+x+y; }
Index: gcc/testsuite/g++.dg/pph/c1eabi1.h
===================================================================
--- gcc/testsuite/g++.dg/pph/c1eabi1.h	(revision 174608)
+++ gcc/testsuite/g++.dg/pph/c1eabi1.h	(working copy)
@@ -1,8 +1,12 @@
+// { dg-xfail-if "ICE" { "*-*-*" } { "-fpph-map=pph.map" } }
+// { dg-bogus "mathcalls.h:365:1: internal compiler error: Segmentation fault"
"" { xfail *-*-* } 0 }
+// { dg-prune-output "In file included " }
+// { dg-prune-output "                 from " }
+// { dg-options "-w -fpermissive" }
+
 #ifndef __PPH_GUARD_H
 #define __PPH_GUARD_H
-/* { dg-options "-w -fpermissive" } */
-/* { dg-xfail-if "ICEs the compiler during PPH read" { "*-*-*" } } */
-/* { dg-prune-output "In file included.*" } */
+
 /* This file tests most of the non-C++ run-time helper functions
    described in Section 4 of the "Run-Time ABI for the ARM
    Architecture".  These are basic tests; they do not try to validate
Index: gcc/testsuite/g++.dg/pph/c1struct.cc
===================================================================
--- gcc/testsuite/g++.dg/pph/c1struct.cc	(revision 174608)
+++ gcc/testsuite/g++.dg/pph/c1struct.cc	(working copy)
@@ -1,3 +1,6 @@
+// pph asm xdiff
+
 #include "c1struct.h"
+
 thing var1;
 thing var2 = { 3 };
Index: gcc/testsuite/g++.dg/pph/x1typerefs.cc
===================================================================
--- gcc/testsuite/g++.dg/pph/x1typerefs.cc	(revision 174608)
+++ gcc/testsuite/g++.dg/pph/x1typerefs.cc	(working copy)
@@ -1,5 +1,11 @@
+// { dg-xfail-if "ICE" { "*-*-*" } { "-fpph-map=pph.map" } }
+// { dg-bogus "c1typerefs.h:11:18: internal compiler error: Segmentation fault"
"" { xfail *-*-* } 0 }
+// { dg-prune-output "In file included from " }
+
 #include "x1typerefs.h"
+
 int derived::method() {
     return field;
 }
+
 vderived variable;
Index: gcc/testsuite/g++.dg/pph/x1struct2.cc
===================================================================
--- gcc/testsuite/g++.dg/pph/x1struct2.cc	(revision 174608)
+++ gcc/testsuite/g++.dg/pph/x1struct2.cc	(working copy)
@@ -1,7 +1,11 @@
+// pph asm xdiff
+
 #include "x1struct2.h"
+
 type D::method()
 { static int x = 2;
   return fld + mbr + gbl; }
+
 type D::mbr = 4;
 typedef D D2;
 D2 var1;
Index: gcc/testsuite/g++.dg/pph/p1mean.cc
===================================================================
--- gcc/testsuite/g++.dg/pph/p1mean.cc	(revision 174608)
+++ gcc/testsuite/g++.dg/pph/p1mean.cc	(working copy)
@@ -1,10 +1,11 @@
+// { dg-timeout 2 { target *-*-* } }
+// { dg-xfail-if "INFINITE" { "*-*-*" } { "-fpph-map=pph.map" } }
+
 #include <stdlib.h>
 #include <stdio.h>
 #include <math.h>
 #include <string.h>
 
-SYNTAX ERROR TO AVOID COMPILING THIS - IT CAUSES INFINITE RECURSION.
-
 static unsigned long long MAX_ITEMS = 10000;
 
 static int
Index: gcc/testsuite/g++.dg/pph/x1autometh.cc
===================================================================
--- gcc/testsuite/g++.dg/pph/x1autometh.cc	(revision 174608)
+++ gcc/testsuite/g++.dg/pph/x1autometh.cc	(working copy)
@@ -1,4 +1,9 @@
+// { dg-xfail-if "ICE" { "*-*-*" } { "-fpph-map=pph.map" } }
+// { dg-bogus "x1autometh.h:8:1: internal compiler error: Segmentation fault"
"" { xfail *-*-* } 0 }
+// { dg-prune-output "In file included from " }
+
 #include "x1autometh.h"
+
 void function() {
     base var1;
     base var2( var1 );
Index: gcc/testsuite/g++.dg/pph/x1tmplfunc.cc
===================================================================
--- gcc/testsuite/g++.dg/pph/x1tmplfunc.cc	(revision 174608)
+++ gcc/testsuite/g++.dg/pph/x1tmplfunc.cc	(working copy)
@@ -1,4 +1,9 @@
+// { dg-xfail-if "ICE" { "*-*-*" } { "-fpph-map=pph.map" } }
+// { dg-bogus "x1tmplfunc.h:12:30: internal compiler error: Segmentation fault"
"" { xfail *-*-* } 0 }
+// { dg-prune-output "In file included from " }
+
 #include "x1tmplfunc.h"
+
 type val = 3;
 
 template<>
Index: gcc/testsuite/g++.dg/pph/c1return-5.h
===================================================================
--- gcc/testsuite/g++.dg/pph/c1return-5.h	(revision 174608)
+++ gcc/testsuite/g++.dg/pph/c1return-5.h	(working copy)
@@ -1,7 +1,7 @@
+// { dg-options "-mpreferred-stack-boundary=4" }
+
 #ifndef __PPH_GUARD_H
 #define __PPH_GUARD_H
-/* { dg-options "-mpreferred-stack-boundary=4" } */
-/* { dg-final { scan-assembler-not "and\[lq\]?\[^\\n\]*-64,\[^\\n\]*sp" } } */
 
 /* This compile only test is to detect an assertion failure in stack branch
    development.  */
Index: gcc/testsuite/g++.dg/pph/x1struct1.cc
===================================================================
--- gcc/testsuite/g++.dg/pph/x1struct1.cc	(revision 174608)
+++ gcc/testsuite/g++.dg/pph/x1struct1.cc	(working copy)
@@ -1,7 +1,11 @@
+// pph asm xdiff
+
 #include "x1struct1.h"
+
 type D::method()
 { static int x = 2;
   return fld + mbr; }
+
 type D::mbr = 4;
 typedef D D2;
 D2 var1;
Index: gcc/testsuite/g++.dg/pph/c120060625-1.cc
===================================================================
--- gcc/testsuite/g++.dg/pph/c120060625-1.cc	(revision 174608)
+++ gcc/testsuite/g++.dg/pph/c120060625-1.cc	(working copy)
@@ -1 +1,5 @@
+// { dg-xfail-if "ICE" { "*-*-*" } { "-fpph-map=pph.map" } }
+// { dg-bogus "c120060625-1.h:14:22: internal compiler error: invalid built-in
macro .__FLT_MAX__." "" { xfail *-*-* } 0 }
+// { dg-prune-output "In file included from " }
+
 #include "c120060625-1.h"
Index: gcc/testsuite/g++.dg/pph/x1special.cc
===================================================================
--- gcc/testsuite/g++.dg/pph/x1special.cc	(revision 174608)
+++ gcc/testsuite/g++.dg/pph/x1special.cc	(working copy)
@@ -1,4 +1,9 @@
+// { dg-xfail-if "ICE" { "*-*-*" } { "-fpph-map=pph.map" } }
+// { dg-bogus "x1special.h:19:6: internal compiler error: tree check: expected
tree that contains 'decl minimal' structure, have 'overload' in
context_for_name_lookup, at cp/search.c:570" "" { xfail *-*-* } 0 }
+// { dg-prune-output "In file included from " }
+
 #include "x1special.h"
+
 B b(1);
 
 int H() {
Index: gcc/testsuite/g++.dg/pph/c1variables.cc
===================================================================
--- gcc/testsuite/g++.dg/pph/c1variables.cc	(revision 174608)
+++ gcc/testsuite/g++.dg/pph/c1variables.cc	(working copy)
@@ -1,4 +1,7 @@
+// pph asm xdiff
+
 #include "c1variables.h"
+
 int gbl_initial = 1;
 const float gbl_init_const = 1.5;
 const int gbl_manifest = 2;
Index: gcc/testsuite/g++.dg/pph/x2nontrivinit.cc
===================================================================
--- gcc/testsuite/g++.dg/pph/x2nontrivinit.cc	(revision 174608)
+++ gcc/testsuite/g++.dg/pph/x2nontrivinit.cc	(working copy)
@@ -1 +1,3 @@
+// pph asm xdiff
+
 #include "x2nontrivinit.h"
Index: gcc/testsuite/g++.dg/pph/c1eabi1.cc
===================================================================
--- gcc/testsuite/g++.dg/pph/c1eabi1.cc	(revision 174608)
+++ gcc/testsuite/g++.dg/pph/c1eabi1.cc	(working copy)
@@ -1,4 +1,7 @@
-/* { dg-options "-w -fpermissive" } */
-/* { dg-xfail-if "ICEs the compiler during PPH read" { "*-*-*" } } */
-/* { dg-prune-output "In file included.*" } */
+// { dg-xfail-if "ICE" { "*-*-*" } { "-fpph-map=pph.map" } }
+// { dg-bogus "mathcalls.h:365:1: internal compiler error: Segmentation fault"
"" { xfail *-*-* } 0 }
+// { dg-prune-output "In file included from " }
+// { dg-prune-output "                 from " }
+// { dg-options "-w -fpermissive" }
+
 #include "c1eabi1.h"
Index: gcc/testsuite/g++.dg/pph/c1simple.cc
===================================================================
--- gcc/testsuite/g++.dg/pph/c1simple.cc	(revision 174608)
+++ gcc/testsuite/g++.dg/pph/c1simple.cc	(working copy)
@@ -1,5 +1,9 @@
+// pph asm xdiff
+
 /* comment */
+
 #include "c1simple2.h"
+
 int main()
 {
 	return foo();
Index: gcc/testsuite/g++.dg/pph/x1functions.cc
===================================================================
--- gcc/testsuite/g++.dg/pph/x1functions.cc	(revision 174608)
+++ gcc/testsuite/g++.dg/pph/x1functions.cc	(working copy)
@@ -1,5 +1,14 @@
+// { dg-xfail-if "ICE" { "*-*-*" } { "-fpph-map=pph.map" } }
+// { dg-bogus "'mbr_decl_inline' was not declared in this scope" "" { xfail
*-*-* } 0 }
+// { dg-bogus "c1functions.h:8:34: internal compiler error: Segmentation fault"
"" { xfail *-*-* } 0 }
+// { dg-prune-output "In file included from " }
+// { dg-prune-output "In member function " }
+// { dg-prune-output "At global scope:" }
+
 #include "x1functions.h"
+
 int type::mbr_decl_then_def(int i)      // need body
 { return mbr_decl_inline( i ); }
+
 int type::mbr_inl_then_def(int i)       // lazy body
 { return mbr_decl_then_def( i ); }
Index: gcc/testsuite/g++.dg/pph/x1template.cc
===================================================================
--- gcc/testsuite/g++.dg/pph/x1template.cc	(revision 174608)
+++ gcc/testsuite/g++.dg/pph/x1template.cc	(working copy)
@@ -1,9 +1,17 @@
+// { dg-xfail-if "ICE" { "*-*-*" } { "-fpph-map=pph.map" } }
+// { dg-bogus "x1template.h:18:13: internal compiler error: in resume_scope, at
cp/name-lookup.c:1573" "" { xfail *-*-* } 0 }
+// { dg-prune-output "In file included from " }
+
 #include "x1template.h"
+
 namespace A {
 int x = 3;
 } // namespace A
+
 int y = 4;
+
 int D::method()
 { return y; }
+
 int main()
 { }
Index: gcc/testsuite/g++.dg/pph/c120060625-1.h
===================================================================
--- gcc/testsuite/g++.dg/pph/c120060625-1.h	(revision 174608)
+++ gcc/testsuite/g++.dg/pph/c120060625-1.h	(working copy)
@@ -1,5 +1,9 @@
+// { dg-xfail-if "ICE" { "*-*-*" } { "-fpph-map=pph.map" } }
+// { dg-bogus "c120060625-1.h:14:22: internal compiler error: invalid built-in
macro .__FLT_MAX__." "" { xfail *-*-* } 0 }
+
 #ifndef __PPH_GUARD_H
 #define __PPH_GUARD_H
+
 /* PR middle-end/28151 */
 /* Testcase by Steven Bosscher <stevenb.gcc@gmail.com> */
 
@@ -11,3 +15,4 @@ void foo (void)
   b = __FLT_MAX__ + a;
 }
 #endif
+
Index: gcc/testsuite/g++.dg/pph/c1return-5.cc
===================================================================
--- gcc/testsuite/g++.dg/pph/c1return-5.cc	(revision 174608)
+++ gcc/testsuite/g++.dg/pph/c1return-5.cc	(working copy)
@@ -1 +1,4 @@
+// { dg-options "-mpreferred-stack-boundary=4" }
+// { dg-final { scan-assembler-not "and\[lq\]?\[^\\n\]*-64,\[^\\n\]*sp" } }
+
 #include "c1return-5.h"
Index: gcc/testsuite/g++.dg/pph/x1tmplclass.cc
===================================================================
--- gcc/testsuite/g++.dg/pph/x1tmplclass.cc	(revision 174608)
+++ gcc/testsuite/g++.dg/pph/x1tmplclass.cc	(working copy)
@@ -1,4 +1,9 @@
+// { dg-xfail-if "ICE" { "*-*-*" } { "-fpph-map=pph.map" } }
+// { dg-bogus "x1tmplclass.h:14:17: internal compiler error: Segmentation
fault" "" { xfail *-*-* } 0 }
+// { dg-prune-output "In file included from " }
+
 #include "x1tmplclass.h"
+
 int wrapper<char>::cache = 2;
 
 template
Index: gcc/testsuite/g++.dg/pph/c2builtin1.cc
===================================================================
--- gcc/testsuite/g++.dg/pph/c2builtin1.cc	(revision 174608)
+++ gcc/testsuite/g++.dg/pph/c2builtin1.cc	(working copy)
@@ -1,3 +1,5 @@
+// pph asm xdiff
+
 #include "c2builtin1.h"
 #include "c2builtin2.h"
 #include "c2builtin3.h"
Index: gcc/testsuite/g++.dg/pph/x1hardlookup.cc
===================================================================
--- gcc/testsuite/g++.dg/pph/x1hardlookup.cc	(revision 174608)
+++ gcc/testsuite/g++.dg/pph/x1hardlookup.cc	(working copy)
@@ -1,5 +1,9 @@
+// pph asm xdiff
+
 #include "x1hardlookup.h"
+
 D P;
+
 int I(int arg)
 { return arg + V; }
 
Index: gcc/testsuite/g++.dg/pph/x1variables.cc
===================================================================
--- gcc/testsuite/g++.dg/pph/x1variables.cc	(revision 174608)
+++ gcc/testsuite/g++.dg/pph/x1variables.cc	(working copy)
@@ -1,4 +1,10 @@
+// { dg-xfail-if "BOGUS" { "*-*-*" } { "-fpph-map=pph.map" } }
+// { dg-bogus "c1variables.h:4:8: error: 'int D::mbr_uninit_plain' is not a
static member of 'struct D'" "" { xfail *-*-* } 0 }
+// { dg-bogus "c1variables.h:5:14: error: 'const int D::mbr_init_const' is not
a static member of 'struct D'" "" { xfail *-*-* } 0 }
+// { dg-prune-output "In file included from " }
+
 #include "x1variables.h"
+
 int D::mbr_init_plain = 4;
 int D::mbr_uninit_plain;
 const int D::mbr_init_const = 5;

--
This patch is available for review at http://codereview.appspot.com/4572042

[Diego Novillo, 2011-06-06 14:21:21]

So, I'm getting this:

Running /home/dnovillo/pph/svn/src/gcc/testsuite/g++.dg/pph/pph.exp ...
XPASS: g++.dg/pph/c120060625-1.cc  -I.  (test for bogus messages, line )
XPASS: g++.dg/pph/c1eabi1.cc  -I.  (test for bogus messages, line )
XPASS: g++.dg/pph/x1autometh.cc  -I.  (test for bogus messages, line )
XPASS: g++.dg/pph/x1functions.cc  -I.  (test for bogus messages, line )
XPASS: g++.dg/pph/x1functions.cc  -I.  (test for bogus messages, line )
XPASS: g++.dg/pph/x1special.cc  -I.  (test for bogus messages, line )
XPASS: g++.dg/pph/x1special.cc  -fpph-map=pph.map -I.  (test for bogus
messages, line )
FAIL: g++.dg/pph/x1special.cc  -fpph-map=pph.map -I. (test for excess errors)
XPASS: g++.dg/pph/x1template.cc  -I.  (test for bogus messages, line )
XPASS: g++.dg/pph/x1template.cc  -fpph-map=pph.map -I.  (test for
bogus messages, line )
FAIL: g++.dg/pph/x1template.cc  -fpph-map=pph.map -I. (test for excess errors)
XPASS: g++.dg/pph/x1tmplclass.cc  -I.  (test for bogus messages, line )
XPASS: g++.dg/pph/x1tmplfunc.cc  -I.  (test for bogus messages, line )
XPASS: g++.dg/pph/x1tmplfunc.cc  -fpph-map=pph.map -I.  (test for
bogus messages, line )
FAIL: g++.dg/pph/x1tmplfunc.cc  -fpph-map=pph.map -I. (test for excess errors)
XPASS: g++.dg/pph/x1typerefs.cc  -I.  (test for bogus messages, line )
XPASS: g++.dg/pph/x1variables.cc  -I.  (test for bogus messages, line )
XPASS: g++.dg/pph/x1variables.cc  -I.  (test for bogus messages, line )
XPASS: g++.dg/pph/x1variables.cc  -fpph-map=pph.map -I.  (test for
bogus messages, line )
XPASS: g++.dg/pph/x1variables.cc  -fpph-map=pph.map -I.  (test for
bogus messages, line )

                === g++ Summary ===

# of expected passes            176
# of unexpected failures        3
# of unexpected successes       17
# of expected failures          45

The three FAILs are expected then?  Those are the ones you mentioned
can't be easily XFAIL'd?


Diego.

On Sun, Jun 5, 2011 at 18:46, Lawrence Crowl <crowl@google.com> wrote:
> The purpose of this patch is to make PPH testing clean.
>
> Some tests have assembly comparison failures between regular compiles and PPH
> compiles.  When this miscompare is exected, we now mark the test with a
> comment "pph asm xdiff".  The driver lib/dg-pph.exp looks for this comment to
> report either FAIL or XFAIL.
>
> Test p1mean.cc enters infinite recursion.  Instead of disabling the test with
> a syntax error, we now run it under a dg-timeout.
>
> Test c1return-5.h was trying to grep assembly, but since we produce pre-parsed
> headers, not assembly, the test was in error.  We have moved the assembly
> check to c1return-5.cc where it belongs.
>
> All PPH tests expected to have PPH compilation failures are marked with
> // { dg-xfail-if "comment" { "*-*-*" } { "-fpph-map=pph.map" } }
> The regular compiles do not have the -fpph-map option, and hence are not
> expected to fail.  One test has a comment INFINITE; one test has a comment
> BOGUS; and the rest have a comment ICE.
>
> The driver lib/dg-pph.exp test control logic now stops a test when any
> compilation fails to produce assembly output.  If the test file contains the
> above directive, that compilation failure is expected and reported as such.
> Otherwise, it is reported as an unexpected failure.
>
> PPH is wrongly producing ICEs and errors.  We mark applicable messages with
> dg-bogus.  Because the line/column information is often wrong, we mark the
> with a line of "0", which avoids line comparisons.  We also prune output as
> needed.
>
> All tests are now either PASS or XFAIL, with the exception of unavoidable
> bogus XPASS messages for "test for bogus messages" and "test for excess
> errors".  These are unavoidable because they cannot be conditionalized based
> on compiler options, and they do not appear in regular compiles.
>
> If you run the output of
>
>    make check-c++ RUNTESTFLAGS=pph.exp
>
> through
>
>    sed -e '
>        /^XPASS: .*test for bogus messages/     d
>        /^XPASS: .*test for excess errors/      d
>
>        /^#/            p
>        /^ERROR: /      p
>        /^XFAIL: /      p
>        /^XPASS: /      p
>        /^FAIL: /       p
>
>        d
>    '
>
> you should get no individual tests listed.
>
>
> Index: gcc/testsuite/ChangeLog.pph
>
> 2011-06-05  Lawrence Crowl  <crowl@google.com>
>
>        * lib/dg-pph.exp (dg-pph-hdr): Add blank lines to logfile.
>        Format if statements and calls consistently.
>        (dg-pph-neg): Remove extraneous assembly files. Add blank
>        lines to logfile. Format if statements and calls consistently.
>        (dg-pph-pos): Stop test when a compilation fails. Recognize
>        expected failing compilations. Recognize expected assembly
>        miscompares. Parenthesize failure notes. Remove extraneous
>        assembly files. Add blank lines to logfile. Format if
>        statements and calls consistently.
>        * g++.dg/pph/c1simple.cc: Add comment "pph asm xdiff"
>        indicating expected assembly difference failure. Add line
>        spacing to test.
>        * g++.dg/pph/c1struct.cc: Likewise.
>        * g++.dg/pph/c1variables.cc: Likewise.
>        * g++.dg/pph/c2builtin1.cc: Likewise.
>        * g++.dg/pph/x1globalref.cc: Likewise.
>        * g++.dg/pph/x1hardlookup.cc: Likewise.
>        * g++.dg/pph/x1struct1.cc: Likewise.
>        * g++.dg/pph/x1struct2.cc: Likewise.
>        * g++.dg/pph/x2nontrivinit.cc: Likewise.
>        * c120060625-1.h: Add dg-xfail-if for -fpph-map. Add bogus
>        message for ICE. Prune output. Add line spacing to text.
>        * g++.dg/pph/c120060625-1.cc: Likewise.
>        * g++.dg/pph/c1eabi1.h: Likewise.
>        * g++.dg/pph/c1eabi1.cc: Likewise.
>        * g++.dg/pph/x1autometh.cc: Likewise.
>        * g++.dg/pph/x1functions.cc: Likewise.
>        * g++.dg/pph/x1special.cc: Likewise.
>        * g++.dg/pph/x1template.cc: Likewise.
>        * g++.dg/pph/x1tmplclass.cc: Likewise.
>        * g++.dg/pph/x1tmplfunc.cc: Likewise.
>        * g++.dg/pph/x1typerefs.cc: Likewise.
>        * g++.dg/pph/x1variables.cc: Likewise.
>        * g++.dg/pph/p1mean.cc: Change syntax error preventing infinite
>        recursion to a dg-timeout. Add dg-xfail-if for -fpph-map.
>        * g++.dg/pph/c1return-5.h: Remove dg-final as we do not generate
>        assembly on compiling headers. Change dg directive location.
>        * g++.dg/pph/c1return-5.cc: Add dg-final as we _do_ generate
>        assembly on compiling sources. Copy up dg-options from
>        c1return-5.h as well.
>
>
> Index: gcc/testsuite/lib/dg-pph.exp
> ===================================================================
> --- gcc/testsuite/lib/dg-pph.exp        (revision 174608)
> +++ gcc/testsuite/lib/dg-pph.exp        (working copy)
> @@ -24,13 +24,14 @@ proc dg-pph-hdr { subdir test options ma
>     global runtests dg-do-what-default
>
>     # If we're only testing specific files and this isn't one of them, skip
it.
> -    if ![runtest_file_p $runtests $test] {
> +    if { ![runtest_file_p $runtests $test] } {
>        return
>     }
>
>     set nshort "$subdir/[file tail $test]"
>     set bname "[file rootname [file tail $nshort]]"
> -    verbose "Testing $nshort, $options" 1
> +    verbose -log "\nTesting $nshort, $options"
> +
>     set dg-do-what-default preparse
>     dg-test -keep-output $test "$options $mapflag -I." ""
>
> @@ -41,16 +42,19 @@ proc dg-pph-neg { subdir test options ma
>     global runtests dg-do-what-default
>
>     # If we're only testing specific files and this isn't one of them, skip
it.
> -    if ![runtest_file_p $runtests $test] {
> +    if { ![runtest_file_p $runtests $test] } {
>        return
>     }
>
>     set nshort "$subdir/[file tail $test]"
>     set bname "[file rootname [file tail $nshort]]"
> -    verbose "Testing $nshort, $options" 1
> +    verbose -log "\nTesting $nshort, $options"
> +
>     set dg-do-what-default compile
>     dg-test -keep-output $test "$options $mapflag -I." ""
> -    file_on_host delete "$bname.s"
> +    if { ![file_on_host exists "$bname.s"] } {
> +       file_on_host delete "$bname.s"
> +    }
>  }
>
>  proc dg-pph-pos { subdir test options mapflag suffix } {
> @@ -58,64 +62,72 @@ proc dg-pph-pos { subdir test options ma
>     global runtests dg-do-what-default
>
>     # If we're only testing specific files and this isn't one of them, skip
it.
> -    if ![runtest_file_p $runtests $test] {
> +    if { ![runtest_file_p $runtests $test] } {
>        return
>     }
>
>     set nshort "$subdir/[file tail $test]"
>     set bname "[file rootname [file tail $nshort]]"
> -    verbose "Testing $nshort, $options" 1
> -    set dg-do-what-default compile
> +    verbose -log "\nTesting $nshort, $options"
>
> -    set have_errs [llength [grep $test "{\[ \t\]\+dg-error\[ \t\]\+.*\[
\t\]\+}"]]
>     # Compile the file the first time for a base case.
> +    set dg-do-what-default compile
>     dg-test -keep-output $test "$options -I." ""
>
> -    if { $have_errs } {
> -       verbose -log "regular compilation failed"
> -       fail "$nshort $options, regular compilation failed"
> -       return
> -    }
> -
> -    if { ! [ file_on_host exists "$bname.s" ] } {
> -       verbose -log "regular assembly file '$bname.s' missing"
> -       fail "$nshort $options, regular assembly missing"
> +    # Quit if it did not compile successfully.
> +    if { ![file_on_host exists "$bname.s"] } {
> +       # All regular compiles should pass.
> +       fail "$nshort $options (regular assembly missing)"
>        return
>     }
>
>     # Rename the .s file into .s-pph to compare it after the second build.
>     remote_upload host "$bname.s" "$bname.s-pph"
>     remote_download host "$bname.s-pph"
> +    file_on_host delete "$bname.s"
> +
> +    verbose -log ""
>
>     # Compile a second time using the pph files.
>     dg-test -keep-output $test "$options $mapflag -I." ""
>
> -    if { $have_errs } {
> -       verbose -log "PPH compilation failed"
> -       fail "$nshort $options, PPH compilation failed"
> -       return
> -    }
> -
> -    if { ! [ file_on_host exists "$bname.s" ] } {
> -       verbose -log "PPH assembly file '$bname.s' missing"
> -       fail "$nshort $options, PPH assembly missing"
> +    # Quit if it did not compile successfully.
> +    if { ![file_on_host exists "$bname.s"] } {
> +       # Expect assembly to be missing when the compile is an expected fail.
> +       if { [llength [grep $test "dg-xfail-if.*-fpph-map"]] } {
> +           xfail "$nshort $options (pph assembly missing)"
> +       } else {
> +           fail "$nshort $options (pph assembly missing)"
> +       }
>        return
>     }
>
>     # Rename the .s file into .s+pph to compare it.
>     remote_upload host "$bname.s" "$bname.s+pph"
>     remote_download host "$bname.s+pph"
> +    file_on_host delete "$bname.s"
> +
> +    verbose -log ""
>
>     # Compare the two assembly files.  They should be identical.
> -    set tmp [ diff "$bname.s-pph" "$bname.s+pph" ]
> -    if { $tmp == 0 } {
> -       verbose -log "assembly file '$bname.s-pph', '$bname.s+pph' comparison
error"
> -       fail "$nshort $options assembly comparison"
> -    } elseif { $tmp == 1 } {
> -       pass "$nshort $options assembly comparison"
> +    set adiff [diff "$bname.s-pph" "$bname.s+pph"]
> +    # The sources mark when they expect the comparison to differ.
> +    set xdiff [llength [grep $test "pph asm xdiff"]]
> +    if { $adiff == 0 } {
> +       fail "$nshort $options comparison failure"
> +    } elseif { $adiff == 1 } {
> +       if { $xdiff } {
> +           xpass "$nshort $options (assembly identical)"
> +       } else {
> +           pass "$nshort $options (assembly identical)"
> +       }
>        file_on_host delete "$bname.s-pph"
>        file_on_host delete "$bname.s+pph"
>     } else {
> -       fail "$nshort $options assembly comparison"
> +       if { $xdiff } {
> +           xfail "$nshort $options (assembly mismatch)"
> +       } else {
> +           fail "$nshort $options (assembly mismatch)"
> +       }
>     }
>  }
> Index: gcc/testsuite/g++.dg/pph/x1globalref.cc
> ===================================================================
> --- gcc/testsuite/g++.dg/pph/x1globalref.cc     (revision 174608)
> +++ gcc/testsuite/g++.dg/pph/x1globalref.cc     (working copy)
> @@ -1,4 +1,7 @@
> +// pph asm xdiff
> +
>  #include "x1globalref.h"
> +
>  type x = 2;
>  type y = x;
>  type f() { return n+m+a[3]+x+y; }
> Index: gcc/testsuite/g++.dg/pph/c1eabi1.h
> ===================================================================
> --- gcc/testsuite/g++.dg/pph/c1eabi1.h  (revision 174608)
> +++ gcc/testsuite/g++.dg/pph/c1eabi1.h  (working copy)
> @@ -1,8 +1,12 @@
> +// { dg-xfail-if "ICE" { "*-*-*" } { "-fpph-map=pph.map" } }
> +// { dg-bogus "mathcalls.h:365:1: internal compiler error: Segmentation
fault" "" { xfail *-*-* } 0 }
> +// { dg-prune-output "In file included " }
> +// { dg-prune-output "                 from " }
> +// { dg-options "-w -fpermissive" }
> +
>  #ifndef __PPH_GUARD_H
>  #define __PPH_GUARD_H
> -/* { dg-options "-w -fpermissive" } */
> -/* { dg-xfail-if "ICEs the compiler during PPH read" { "*-*-*" } } */
> -/* { dg-prune-output "In file included.*" } */
> +
>  /* This file tests most of the non-C++ run-time helper functions
>    described in Section 4 of the "Run-Time ABI for the ARM
>    Architecture".  These are basic tests; they do not try to validate
> Index: gcc/testsuite/g++.dg/pph/c1struct.cc
> ===================================================================
> --- gcc/testsuite/g++.dg/pph/c1struct.cc        (revision 174608)
> +++ gcc/testsuite/g++.dg/pph/c1struct.cc        (working copy)
> @@ -1,3 +1,6 @@
> +// pph asm xdiff
> +
>  #include "c1struct.h"
> +
>  thing var1;
>  thing var2 = { 3 };
> Index: gcc/testsuite/g++.dg/pph/x1typerefs.cc
> ===================================================================
> --- gcc/testsuite/g++.dg/pph/x1typerefs.cc      (revision 174608)
> +++ gcc/testsuite/g++.dg/pph/x1typerefs.cc      (working copy)
> @@ -1,5 +1,11 @@
> +// { dg-xfail-if "ICE" { "*-*-*" } { "-fpph-map=pph.map" } }
> +// { dg-bogus "c1typerefs.h:11:18: internal compiler error: Segmentation
fault" "" { xfail *-*-* } 0 }
> +// { dg-prune-output "In file included from " }
> +
>  #include "x1typerefs.h"
> +
>  int derived::method() {
>     return field;
>  }
> +
>  vderived variable;
> Index: gcc/testsuite/g++.dg/pph/x1struct2.cc
> ===================================================================
> --- gcc/testsuite/g++.dg/pph/x1struct2.cc       (revision 174608)
> +++ gcc/testsuite/g++.dg/pph/x1struct2.cc       (working copy)
> @@ -1,7 +1,11 @@
> +// pph asm xdiff
> +
>  #include "x1struct2.h"
> +
>  type D::method()
>  { static int x = 2;
>   return fld + mbr + gbl; }
> +
>  type D::mbr = 4;
>  typedef D D2;
>  D2 var1;
> Index: gcc/testsuite/g++.dg/pph/p1mean.cc
> ===================================================================
> --- gcc/testsuite/g++.dg/pph/p1mean.cc  (revision 174608)
> +++ gcc/testsuite/g++.dg/pph/p1mean.cc  (working copy)
> @@ -1,10 +1,11 @@
> +// { dg-timeout 2 { target *-*-* } }
> +// { dg-xfail-if "INFINITE" { "*-*-*" } { "-fpph-map=pph.map" } }
> +
>  #include <stdlib.h>
>  #include <stdio.h>
>  #include <math.h>
>  #include <string.h>
>
> -SYNTAX ERROR TO AVOID COMPILING THIS - IT CAUSES INFINITE RECURSION.
> -
>  static unsigned long long MAX_ITEMS = 10000;
>
>  static int
> Index: gcc/testsuite/g++.dg/pph/x1autometh.cc
> ===================================================================
> --- gcc/testsuite/g++.dg/pph/x1autometh.cc      (revision 174608)
> +++ gcc/testsuite/g++.dg/pph/x1autometh.cc      (working copy)
> @@ -1,4 +1,9 @@
> +// { dg-xfail-if "ICE" { "*-*-*" } { "-fpph-map=pph.map" } }
> +// { dg-bogus "x1autometh.h:8:1: internal compiler error: Segmentation fault"
"" { xfail *-*-* } 0 }
> +// { dg-prune-output "In file included from " }
> +
>  #include "x1autometh.h"
> +
>  void function() {
>     base var1;
>     base var2( var1 );
> Index: gcc/testsuite/g++.dg/pph/x1tmplfunc.cc
> ===================================================================
> --- gcc/testsuite/g++.dg/pph/x1tmplfunc.cc      (revision 174608)
> +++ gcc/testsuite/g++.dg/pph/x1tmplfunc.cc      (working copy)
> @@ -1,4 +1,9 @@
> +// { dg-xfail-if "ICE" { "*-*-*" } { "-fpph-map=pph.map" } }
> +// { dg-bogus "x1tmplfunc.h:12:30: internal compiler error: Segmentation
fault" "" { xfail *-*-* } 0 }
> +// { dg-prune-output "In file included from " }
> +
>  #include "x1tmplfunc.h"
> +
>  type val = 3;
>
>  template<>
> Index: gcc/testsuite/g++.dg/pph/c1return-5.h
> ===================================================================
> --- gcc/testsuite/g++.dg/pph/c1return-5.h       (revision 174608)
> +++ gcc/testsuite/g++.dg/pph/c1return-5.h       (working copy)
> @@ -1,7 +1,7 @@
> +// { dg-options "-mpreferred-stack-boundary=4" }
> +
>  #ifndef __PPH_GUARD_H
>  #define __PPH_GUARD_H
> -/* { dg-options "-mpreferred-stack-boundary=4" } */
> -/* { dg-final { scan-assembler-not "and\[lq\]?\[^\\n\]*-64,\[^\\n\]*sp" } }
*/
>
>  /* This compile only test is to detect an assertion failure in stack branch
>    development.  */
> Index: gcc/testsuite/g++.dg/pph/x1struct1.cc
> ===================================================================
> --- gcc/testsuite/g++.dg/pph/x1struct1.cc       (revision 174608)
> +++ gcc/testsuite/g++.dg/pph/x1struct1.cc       (working copy)
> @@ -1,7 +1,11 @@
> +// pph asm xdiff
> +
>  #include "x1struct1.h"
> +
>  type D::method()
>  { static int x = 2;
>   return fld + mbr; }
> +
>  type D::mbr = 4;
>  typedef D D2;
>  D2 var1;
> Index: gcc/testsuite/g++.dg/pph/c120060625-1.cc
> ===================================================================
> --- gcc/testsuite/g++.dg/pph/c120060625-1.cc    (revision 174608)
> +++ gcc/testsuite/g++.dg/pph/c120060625-1.cc    (working copy)
> @@ -1 +1,5 @@
> +// { dg-xfail-if "ICE" { "*-*-*" } { "-fpph-map=pph.map" } }
> +// { dg-bogus "c120060625-1.h:14:22: internal compiler error: invalid
built-in macro .__FLT_MAX__." "" { xfail *-*-* } 0 }
> +// { dg-prune-output "In file included from " }
> +
>  #include "c120060625-1.h"
> Index: gcc/testsuite/g++.dg/pph/x1special.cc
> ===================================================================
> --- gcc/testsuite/g++.dg/pph/x1special.cc       (revision 174608)
> +++ gcc/testsuite/g++.dg/pph/x1special.cc       (working copy)
> @@ -1,4 +1,9 @@
> +// { dg-xfail-if "ICE" { "*-*-*" } { "-fpph-map=pph.map" } }
> +// { dg-bogus "x1special.h:19:6: internal compiler error: tree check:
expected tree that contains 'decl minimal' structure, have 'overload' in
context_for_name_lookup, at cp/search.c:570" "" { xfail *-*-* } 0 }
> +// { dg-prune-output "In file included from " }
> +
>  #include "x1special.h"
> +
>  B b(1);
>
>  int H() {
> Index: gcc/testsuite/g++.dg/pph/c1variables.cc
> ===================================================================
> --- gcc/testsuite/g++.dg/pph/c1variables.cc     (revision 174608)
> +++ gcc/testsuite/g++.dg/pph/c1variables.cc     (working copy)
> @@ -1,4 +1,7 @@
> +// pph asm xdiff
> +
>  #include "c1variables.h"
> +
>  int gbl_initial = 1;
>  const float gbl_init_const = 1.5;
>  const int gbl_manifest = 2;
> Index: gcc/testsuite/g++.dg/pph/x2nontrivinit.cc
> ===================================================================
> --- gcc/testsuite/g++.dg/pph/x2nontrivinit.cc   (revision 174608)
> +++ gcc/testsuite/g++.dg/pph/x2nontrivinit.cc   (working copy)
> @@ -1 +1,3 @@
> +// pph asm xdiff
> +
>  #include "x2nontrivinit.h"
> Index: gcc/testsuite/g++.dg/pph/c1eabi1.cc
> ===================================================================
> --- gcc/testsuite/g++.dg/pph/c1eabi1.cc (revision 174608)
> +++ gcc/testsuite/g++.dg/pph/c1eabi1.cc (working copy)
> @@ -1,4 +1,7 @@
> -/* { dg-options "-w -fpermissive" } */
> -/* { dg-xfail-if "ICEs the compiler during PPH read" { "*-*-*" } } */
> -/* { dg-prune-output "In file included.*" } */
> +// { dg-xfail-if "ICE" { "*-*-*" } { "-fpph-map=pph.map" } }
> +// { dg-bogus "mathcalls.h:365:1: internal compiler error: Segmentation
fault" "" { xfail *-*-* } 0 }
> +// { dg-prune-output "In file included from " }
> +// { dg-prune-output "                 from " }
> +// { dg-options "-w -fpermissive" }
> +
>  #include "c1eabi1.h"
> Index: gcc/testsuite/g++.dg/pph/c1simple.cc
> ===================================================================
> --- gcc/testsuite/g++.dg/pph/c1simple.cc        (revision 174608)
> +++ gcc/testsuite/g++.dg/pph/c1simple.cc        (working copy)
> @@ -1,5 +1,9 @@
> +// pph asm xdiff
> +
>  /* comment */
> +
>  #include "c1simple2.h"
> +
>  int main()
>  {
>        return foo();
> Index: gcc/testsuite/g++.dg/pph/x1functions.cc
> ===================================================================
> --- gcc/testsuite/g++.dg/pph/x1functions.cc     (revision 174608)
> +++ gcc/testsuite/g++.dg/pph/x1functions.cc     (working copy)
> @@ -1,5 +1,14 @@
> +// { dg-xfail-if "ICE" { "*-*-*" } { "-fpph-map=pph.map" } }
> +// { dg-bogus "'mbr_decl_inline' was not declared in this scope" "" { xfail
*-*-* } 0 }
> +// { dg-bogus "c1functions.h:8:34: internal compiler error: Segmentation
fault" "" { xfail *-*-* } 0 }
> +// { dg-prune-output "In file included from " }
> +// { dg-prune-output "In member function " }
> +// { dg-prune-output "At global scope:" }
> +
>  #include "x1functions.h"
> +
>  int type::mbr_decl_then_def(int i)      // need body
>  { return mbr_decl_inline( i ); }
> +
>  int type::mbr_inl_then_def(int i)       // lazy body
>  { return mbr_decl_then_def( i ); }
> Index: gcc/testsuite/g++.dg/pph/x1template.cc
> ===================================================================
> --- gcc/testsuite/g++.dg/pph/x1template.cc      (revision 174608)
> +++ gcc/testsuite/g++.dg/pph/x1template.cc      (working copy)
> @@ -1,9 +1,17 @@
> +// { dg-xfail-if "ICE" { "*-*-*" } { "-fpph-map=pph.map" } }
> +// { dg-bogus "x1template.h:18:13: internal compiler error: in resume_scope,
at cp/name-lookup.c:1573" "" { xfail *-*-* } 0 }
> +// { dg-prune-output "In file included from " }
> +
>  #include "x1template.h"
> +
>  namespace A {
>  int x = 3;
>  } // namespace A
> +
>  int y = 4;
> +
>  int D::method()
>  { return y; }
> +
>  int main()
>  { }
> Index: gcc/testsuite/g++.dg/pph/c120060625-1.h
> ===================================================================
> --- gcc/testsuite/g++.dg/pph/c120060625-1.h     (revision 174608)
> +++ gcc/testsuite/g++.dg/pph/c120060625-1.h     (working copy)
> @@ -1,5 +1,9 @@
> +// { dg-xfail-if "ICE" { "*-*-*" } { "-fpph-map=pph.map" } }
> +// { dg-bogus "c120060625-1.h:14:22: internal compiler error: invalid
built-in macro .__FLT_MAX__." "" { xfail *-*-* } 0 }
> +
>  #ifndef __PPH_GUARD_H
>  #define __PPH_GUARD_H
> +
>  /* PR middle-end/28151 */
>  /* Testcase by Steven Bosscher <stevenb.gcc@gmail.com> */
>
> @@ -11,3 +15,4 @@ void foo (void)
>   b = __FLT_MAX__ + a;
>  }
>  #endif
> +
> Index: gcc/testsuite/g++.dg/pph/c1return-5.cc
> ===================================================================
> --- gcc/testsuite/g++.dg/pph/c1return-5.cc      (revision 174608)
> +++ gcc/testsuite/g++.dg/pph/c1return-5.cc      (working copy)
> @@ -1 +1,4 @@
> +// { dg-options "-mpreferred-stack-boundary=4" }
> +// { dg-final { scan-assembler-not "and\[lq\]?\[^\\n\]*-64,\[^\\n\]*sp" } }
> +
>  #include "c1return-5.h"
> Index: gcc/testsuite/g++.dg/pph/x1tmplclass.cc
> ===================================================================
> --- gcc/testsuite/g++.dg/pph/x1tmplclass.cc     (revision 174608)
> +++ gcc/testsuite/g++.dg/pph/x1tmplclass.cc     (working copy)
> @@ -1,4 +1,9 @@
> +// { dg-xfail-if "ICE" { "*-*-*" } { "-fpph-map=pph.map" } }
> +// { dg-bogus "x1tmplclass.h:14:17: internal compiler error: Segmentation
fault" "" { xfail *-*-* } 0 }
> +// { dg-prune-output "In file included from " }
> +
>  #include "x1tmplclass.h"
> +
>  int wrapper<char>::cache = 2;
>
>  template
> Index: gcc/testsuite/g++.dg/pph/c2builtin1.cc
> ===================================================================
> --- gcc/testsuite/g++.dg/pph/c2builtin1.cc      (revision 174608)
> +++ gcc/testsuite/g++.dg/pph/c2builtin1.cc      (working copy)
> @@ -1,3 +1,5 @@
> +// pph asm xdiff
> +
>  #include "c2builtin1.h"
>  #include "c2builtin2.h"
>  #include "c2builtin3.h"
> Index: gcc/testsuite/g++.dg/pph/x1hardlookup.cc
> ===================================================================
> --- gcc/testsuite/g++.dg/pph/x1hardlookup.cc    (revision 174608)
> +++ gcc/testsuite/g++.dg/pph/x1hardlookup.cc    (working copy)
> @@ -1,5 +1,9 @@
> +// pph asm xdiff
> +
>  #include "x1hardlookup.h"
> +
>  D P;
> +
>  int I(int arg)
>  { return arg + V; }
>
> Index: gcc/testsuite/g++.dg/pph/x1variables.cc
> ===================================================================
> --- gcc/testsuite/g++.dg/pph/x1variables.cc     (revision 174608)
> +++ gcc/testsuite/g++.dg/pph/x1variables.cc     (working copy)
> @@ -1,4 +1,10 @@
> +// { dg-xfail-if "BOGUS" { "*-*-*" } { "-fpph-map=pph.map" } }
> +// { dg-bogus "c1variables.h:4:8: error: 'int D::mbr_uninit_plain' is not a
static member of 'struct D'" "" { xfail *-*-* } 0 }
> +// { dg-bogus "c1variables.h:5:14: error: 'const int D::mbr_init_const' is
not a static member of 'struct D'" "" { xfail *-*-* } 0 }
> +// { dg-prune-output "In file included from " }
> +
>  #include "x1variables.h"
> +
>  int D::mbr_init_plain = 4;
>  int D::mbr_uninit_plain;
>  const int D::mbr_init_const = 5;
>
> --
> This patch is available for review at http://codereview.appspot.com/4572042
>

[Lawrence Crowl, 2011-06-06 18:33:40]

On 6/6/11, Diego Novillo <dnovillo@google.com> wrote:
> So, I'm getting this:
>
> Running /home/dnovillo/pph/svn/src/gcc/testsuite/g++.dg/pph/pph.exp ...
> XPASS: g++.dg/pph/c120060625-1.cc  -I.  (test for bogus messages, line )
> XPASS: g++.dg/pph/c1eabi1.cc  -I.  (test for bogus messages, line )
> XPASS: g++.dg/pph/x1autometh.cc  -I.  (test for bogus messages, line )
> XPASS: g++.dg/pph/x1functions.cc  -I.  (test for bogus messages, line )
> XPASS: g++.dg/pph/x1functions.cc  -I.  (test for bogus messages, line )
> XPASS: g++.dg/pph/x1special.cc  -I.  (test for bogus messages, line )
> XPASS: g++.dg/pph/x1special.cc  -fpph-map=pph.map -I.  (test for bogus
> messages, line )
> FAIL: g++.dg/pph/x1special.cc  -fpph-map=pph.map -I. (test for excess
> errors)
> XPASS: g++.dg/pph/x1template.cc  -I.  (test for bogus messages, line )
> XPASS: g++.dg/pph/x1template.cc  -fpph-map=pph.map -I.  (test for
> bogus messages, line )
> FAIL: g++.dg/pph/x1template.cc  -fpph-map=pph.map -I. (test for excess
> errors)
> XPASS: g++.dg/pph/x1tmplclass.cc  -I.  (test for bogus messages, line )
> XPASS: g++.dg/pph/x1tmplfunc.cc  -I.  (test for bogus messages, line )
> XPASS: g++.dg/pph/x1tmplfunc.cc  -fpph-map=pph.map -I.  (test for
> bogus messages, line )
> FAIL: g++.dg/pph/x1tmplfunc.cc  -fpph-map=pph.map -I. (test for excess
> errors)
> XPASS: g++.dg/pph/x1typerefs.cc  -I.  (test for bogus messages, line )
> XPASS: g++.dg/pph/x1variables.cc  -I.  (test for bogus messages, line )
> XPASS: g++.dg/pph/x1variables.cc  -I.  (test for bogus messages, line )
> XPASS: g++.dg/pph/x1variables.cc  -fpph-map=pph.map -I.  (test for
> bogus messages, line )
> XPASS: g++.dg/pph/x1variables.cc  -fpph-map=pph.map -I.  (test for
> bogus messages, line )
>
>                 === g++ Summary ===
>
> # of expected passes            176
> # of unexpected failures        3
> # of unexpected successes       17
> # of expected failures          45
>
> The three FAILs are expected then?  Those are the ones you mentioned
> can't be easily XFAIL'd?

These are probably changes in the line number of the ICE,
and so the dg-bogus needs updating.

-- 
Lawrence Crowl

[Diego Novillo, 2011-06-07 18:43:31]

After getting new failures due to an unrelated fix, I think this will
be more trouble than it's worth.

First, we can't get rid of the XPASSes, so those will always be noisy.

Second, some XPASSes will need to be unmarked because we just fixed
the underlying problem.

Third, we are at such an early stage, that fixes to a test case will
generally expose failures in other already failing tests, but these
failures will be in a different place.  So more noise.

I really think that for now the easiest way to keep track of this is
to have a clean build to compare against.


Diego.

[Lawrence Crowl, 2011-06-08 21:09:34]

On 6/7/11, Diego Novillo <dnovillo@google.com> wrote:
> After getting new failures due to an unrelated fix, I think this will
> be more trouble than it's worth.
>
> First, we can't get rid of the XPASSes, so those will always be noisy.

Remove all the noise with:

sed -e '
        /-fpph-map=pph.map/     ! {
                /^XPASS: .*test for bogus messages/     d
                /^XPASS: .*test for excess errors/      d
        }

        /^#/            p
        /^ERROR: /      p
        /^XFAIL: /      p
        /^XPASS: /      p
        /^FAIL: /       p

        d
'

>
> Second, some XPASSes will need to be unmarked because we just fixed
> the underlying problem.

That is how we know that what progress we have made.  These should
be reflected in the work list.

>
> Third, we are at such an early stage, that fixes to a test case will
> generally expose failures in other already failing tests, but these
> failures will be in a different place.  So more noise.

That is now we expose remaining tasks.  These should be reflected
in the work list.

>
> I really think that for now the easiest way to keep track of this is
> to have a clean build to compare against.

But the clean build isn't a decent comparison.  The existing tools
only compare against whether or not tests fail.  They fail to
report that a test failed for different reasons.  Given that there
are only twelve tests for which there is a concern, I think we are
better off know where we are.

-- 
Lawrence Crowl

[Lawrence Crowl, 2011-06-08 21:16:49]

On 6/7/11, Diego Novillo <dnovillo@google.com> wrote:
> After getting new failures due to an unrelated fix, I think this will
> be more trouble than it's worth.
>
> First, we can't get rid of the XPASSes, so those will always be noisy.

Remove all the noise with:

sed -e '
        /-fpph-map=pph.map/     ! {
                /^XPASS: .*test for bogus messages/     d
                /^XPASS: .*test for excess errors/      d
        }

        /^#/            p
        /^ERROR: /      p
        /^XFAIL: /      p
        /^XPASS: /      p
        /^FAIL: /       p

        d
'

>
> Second, some XPASSes will need to be unmarked because we just fixed
> the underlying problem.

That is how we know that what progress we have made.  These should
be reflected in the work list.

>
> Third, we are at such an early stage, that fixes to a test case will
> generally expose failures in other already failing tests, but these
> failures will be in a different place.  So more noise.

That is now we expose remaining tasks.  These should be reflected
in the work list.

>
> I really think that for now the easiest way to keep track of this is
> to have a clean build to compare against.

But the clean build isn't a decent comparison.  The existing tools
only compare against whether or not tests fail.  They fail to
report that a test failed for different reasons.  Given that there
are only twelve tests for which there is a concern, I think we are
better off know where we are.

-- 
Lawrence Crowl