glibc/scripts/evaluate-test.sh

46 lines
1.1 KiB
Bash
Raw Normal View History

Generate .test-result files for ordinary tests. This patch, an updated version of <https://sourceware.org/ml/libc-alpha/2014-01/msg00193.html>, starts the process of generating explicit PASS or FAIL status for individual glibc tests. It's based on Tomas Dohnalek's patch <https://sourceware.org/ml/libc-alpha/2012-10/msg00278.html>, but is deliberately more minimal: it doesn't try to cover any tests outside of $(tests) / $(xtests) (that's for a later patch), nor does it put the result together in an overall summary file (again, a later patch): it just generates the .test-result files. Thus, this patch keeps the overall logic for when a testsuite run finishes completely unchanged: a test failing will terminate the run. I think we *should* move to a more conventional approach where plain "make check" does not terminate for an individual test failure, unless e.g. you say "make stop-on-test-failure=y check", but that sort of policy change is best done as a separate patch once the infrastructure is in place to generate summary files for completed test runs (which will entirely consist of PASS and XFAIL lines if the testsuite run reaches the point of generating them, until such a policy change is made). Tested x86_64. 2014-02-14 Tomas Dohnalek <tdohnale@redhat.com> Joseph Myers <joseph@codesourcery.com> * Makeconfig (test-name): New variable. (evaluate-test): Likewise. * Makerules (do-test-clean): Remove .test-result files. (common-mostlyclean): Likewise. * Rules ($(objpfx)%.out): Use $(evaluate-test) in both rules. * scripts/evaluate-test.sh: New file.
2014-02-15 01:04:57 +00:00
#! /bin/sh
# Output a test status line.
# Copyright (C) 2012-2015 Free Software Foundation, Inc.
Generate .test-result files for ordinary tests. This patch, an updated version of <https://sourceware.org/ml/libc-alpha/2014-01/msg00193.html>, starts the process of generating explicit PASS or FAIL status for individual glibc tests. It's based on Tomas Dohnalek's patch <https://sourceware.org/ml/libc-alpha/2012-10/msg00278.html>, but is deliberately more minimal: it doesn't try to cover any tests outside of $(tests) / $(xtests) (that's for a later patch), nor does it put the result together in an overall summary file (again, a later patch): it just generates the .test-result files. Thus, this patch keeps the overall logic for when a testsuite run finishes completely unchanged: a test failing will terminate the run. I think we *should* move to a more conventional approach where plain "make check" does not terminate for an individual test failure, unless e.g. you say "make stop-on-test-failure=y check", but that sort of policy change is best done as a separate patch once the infrastructure is in place to generate summary files for completed test runs (which will entirely consist of PASS and XFAIL lines if the testsuite run reaches the point of generating them, until such a policy change is made). Tested x86_64. 2014-02-14 Tomas Dohnalek <tdohnale@redhat.com> Joseph Myers <joseph@codesourcery.com> * Makeconfig (test-name): New variable. (evaluate-test): Likewise. * Makerules (do-test-clean): Remove .test-result files. (common-mostlyclean): Likewise. * Rules ($(objpfx)%.out): Use $(evaluate-test) in both rules. * scripts/evaluate-test.sh: New file.
2014-02-15 01:04:57 +00:00
# This file is part of the GNU C Library.
# The GNU C Library is free software; you can redistribute it and/or
# modify it under the terms of the GNU Lesser General Public
# License as published by the Free Software Foundation; either
# version 2.1 of the License, or (at your option) any later version.
# The GNU C Library is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# Lesser General Public License for more details.
# You should have received a copy of the GNU Lesser General Public
# License along with the GNU C Library; if not, see
# <http://www.gnu.org/licenses/>.
Do not terminate default test runs on test failure. This patch is an updated version of <https://sourceware.org/ml/libc-alpha/2014-01/msg00198.html> and <https://sourceware.org/ml/libc-alpha/2014-03/msg00180.html>. Normal practice for software testsuites is that rather than terminating immediately when a test fails, they continue running and report at the end on how many tests passed or failed. The principle behind the glibc testsuite stopping on failure was probably that the expected state is no failures and so any failure indicates a problem such as miscompilation. In practice, while this is fairly close to true for native testing on x86_64 and x86 (kernel bugs and race conditions can still cause intermittent failures), it's less likely to be the case on other platforms, and so people testing glibc run the testsuite with "make -k" and then examine the logs to determine whether the failures are what they expect to fail on that platform, possibly with some automation for the comparison. This patch switches the glibc testsuite to the normal convention of not stopping on failure - unless you use stop-on-test-failure=y, in which case it behaves essentially as it did before (and does not generate overall test summaries on failure). Instead, the summary tests.sum may contain tests that FAILed. At the end of the test run, any FAIL or ERROR lines from tests.sum are printed, and then it exits with error status if there were any such lines. In addition, build failures will also cause the test run to stop - this has the justification that those *do* indicate serious problems that should be promptly fixed and aren't generally hard to fix (but apart from that, avoiding the build stopping on those failures seems harder). Note that unlike the previous patches in this series, this *does* require people with automation around testing glibc to change their processes - either to start using tests.sum / xtests.sum to track failures and compare them with expectations (with or without also using "make -k" and examining "make" logs to identify build failures), or else to use stop-on-test-failure=y and ignore the new tests.sum / xtests.sum mechanism. (If all you check is the exit status from "make check", no changes are needed unless you want to avoid test runs continuing after the first failure.) Tested x86_64. * scripts/evaluate-test.sh: Handle fourth argument to determine whether test run should stop on failure. * Makeconfig (stop-on-test-failure): New variable. (evaluate-test): Pass fourth argument to evaluate-test.sh based on $(stop-on-test-failure). * Makefile (tests): Give a summary of results from testing and exit with failure status if they include an ERROR or FAIL. (xtests): Likewise. * manual/install.texi (Configuring and compiling): Mention stop-on-test-failure=y. * INSTALL: Regenerated.
2014-03-14 21:02:40 +00:00
# usage: evaluate-test.sh test_name rc xfail stop_on_failure
Generate .test-result files for ordinary tests. This patch, an updated version of <https://sourceware.org/ml/libc-alpha/2014-01/msg00193.html>, starts the process of generating explicit PASS or FAIL status for individual glibc tests. It's based on Tomas Dohnalek's patch <https://sourceware.org/ml/libc-alpha/2012-10/msg00278.html>, but is deliberately more minimal: it doesn't try to cover any tests outside of $(tests) / $(xtests) (that's for a later patch), nor does it put the result together in an overall summary file (again, a later patch): it just generates the .test-result files. Thus, this patch keeps the overall logic for when a testsuite run finishes completely unchanged: a test failing will terminate the run. I think we *should* move to a more conventional approach where plain "make check" does not terminate for an individual test failure, unless e.g. you say "make stop-on-test-failure=y check", but that sort of policy change is best done as a separate patch once the infrastructure is in place to generate summary files for completed test runs (which will entirely consist of PASS and XFAIL lines if the testsuite run reaches the point of generating them, until such a policy change is made). Tested x86_64. 2014-02-14 Tomas Dohnalek <tdohnale@redhat.com> Joseph Myers <joseph@codesourcery.com> * Makeconfig (test-name): New variable. (evaluate-test): Likewise. * Makerules (do-test-clean): Remove .test-result files. (common-mostlyclean): Likewise. * Rules ($(objpfx)%.out): Use $(evaluate-test) in both rules. * scripts/evaluate-test.sh: New file.
2014-02-15 01:04:57 +00:00
test_name=$1
rc=$2
orig_rc=$rc
xfail=$3
Do not terminate default test runs on test failure. This patch is an updated version of <https://sourceware.org/ml/libc-alpha/2014-01/msg00198.html> and <https://sourceware.org/ml/libc-alpha/2014-03/msg00180.html>. Normal practice for software testsuites is that rather than terminating immediately when a test fails, they continue running and report at the end on how many tests passed or failed. The principle behind the glibc testsuite stopping on failure was probably that the expected state is no failures and so any failure indicates a problem such as miscompilation. In practice, while this is fairly close to true for native testing on x86_64 and x86 (kernel bugs and race conditions can still cause intermittent failures), it's less likely to be the case on other platforms, and so people testing glibc run the testsuite with "make -k" and then examine the logs to determine whether the failures are what they expect to fail on that platform, possibly with some automation for the comparison. This patch switches the glibc testsuite to the normal convention of not stopping on failure - unless you use stop-on-test-failure=y, in which case it behaves essentially as it did before (and does not generate overall test summaries on failure). Instead, the summary tests.sum may contain tests that FAILed. At the end of the test run, any FAIL or ERROR lines from tests.sum are printed, and then it exits with error status if there were any such lines. In addition, build failures will also cause the test run to stop - this has the justification that those *do* indicate serious problems that should be promptly fixed and aren't generally hard to fix (but apart from that, avoiding the build stopping on those failures seems harder). Note that unlike the previous patches in this series, this *does* require people with automation around testing glibc to change their processes - either to start using tests.sum / xtests.sum to track failures and compare them with expectations (with or without also using "make -k" and examining "make" logs to identify build failures), or else to use stop-on-test-failure=y and ignore the new tests.sum / xtests.sum mechanism. (If all you check is the exit status from "make check", no changes are needed unless you want to avoid test runs continuing after the first failure.) Tested x86_64. * scripts/evaluate-test.sh: Handle fourth argument to determine whether test run should stop on failure. * Makeconfig (stop-on-test-failure): New variable. (evaluate-test): Pass fourth argument to evaluate-test.sh based on $(stop-on-test-failure). * Makefile (tests): Give a summary of results from testing and exit with failure status if they include an ERROR or FAIL. (xtests): Likewise. * manual/install.texi (Configuring and compiling): Mention stop-on-test-failure=y. * INSTALL: Regenerated.
2014-03-14 21:02:40 +00:00
stop_on_failure=$4
Generate .test-result files for ordinary tests. This patch, an updated version of <https://sourceware.org/ml/libc-alpha/2014-01/msg00193.html>, starts the process of generating explicit PASS or FAIL status for individual glibc tests. It's based on Tomas Dohnalek's patch <https://sourceware.org/ml/libc-alpha/2012-10/msg00278.html>, but is deliberately more minimal: it doesn't try to cover any tests outside of $(tests) / $(xtests) (that's for a later patch), nor does it put the result together in an overall summary file (again, a later patch): it just generates the .test-result files. Thus, this patch keeps the overall logic for when a testsuite run finishes completely unchanged: a test failing will terminate the run. I think we *should* move to a more conventional approach where plain "make check" does not terminate for an individual test failure, unless e.g. you say "make stop-on-test-failure=y check", but that sort of policy change is best done as a separate patch once the infrastructure is in place to generate summary files for completed test runs (which will entirely consist of PASS and XFAIL lines if the testsuite run reaches the point of generating them, until such a policy change is made). Tested x86_64. 2014-02-14 Tomas Dohnalek <tdohnale@redhat.com> Joseph Myers <joseph@codesourcery.com> * Makeconfig (test-name): New variable. (evaluate-test): Likewise. * Makerules (do-test-clean): Remove .test-result files. (common-mostlyclean): Likewise. * Rules ($(objpfx)%.out): Use $(evaluate-test) in both rules. * scripts/evaluate-test.sh: New file.
2014-02-15 01:04:57 +00:00
if [ $rc -eq 0 ]; then
result="PASS"
else
result="FAIL"
fi
if $xfail; then
result="X$result"
rc=0
fi
Generate .test-result files for ordinary tests. This patch, an updated version of <https://sourceware.org/ml/libc-alpha/2014-01/msg00193.html>, starts the process of generating explicit PASS or FAIL status for individual glibc tests. It's based on Tomas Dohnalek's patch <https://sourceware.org/ml/libc-alpha/2012-10/msg00278.html>, but is deliberately more minimal: it doesn't try to cover any tests outside of $(tests) / $(xtests) (that's for a later patch), nor does it put the result together in an overall summary file (again, a later patch): it just generates the .test-result files. Thus, this patch keeps the overall logic for when a testsuite run finishes completely unchanged: a test failing will terminate the run. I think we *should* move to a more conventional approach where plain "make check" does not terminate for an individual test failure, unless e.g. you say "make stop-on-test-failure=y check", but that sort of policy change is best done as a separate patch once the infrastructure is in place to generate summary files for completed test runs (which will entirely consist of PASS and XFAIL lines if the testsuite run reaches the point of generating them, until such a policy change is made). Tested x86_64. 2014-02-14 Tomas Dohnalek <tdohnale@redhat.com> Joseph Myers <joseph@codesourcery.com> * Makeconfig (test-name): New variable. (evaluate-test): Likewise. * Makerules (do-test-clean): Remove .test-result files. (common-mostlyclean): Likewise. * Rules ($(objpfx)%.out): Use $(evaluate-test) in both rules. * scripts/evaluate-test.sh: New file.
2014-02-15 01:04:57 +00:00
echo "$result: $test_name"
echo "original exit status $orig_rc"
Do not terminate default test runs on test failure. This patch is an updated version of <https://sourceware.org/ml/libc-alpha/2014-01/msg00198.html> and <https://sourceware.org/ml/libc-alpha/2014-03/msg00180.html>. Normal practice for software testsuites is that rather than terminating immediately when a test fails, they continue running and report at the end on how many tests passed or failed. The principle behind the glibc testsuite stopping on failure was probably that the expected state is no failures and so any failure indicates a problem such as miscompilation. In practice, while this is fairly close to true for native testing on x86_64 and x86 (kernel bugs and race conditions can still cause intermittent failures), it's less likely to be the case on other platforms, and so people testing glibc run the testsuite with "make -k" and then examine the logs to determine whether the failures are what they expect to fail on that platform, possibly with some automation for the comparison. This patch switches the glibc testsuite to the normal convention of not stopping on failure - unless you use stop-on-test-failure=y, in which case it behaves essentially as it did before (and does not generate overall test summaries on failure). Instead, the summary tests.sum may contain tests that FAILed. At the end of the test run, any FAIL or ERROR lines from tests.sum are printed, and then it exits with error status if there were any such lines. In addition, build failures will also cause the test run to stop - this has the justification that those *do* indicate serious problems that should be promptly fixed and aren't generally hard to fix (but apart from that, avoiding the build stopping on those failures seems harder). Note that unlike the previous patches in this series, this *does* require people with automation around testing glibc to change their processes - either to start using tests.sum / xtests.sum to track failures and compare them with expectations (with or without also using "make -k" and examining "make" logs to identify build failures), or else to use stop-on-test-failure=y and ignore the new tests.sum / xtests.sum mechanism. (If all you check is the exit status from "make check", no changes are needed unless you want to avoid test runs continuing after the first failure.) Tested x86_64. * scripts/evaluate-test.sh: Handle fourth argument to determine whether test run should stop on failure. * Makeconfig (stop-on-test-failure): New variable. (evaluate-test): Pass fourth argument to evaluate-test.sh based on $(stop-on-test-failure). * Makefile (tests): Give a summary of results from testing and exit with failure status if they include an ERROR or FAIL. (xtests): Likewise. * manual/install.texi (Configuring and compiling): Mention stop-on-test-failure=y. * INSTALL: Regenerated.
2014-03-14 21:02:40 +00:00
if $stop_on_failure; then
exit $rc
else
exit 0
fi