The feature was removed from the bots a while ago. It was
superseeded by the flaky-test detection which reruns tests.
Remaining tests still marked as flaky most certainly pass
since a while.
Referencing all the bugs whose expectations lines get
removed by this.
BUG=v8:3838,v8:3525,v8:3125
LOG=n
Review URL: https://codereview.chromium.org/1802983002
Cr-Commit-Position: refs/heads/master@{#34804}
test262 "negative" test expectations list which exception is thrown. The ES2017
draft specification is very specific about which exception class is thrown
from which path, and V8 works hard to be correct with respect to that spec.
Previously, the test262 test runner would accept any nonzero status code,
such as from a crash, or a FAIL printed out, for a negative test. This
patch makes negative tests check for the right answer using a quick-and-dirty
parsing of the exception printing from d8 to find the exception class.
It invokes d8 in a way to get a status code of 0 from thrown exceptions
so that 'negative' tests aren't actually implemented by negating the output.
Amazingly, this didn't catch any test262 failures, but I verified the extra
checking interactively by changing a negative test to expect a different type
and saw it fail.
BUG=v8:4803
R=machenbach
LOG=Y
Review URL: https://codereview.chromium.org/1766503002
Cr-Commit-Position: refs/heads/master@{#34763}
Changes include:
- better test coverage for builds with snapshot
- write snapshot blobs to buffer instead of test serialization files
- renamed tests
R=machenbach@chromium.org
Review URL: https://codereview.chromium.org/1777213002
Cr-Commit-Position: refs/heads/master@{#34657}
This adds sanitizer-coverage compilation, test-runner
features and post processing.
Sanitizer coverage is expected to be used together with
asan.
During test runner execution, the produced sancov files
are disambiguated and match the pattern:
<executable name>.test.<test id>.sancov.
Two additional scripts are added for merging raw sancov
files and for generating json data containing all
instrumented lines + all covered lines from merged sancov
files. Both scripts use multiprocessing for speed.
The json data will later be uploaded to google storage
for further use, e.g. to show coverage data in rietveld.
Sancov documentation:
http://clang.llvm.org/docs/SanitizerCoverage.html
BUG=chromium:568949
LOG=n
NOTRY=true
TEST=python -m unittest sancov_formatter_test
TEST=python -m unittest sancov_merger_test
Review URL: https://codereview.chromium.org/1737263003
Cr-Commit-Position: refs/heads/master@{#34578}
This returns a proper test result (failure) when files are
missing for setting up the command.
Review URL: https://codereview.chromium.org/1696713002
Cr-Commit-Position: refs/heads/master@{#33946}
The current support for try-catch in the interpreter can handle most of
the cases appearing in our test suite. Also the flag in question did not
detect try-finally constructs. This removes the flag and instead extends
the test expectations.
R=rmcilroy@chromium.org
BUG=v8:4674
LOG=n
Review URL: https://codereview.chromium.org/1631593003
Cr-Commit-Position: refs/heads/master@{#33494}
This removes the above flag definition. The flag is no longer needed as
the default implementation is more than capable of faking presence of
handling of try-catch and try-finally constructs by now.
R=rmcilroy@chromium.org
BUG=v8:4674
LOG=n
Review URL: https://codereview.chromium.org/1603063003
Cr-Commit-Position: refs/heads/master@{#33402}
Now that we support eval in Ignition, remove the fallback for eval checks
and make the flag only fallback on catch blocks.
BUG=v8:4280,v8:4676
LOG=N
Review URL: https://codereview.chromium.org/1595223004
Cr-Commit-Position: refs/heads/master@{#33384}
Adds --turbo to the set of flags run with ignition (making the
configuration pure Ignition+TurboFan except for fallbacks to
full-codegen for catch / eval). Also changes the default
--ignition-filter to allow everything, rather than omit everything.
BUG=v8:4280
LOG=N
Review URL: https://codereview.chromium.org/1537433003
Cr-Commit-Position: refs/heads/master@{#33090}
This will allow moving the test outcome check beyond the
multi-process boundary in a follow up. It'll allow wrapping
more complex test jobs like predicable mode on the multi-
process side, which will make the code easier to maintain.
BUG=
Review URL: https://codereview.chromium.org/1469833002
Cr-Commit-Position: refs/heads/master@{#32373}
This will make sure that message tests cover both the parser and preparser
paths, just as we do for parsing-related cctests.
BUG=v8:4372
LOG=n
Review URL: https://codereview.chromium.org/1469383004
Cr-Commit-Position: refs/heads/master@{#32307}
This loads all test suites and status files to catch subtle
syntax errors. It also checks basic status file integrity
and common mistakes.
NOTRY=true
Review URL: https://codereview.chromium.org/1475663002
Cr-Commit-Position: refs/heads/master@{#32271}
The feature sometimes prevents subsequent swarming bots to
delete the work directory on windows.
The data file is not kept on swarming bots anyways, therefore
this switches off the feature completely.
BUG=chromium:535160
LOG=n
Review URL: https://codereview.chromium.org/1468933002
Cr-Commit-Position: refs/heads/master@{#32168}
Adds a blacklist of tests which are currently unsupported or broken in Ignition to
the mjsunit and test262 test status.
Also removes --ignition-script-filter flag, and adds a
--ignition_fallback_on_eval_and_catch flag which fallsback to fullcodegen for
functions which call eval or contain a catch block.
BUG=v8:4280
LOG=N
Review URL: https://codereview.chromium.org/1420963009
Cr-Commit-Position: refs/heads/master@{#31864}
The flake detection is done on the infra-side according to
the contents of the json test results. We don't want the
runner to fail after flakes.
This was controlled on the infra side by accepting any exit
codes so far. After the swarming switch, this is more
difficult, because the runner is wrapped by the swarming
collect script. There, failing exit codes can mean many
things, including network failures. Therefore, we now
force exit code 0 with test failures if those failures
are reported in the formal test results json.
The infrastructure will take care of reporting the flakes
and failures accordingly.
BUG=chromium:535160
LOG=n
Review URL: https://codereview.chromium.org/1416373005
Cr-Commit-Position: refs/heads/master@{#31740}
Adds an ignition variant to the test runner and adds support to test262 for
filtering such that only test scripts (not the test harness) get run by
the interpreter.
BUG=v8:4280
LOG=N
Review URL: https://codereview.chromium.org/1379093002
Cr-Commit-Position: refs/heads/master@{#31123}
Swarming won't support an incremental data file. By just
sorting the lowest hanging fruits to the beginning we
already get a big bang for the buck (>80% of the improvement
we get otherwise).
This will require semi-regular manual updates of the
slowest tests.
BUG=chromium:535160
LOG=n
Review URL: https://codereview.chromium.org/1385443002
Cr-Commit-Position: refs/heads/master@{#31061}
[run-tests.py]
- adding more detailed information about the flags
- show more detailed error message on failing Popen commands
BUG=
Review URL: https://codereview.chromium.org/1369343002
Cr-Commit-Position: refs/heads/master@{#30997}
Now run-tests.py understands "suite/foo/bar" with forward slashes for
command-line test selection on all test suites on all platforms.
Previously, file-based suites like mjsunit also accepted "mjsunit/foo\bar";
that behavior is sacrificed here in favor of unification. For the cctest
suite, OTOH, it wasn't possible on Windows to select specific tests at all.
Original review: https://codereview.chromium.org/1348653003/
This reverts commit 5f44a91059.
NOTRY=true
Review URL: https://codereview.chromium.org/1356613002
Cr-Commit-Position: refs/heads/master@{#30798}
Reason for revert:
mozilla tests are failing on Windows
Original issue's description:
> [test] Fix cctest path separators on Windows
>
> Now run-tests.py understands "suite/foo/bar" with forward slashes for
> command-line test selection on all test suites on all platforms.
>
> Previously, file-based suites like mjsunit also accepted "mjsunit/foo\bar";
> that behavior is sacrificed here in favor of unification. For the cctest
> suite, OTOH, it wasn't possible on Windows to select specific tests at all.
>
> Committed: https://crrev.com/b36cfdb39ae648b49a1396c4f669df9b1f57996c
> Cr-Commit-Position: refs/heads/master@{#30794}
TBR=machenbach@google.com,machenbach@chromium.org
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
Review URL: https://codereview.chromium.org/1349163002
Cr-Commit-Position: refs/heads/master@{#30795}
Now run-tests.py understands "suite/foo/bar" with forward slashes for
command-line test selection on all test suites on all platforms.
Previously, file-based suites like mjsunit also accepted "mjsunit/foo\bar";
that behavior is sacrificed here in favor of unification. For the cctest
suite, OTOH, it wasn't possible on Windows to select specific tests at all.
Review URL: https://codereview.chromium.org/1348653003
Cr-Commit-Position: refs/heads/master@{#30794}
This information can be used on the buildbot side to only
rebuild the failing target for bisection.
BUG=chromium:511215
LOG=n
NOTRY=true
Review URL: https://codereview.chromium.org/1313353006
Cr-Commit-Position: refs/heads/master@{#30617}
Use the same path separator for test cases on the cmd-line
on all platforms as in testcase.GetLabel, which is used to
report failures.
BUG=chromium:511215
LOG=n
NOTRY=true
Review URL: https://codereview.chromium.org/1281453003
Cr-Commit-Position: refs/heads/master@{#30045}
This allows variants to be named on test failures (follow
up) and then to be used in the test runner for a repro.
This also speeds up variant iteration for test262 and fixes
a bug with variants for benchmarks.
BUG=chromium:511215
NOTREECHECKS=true
LOG=n
Review URL: https://codereview.chromium.org/1245623005
Cr-Commit-Position: refs/heads/master@{#29899}
There are many test names in the v8 code base that prefix
others, which makes it hard to only run those tests.
BUG=chromium:511215
LOG=n
NOTRY=true
Review URL: https://codereview.chromium.org/1251363002
Cr-Commit-Position: refs/heads/master@{#29809}
Reason for revert:
All build problems had other reasons.
Original issue's description:
> Revert of [test] Refactoring - Let runner handle test IDs. (patchset #1 id:1 of https://codereview.chromium.org/1168303007/)
>
> Reason for revert:
> [Sheriff] Revert until the tree is in a better state.
>
> Original issue's description:
> > [test] Refactoring - Let runner handle test IDs.
> >
> > This prepares for properly rerunning tests. Currently when
> > tests are rerun, the same test object is reused. This
> > will be changed in a follow up.
> >
> > Committed: https://crrev.com/f41a81b8a513fc360c500c066b74f223bc9c0223
> > Cr-Commit-Position: refs/heads/master@{#28864}
>
> TBR=jkummerow@chromium.org,tandrii@chromium.org
> NOPRESUBMIT=true
> NOTREECHECKS=true
> NOTRY=true
>
> Committed: https://crrev.com/f83444a5f24381ec9ccc28b1e18e1e9370415bb2
> Cr-Commit-Position: refs/heads/master@{#28891}
TBR=jkummerow@chromium.org,tandrii@chromium.org
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
Review URL: https://codereview.chromium.org/1179483002
Cr-Commit-Position: refs/heads/master@{#28903}
Reason for revert:
All build problems had other causes.
Original issue's description:
> Revert of [test] Refactoring - Use subject/observer pattern for progress indicators. (patchset #3 id:40001 of https://codereview.chromium.org/1171943002/)
>
> Reason for revert:
> might break stuff
>
> Original issue's description:
> > [test] Refactoring - Use subject/observer pattern for progress indicators.
> >
> > This should prevent bugs caused by missing super calls in
> > overridden methods. The assumption is that methods of
> > different indicators are independent.
> >
> > Committed: https://crrev.com/fbe973ff1722a6158a5b2babce9c1a32d26a1d3b
> > Cr-Commit-Position: refs/heads/master@{#28866}
>
> TBR=jkummerow@chromium.org,tandrii@chromium.org
> NOPRESUBMIT=true
> NOTREECHECKS=true
> NOTRY=true
>
> Committed: https://crrev.com/2a3962d9d2a7415378811f4d56522531332d5a3e
> Cr-Commit-Position: refs/heads/master@{#28869}
TBR=jkummerow@chromium.org,tandrii@chromium.org
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
Review URL: https://codereview.chromium.org/1176863002
Cr-Commit-Position: refs/heads/master@{#28901}
Reason for revert:
[Sheriff] Revert until the tree is in a better state.
Original issue's description:
> [test] Refactoring - Let runner handle test IDs.
>
> This prepares for properly rerunning tests. Currently when
> tests are rerun, the same test object is reused. This
> will be changed in a follow up.
>
> Committed: https://crrev.com/f41a81b8a513fc360c500c066b74f223bc9c0223
> Cr-Commit-Position: refs/heads/master@{#28864}
TBR=jkummerow@chromium.org,tandrii@chromium.org
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
Review URL: https://codereview.chromium.org/1167343003
Cr-Commit-Position: refs/heads/master@{#28891}
Reason for revert:
[Sheriff] Still lots of breakages on some builders. E.g. http://build.chromium.org/p/client.v8/builders/V8%20Linux%20-%20nosnap/builds/3313
Original issue's description:
> Update Test262 to 5/30
>
> This updates the test runner to run all tests in both sloppy
> and strict mode.
>
> It also marks the test in the status file as failing in
> sloppy mode even if all it does is marking the test as
> [PASS, FAIL]. A future CL will have to validate that the
> tests pass and fail in the correct mode.
>
> Committed: https://crrev.com/d869f4a4801d4ef6868c266c07f9e29d2e29cba5
> Cr-Commit-Position: refs/heads/master@{#28879}
TBR=rossberg@chromium.org,arv@chromium.org
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
Review URL: https://codereview.chromium.org/1173933002
Cr-Commit-Position: refs/heads/master@{#28890}
This updates the test runner to run all tests in both sloppy
and strict mode.
It also marks the test in the status file as failing in
sloppy mode even if all it does is marking the test as
[PASS, FAIL]. A future CL will have to validate that the
tests pass and fail in the correct mode.
Review URL: https://codereview.chromium.org/1136553008
Cr-Commit-Position: refs/heads/master@{#28879}
Reason for revert:
might break stuff
Original issue's description:
> [test] Refactoring - Use subject/observer pattern for progress indicators.
>
> This should prevent bugs caused by missing super calls in
> overridden methods. The assumption is that methods of
> different indicators are independent.
>
> Committed: https://crrev.com/fbe973ff1722a6158a5b2babce9c1a32d26a1d3b
> Cr-Commit-Position: refs/heads/master@{#28866}
TBR=jkummerow@chromium.org,tandrii@chromium.org
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
Review URL: https://codereview.chromium.org/1163373005
Cr-Commit-Position: refs/heads/master@{#28869}
This should prevent bugs caused by missing super calls in
overridden methods. The assumption is that methods of
different indicators are independent.
Review URL: https://codereview.chromium.org/1171943002
Cr-Commit-Position: refs/heads/master@{#28866}
This prepares for properly rerunning tests. Currently when
tests are rerun, the same test object is reused. This
will be changed in a follow up.
Review URL: https://codereview.chromium.org/1168303007
Cr-Commit-Position: refs/heads/master@{#28864}
Without this change, wildcards always overwrite the outcomes
of more specific rules. Now we always merge.
Review URL: https://codereview.chromium.org/1153073002
Cr-Commit-Position: refs/heads/master@{#28629}
This changes the unittests suite to be run in only the default variant,
because unit tests are (and should be) written to be independent from
global flags.
R=machenbach@chromium.org
Review URL: https://codereview.chromium.org/1067863007
Cr-Commit-Position: refs/heads/master@{#28054}
This flag is intended as a staging flag for TurboFan. It serves as a
single flag that always enables a most recent configuration of TurboFan
for test suites and benchmarks, without needing to update test drivers.
R=titzer@chromium.org,machenbach@chromium.org
Review URL: https://codereview.chromium.org/1094573002
Cr-Commit-Position: refs/heads/master@{#27896}
Second try. Disabled the tests that were failing due to
https://github.com/tc39/test262/issues/215
This updates test262 to revision d24fd10 (2015/03/11).
The files moved around in the test repo and a lot of new tests are
failing.
BUG=None
LOG=N
R=adamk, rossberg
Review URL: https://codereview.chromium.org/1040093003
Cr-Commit-Position: refs/heads/master@{#27543}
Reason for revert:
Bot failed to include verifyNotEnumerable function for some screwed up reason.
Original issue's description:
> [es6] Update test262 tests
>
> This updates test262 to revision d24fd10 (2015/03/11).
>
> The files moved around in the test repo and a lot of new tests are
> failing.
>
> BUG=None
> LOG=N
> R=rossberg
>
> Committed: https://crrev.com/4f2fb3835feff3663146f12be42b01a226d0065e
> Cr-Commit-Position: refs/heads/master@{#27522}
TBR=rossberg@chromium.org
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
BUG=None
Review URL: https://codereview.chromium.org/1047713002
Cr-Commit-Position: refs/heads/master@{#27525}
This updates test262 to revision d24fd10 (2015/03/11).
The files moved around in the test repo and a lot of new tests are
failing.
BUG=None
LOG=N
R=rossberg
Review URL: https://codereview.chromium.org/1025043002
Cr-Commit-Position: refs/heads/master@{#27522}
Contribution of PowerPC port (continuation of 422063005 and 817143002). This patch covers
the key changes needed to the common files needed to support AIX. Subsequent
patches will cover:
- changes to update the ppc directories so they are current with the changes
in the rest of the project.
- remaining AIX changes not resolved by 4.8 compiler
- individual optimizations for PPC
This is based off of the GitHub repository
https://github.com/andrewlow/v8ppcR=danno@chromium.org, svenpanne@chromium.org
BUG=
Review URL: https://codereview.chromium.org/866843003
Cr-Commit-Position: refs/heads/master@{#26343}
There are separated JS and C stacks on simulators so for some stack
extensive tests (like mozilla/js1_5/extensions/regress-355497) might
cause a C stack overflow and that overflow is not caught by V8. It is
not an issue on real HW. Increasing the C stack also solves the problem
but we have already FLAG_sim_stack_size flag to control the JS stack
size.
This patch makes it possible to add flags to tests conditionally in
.status files.
TEST=mozilla/js1_5/extensions/regress-355497
BUG=v8:3152
LOG=N
Review URL: https://codereview.chromium.org/735723006
Cr-Commit-Position: refs/heads/master@{#25434}
