The feature was removed from the bots a while ago. It was
superseeded by the flaky-test detection which reruns tests.
Remaining tests still marked as flaky most certainly pass
since a while.
Referencing all the bugs whose expectations lines get
removed by this.
BUG=v8:3838,v8:3525,v8:3125
LOG=n
Review URL: https://codereview.chromium.org/1802983002
Cr-Commit-Position: refs/heads/master@{#34804}
This also runs the message test suite against Ignition. By now most of
the source positions (and exception messages) are accurate, the failing
ones have been blacklisted.
R=machenbach@chromium.org
Review URL: https://codereview.chromium.org/1783773003
Cr-Commit-Position: refs/heads/master@{#34708}
Changes include:
- better test coverage for builds with snapshot
- write snapshot blobs to buffer instead of test serialization files
- renamed tests
R=machenbach@chromium.org
Review URL: https://codereview.chromium.org/1777213002
Cr-Commit-Position: refs/heads/master@{#34657}
With this, the test runner automatically merges sancov
files after testing. There's no need to do this by some
external infrastructure.
In a future CL, we could even merge during testing to lift
harddisk pressure.
BUG=chromium:568949
LOG=n
NOTRY=true
Review URL: https://codereview.chromium.org/1776123002
Cr-Commit-Position: refs/heads/master@{#34606}
This adds sanitizer-coverage compilation, test-runner
features and post processing.
Sanitizer coverage is expected to be used together with
asan.
During test runner execution, the produced sancov files
are disambiguated and match the pattern:
<executable name>.test.<test id>.sancov.
Two additional scripts are added for merging raw sancov
files and for generating json data containing all
instrumented lines + all covered lines from merged sancov
files. Both scripts use multiprocessing for speed.
The json data will later be uploaded to google storage
for further use, e.g. to show coverage data in rietveld.
Sancov documentation:
http://clang.llvm.org/docs/SanitizerCoverage.html
BUG=chromium:568949
LOG=n
NOTRY=true
TEST=python -m unittest sancov_formatter_test
TEST=python -m unittest sancov_merger_test
Review URL: https://codereview.chromium.org/1737263003
Cr-Commit-Position: refs/heads/master@{#34578}
Add control for the ignition test suites. This will be
followed by a change on the infra side that targets the
ignition set instead of the hardcoded mjsunit.
After that, cctest can be enabled in a separate CL. This CL
already includes the logic for running cctest for ignition
on swarming.
BUG=v8:4280
LOG=N
Review URL: https://codereview.chromium.org/1559943002
Cr-Commit-Position: refs/heads/master@{#33089}
This will allow moving the test outcome check beyond the
multi-process boundary in a follow up. It'll allow wrapping
more complex test jobs like predicable mode on the multi-
process side, which will make the code easier to maintain.
BUG=
Review URL: https://codereview.chromium.org/1469833002
Cr-Commit-Position: refs/heads/master@{#32373}
The feature sometimes prevents subsequent swarming bots to
delete the work directory on windows.
The data file is not kept on swarming bots anyways, therefore
this switches off the feature completely.
BUG=chromium:535160
LOG=n
Review URL: https://codereview.chromium.org/1468933002
Cr-Commit-Position: refs/heads/master@{#32168}
Adds a blacklist of tests which are currently unsupported or broken in Ignition to
the mjsunit and test262 test status.
Also removes --ignition-script-filter flag, and adds a
--ignition_fallback_on_eval_and_catch flag which fallsback to fullcodegen for
functions which call eval or contain a catch block.
BUG=v8:4280
LOG=N
Review URL: https://codereview.chromium.org/1420963009
Cr-Commit-Position: refs/heads/master@{#31864}
The flake detection is done on the infra-side according to
the contents of the json test results. We don't want the
runner to fail after flakes.
This was controlled on the infra side by accepting any exit
codes so far. After the swarming switch, this is more
difficult, because the runner is wrapped by the swarming
collect script. There, failing exit codes can mean many
things, including network failures. Therefore, we now
force exit code 0 with test failures if those failures
are reported in the formal test results json.
The infrastructure will take care of reporting the flakes
and failures accordingly.
BUG=chromium:535160
LOG=n
Review URL: https://codereview.chromium.org/1416373005
Cr-Commit-Position: refs/heads/master@{#31740}
This is currently hardcoded in the infra-side test runner
setup, but will be removed there, so that swarming triggers
don't need a custom environment.
BUG=chromium:535160
LOG=n
Review URL: https://codereview.chromium.org/1411733008
Cr-Commit-Position: refs/heads/master@{#31682}
The test suite is ran in 60% of the bots anyway and the
step is very short. For swarming, it's better to run this
together in one step as each step triggers a different bot.
BUG=chromium:535160
LOG=n
NOTRY=true
Review URL: https://codereview.chromium.org/1413023002
Cr-Commit-Position: refs/heads/master@{#31360}
This adds the unittests to the "default" test set. Now that
the "default" and the DEFAULT_TESTS (i.e. runner with
no arguments) are the same, removed DEFAULT_TESTS and use
TEST_MAP["default"] instead.
On the bots, where unittests and default were run in
separation before, the explicit unittests step should now
be skipped.
This is necessary for swarming, as the unittests step is
too small to justify its own swarming job.
BUG=chromium:535160
LOG=n
Review URL: https://codereview.chromium.org/1374733006
Cr-Commit-Position: refs/heads/master@{#31122}
[run-tests.py]
- adding more detailed information about the flags
- show more detailed error message on failing Popen commands
BUG=
Review URL: https://codereview.chromium.org/1369343002
Cr-Commit-Position: refs/heads/master@{#30997}
The previous approach to this (in 1b1de2d22) did not work well with
bots that intentionally run no tests. But I think this warning message
is still useful for developers.
Review URL: https://codereview.chromium.org/1281313004
Cr-Commit-Position: refs/heads/master@{#30116}
Reason for revert:
Some bots that don't run tests (by design) don't return the test results json anymore which makes the infrastructure side fail now.
Original issue's description:
> Make run-tests.py warn when it's not testing anything
>
> I've often been confused by an "all passed" run of the test script
> when it turned out that either all the tests I cared about were skipped
> or, more likely, I mistyped the name(s) of the tests I wanted to run.
> This patch aims to fix that (and gives a useful diagnostic for the
> "all matched tests were skipped" case).
>
> R=machenbach@chromium.org
> NOTRY=true
>
> Committed: https://crrev.com/1b1de2d221a0ed23e529e91bf9fa3dc287acb2ca
> Cr-Commit-Position: refs/heads/master@{#30095}
TBR=jkummerow@chromium.org,adamk@chromium.org
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
Review URL: https://codereview.chromium.org/1283593005
Cr-Commit-Position: refs/heads/master@{#30097}
I've often been confused by an "all passed" run of the test script
when it turned out that either all the tests I cared about were skipped
or, more likely, I mistyped the name(s) of the tests I wanted to run.
This patch aims to fix that (and gives a useful diagnostic for the
"all matched tests were skipped" case).
R=machenbach@chromium.org
NOTRY=true
Review URL: https://codereview.chromium.org/1283513003
Cr-Commit-Position: refs/heads/master@{#30095}
Use the same path separator for test cases on the cmd-line
on all platforms as in testcase.GetLabel, which is used to
report failures.
BUG=chromium:511215
LOG=n
NOTRY=true
Review URL: https://codereview.chromium.org/1281453003
Cr-Commit-Position: refs/heads/master@{#30045}
This allows variants to be named on test failures (follow
up) and then to be used in the test runner for a repro.
This also speeds up variant iteration for test262 and fixes
a bug with variants for benchmarks.
BUG=chromium:511215
NOTREECHECKS=true
LOG=n
Review URL: https://codereview.chromium.org/1245623005
Cr-Commit-Position: refs/heads/master@{#29899}
Reason for revert:
All build problems had other reasons.
Original issue's description:
> Revert of [test] Refactoring - Let runner handle test IDs. (patchset #1 id:1 of https://codereview.chromium.org/1168303007/)
>
> Reason for revert:
> [Sheriff] Revert until the tree is in a better state.
>
> Original issue's description:
> > [test] Refactoring - Let runner handle test IDs.
> >
> > This prepares for properly rerunning tests. Currently when
> > tests are rerun, the same test object is reused. This
> > will be changed in a follow up.
> >
> > Committed: https://crrev.com/f41a81b8a513fc360c500c066b74f223bc9c0223
> > Cr-Commit-Position: refs/heads/master@{#28864}
>
> TBR=jkummerow@chromium.org,tandrii@chromium.org
> NOPRESUBMIT=true
> NOTREECHECKS=true
> NOTRY=true
>
> Committed: https://crrev.com/f83444a5f24381ec9ccc28b1e18e1e9370415bb2
> Cr-Commit-Position: refs/heads/master@{#28891}
TBR=jkummerow@chromium.org,tandrii@chromium.org
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
Review URL: https://codereview.chromium.org/1179483002
Cr-Commit-Position: refs/heads/master@{#28903}
Reason for revert:
All build problems had other causes.
Original issue's description:
> Revert of [test] Refactoring - Use subject/observer pattern for progress indicators. (patchset #3 id:40001 of https://codereview.chromium.org/1171943002/)
>
> Reason for revert:
> might break stuff
>
> Original issue's description:
> > [test] Refactoring - Use subject/observer pattern for progress indicators.
> >
> > This should prevent bugs caused by missing super calls in
> > overridden methods. The assumption is that methods of
> > different indicators are independent.
> >
> > Committed: https://crrev.com/fbe973ff1722a6158a5b2babce9c1a32d26a1d3b
> > Cr-Commit-Position: refs/heads/master@{#28866}
>
> TBR=jkummerow@chromium.org,tandrii@chromium.org
> NOPRESUBMIT=true
> NOTREECHECKS=true
> NOTRY=true
>
> Committed: https://crrev.com/2a3962d9d2a7415378811f4d56522531332d5a3e
> Cr-Commit-Position: refs/heads/master@{#28869}
TBR=jkummerow@chromium.org,tandrii@chromium.org
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
Review URL: https://codereview.chromium.org/1176863002
Cr-Commit-Position: refs/heads/master@{#28901}
Reason for revert:
[Sheriff] Revert until the tree is in a better state.
Original issue's description:
> [test] Refactoring - Let runner handle test IDs.
>
> This prepares for properly rerunning tests. Currently when
> tests are rerun, the same test object is reused. This
> will be changed in a follow up.
>
> Committed: https://crrev.com/f41a81b8a513fc360c500c066b74f223bc9c0223
> Cr-Commit-Position: refs/heads/master@{#28864}
TBR=jkummerow@chromium.org,tandrii@chromium.org
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
Review URL: https://codereview.chromium.org/1167343003
Cr-Commit-Position: refs/heads/master@{#28891}
Reason for revert:
might break stuff
Original issue's description:
> [test] Refactoring - Use subject/observer pattern for progress indicators.
>
> This should prevent bugs caused by missing super calls in
> overridden methods. The assumption is that methods of
> different indicators are independent.
>
> Committed: https://crrev.com/fbe973ff1722a6158a5b2babce9c1a32d26a1d3b
> Cr-Commit-Position: refs/heads/master@{#28866}
TBR=jkummerow@chromium.org,tandrii@chromium.org
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
Review URL: https://codereview.chromium.org/1163373005
Cr-Commit-Position: refs/heads/master@{#28869}
This should prevent bugs caused by missing super calls in
overridden methods. The assumption is that methods of
different indicators are independent.
Review URL: https://codereview.chromium.org/1171943002
Cr-Commit-Position: refs/heads/master@{#28866}
This prepares for properly rerunning tests. Currently when
tests are rerun, the same test object is reused. This
will be changed in a follow up.
Review URL: https://codereview.chromium.org/1168303007
Cr-Commit-Position: refs/heads/master@{#28864}
The simd.js benchmarks reside in the same repository
that holds the compliance tests.
Adding an option to run-test.py to download test data
and exit, so that a recipe to run the simd.js
benchmarks can call run-test.py with this option
to fetch the test data.
BUG=https://code.google.com/p/v8/issues/detail?id=4124
LOG=N
TEST=manual verification it works.
NOTRY=true
R=machenbach@chromium.org
Review URL: https://codereview.chromium.org/1140353002
Cr-Commit-Position: refs/heads/master@{#28446}
