The flake detection is done on the infra-side according to
the contents of the json test results. We don't want the
runner to fail after flakes.
This was controlled on the infra side by accepting any exit
codes so far. After the swarming switch, this is more
difficult, because the runner is wrapped by the swarming
collect script. There, failing exit codes can mean many
things, including network failures. Therefore, we now
force exit code 0 with test failures if those failures
are reported in the formal test results json.
The infrastructure will take care of reporting the flakes
and failures accordingly.
BUG=chromium:535160
LOG=n
Review URL: https://codereview.chromium.org/1416373005
Cr-Commit-Position: refs/heads/master@{#31740}
Adds an ignition variant to the test runner and adds support to test262 for
filtering such that only test scripts (not the test harness) get run by
the interpreter.
BUG=v8:4280
LOG=N
Review URL: https://codereview.chromium.org/1379093002
Cr-Commit-Position: refs/heads/master@{#31123}
Reason for revert:
Prime suspect in breakage of V8 Linux -- no snap
Original issue's description:
> [swarming] Isolate v8 testing.
>
> Add gyp support and isolates for default test suites.
> Add two default isolates, one (default) for using the
> test suite collection we call "default" on the bots. One
> (developer_default) for also supporting the way developers
> call the driver (i.e. without argument, which includes
> the unittests).
>
> BUG=chromium:535160
> LOG=n
>
> Committed: https://crrev.com/9bd83f58f29ab0c7c5b71b00bcb1df3a9e641f05
> Cr-Commit-Position: refs/heads/master@{#31081}
TBR=tandrii@chromium.org,jochen@chromium.org,maruel@chromium.org,machenbach@chromium.org
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
BUG=chromium:535160
Review URL: https://codereview.chromium.org/1370993008
Cr-Commit-Position: refs/heads/master@{#31084}
Add gyp support and isolates for default test suites.
Add two default isolates, one (default) for using the
test suite collection we call "default" on the bots. One
(developer_default) for also supporting the way developers
call the driver (i.e. without argument, which includes
the unittests).
BUG=chromium:535160
LOG=n
Review URL: https://codereview.chromium.org/1380593002
Cr-Commit-Position: refs/heads/master@{#31081}
Swarming won't support an incremental data file. By just
sorting the lowest hanging fruits to the beginning we
already get a big bang for the buck (>80% of the improvement
we get otherwise).
This will require semi-regular manual updates of the
slowest tests.
BUG=chromium:535160
LOG=n
Review URL: https://codereview.chromium.org/1385443002
Cr-Commit-Position: refs/heads/master@{#31061}
[run-tests.py]
- adding more detailed information about the flags
- show more detailed error message on failing Popen commands
BUG=
Review URL: https://codereview.chromium.org/1369343002
Cr-Commit-Position: refs/heads/master@{#30997}
Now run-tests.py understands "suite/foo/bar" with forward slashes for
command-line test selection on all test suites on all platforms.
Previously, file-based suites like mjsunit also accepted "mjsunit/foo\bar";
that behavior is sacrificed here in favor of unification. For the cctest
suite, OTOH, it wasn't possible on Windows to select specific tests at all.
Original review: https://codereview.chromium.org/1348653003/
This reverts commit 5f44a91059.
NOTRY=true
Review URL: https://codereview.chromium.org/1356613002
Cr-Commit-Position: refs/heads/master@{#30798}
Reason for revert:
mozilla tests are failing on Windows
Original issue's description:
> [test] Fix cctest path separators on Windows
>
> Now run-tests.py understands "suite/foo/bar" with forward slashes for
> command-line test selection on all test suites on all platforms.
>
> Previously, file-based suites like mjsunit also accepted "mjsunit/foo\bar";
> that behavior is sacrificed here in favor of unification. For the cctest
> suite, OTOH, it wasn't possible on Windows to select specific tests at all.
>
> Committed: https://crrev.com/b36cfdb39ae648b49a1396c4f669df9b1f57996c
> Cr-Commit-Position: refs/heads/master@{#30794}
TBR=machenbach@google.com,machenbach@chromium.org
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
Review URL: https://codereview.chromium.org/1349163002
Cr-Commit-Position: refs/heads/master@{#30795}
Now run-tests.py understands "suite/foo/bar" with forward slashes for
command-line test selection on all test suites on all platforms.
Previously, file-based suites like mjsunit also accepted "mjsunit/foo\bar";
that behavior is sacrificed here in favor of unification. For the cctest
suite, OTOH, it wasn't possible on Windows to select specific tests at all.
Review URL: https://codereview.chromium.org/1348653003
Cr-Commit-Position: refs/heads/master@{#30794}
This information can be used on the buildbot side to only
rebuild the failing target for bisection.
BUG=chromium:511215
LOG=n
NOTRY=true
Review URL: https://codereview.chromium.org/1313353006
Cr-Commit-Position: refs/heads/master@{#30617}
Use the same path separator for test cases on the cmd-line
on all platforms as in testcase.GetLabel, which is used to
report failures.
BUG=chromium:511215
LOG=n
NOTRY=true
Review URL: https://codereview.chromium.org/1281453003
Cr-Commit-Position: refs/heads/master@{#30045}
This allows variants to be named on test failures (follow
up) and then to be used in the test runner for a repro.
This also speeds up variant iteration for test262 and fixes
a bug with variants for benchmarks.
BUG=chromium:511215
NOTREECHECKS=true
LOG=n
Review URL: https://codereview.chromium.org/1245623005
Cr-Commit-Position: refs/heads/master@{#29899}
There are many test names in the v8 code base that prefix
others, which makes it hard to only run those tests.
BUG=chromium:511215
LOG=n
NOTRY=true
Review URL: https://codereview.chromium.org/1251363002
Cr-Commit-Position: refs/heads/master@{#29809}
Reason for revert:
All build problems had other reasons.
Original issue's description:
> Revert of [test] Refactoring - Let runner handle test IDs. (patchset #1 id:1 of https://codereview.chromium.org/1168303007/)
>
> Reason for revert:
> [Sheriff] Revert until the tree is in a better state.
>
> Original issue's description:
> > [test] Refactoring - Let runner handle test IDs.
> >
> > This prepares for properly rerunning tests. Currently when
> > tests are rerun, the same test object is reused. This
> > will be changed in a follow up.
> >
> > Committed: https://crrev.com/f41a81b8a513fc360c500c066b74f223bc9c0223
> > Cr-Commit-Position: refs/heads/master@{#28864}
>
> TBR=jkummerow@chromium.org,tandrii@chromium.org
> NOPRESUBMIT=true
> NOTREECHECKS=true
> NOTRY=true
>
> Committed: https://crrev.com/f83444a5f24381ec9ccc28b1e18e1e9370415bb2
> Cr-Commit-Position: refs/heads/master@{#28891}
TBR=jkummerow@chromium.org,tandrii@chromium.org
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
Review URL: https://codereview.chromium.org/1179483002
Cr-Commit-Position: refs/heads/master@{#28903}
Reason for revert:
All build problems had other causes.
Original issue's description:
> Revert of [test] Refactoring - Use subject/observer pattern for progress indicators. (patchset #3 id:40001 of https://codereview.chromium.org/1171943002/)
>
> Reason for revert:
> might break stuff
>
> Original issue's description:
> > [test] Refactoring - Use subject/observer pattern for progress indicators.
> >
> > This should prevent bugs caused by missing super calls in
> > overridden methods. The assumption is that methods of
> > different indicators are independent.
> >
> > Committed: https://crrev.com/fbe973ff1722a6158a5b2babce9c1a32d26a1d3b
> > Cr-Commit-Position: refs/heads/master@{#28866}
>
> TBR=jkummerow@chromium.org,tandrii@chromium.org
> NOPRESUBMIT=true
> NOTREECHECKS=true
> NOTRY=true
>
> Committed: https://crrev.com/2a3962d9d2a7415378811f4d56522531332d5a3e
> Cr-Commit-Position: refs/heads/master@{#28869}
TBR=jkummerow@chromium.org,tandrii@chromium.org
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
Review URL: https://codereview.chromium.org/1176863002
Cr-Commit-Position: refs/heads/master@{#28901}
Reason for revert:
[Sheriff] Revert until the tree is in a better state.
Original issue's description:
> [test] Refactoring - Let runner handle test IDs.
>
> This prepares for properly rerunning tests. Currently when
> tests are rerun, the same test object is reused. This
> will be changed in a follow up.
>
> Committed: https://crrev.com/f41a81b8a513fc360c500c066b74f223bc9c0223
> Cr-Commit-Position: refs/heads/master@{#28864}
TBR=jkummerow@chromium.org,tandrii@chromium.org
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
Review URL: https://codereview.chromium.org/1167343003
Cr-Commit-Position: refs/heads/master@{#28891}
Reason for revert:
[Sheriff] Still lots of breakages on some builders. E.g. http://build.chromium.org/p/client.v8/builders/V8%20Linux%20-%20nosnap/builds/3313
Original issue's description:
> Update Test262 to 5/30
>
> This updates the test runner to run all tests in both sloppy
> and strict mode.
>
> It also marks the test in the status file as failing in
> sloppy mode even if all it does is marking the test as
> [PASS, FAIL]. A future CL will have to validate that the
> tests pass and fail in the correct mode.
>
> Committed: https://crrev.com/d869f4a4801d4ef6868c266c07f9e29d2e29cba5
> Cr-Commit-Position: refs/heads/master@{#28879}
TBR=rossberg@chromium.org,arv@chromium.org
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
Review URL: https://codereview.chromium.org/1173933002
Cr-Commit-Position: refs/heads/master@{#28890}
This updates the test runner to run all tests in both sloppy
and strict mode.
It also marks the test in the status file as failing in
sloppy mode even if all it does is marking the test as
[PASS, FAIL]. A future CL will have to validate that the
tests pass and fail in the correct mode.
Review URL: https://codereview.chromium.org/1136553008
Cr-Commit-Position: refs/heads/master@{#28879}
Reason for revert:
might break stuff
Original issue's description:
> [test] Refactoring - Use subject/observer pattern for progress indicators.
>
> This should prevent bugs caused by missing super calls in
> overridden methods. The assumption is that methods of
> different indicators are independent.
>
> Committed: https://crrev.com/fbe973ff1722a6158a5b2babce9c1a32d26a1d3b
> Cr-Commit-Position: refs/heads/master@{#28866}
TBR=jkummerow@chromium.org,tandrii@chromium.org
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
Review URL: https://codereview.chromium.org/1163373005
Cr-Commit-Position: refs/heads/master@{#28869}
This should prevent bugs caused by missing super calls in
overridden methods. The assumption is that methods of
different indicators are independent.
Review URL: https://codereview.chromium.org/1171943002
Cr-Commit-Position: refs/heads/master@{#28866}
This prepares for properly rerunning tests. Currently when
tests are rerun, the same test object is reused. This
will be changed in a follow up.
Review URL: https://codereview.chromium.org/1168303007
Cr-Commit-Position: refs/heads/master@{#28864}
Without this change, wildcards always overwrite the outcomes
of more specific rules. Now we always merge.
Review URL: https://codereview.chromium.org/1153073002
Cr-Commit-Position: refs/heads/master@{#28629}
This changes the unittests suite to be run in only the default variant,
because unit tests are (and should be) written to be independent from
global flags.
R=machenbach@chromium.org
Review URL: https://codereview.chromium.org/1067863007
Cr-Commit-Position: refs/heads/master@{#28054}
This flag is intended as a staging flag for TurboFan. It serves as a
single flag that always enables a most recent configuration of TurboFan
for test suites and benchmarks, without needing to update test drivers.
R=titzer@chromium.org,machenbach@chromium.org
Review URL: https://codereview.chromium.org/1094573002
Cr-Commit-Position: refs/heads/master@{#27896}
Second try. Disabled the tests that were failing due to
https://github.com/tc39/test262/issues/215
This updates test262 to revision d24fd10 (2015/03/11).
The files moved around in the test repo and a lot of new tests are
failing.
BUG=None
LOG=N
R=adamk, rossberg
Review URL: https://codereview.chromium.org/1040093003
Cr-Commit-Position: refs/heads/master@{#27543}
Reason for revert:
Bot failed to include verifyNotEnumerable function for some screwed up reason.
Original issue's description:
> [es6] Update test262 tests
>
> This updates test262 to revision d24fd10 (2015/03/11).
>
> The files moved around in the test repo and a lot of new tests are
> failing.
>
> BUG=None
> LOG=N
> R=rossberg
>
> Committed: https://crrev.com/4f2fb3835feff3663146f12be42b01a226d0065e
> Cr-Commit-Position: refs/heads/master@{#27522}
TBR=rossberg@chromium.org
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
BUG=None
Review URL: https://codereview.chromium.org/1047713002
Cr-Commit-Position: refs/heads/master@{#27525}
This updates test262 to revision d24fd10 (2015/03/11).
The files moved around in the test repo and a lot of new tests are
failing.
BUG=None
LOG=N
R=rossberg
Review URL: https://codereview.chromium.org/1025043002
Cr-Commit-Position: refs/heads/master@{#27522}
