Commit Graph

88 Commits

Author SHA1 Message Date
Michael Achenbach
d64a48dbdf [testrunner] Prevent erroneous overriding of signal handlers
When an overall timeout is reached, swarming sends a SIGTERM to
terminate the test runner. The test runner has a signal handler on the
main process to terminate all workers gracefully.

Additionally, every worker process installs a signal handler for
terminating ongoing tests wrapped by command.Command.

Also, command.Command is used on the main process to list tests for
cctest and gtest executables, which led to overriding the test runner's
main signal handler.

This CL disables using signal handlers in commands by default and only
explicitly enables it in safe source locations.

Bug: v8:8292
Change-Id: Ifceadaff75bdd2b77e761498bccbe00b6a3e265c
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2002528
Reviewed-by: Tamer Tas <tmrts@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#65815}
2020-01-16 14:44:14 +00:00
Liviu Rau
b990d2a4ad [testing] Show exit code hex value for failing tests
When a test fails the exit code of the process is printed
in the logs in the decimal format. On Windows searching for
the actual error is pretty difficult in this format since
we are dealing with negative values errors.
We print now the hex format of the same exit value next to
the original one.

Example failure of a test:
https://ci.chromium.org/p/v8/builders/try/v8_win64_dbg/b8897355488287117696
https://logs.chromium.org/logs/v8/buildbucket/cr-buildbucket.appspot.com/8897355488287117696/+/steps/Check/0/stdout
https://logs.chromium.org/logs/v8/buildbucket/cr-buildbucket.appspot.com/8897355488287117696/+/steps/Check/0/logs/RunWasmInterpreter_Int32Const_many/0

Bug: v8:9810
Change-Id: I51b65d6105bda6516707a764dbd41fd36c9dfc80
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1903975
Commit-Queue: Liviu Rau <liviurau@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Tamer Tas <tmrts@chromium.org>
Cr-Commit-Position: refs/heads/master@{#64878}
2019-11-11 11:15:41 +00:00
Michael Achenbach
393f788c1f Reland "[test] Remove longer timeout on bots"
This is a reland of e632f8f45b

Original change's description:
> [test] Remove longer timeout on bots
>
> CQ_INCLUDE_TRYBOTS=luci.v8.try:v8_linux_gc_stress_dbg
> CQ_INCLUDE_TRYBOTS=luci.v8.try:v8_linux_gcc_rel
> CQ_INCLUDE_TRYBOTS=luci.v8.try:v8_linux64_msan_rel
> CQ_INCLUDE_TRYBOTS=luci.v8.try:v8_linux64_tsan_rel
> CQ_INCLUDE_TRYBOTS=luci.v8.try:v8_linux_arm64_dbg
> CQ_INCLUDE_TRYBOTS=luci.v8.try:v8_mac64_asan_rel
> CQ_INCLUDE_TRYBOTS=luci.v8.try:v8_win64_dbg
> CQ_INCLUDE_TRYBOTS=luci.v8.try:v8_android_arm64_n5x_rel_ng
> CQ_INCLUDE_TRYBOTS=luci.v8.try:v8_linux64_cfi_rel_ng
> CQ_INCLUDE_TRYBOTS=luci.v8.try:v8_linux64_ubsan_rel_ng
> CQ_INCLUDE_TRYBOTS=luci.v8.try:v8_linux_arm_lite_rel_ng
> CQ_INCLUDE_TRYBOTS=luci.v8.try:v8_linux_noi18n_rel_ng
> CQ_INCLUDE_TRYBOTS=luci.v8.try:v8_mac64_dbg_ng
>
> Bug: v8:9145
> Change-Id: I6efee8579d9d9e0aad0431f6b87c152141d4ec7f
> Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1581261
> Commit-Queue: Michael Achenbach <machenbach@chromium.org>
> Reviewed-by: Tamer Tas <tmrts@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#61107}

Bug: v8:9145
Change-Id: Id8f0468bbf2d67f62141e7f7c42417d351c3fffa
Cq-Include-Trybots: luci.v8.try:v8_linux_gc_stress_dbg
Cq-Include-Trybots: luci.v8.try:v8_linux_gcc_rel
Cq-Include-Trybots: luci.v8.try:v8_linux64_msan_rel
Cq-Include-Trybots: luci.v8.try:v8_linux64_tsan_rel
Cq-Include-Trybots: luci.v8.try:v8_linux_arm64_dbg
Cq-Include-Trybots: luci.v8.try:v8_mac64_asan_rel
Cq-Include-Trybots: luci.v8.try:v8_win64_dbg
Cq-Include-Trybots: luci.v8.try:v8_android_arm64_n5x_rel_ng
Cq-Include-Trybots: luci.v8.try:v8_linux64_cfi_rel_ng
Cq-Include-Trybots: luci.v8.try:v8_linux64_ubsan_rel_ng
Cq-Include-Trybots: luci.v8.try:v8_linux_arm_lite_rel_ng
Cq-Include-Trybots: luci.v8.try:v8_linux_noi18n_rel_ng
Cq-Include-Trybots: luci.v8.try:v8_mac64_dbg_ng
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1588429
Reviewed-by: Tamer Tas <tmrts@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#61167}
2019-05-02 13:04:29 +00:00
Yang Guo
2e6b9f576c Revert "[test] Remove longer timeout on bots"
This reverts commit e632f8f45b.

Reason for revert: Seems to cause some timeouts to trigger failures: https://ci.chromium.org/p/v8/builders/ci/V8%20Linux64%20TSAN/26129

Original change's description:
> [test] Remove longer timeout on bots
> 
> CQ_INCLUDE_TRYBOTS=luci.v8.try:v8_linux_gc_stress_dbg
> CQ_INCLUDE_TRYBOTS=luci.v8.try:v8_linux_gcc_rel
> CQ_INCLUDE_TRYBOTS=luci.v8.try:v8_linux64_msan_rel
> CQ_INCLUDE_TRYBOTS=luci.v8.try:v8_linux64_tsan_rel
> CQ_INCLUDE_TRYBOTS=luci.v8.try:v8_linux_arm64_dbg
> CQ_INCLUDE_TRYBOTS=luci.v8.try:v8_mac64_asan_rel
> CQ_INCLUDE_TRYBOTS=luci.v8.try:v8_win64_dbg
> CQ_INCLUDE_TRYBOTS=luci.v8.try:v8_android_arm64_n5x_rel_ng
> CQ_INCLUDE_TRYBOTS=luci.v8.try:v8_linux64_cfi_rel_ng
> CQ_INCLUDE_TRYBOTS=luci.v8.try:v8_linux64_ubsan_rel_ng
> CQ_INCLUDE_TRYBOTS=luci.v8.try:v8_linux_arm_lite_rel_ng
> CQ_INCLUDE_TRYBOTS=luci.v8.try:v8_linux_noi18n_rel_ng
> CQ_INCLUDE_TRYBOTS=luci.v8.try:v8_mac64_dbg_ng
> 
> Bug: v8:9145
> Change-Id: I6efee8579d9d9e0aad0431f6b87c152141d4ec7f
> Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1581261
> Commit-Queue: Michael Achenbach <machenbach@chromium.org>
> Reviewed-by: Tamer Tas <tmrts@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#61107}

TBR=machenbach@chromium.org,tmrts@chromium.org

Change-Id: I5e23aebc41e6ab15baa57093f7f056cdabd58f21
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: v8:9145
Cq-Include-Trybots: luci.v8.try:v8_linux_gc_stress_dbg, luci.v8.try:v8_linux_gcc_rel, luci.v8.try:v8_linux64_msan_rel, luci.v8.try:v8_linux64_tsan_rel, luci.v8.try:v8_linux_arm64_dbg, luci.v8.try:v8_mac64_asan_rel, luci.v8.try:v8_win64_dbg, luci.v8.try:v8_android_arm64_n5x_rel_ng, luci.v8.try:v8_linux64_cfi_rel_ng, luci.v8.try:v8_linux64_ubsan_rel_ng, luci.v8.try:v8_linux_arm_lite_rel_ng, luci.v8.try:v8_linux_noi18n_rel_ng, luci.v8.try:v8_mac64_dbg_ng
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1588425
Reviewed-by: Yang Guo <yangguo@chromium.org>
Cr-Commit-Position: refs/heads/master@{#61111}
2019-04-30 11:01:39 +00:00
Michael Achenbach
e632f8f45b [test] Remove longer timeout on bots
CQ_INCLUDE_TRYBOTS=luci.v8.try:v8_linux_gc_stress_dbg
CQ_INCLUDE_TRYBOTS=luci.v8.try:v8_linux_gcc_rel
CQ_INCLUDE_TRYBOTS=luci.v8.try:v8_linux64_msan_rel
CQ_INCLUDE_TRYBOTS=luci.v8.try:v8_linux64_tsan_rel
CQ_INCLUDE_TRYBOTS=luci.v8.try:v8_linux_arm64_dbg
CQ_INCLUDE_TRYBOTS=luci.v8.try:v8_mac64_asan_rel
CQ_INCLUDE_TRYBOTS=luci.v8.try:v8_win64_dbg
CQ_INCLUDE_TRYBOTS=luci.v8.try:v8_android_arm64_n5x_rel_ng
CQ_INCLUDE_TRYBOTS=luci.v8.try:v8_linux64_cfi_rel_ng
CQ_INCLUDE_TRYBOTS=luci.v8.try:v8_linux64_ubsan_rel_ng
CQ_INCLUDE_TRYBOTS=luci.v8.try:v8_linux_arm_lite_rel_ng
CQ_INCLUDE_TRYBOTS=luci.v8.try:v8_linux_noi18n_rel_ng
CQ_INCLUDE_TRYBOTS=luci.v8.try:v8_mac64_dbg_ng

Bug: v8:9145
Change-Id: I6efee8579d9d9e0aad0431f6b87c152141d4ec7f
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1581261
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Tamer Tas <tmrts@chromium.org>
Cr-Commit-Position: refs/heads/master@{#61107}
2019-04-30 09:50:26 +00:00
Sergiy Belozorov
c8b5feec0f [tools] Remove TestFailedError and generate Output object for all cases
This is part of the refactoring to allow exporting more information about test
execution to the recipes and upload this information to ChromePerf.

This fixes running secondary even after primary run fails, which will allow us
to differentiate between test and infra failures as latter ones will also affect
refbuilds and re-runs without patch.

R=machenbach@chromium.org, tmrts@chromium.org

Bug: chromium:841700
Change-Id: I29ce49d2f8c5e73158f1d41a73c51f2b35929f36
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1570006
Commit-Queue: Sergiy Belozorov <sergiyb@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#60946}
2019-04-23 12:46:31 +00:00
Sven Sauleau
a427f31332 [testrunner] handle timeout param in file
WebAssembly's specification tests use a timeout annotation.
This change allows the shared testrunner to use it to calculate
the testcase timeout.

Currently, the allowed timeout values are: long. Other values will
emit a warning.

Change-Id: Id7f453f5fd49854c8f53ff86ef2ec58aa0ae8748
Reviewed-on: https://chromium-review.googlesource.com/c/1480376
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Sven Sauleau <ssauleau@igalia.com>
Cr-Commit-Position: refs/heads/master@{#59919}
2019-02-27 17:49:17 +00:00
Vadim Gorbachev (bmsdave)
6cad46b4f3 Preparing v8 to use with python3 /tools/testrunner
There are now less that 400 days until the end of life
of Python 2(aka _legacy_ Python) https://pythonclock.org/ .
The code compatibility check for python2 and python3
used the following tools: futurize, flake8
You can see the reports here: https://travis-ci.com/bmsdave/v8/builds

This CL was uploaded by git cl split.

Bug: v8:8594
Change-Id: I2a90aaecb270f03aed1c0fc92da1a0e2621b0eb2
Reviewed-on: https://chromium-review.googlesource.com/c/1470101
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Sergiy Belozorov <sergiyb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#59678}
2019-02-19 09:23:11 +00:00
Sergiy Belozorov
996cf21954 [tools] Generate additional variant for FAIL_PHASE_ONLY tests
The additional variant does not use wrapper disabling phase tests and negated
outcome processor. This allows to ensure that tests marked FAIL_PHASE_ONLY, do
actually fail without it.

R=machenbach@chromium.org

Bug: v8:8467
Change-Id: I66e07bd7107520872cc013bf0f33fdc6664baf56
Reviewed-on: https://chromium-review.googlesource.com/c/1361164
Commit-Queue: Sergiy Belozorov <sergiyb@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#58211}
2018-12-13 11:28:57 +00:00
Sergiy Byelozyorov
948cf2f642 [tools] Add FAIL_PHASE_ONLY status file modifier for test262 tests
R=machenbach@chromium.org, mathias@chromium.org

Bug: v8:8467
Change-Id: I7369fa993d0762cf9c3c6f55361d074d067a0eb5
Reviewed-on: https://chromium-review.googlesource.com/c/1338347
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Sergiy Belozorov <sergiyb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#57734}
2018-11-22 12:43:05 +00:00
Michael Achenbach
ab18455532 [test] Add feature to keep running skipped tests.
This commit adds a --run-skipped flag to the test runner that will
bypass the 'SKIP' status.

Bug: v8:8485
Change-Id: Iac012bdaf2de6b0f8e44ed3a65bc9330709527bb
Reviewed-on: https://chromium-review.googlesource.com/c/1346490
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Maya Lekova <mslekova@chromium.org>
Reviewed-by: Sigurd Schneider <sigurds@chromium.org>
Cr-Commit-Position: refs/heads/master@{#57710}
2018-11-22 08:12:21 +00:00
Michael Achenbach
eb7c56c8a7 [test] Fix resource regexp for Android
The regexp for finding resources to be pushed to Android devices was
too lax. On empty strings it tried to check for more resources on a
directory and hung.

The last test262 roll contains tests with empty imports that started
hanging in this way.

TBR=neis@chromium.org
NOTRY=true

Bug: v8:7834
Change-Id: Ie58f1b18bdd99b7b40c1fb39b25e2f481932e0f3
Reviewed-on: https://chromium-review.googlesource.com/c/1270579
Reviewed-by: Georg Neis <neis@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#56465}
2018-10-09 08:35:50 +00:00
Michael Achenbach
0dba4b907b Reland "[test] Increase coverage of d8_default test suites"
This is a reland of 8ac91f6c6a

Skips failing tests on gc stress and fixes predictable testing.

Original change's description:
> [test] Increase coverage of d8_default test suites
>
> NOTRY=true
>
> Bug: v8:7285,v8:8140
> Cq-Include-Trybots: luci.v8.try:v8_linux_noi18n_rel_ng
> Change-Id: I9cb216de302bc787189f8f12f5b254909b0f5773
> Reviewed-on: https://chromium-review.googlesource.com/1208496
> Commit-Queue: Michael Achenbach <machenbach@chromium.org>
> Reviewed-by: Ross McIlroy <rmcilroy@chromium.org>
> Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#55682}

Bug: v8:7285, v8:8140, v8:8141
Change-Id: Ia7a437b874d5c8712f6def30382404e527145610
Cq-Include-Trybots: luci.v8.try:v8_linux_noi18n_rel_ng
Cq-Include-Trybots: luci.v8.try:v8_android_arm64_n5x_rel_ng
Cq-Include-Trybots: luci.v8.try:v8_linux_optional_rel_ng
Cq-Include-Trybots: luci.v8.try:v8_linux_gc_stress_dbg
Cq-Include-Trybots: luci.v8.try:v8_mac64_gc_stress_dbg
Reviewed-on: https://chromium-review.googlesource.com/1209762
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Reviewed-by: Yang Guo <yangguo@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#55701}
2018-09-07 07:53:29 +00:00
Michael Achenbach
217cced963 [test] Enable Andoid testing on more JS test suites
Bug: chromium:866862
Change-Id: Ib6c89d88344c2348de155fd54eafa03cf18ea88d
Reviewed-on: https://chromium-review.googlesource.com/1206631
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#55674}
2018-09-06 07:26:22 +00:00
Michael Achenbach
beb7778d87 [test] Fix resource-fetching logic for imported files
Path names in import/export statements are relative to the file they are in.

This fixes the logic and unblocks using the messages test suite on Android,
which has cases importing files from mjsunit, which import more files from
there.

Bug: chromium:866862
Change-Id: I8d2ff645f69b67fbdaf4a622d06308e55298b0ce
Reviewed-on: https://chromium-review.googlesource.com/1206570
Reviewed-by: Georg Neis <neis@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#55643}
2018-09-05 10:27:07 +00:00
Michael Achenbach
6206a3e362 [test] Share resource-fetching logic with all d8 test cases
This shares logic for finding additional resources in JS source code.
Previously the logic was implemented for mjsunit, now it will be used
across all d8-based test cases.

This'll enable adding those test suites for Android testing.

Bug: chromium:866862
Cq-Include-Trybots: luci.v8.try:v8_linux_noi18n_rel_ng
Change-Id: I7c89ba141483aaf692a03c0e168edb61bbb7b010
Reviewed-on: https://chromium-review.googlesource.com/1193873
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Jakob Gruber <jgruber@chromium.org>
Reviewed-by: Georg Neis <neis@chromium.org>
Cr-Commit-Position: refs/heads/master@{#55503}
2018-08-29 17:51:43 +00:00
Michael Achenbach
b2537f219d Reland "[test] Add logic to run tests on Android"
This is a reland of 4c0943424c

Original change's description:
> [test] Add logic to run tests on Android
> 
> This adds a new command abstraction for running commands on Android
> using dockered devices on swarming.
> 
> The new abstraction handles pushing all required files to the device.
> The logic used for pushing and running is reused from the perf runner.
> 
> This adds only the mjsunit test suite. Others will be handled in
> follow up CLs. The suite logic is enhanced with auto-detection of files
> to be pushed to devices, for e.g. load or import statements.
> 
> Some test cases need an extra resource section for specifying required
> files.
> 
> Remaining failing tests are marked in the status files for later
> triage.
> 
> Bug: chromium:866862
> Change-Id: I2b957559f07fdcd8c1bd2f7034f5ba7754a31fb7
> Reviewed-on: https://chromium-review.googlesource.com/1150153
> Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
> Commit-Queue: Michael Achenbach <machenbach@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#55041}

Bug: chromium:866862
Change-Id: Icf7e04c75d4abeab7254d10ba21240e46b0022ae
Reviewed-on: https://chromium-review.googlesource.com/1170643
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#55059}
2018-08-10 17:56:12 +00:00
Sergiy Byelozyorov
2071051e4b Revert "[test] Add logic to run tests on Android"
This reverts commit 4c0943424c.

Reason for revert: Unfortunately this broke all perf builders.

Original change's description:
> [test] Add logic to run tests on Android
> 
> This adds a new command abstraction for running commands on Android
> using dockered devices on swarming.
> 
> The new abstraction handles pushing all required files to the device.
> The logic used for pushing and running is reused from the perf runner.
> 
> This adds only the mjsunit test suite. Others will be handled in
> follow up CLs. The suite logic is enhanced with auto-detection of files
> to be pushed to devices, for e.g. load or import statements.
> 
> Some test cases need an extra resource section for specifying required
> files.
> 
> Remaining failing tests are marked in the status files for later
> triage.
> 
> Bug: chromium:866862
> Change-Id: I2b957559f07fdcd8c1bd2f7034f5ba7754a31fb7
> Reviewed-on: https://chromium-review.googlesource.com/1150153
> Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
> Commit-Queue: Michael Achenbach <machenbach@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#55041}

TBR=machenbach@chromium.org,yangguo@chromium.org,sergiyb@chromium.org

Change-Id: If80129810586b709dab762c9b5724888e15daec2
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: chromium:866862
Reviewed-on: https://chromium-review.googlesource.com/1170962
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Commit-Queue: Sergiy Byelozyorov <sergiyb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#55058}
2018-08-10 16:16:16 +00:00
Michael Achenbach
4c0943424c [test] Add logic to run tests on Android
This adds a new command abstraction for running commands on Android
using dockered devices on swarming.

The new abstraction handles pushing all required files to the device.
The logic used for pushing and running is reused from the perf runner.

This adds only the mjsunit test suite. Others will be handled in
follow up CLs. The suite logic is enhanced with auto-detection of files
to be pushed to devices, for e.g. load or import statements.

Some test cases need an extra resource section for specifying required
files.

Remaining failing tests are marked in the status files for later
triage.

Bug: chromium:866862
Change-Id: I2b957559f07fdcd8c1bd2f7034f5ba7754a31fb7
Reviewed-on: https://chromium-review.googlesource.com/1150153
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#55041}
2018-08-10 10:11:53 +00:00
Michael Achenbach
eb086bb6b2 [test] Change precedence of statusfile flags
Bug: v8:7812
Change-Id: Ia5a9c68e0aeb9d993ca2cd5f855ff39e172890e8
Reviewed-on: https://chromium-review.googlesource.com/1090282
Reviewed-by: Sathya Gunasekaran <gsathya@chromium.org>
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#53601}
2018-06-08 08:09:22 +00:00
Michal Majewski
2fe1af4787 [test] Remove dead code from suites and tests
Bug: v8:6917
Cq-Include-Trybots: luci.v8.try:v8_linux_noi18n_rel_ng
Change-Id: I6bfea537347fb236d41b31f0c6c64c108928bf28
Reviewed-on: https://chromium-review.googlesource.com/897784
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michał Majewski <majeski@google.com>
Cr-Commit-Position: refs/heads/master@{#51033}
2018-02-01 14:12:49 +00:00
Michal Majewski
e64f546140 [test] Remove old code from run-tests
Only test processors code left. It enabled to move more stuff to
the base runner, like progress indicators creation.

Bug: v8:6917
Change-Id: Ie6dd211cec561a07d92bcc4431ea88eb1842c8fa
Reviewed-on: https://chromium-review.googlesource.com/897624
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michał Majewski <majeski@google.com>
Cr-Commit-Position: refs/heads/master@{#51030}
2018-02-01 13:24:57 +00:00
Michal Majewski
d3083b46ee [test] Move options to test config
Bug: v8:6917
Cq-Include-Trybots: luci.v8.try:v8_linux_noi18n_rel_ng
Change-Id: I254d2e545709029346f585b02a9edf91d3f27893
Reviewed-on: https://chromium-review.googlesource.com/893321
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michał Majewski <majeski@google.com>
Cr-Commit-Position: refs/heads/master@{#50996}
2018-01-31 13:46:29 +00:00
Michal Majewski
35ca0a01c3 [test] Filter out incompatible tests in predictable mode
Bug: v8:6917
Change-Id: Ie18390ae0849e8d5341e4c2a7b757fae886c2986
Reviewed-on: https://chromium-review.googlesource.com/895582
Commit-Queue: Michał Majewski <majeski@google.com>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#50994}
2018-01-31 13:32:41 +00:00
Michal Majewski
5ede3cb533 Reland "[test] Random seed processor"
This is a reland of 0db74d4974.

Original change's description:
> [test] Random seed processor
> 
> 1. --total-timeout-sec now available for ./run-tests.py. It can be
> useful with infinite seed stressing
> 2. random seed dropped from the context. Now JSON progress indicator
> gets it from the list of command args.
> 
> Bug: v8:6917
> Cq-Include-Trybots: luci.v8.try:v8_linux_noi18n_rel_ng
> Change-Id: I73e535bc8face9b913c696b8d5e3a246fa231004
> Reviewed-on: https://chromium-review.googlesource.com/888524
> Commit-Queue: Michał Majewski <majeski@google.com>
> Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#50964}

Bug: v8:6917
Change-Id: I1ea376a4abffce5ab65f4834ea7e6d6011765ffa
Cq-Include-Trybots: luci.v8.try:v8_linux_noi18n_rel_ng
Reviewed-on: https://chromium-review.googlesource.com/894204
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michał Majewski <majeski@google.com>
Cr-Commit-Position: refs/heads/master@{#50978}
2018-01-31 09:53:34 +00:00
Michael Achenbach
ba30988cbc Revert "[test] Random seed processor"
This reverts commit 0db74d4974.

Reason for revert: https://chromium-swarm.appspot.com/task?id=3b609f9976bac610&refresh=10&show_raw=1

Original change's description:
> [test] Random seed processor
> 
> 1. --total-timeout-sec now available for ./run-tests.py. It can be
> useful with infinite seed stressing
> 2. random seed dropped from the context. Now JSON progress indicator
> gets it from the list of command args.
> 
> Bug: v8:6917
> Cq-Include-Trybots: luci.v8.try:v8_linux_noi18n_rel_ng
> Change-Id: I73e535bc8face9b913c696b8d5e3a246fa231004
> Reviewed-on: https://chromium-review.googlesource.com/888524
> Commit-Queue: Michał Majewski <majeski@google.com>
> Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#50964}

TBR=machenbach@chromium.org,sergiyb@chromium.org,majeski@google.com

Change-Id: I2d96ea328cda2d09b01ff455e47c77d567fafe00
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: v8:6917
Cq-Include-Trybots: luci.v8.try:v8_linux_noi18n_rel_ng
Reviewed-on: https://chromium-review.googlesource.com/894522
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#50970}
2018-01-30 22:05:03 +00:00
Michal Majewski
0db74d4974 [test] Random seed processor
1. --total-timeout-sec now available for ./run-tests.py. It can be
useful with infinite seed stressing
2. random seed dropped from the context. Now JSON progress indicator
gets it from the list of command args.

Bug: v8:6917
Cq-Include-Trybots: luci.v8.try:v8_linux_noi18n_rel_ng
Change-Id: I73e535bc8face9b913c696b8d5e3a246fa231004
Reviewed-on: https://chromium-review.googlesource.com/888524
Commit-Queue: Michał Majewski <majeski@google.com>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#50964}
2018-01-30 15:50:49 +00:00
Michal Majewski
0998eda9a4 [test] Disable reduce result on the main process
Since we're not winning anything by changing the result between
processors on the main process, reduce is noop there and result is
immutable.

Bug: v8:6917
Change-Id: Ieb282e7abd4ab31162aee6b52493a6e1b6a25109
Cq-Include-Trybots: luci.v8.try:v8_linux64_fyi_rel_ng
Reviewed-on: https://chromium-review.googlesource.com/878239
Commit-Queue: Michał Majewski <majeski@google.com>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#50784}
2018-01-23 08:58:20 +00:00
Michael Achenbach
7497fa1ab6 [test] Add rerun and json output to numfuzz
This adds flags to numfuzz for rerunning flaky tests and for storing
json test results. With those flags added, the infra-side can call
numfuzz with the same API as the standard test-runner.

TBR=sergiyb@chromium.org

Bug: v8:6917
Change-Id: I02d1cd02d90677c83f10e072383f3650c041cab1
Reviewed-on: https://chromium-review.googlesource.com/877890
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#50753}
2018-01-22 12:26:36 +00:00
Michal Majewski
50a91fe9ee [test] Fix subtest creation
Update flags before recalculating outcomes.

Bug: v8:6917
Change-Id: I5f54f7d14fd60f7e35a976e5200d3f7f0e74a5b7
Cq-Include-Trybots: luci.v8.try:v8_linux64_fyi_rel_ng
Reviewed-on: https://chromium-review.googlesource.com/876364
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michał Majewski <majeski@google.com>
Cr-Commit-Position: refs/heads/master@{#50731}
2018-01-19 18:37:08 +00:00
Michal Majewski
61c562b026 [test] Implement gc fuzzer with test processors
Bug: v8:6917
Change-Id: I2a7ecc6897c8ccd6ed862cf2b0b484673ee359f6
Reviewed-on: https://chromium-review.googlesource.com/871310
Commit-Queue: Michał Majewski <majeski@google.com>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#50728}
2018-01-19 16:58:49 +00:00
Michal Majewski
0bfc1b2562 [test] Do not pass unnecessary results
Bug: v8:6917
Change-Id: I1a355bdfe3f873091a7d7c32a937a533a7d8b3d4
Cq-Include-Trybots: luci.v8.try:v8_linux64_fyi_rel_ng
Reviewed-on: https://chromium-review.googlesource.com/867053
Commit-Queue: Michał Majewski <majeski@google.com>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#50650}
2018-01-17 11:32:12 +00:00
Michael Achenbach
56fa0a535b [test] Remove deprecated FAST_VARIANTS modifier
Bug: v8:7264
Cq-Include-Trybots: luci.v8.try:v8_linux_noi18n_rel_ng
Change-Id: I0c07c92955c0b6c145c7b72e5014f828eaf59c9a
Reviewed-on: https://chromium-review.googlesource.com/865903
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#50582}
2018-01-15 14:30:47 +00:00
Michal Majewski
60c17bf983 [test] Simplify passing results between test processors.
Bug: v8:6917
Change-Id: Id73e4892a0d1b3b9c5bdd70ccc136e7bd2edf360
Cq-Include-Trybots: luci.v8.try:v8_linux64_fyi_rel_ng
Reviewed-on: https://chromium-review.googlesource.com/863603
Commit-Queue: Michał Majewski <majeski@google.com>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#50565}
2018-01-15 08:51:40 +00:00
Michal Majewski
7f01c9303f [test] Implement variant generators as processors
Bug: v8:6917
Cq-Include-Trybots: luci.v8.try:v8_linux64_fyi_rel_ng;luci.v8.try:v8_linux_noi18n_rel_ng
Change-Id: Ib5bfdf4d6fee6102f62c7334a1b22146f1a1fc5b
Reviewed-on: https://chromium-review.googlesource.com/857376
Commit-Queue: Michał Majewski <majeski@google.com>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#50525}
2018-01-12 10:39:20 +00:00
Michal Majewski
ed619fa6c6 [test] Make variant names unique
Temporary workaround to avoid the same variant name for multiple
variants of the same test.

Bug: v8:6917
Change-Id: I9a25dcaf81d35da0dc2617c089cb4811c2a958cb
Cq-Include-Trybots: luci.v8.try:v8_linux64_fyi_rel_ng
Reviewed-on: https://chromium-review.googlesource.com/852833
Commit-Queue: Michał Majewski <majeski@google.com>
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#50428}
2018-01-09 11:18:40 +00:00
Michal Majewski
da3416f53a [test] Introduce test processors
Test processors can be enabled with --infra-staging flag.

Rerunning tests, execution and verbose progress indicator already
work as test processors.

Bug: v8:6917
Change-Id: I40fc42db94dbc8629e8000a3d363030045532fe3
Reviewed-on: https://chromium-review.googlesource.com/850398
Commit-Queue: Michał Majewski <majeski@google.com>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#50399}
2018-01-08 11:27:47 +00:00
Michal Majewski
9a3773e665 [test] Fix predictable mode with output processors
Bug: v8:6917
Change-Id: I7bc5d09ff19a9cf7cb4bee68ca6cf34f5cfa8b19
Reviewed-on: https://chromium-review.googlesource.com/850874
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#50365}
2018-01-04 19:16:45 +00:00
Michal Majewski
9f7d440e98 [test] Check output on the worker process.
I added additional exception logging in the execution.py since
errors in processing results were really difficult to debug.

There is a problem on Windows with class serialization when
it comes from dynamically loaded module. To fix it I moved all
output processors to the tools/testrunner/outproc/ and import
them in test/*/testcfg.py.

Bug: v8:6917
Change-Id: Ida604641d659b006e91faf1d56a37769ec47f5f3
Reviewed-on: https://chromium-review.googlesource.com/842784
Commit-Queue: Michał Majewski <majeski@google.com>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#50361}
2018-01-04 14:12:31 +00:00
Michael Achenbach
c1e9bc02f8 Revert "Revert "[test] Move has unexpected output to outproc.""
This reverts commit 1685b5d27a.

Reason for revert: Was probably caused by infra change:
https://crrev.com/c/845781

Original change's description:
> Revert "[test] Move has unexpected output to outproc."
> 
> This reverts commit 71605b3ea4.
> 
> Reason for revert: Seems to break static-initializers step:
> https://build.chromium.org/p/client.v8/builders/V8%20Linux64/builds/22156
> 
> Original change's description:
> > [test] Move has unexpected output to outproc.
> > 
> > Expected outcomes optimized to serialize [PASS] as None.
> > 
> > Keeping expected outcomes inside output processors should be
> > optimized in the future. Few possible optimizations:
> > - separate classes for tests that are expected to PASS - done as
> > an example in mozilla test suite.
> > - cache output processors inside testcase.
> > - share output processors between copies of the same test - needs
> > some updates to the create_variant to update outproc only if
> > expected outcomes changed.
> > 
> > Bug: v8:6917
> > Change-Id: Ie73f1dcdf17fdfc65bce27228f818b1dd1e420c9
> > Reviewed-on: https://chromium-review.googlesource.com/843025
> > Commit-Queue: Michael Achenbach <machenbach@chromium.org>
> > Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> > Cr-Commit-Position: refs/heads/master@{#50347}
> 
> TBR=machenbach@chromium.org,sergiyb@chromium.org,majeski@google.com
> 
> Change-Id: Ice1f3aee0a26f7f38996459d38fd6e0bd964113d
> No-Presubmit: true
> No-Tree-Checks: true
> No-Try: true
> Bug: v8:6917
> Reviewed-on: https://chromium-review.googlesource.com/849572
> Reviewed-by: Bill Budge <bbudge@chromium.org>
> Commit-Queue: Bill Budge <bbudge@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#50348}

TBR=bbudge@chromium.org,machenbach@chromium.org,sergiyb@chromium.org,majeski@google.com

Change-Id: I7a522b6487de6e96985d223524533493eb9171f0
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: v8:6917
Reviewed-on: https://chromium-review.googlesource.com/848975
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#50350}
2018-01-04 07:39:21 +00:00
Bill Budge
1685b5d27a Revert "[test] Move has unexpected output to outproc."
This reverts commit 71605b3ea4.

Reason for revert: Seems to break static-initializers step:
https://build.chromium.org/p/client.v8/builders/V8%20Linux64/builds/22156

Original change's description:
> [test] Move has unexpected output to outproc.
> 
> Expected outcomes optimized to serialize [PASS] as None.
> 
> Keeping expected outcomes inside output processors should be
> optimized in the future. Few possible optimizations:
> - separate classes for tests that are expected to PASS - done as
> an example in mozilla test suite.
> - cache output processors inside testcase.
> - share output processors between copies of the same test - needs
> some updates to the create_variant to update outproc only if
> expected outcomes changed.
> 
> Bug: v8:6917
> Change-Id: Ie73f1dcdf17fdfc65bce27228f818b1dd1e420c9
> Reviewed-on: https://chromium-review.googlesource.com/843025
> Commit-Queue: Michael Achenbach <machenbach@chromium.org>
> Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#50347}

TBR=machenbach@chromium.org,sergiyb@chromium.org,majeski@google.com

Change-Id: Ice1f3aee0a26f7f38996459d38fd6e0bd964113d
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: v8:6917
Reviewed-on: https://chromium-review.googlesource.com/849572
Reviewed-by: Bill Budge <bbudge@chromium.org>
Commit-Queue: Bill Budge <bbudge@chromium.org>
Cr-Commit-Position: refs/heads/master@{#50348}
2018-01-03 23:01:11 +00:00
Michal Majewski
71605b3ea4 [test] Move has unexpected output to outproc.
Expected outcomes optimized to serialize [PASS] as None.

Keeping expected outcomes inside output processors should be
optimized in the future. Few possible optimizations:
- separate classes for tests that are expected to PASS - done as
an example in mozilla test suite.
- cache output processors inside testcase.
- share output processors between copies of the same test - needs
some updates to the create_variant to update outproc only if
expected outcomes changed.

Bug: v8:6917
Change-Id: Ie73f1dcdf17fdfc65bce27228f818b1dd1e420c9
Reviewed-on: https://chromium-review.googlesource.com/843025
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#50347}
2018-01-03 22:19:49 +00:00
Michal Majewski
a130438e13 [test] Removed old output parsing functions.
All get outcome logic moved out from testsuite which makes output
processors serializable. Output processors that are not global
are stored inside testcase to share them with all test copies.

Bug: v8:6917
Cq-Include-Trybots: master.tryserver.v8:v8_linux_noi18n_rel_ng
Change-Id: I625466b2b00565d885556f96cab03d8d16fa7ff4
Reviewed-on: https://chromium-review.googlesource.com/837069
Commit-Queue: Michał Majewski <majeski@google.com>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#50275}
2017-12-21 15:25:33 +00:00
Michal Majewski
848487ad61 [test] Output processor for message
Bug: v8:6917
Change-Id: I51ec0be7ea961d3c81e03675cc65d03ecf526c3d
Reviewed-on: https://chromium-review.googlesource.com/836988
Commit-Queue: Michał Majewski <majeski@google.com>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#50268}
2017-12-21 14:38:28 +00:00
Michal Majewski
b0db2dc6ae [test] Output processors for inspector and webkit
Bug: v8:6917
Change-Id: I81bff2190766b3ccbc5da43ff2f3105c6c95da67
Reviewed-on: https://chromium-review.googlesource.com/836557
Commit-Queue: Michał Majewski <majeski@google.com>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#50263}
2017-12-21 12:55:35 +00:00
Michal Majewski
d705e15857 [test] Prepare initial version of output processor
First CL moving parsing output to a separate class.
Following CLs will move (and deduplicate) code from
testcases by sharing common features like comparing
stdout to the expected output.

Bug: v8:6917
Change-Id: I1174a985949a0b946cefbf7e7c91e8815cbb4f15
Reviewed-on: https://chromium-review.googlesource.com/834130
Commit-Queue: Michał Majewski <majeski@google.com>
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#50228}
2017-12-20 11:24:07 +00:00
Michal Majewski
56b2b3be5e [test] Fix --report output.
Fix report mode and make it use testcase properties so statusfile
outcomes can be private.

Bug: v8:6917
Change-Id: Id38e89e0ba427c3bbb7ad12ba93e02beb7e46219
Reviewed-on: https://chromium-review.googlesource.com/833909
Commit-Queue: Michał Majewski <majeski@google.com>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#50209}
2017-12-19 16:35:39 +00:00
Michal Majewski
5e6fd798f9 [test] Fixed target_name in json progress indicator
Pass shell name instead of an absolute path.

Bug: v8:796166
Change-Id: Ia9472e893fd2cb3fde2a94997f3e9daf30da06ea
Reviewed-on: https://chromium-review.googlesource.com/833917
Commit-Queue: Michał Majewski <majeski@google.com>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#50207}
2017-12-19 15:23:54 +00:00
Michal Majewski
4695e97905 [test] Move getting outcomes to the statusfile
Bug: v8:6917
Change-Id: I175fa426546f2f3775a35f1094dfb19e06b2185d
Reviewed-on: https://chromium-review.googlesource.com/832394
Commit-Queue: Michał Majewski <majeski@google.com>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#50203}
2017-12-19 14:35:16 +00:00
Michal Majewski
1f60466659 [test] Store outcomes in the testcase
Bug: v8:6917
Cq-Include-Trybots: master.tryserver.v8:v8_linux_noi18n_rel_ng
Change-Id: Ia52d4bedbeff5b93915ef69a2dc78f6d92669061
Reviewed-on: https://chromium-review.googlesource.com/832467
Commit-Queue: Michał Majewski <majeski@google.com>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#50200}
2017-12-19 14:10:06 +00:00