Commit Graph

60 Commits

Author SHA1 Message Date
Michael Achenbach
0dba4b907b Reland "[test] Increase coverage of d8_default test suites"
This is a reland of 8ac91f6c6a

Skips failing tests on gc stress and fixes predictable testing.

Original change's description:
> [test] Increase coverage of d8_default test suites
>
> NOTRY=true
>
> Bug: v8:7285,v8:8140
> Cq-Include-Trybots: luci.v8.try:v8_linux_noi18n_rel_ng
> Change-Id: I9cb216de302bc787189f8f12f5b254909b0f5773
> Reviewed-on: https://chromium-review.googlesource.com/1208496
> Commit-Queue: Michael Achenbach <machenbach@chromium.org>
> Reviewed-by: Ross McIlroy <rmcilroy@chromium.org>
> Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#55682}

Bug: v8:7285, v8:8140, v8:8141
Change-Id: Ia7a437b874d5c8712f6def30382404e527145610
Cq-Include-Trybots: luci.v8.try:v8_linux_noi18n_rel_ng
Cq-Include-Trybots: luci.v8.try:v8_android_arm64_n5x_rel_ng
Cq-Include-Trybots: luci.v8.try:v8_linux_optional_rel_ng
Cq-Include-Trybots: luci.v8.try:v8_linux_gc_stress_dbg
Cq-Include-Trybots: luci.v8.try:v8_mac64_gc_stress_dbg
Reviewed-on: https://chromium-review.googlesource.com/1209762
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Reviewed-by: Yang Guo <yangguo@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#55701}
2018-09-07 07:53:29 +00:00
Michael Achenbach
b2537f219d Reland "[test] Add logic to run tests on Android"
This is a reland of 4c0943424c

Original change's description:
> [test] Add logic to run tests on Android
> 
> This adds a new command abstraction for running commands on Android
> using dockered devices on swarming.
> 
> The new abstraction handles pushing all required files to the device.
> The logic used for pushing and running is reused from the perf runner.
> 
> This adds only the mjsunit test suite. Others will be handled in
> follow up CLs. The suite logic is enhanced with auto-detection of files
> to be pushed to devices, for e.g. load or import statements.
> 
> Some test cases need an extra resource section for specifying required
> files.
> 
> Remaining failing tests are marked in the status files for later
> triage.
> 
> Bug: chromium:866862
> Change-Id: I2b957559f07fdcd8c1bd2f7034f5ba7754a31fb7
> Reviewed-on: https://chromium-review.googlesource.com/1150153
> Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
> Commit-Queue: Michael Achenbach <machenbach@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#55041}

Bug: chromium:866862
Change-Id: Icf7e04c75d4abeab7254d10ba21240e46b0022ae
Reviewed-on: https://chromium-review.googlesource.com/1170643
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#55059}
2018-08-10 17:56:12 +00:00
Sergiy Byelozyorov
2071051e4b Revert "[test] Add logic to run tests on Android"
This reverts commit 4c0943424c.

Reason for revert: Unfortunately this broke all perf builders.

Original change's description:
> [test] Add logic to run tests on Android
> 
> This adds a new command abstraction for running commands on Android
> using dockered devices on swarming.
> 
> The new abstraction handles pushing all required files to the device.
> The logic used for pushing and running is reused from the perf runner.
> 
> This adds only the mjsunit test suite. Others will be handled in
> follow up CLs. The suite logic is enhanced with auto-detection of files
> to be pushed to devices, for e.g. load or import statements.
> 
> Some test cases need an extra resource section for specifying required
> files.
> 
> Remaining failing tests are marked in the status files for later
> triage.
> 
> Bug: chromium:866862
> Change-Id: I2b957559f07fdcd8c1bd2f7034f5ba7754a31fb7
> Reviewed-on: https://chromium-review.googlesource.com/1150153
> Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
> Commit-Queue: Michael Achenbach <machenbach@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#55041}

TBR=machenbach@chromium.org,yangguo@chromium.org,sergiyb@chromium.org

Change-Id: If80129810586b709dab762c9b5724888e15daec2
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: chromium:866862
Reviewed-on: https://chromium-review.googlesource.com/1170962
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Commit-Queue: Sergiy Byelozyorov <sergiyb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#55058}
2018-08-10 16:16:16 +00:00
Michael Achenbach
4c0943424c [test] Add logic to run tests on Android
This adds a new command abstraction for running commands on Android
using dockered devices on swarming.

The new abstraction handles pushing all required files to the device.
The logic used for pushing and running is reused from the perf runner.

This adds only the mjsunit test suite. Others will be handled in
follow up CLs. The suite logic is enhanced with auto-detection of files
to be pushed to devices, for e.g. load or import statements.

Some test cases need an extra resource section for specifying required
files.

Remaining failing tests are marked in the status files for later
triage.

Bug: chromium:866862
Change-Id: I2b957559f07fdcd8c1bd2f7034f5ba7754a31fb7
Reviewed-on: https://chromium-review.googlesource.com/1150153
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#55041}
2018-08-10 10:11:53 +00:00
Sergiy Byelozyorov
a82ea31afb [tools] Detect location of android build tools automatically
R=machenbach@chromium.org

No-Try: true
Bug: chromium:838864
Change-Id: Ifbdd9fe986e84197fd1ff6441c1565fed44c2c6f
Reviewed-on: https://chromium-review.googlesource.com/1149869
Commit-Queue: Sergiy Byelozyorov <sergiyb@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#54689}
2018-07-25 11:23:55 +00:00
Michael Achenbach
4a2d9b1b20 [test] Abort testing immediately on interrupt and sigterm
This is a partial revert of:
https://crrev.com/c/890938 and https://crrev.com/c/893982

Before this CL, the test runner blocked on ongoing tests in order to
process their results after an internal timeout. However, the logic
required for this feature was overly complicated and prevented an
acceptable implementation for fast aborts. Furthermore, also the fuzzers
suffered from timeouts on swarming due to hanging tests.

Instead, we now abort immediately on internal timeout (used on
fuzzers), SIGINT (Ctrl-C) and SIGTERM. Ongoing tests are immediately
terminated and their results are disregarded. On SIGTERM and SIGINT,
we return with non-zero exit codes, and zero on internal timeout.

This will also properly return json output, when the external hard
timeout is reached on swarming (causes SIGTERM).

TBR=sergiyb@chromium.org

Bug: v8:7423, chromium:813065
Change-Id: Ib20f835f58a0970693bdd3b21dc5d766d8e115d8
Reviewed-on: https://chromium-review.googlesource.com/924852
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#51399}
2018-02-20 18:02:25 +00:00
Michal Majewski
bb9d073a2c [test] Always run with test processors
System tests updated to pass. I will cleanup the code in a
separate CL removing the old code.

Bug: v8:6917
Change-Id: Ie49efe06c43faade846e322b35c4a2068137a88b
Reviewed-on: https://chromium-review.googlesource.com/897565
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michał Majewski <majeski@google.com>
Cr-Commit-Position: refs/heads/master@{#51025}
2018-02-01 10:46:32 +00:00
Michal Majewski
804cc9f385 [test] Remove context usage from test processors
Bug: v8:6917
Cq-Include-Trybots: luci.v8.try:v8_linux_noi18n_rel_ng
Change-Id: Id5a7b41502dfd4be43496b1edb958522a8eb0c31
Reviewed-on: https://chromium-review.googlesource.com/895588
Commit-Queue: Michał Majewski <majeski@google.com>
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#51023}
2018-02-01 10:35:02 +00:00
Michal Majewski
d3083b46ee [test] Move options to test config
Bug: v8:6917
Cq-Include-Trybots: luci.v8.try:v8_linux_noi18n_rel_ng
Change-Id: I254d2e545709029346f585b02a9edf91d3f27893
Reviewed-on: https://chromium-review.googlesource.com/893321
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michał Majewski <majeski@google.com>
Cr-Commit-Position: refs/heads/master@{#50996}
2018-01-31 13:46:29 +00:00
Michal Majewski
5ede3cb533 Reland "[test] Random seed processor"
This is a reland of 0db74d4974.

Original change's description:
> [test] Random seed processor
> 
> 1. --total-timeout-sec now available for ./run-tests.py. It can be
> useful with infinite seed stressing
> 2. random seed dropped from the context. Now JSON progress indicator
> gets it from the list of command args.
> 
> Bug: v8:6917
> Cq-Include-Trybots: luci.v8.try:v8_linux_noi18n_rel_ng
> Change-Id: I73e535bc8face9b913c696b8d5e3a246fa231004
> Reviewed-on: https://chromium-review.googlesource.com/888524
> Commit-Queue: Michał Majewski <majeski@google.com>
> Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#50964}

Bug: v8:6917
Change-Id: I1ea376a4abffce5ab65f4834ea7e6d6011765ffa
Cq-Include-Trybots: luci.v8.try:v8_linux_noi18n_rel_ng
Reviewed-on: https://chromium-review.googlesource.com/894204
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michał Majewski <majeski@google.com>
Cr-Commit-Position: refs/heads/master@{#50978}
2018-01-31 09:53:34 +00:00
Michael Achenbach
ba30988cbc Revert "[test] Random seed processor"
This reverts commit 0db74d4974.

Reason for revert: https://chromium-swarm.appspot.com/task?id=3b609f9976bac610&refresh=10&show_raw=1

Original change's description:
> [test] Random seed processor
> 
> 1. --total-timeout-sec now available for ./run-tests.py. It can be
> useful with infinite seed stressing
> 2. random seed dropped from the context. Now JSON progress indicator
> gets it from the list of command args.
> 
> Bug: v8:6917
> Cq-Include-Trybots: luci.v8.try:v8_linux_noi18n_rel_ng
> Change-Id: I73e535bc8face9b913c696b8d5e3a246fa231004
> Reviewed-on: https://chromium-review.googlesource.com/888524
> Commit-Queue: Michał Majewski <majeski@google.com>
> Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#50964}

TBR=machenbach@chromium.org,sergiyb@chromium.org,majeski@google.com

Change-Id: I2d96ea328cda2d09b01ff455e47c77d567fafe00
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: v8:6917
Cq-Include-Trybots: luci.v8.try:v8_linux_noi18n_rel_ng
Reviewed-on: https://chromium-review.googlesource.com/894522
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#50970}
2018-01-30 22:05:03 +00:00
Michal Majewski
0db74d4974 [test] Random seed processor
1. --total-timeout-sec now available for ./run-tests.py. It can be
useful with infinite seed stressing
2. random seed dropped from the context. Now JSON progress indicator
gets it from the list of command args.

Bug: v8:6917
Cq-Include-Trybots: luci.v8.try:v8_linux_noi18n_rel_ng
Change-Id: I73e535bc8face9b913c696b8d5e3a246fa231004
Reviewed-on: https://chromium-review.googlesource.com/888524
Commit-Queue: Michał Majewski <majeski@google.com>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#50964}
2018-01-30 15:50:49 +00:00
Michael Achenbach
ee4afcbf39 [test] Roll out new test runner to all testing with arch x64
This will affect all manual test runs with x64. Most bots on x64 already
migrated.

TBR=sergiyb@chromium.org
NOTRY=true

Bug: v8:7343
Change-Id: I87f46f1848a813c0b320b3e9901481b9232025a5
Reviewed-on: https://chromium-review.googlesource.com/883101
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#50825}
2018-01-24 09:06:23 +00:00
Michael Achenbach
9d7c40ea76 [test] Fail when no tests are run
TBR=sergiyb@chromium.org

Bug: v8:7337
Change-Id: I1732f6e587305ce4ab41a65f73e943c7eb9e1d15
Reviewed-on: https://chromium-review.googlesource.com/877760
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#50747}
2018-01-22 09:57:53 +00:00
Michael Achenbach
06f0365165 [test] Run test-runner system tests also when tests change.
NOTRY=true

Bug: v8:6917
Change-Id: I08a09ba6f39fc3be7a11e520d2a59ced1354f892
Reviewed-on: https://chromium-review.googlesource.com/868291
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#50659}
2018-01-17 15:01:33 +00:00
Michal Majewski
96d06a6351 [test] Fix args parsing and name filter
Bug: v8:6917
Cq-Include-Trybots: luci.v8.try:v8_linux64_fyi_rel_ng;luci.v8.try:v8_linux_noi18n_rel_ng
Change-Id: I14c7e0a140bfe828ee8199606db36cab12be14ff
Reviewed-on: https://chromium-review.googlesource.com/868135
Commit-Queue: Michał Majewski <majeski@google.com>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#50649}
2018-01-17 10:32:12 +00:00
Michal Majewski
a6973d9fd2 [test] Add sharded system test for test processors
Bug: v8:6917
Change-Id: If9c097548a5879d693174c73e9952db7b25ef2e7
Cq-Include-Trybots: luci.v8.try:v8_linux64_fyi_rel_ng
Reviewed-on: https://chromium-review.googlesource.com/868437
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michał Majewski <majeski@google.com>
Cr-Commit-Position: refs/heads/master@{#50646}
2018-01-17 09:54:09 +00:00
Michal Majewski
00ac7641a6 [test] Drop starting message from indicators
Since test processors create tests dynamically we cannot simply
count how many tests will be run. Instead we count only base tests
that we've loaded, before creating variants.

Bug: v8:6917
Change-Id: Ibc5b9a73f6afad423572afa575f477ca661a99d5
Cq-Include-Trybots: luci.v8.try:v8_linux64_fyi_rel_ng
Reviewed-on: https://chromium-review.googlesource.com/868290
Commit-Queue: Michał Majewski <majeski@google.com>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#50642}
2018-01-17 09:31:19 +00:00
Michael Achenbach
6c72df9ed0 Reland "[test] Add test runner system tests for flaky tests"
This is a reland of 96f5579669
Original change's description:
> [test] Add test runner system tests for flaky tests
>
> This uses a temporary file in the mocked d8 script to let d8 flip
> between FAIL and PASS on rerun.
>
> This adds a separate test root with dedicated test suite for testing
> flakes to no interfere with existing status file configs.
>
> NOTRY=true
>
> Bug: v8:6917
> Change-Id: Id43753650195fb74cceb2a3ee9014100cabad546
> Reviewed-on: https://chromium-review.googlesource.com/867917
> Commit-Queue: Michael Achenbach <machenbach@chromium.org>
> Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#50621}

TBR=sergiyb@chromium.org
NOTRY=true

Bug: v8:6917
Change-Id: I4e7bca272dfd8778fbb8d012fcd6fd8406158e43
Reviewed-on: https://chromium-review.googlesource.com/868433
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#50625}
2018-01-16 15:01:56 +00:00
Michael Achenbach
bc130d803c Revert "[test] Add test runner system tests for flaky tests"
This reverts commit 96f5579669.

Reason for revert: breaks CI presubmit:
https://build.chromium.org/p/client.v8/builders/V8%20Linux%20-%20presubmit/builds/22115

Original change's description:
> [test] Add test runner system tests for flaky tests
> 
> This uses a temporary file in the mocked d8 script to let d8 flip
> between FAIL and PASS on rerun.
> 
> This adds a separate test root with dedicated test suite for testing
> flakes to no interfere with existing status file configs.
> 
> NOTRY=true
> 
> Bug: v8:6917
> Change-Id: Id43753650195fb74cceb2a3ee9014100cabad546
> Reviewed-on: https://chromium-review.googlesource.com/867917
> Commit-Queue: Michael Achenbach <machenbach@chromium.org>
> Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#50621}

TBR=machenbach@chromium.org,sergiyb@chromium.org,majeski@google.com

Change-Id: I103bf8f0e926a3f56921f14bdd7e875ce839ea71
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: v8:6917
Reviewed-on: https://chromium-review.googlesource.com/868530
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#50624}
2018-01-16 14:39:30 +00:00
Michael Achenbach
96f5579669 [test] Add test runner system tests for flaky tests
This uses a temporary file in the mocked d8 script to let d8 flip
between FAIL and PASS on rerun.

This adds a separate test root with dedicated test suite for testing
flakes to no interfere with existing status file configs.

NOTRY=true

Bug: v8:6917
Change-Id: Id43753650195fb74cceb2a3ee9014100cabad546
Reviewed-on: https://chromium-review.googlesource.com/867917
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#50621}
2018-01-16 14:25:39 +00:00
Michal Majewski
d53c4aa065 [test] Implement shard processor
Bug: v8:6917
Change-Id: I5b77e7445ca3a8eb5692659e94d3b8266479b415
Cq-Include-Trybots: luci.v8.try:v8_linux64_fyi_rel_ng
Reviewed-on: https://chromium-review.googlesource.com/866866
Commit-Queue: Michał Majewski <majeski@google.com>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#50615}
2018-01-16 11:33:03 +00:00
Michael Achenbach
56fa0a535b [test] Remove deprecated FAST_VARIANTS modifier
Bug: v8:7264
Cq-Include-Trybots: luci.v8.try:v8_linux_noi18n_rel_ng
Change-Id: I0c07c92955c0b6c145c7b72e5014f828eaf59c9a
Reviewed-on: https://chromium-review.googlesource.com/865903
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#50582}
2018-01-15 14:30:47 +00:00
Michal Majewski
bddfee9822 [test] Add special result for rerun and json indicator.
Bug: v8:6917
Change-Id: I5136f183bd1728a1ab90a9ebb2560d978e17ef28
Cq-Include-Trybots: luci.v8.try:v8_linux64_fyi_rel_ng
Reviewed-on: https://chromium-review.googlesource.com/863623
Commit-Queue: Michał Majewski <majeski@google.com>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#50576}
2018-01-15 12:54:23 +00:00
Michal Majewski
7f01c9303f [test] Implement variant generators as processors
Bug: v8:6917
Cq-Include-Trybots: luci.v8.try:v8_linux64_fyi_rel_ng;luci.v8.try:v8_linux_noi18n_rel_ng
Change-Id: Ib5bfdf4d6fee6102f62c7334a1b22146f1a1fc5b
Reviewed-on: https://chromium-review.googlesource.com/857376
Commit-Queue: Michał Majewski <majeski@google.com>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#50525}
2018-01-12 10:39:20 +00:00
Michal Majewski
463dbab3ec [test] Implemented status file filter as a processor
Bug: v8:6917
Change-Id: I4b10091a40372e1aa614ac26452e20ed481ab686
Cq-Include-Trybots: luci.v8.try:v8_linux64_fyi_rel_ng
Reviewed-on: https://chromium-review.googlesource.com/856498
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Commit-Queue: Michał Majewski <majeski@google.com>
Cr-Commit-Position: refs/heads/master@{#50482}
2018-01-10 16:08:46 +00:00
Michal Majewski
501413b9b9 [test] Implemented indicators as processors
Added simple system tests for different progress indicators.

Bug: v8:6917
Change-Id: I906ddfd06e82cc19d3b2210e09457456be00309b
Cq-Include-Trybots: luci.v8.try:v8_linux64_fyi_rel_ng
Reviewed-on: https://chromium-review.googlesource.com/852495
Commit-Queue: Michał Majewski <majeski@google.com>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#50406}
2018-01-08 13:08:40 +00:00
Michal Majewski
da3416f53a [test] Introduce test processors
Test processors can be enabled with --infra-staging flag.

Rerunning tests, execution and verbose progress indicator already
work as test processors.

Bug: v8:6917
Change-Id: I40fc42db94dbc8629e8000a3d363030045532fe3
Reviewed-on: https://chromium-review.googlesource.com/850398
Commit-Queue: Michał Majewski <majeski@google.com>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#50399}
2018-01-08 11:27:47 +00:00
Michael Achenbach
5a8e1f3f1b [test] Make coverage in test-runner tests work with multiprocessing
This replaces multiprocessing with threading in tests to make python
coverage work.

TBR=sergiyb@chromium.org

Bug: v8:6917
Change-Id: Idff763dfefa4a7fc782133d94089b3a5b00a194d
Reviewed-on: https://chromium-review.googlesource.com/844735
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#50344}
2018-01-03 17:37:05 +00:00
Michael Achenbach
13485a6991 [test] Add basic test-runner system tests
This tests some high-level scenarios of the V8 test runner, with
fake executable, test-suite extension and build configs.

The runners are slightly modified to be easier testable. Args
are passed from the tests now and the V8 root folder can be faked
by the tests.

We support coverage if python coverage 4.0 is installed. Otherwise
we run without it.

Bug: v8:6917
Change-Id: Ib149fd88027cbdc3382bcaea2d82020582f79d2d
Reviewed-on: https://chromium-review.googlesource.com/831506
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#50310}
2017-12-27 18:03:49 +00:00
Michael Achenbach
325a56cdd1 [test] Move predictable testing into a python wrapper
This moves the verify-predictable logic from the test runner into
a python wrapper script.

This revealed two more tests that don't print allocations, which are
now skipped.

Bug: v8:7166, v8:7177
Change-Id: Ie4a541cb2a20900414ffe1caf4b3fccc4a5edb52
Reviewed-on: https://chromium-review.googlesource.com/808971
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#49964}
2017-12-08 14:46:10 +00:00
Michal Majewski
68c5d01ad6 Reland "Reland "[test] Creating command before execution phase.""
This is a reland of 3b06511052
Original change's description:
> Reland "[test] Creating command before execution phase."
> 
> This is a reland of 98cc9e862f
> Original change's description:
> > [test] Creating command before execution phase.
> > 
> > Immutable command class with shell, flags and
> > environment.
> > 
> > Command creation moved from worker to the main
> > process. Because of that there is no need to send
> > test cases beyond process boundaries and load test
> > suites in worker processes.
> > 
> > Bug: v8:6917
> > Change-Id: Ib6a44278095b4f7141eb9b96802fe3e8117678a6
> > Reviewed-on: https://chromium-review.googlesource.com/791710
> > Commit-Queue: Michał Majewski <majeski@google.com>
> > Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> > Cr-Commit-Position: refs/heads/master@{#49746}
> 
> Bug: v8:6917
> Change-Id: I49c29a8db813c47909f2cc45070ac7721a447c7a
> Reviewed-on: https://chromium-review.googlesource.com/800370
> Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> Commit-Queue: Michał Majewski <majeski@google.com>
> Cr-Commit-Position: refs/heads/master@{#49756}

Bug: v8:6917
Change-Id: Ia39010a0a0f63537ad12490dfab17897d70d4930
Reviewed-on: https://chromium-review.googlesource.com/806034
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michał Majewski <majeski@google.com>
Cr-Commit-Position: refs/heads/master@{#49830}
2017-12-04 13:40:29 +00:00
Michael Achenbach
d3a8dec257 Revert "Reland "[test] Creating command before execution phase.""
This reverts commit 3b06511052.

Reason for revert: Broke fuzzers:
https://build.chromium.org/p/client.v8.clusterfuzz/builders/V8%20Deopt%20Fuzzer/builds/384

Original change's description:
> Reland "[test] Creating command before execution phase."
> 
> This is a reland of 98cc9e862f
> Original change's description:
> > [test] Creating command before execution phase.
> > 
> > Immutable command class with shell, flags and
> > environment.
> > 
> > Command creation moved from worker to the main
> > process. Because of that there is no need to send
> > test cases beyond process boundaries and load test
> > suites in worker processes.
> > 
> > Bug: v8:6917
> > Change-Id: Ib6a44278095b4f7141eb9b96802fe3e8117678a6
> > Reviewed-on: https://chromium-review.googlesource.com/791710
> > Commit-Queue: Michał Majewski <majeski@google.com>
> > Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> > Cr-Commit-Position: refs/heads/master@{#49746}
> 
> Bug: v8:6917
> Change-Id: I49c29a8db813c47909f2cc45070ac7721a447c7a
> Reviewed-on: https://chromium-review.googlesource.com/800370
> Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> Commit-Queue: Michał Majewski <majeski@google.com>
> Cr-Commit-Position: refs/heads/master@{#49756}

TBR=machenbach@chromium.org,sergiyb@chromium.org,majeski@google.com

# Not skipping CQ checks because original CL landed > 1 day ago.

Bug: v8:6917
Change-Id: I4938642c4396366be1e13daf6998c4b8538b688b
Reviewed-on: https://chromium-review.googlesource.com/804254
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#49805}
2017-12-01 20:44:50 +00:00
Michal Majewski
3b06511052 Reland "[test] Creating command before execution phase."
This is a reland of 98cc9e862f
Original change's description:
> [test] Creating command before execution phase.
> 
> Immutable command class with shell, flags and
> environment.
> 
> Command creation moved from worker to the main
> process. Because of that there is no need to send
> test cases beyond process boundaries and load test
> suites in worker processes.
> 
> Bug: v8:6917
> Change-Id: Ib6a44278095b4f7141eb9b96802fe3e8117678a6
> Reviewed-on: https://chromium-review.googlesource.com/791710
> Commit-Queue: Michał Majewski <majeski@google.com>
> Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#49746}

Bug: v8:6917
Change-Id: I49c29a8db813c47909f2cc45070ac7721a447c7a
Reviewed-on: https://chromium-review.googlesource.com/800370
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michał Majewski <majeski@google.com>
Cr-Commit-Position: refs/heads/master@{#49756}
2017-11-30 13:47:40 +00:00
Michael Achenbach
d8b369d2be Revert "[test] Creating command before execution phase."
This reverts commit 98cc9e862f.

Reason for revert: Breaks test isolation:
https://build.chromium.org/p/client.v8/builders/V8%20Linux%20-%20builder/builds/29746

Original change's description:
> [test] Creating command before execution phase.
> 
> Immutable command class with shell, flags and
> environment.
> 
> Command creation moved from worker to the main
> process. Because of that there is no need to send
> test cases beyond process boundaries and load test
> suites in worker processes.
> 
> Bug: v8:6917
> Change-Id: Ib6a44278095b4f7141eb9b96802fe3e8117678a6
> Reviewed-on: https://chromium-review.googlesource.com/791710
> Commit-Queue: Michał Majewski <majeski@google.com>
> Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#49746}

TBR=machenbach@chromium.org,sergiyb@chromium.org,majeski@google.com

Change-Id: I44b99468d18fd093833f4185dad067a9eeaf2bc1
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: v8:6917
Reviewed-on: https://chromium-review.googlesource.com/800292
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#49747}
2017-11-30 12:32:31 +00:00
Michal Majewski
98cc9e862f [test] Creating command before execution phase.
Immutable command class with shell, flags and
environment.

Command creation moved from worker to the main
process. Because of that there is no need to send
test cases beyond process boundaries and load test
suites in worker processes.

Bug: v8:6917
Change-Id: Ib6a44278095b4f7141eb9b96802fe3e8117678a6
Reviewed-on: https://chromium-review.googlesource.com/791710
Commit-Queue: Michał Majewski <majeski@google.com>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#49746}
2017-11-30 12:30:09 +00:00
Sergiy Byelozyorov
d5d6229be3 [tools] Refactor patch/no-patch terminology in run_perf.py
The runs are now called as primary (no suffix) and secondary. This is in
preparation to adding secondary builds on CI, which will run tests on the latest
released stable V8 binary (aka as ref builds).

R=machenbach@chromium.org

Bug: chromium:783763
Change-Id: Ie6560012887bd5bb0d948bc8d34a9256d922137c
Reviewed-on: https://chromium-review.googlesource.com/781941
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Sergiy Byelozyorov <sergiyb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#49560}
2017-11-22 07:19:50 +00:00
Michael Achenbach
b593f1a56e [test] Implement results processor for perf runner.
This adds the possibility to specify a python script for post-processing stdout.

This also adds some system tests for testing the new feature.

NOTRY=true

Change-Id: I0383afb3e23513629508feeb639ed2dfce56b54a
Reviewed-on: https://chromium-review.googlesource.com/443449
Reviewed-by: Clemens Hammacher <clemensh@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#43257}
2017-02-16 15:43:19 +00:00
Michael Achenbach
34663b428c [test] Fix unittests for perf runner.
This was omitted in:
https://codereview.chromium.org/2615623003

TBR=clemensh@chromium.org,mtrofin@chromium.org
NOTRY=true

Change-Id: I78449fe72e27976b95a9557e0bd8f986ed8caa64
Reviewed-on: https://chromium-review.googlesource.com/443526
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#43232}
2017-02-16 08:48:33 +00:00
machenbach
4a58b46e55 Revert of [tools] add --pretty switch to run_perf.py (patchset #8 id:140001 of https://codereview.chromium.org/1681283004/ )
Reason for revert:
[Sheriff] Still fails on android.

Original issue's description:
> reland [tools] add --pretty switch to run_perf.py
>
> This CL improves running our internal benchmarks locally by adding the
> --pretty option to tools/run_perf.py. With the flag enabled we print
> the run-time of each benchmark directly and avoid the json output at
> the end.
>
> NOTRY=true
>
> Committed: https://crrev.com/b543c40d78c3aa8283a7d9fccb6c9e26a9a81f90
> Cr-Commit-Position: refs/heads/master@{#34023}

TBR=cbruni@chromium.org
# Skipping CQ checks because original CL landed less than 1 days ago.
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true

Review URL: https://codereview.chromium.org/1700953002

Cr-Commit-Position: refs/heads/master@{#34033}
2016-02-16 12:56:43 +00:00
cbruni
b543c40d78 reland [tools] add --pretty switch to run_perf.py
This CL improves running our internal benchmarks locally by adding the
--pretty option to tools/run_perf.py. With the flag enabled we print
the run-time of each benchmark directly and avoid the json output at
the end.

NOTRY=true

Review URL: https://codereview.chromium.org/1681283004

Cr-Commit-Position: refs/heads/master@{#34023}
2016-02-16 09:51:54 +00:00
machenbach
2f8e33526b Revert of [tools] add --pretty switch to run_perf.py (patchset #7 id:120001 of https://codereview.chromium.org/1681283004/ )
Reason for revert:
[Sheriff] Breaks android runs. Will look later why.

Original issue's description:
> [tools] add --pretty switch to run_perf.py
>
> This CL improves running our internal benchmarks locally by adding the
> --pretty option to tools/run_perf.py. With the flag enabled we print
> the run-time of each benchmark directly and avoid the json output at
> the end.
>
> NOTRY=true
>
> Committed: https://crrev.com/83f69507ab1b9380b56758b747d4f3fabc849e49
> Cr-Commit-Position: refs/heads/master@{#33981}

TBR=cbruni@chromium.org
# Skipping CQ checks because original CL landed less than 1 days ago.
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true

Review URL: https://codereview.chromium.org/1696293002

Cr-Commit-Position: refs/heads/master@{#34017}
2016-02-16 08:19:32 +00:00
cbruni
83f69507ab [tools] add --pretty switch to run_perf.py
This CL improves running our internal benchmarks locally by adding the
--pretty option to tools/run_perf.py. With the flag enabled we print
the run-time of each benchmark directly and avoid the json output at
the end.

NOTRY=true

Review URL: https://codereview.chromium.org/1681283004

Cr-Commit-Position: refs/heads/master@{#33981}
2016-02-15 09:23:26 +00:00
machenbach
18d2c58cfd [test] More robust perf runner with profiler option.
NOTRY=true

Review URL: https://codereview.chromium.org/1341213002

Cr-Commit-Position: refs/heads/master@{#30731}
2015-09-15 08:16:10 +00:00
machenbach
e925e0b69f [test] Let perf runner interleave try executions.
This allows running two executables interleaved to reduce
flakiness of tryjobs. The executables must reside in
different out directories.

BUG=chromium:507213
LOG=n
NOTRY=true
TEST=python -m unittest run_perf_test

Review URL: https://codereview.chromium.org/1215273003

Cr-Commit-Position: refs/heads/master@{#29572}
2015-07-10 13:02:25 +00:00
machenbach
af1eb2258c Make perf test runner more robust.
Convert all results to floats and log errors on failing
conversions. Otherwise these results bubble up into
buildbot.

TBR=tandrii@chromium.org
BUG=chromium:374740
LOG=n
NOTRY=true

Review URL: https://codereview.chromium.org/811483008

Cr-Commit-Position: refs/heads/master@{#25913}
2014-12-21 10:25:32 +00:00
machenbach
6a3460129c Add Android platform to performance runner.
BUG=chromium:374740
LOG=n
TEST=python -m unittest run_perf_test
NOTRY=true

Review URL: https://codereview.chromium.org/768143002

Cr-Commit-Position: refs/heads/master@{#25614}
2014-12-02 14:24:00 +00:00
machenbach
7c92531255 Make resources config ready for android test runner.
This switches off the resources feature. It will be solely
used for configuring files to be copied to android devices
in a follow up CL.

TBR=svenpanne@chromium.org
NOTRY=true

Review URL: https://codereview.chromium.org/764013002

Cr-Commit-Position: refs/heads/master@{#25563}
2014-11-28 11:35:44 +00:00
machenbach
e42fda5ebb Add test flags feature to perf runner.
TBR=svenpanne@chromium.org
NOTRY=true

Review URL: https://codereview.chromium.org/722023006

Cr-Commit-Position: refs/heads/master@{#25559}
2014-11-28 10:34:35 +00:00
machenbach@chromium.org
bb117b4dde Make benchmark runner understand chromium perf output.
BUG=406405
LOG=n
TEST=python -m unittest run_benchmarks_test
R=bmeurer@chromium.org

Review URL: https://codereview.chromium.org/498163002

git-svn-id: https://v8.googlecode.com/svn/branches/bleeding_edge@24509 ce2b1a6d-e550-0410-aec6-3dcde31c8c00
2014-10-10 07:12:38 +00:00