Commit Graph

98 Commits

Author SHA1 Message Date
Sergiy Belozorov
b3e2bcf95a [tools] Default to 4 retries for all perf tests
Overall, total test runtime that was wasted due to timeouts is 3420 seconds in
the last 2 weeks. Even with 4 retries, assuming all of them time out, needed
additional capacity is under 2 hours per week. Based on this analysis, I think
it's safe to land this CL.

Note that this is not intended as a long-term solution of the timeout problem,
but rather a temporary solution to prevent ongoing errors. Proper investigation
and correct long-term solution are still needed and tracked in the bug.

R=machenbach@chromium.org, tmrts@chromium.org

Bug: chromium:841700
Change-Id: Id16e6b784fa85bb9e28ed8c6b267b583636e2dc1
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1593342
Reviewed-by: Tamer Tas <tmrts@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Sergiy Belozorov <sergiyb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#61224}
2019-05-04 18:25:27 +00:00
Sergiy Belozorov
ceaeca9d2c [tools] Record durations also for unsuccesful runs
This is needed to allow deriving timeout and near_timeout runnables on the
recipe side and drop corresponding fields in the output.

We also remove some unused code that was removed in previous refactoring CLs.

R=tmrts@chromium.org

No-Try: true
No-Tree-Checks: true
Bug: chromium:841700
Change-Id: Ib7cec9baceff994fb9b32be09ba4a4079ebff2c8
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1588417
Commit-Queue: Sergiy Belozorov <sergiyb@chromium.org>
Reviewed-by: Tamer Tas <tmrts@chromium.org>
Cr-Commit-Position: refs/heads/master@{#61095}
2019-04-29 16:00:41 +00:00
Sergiy Belozorov
7a70c55dc0 [tools] Replace yield-based output processing with explicit method calls
This also makes processing immediate, i.e. outputs are parsed into results as
soon as test run is finished, which will allow us to implement logic that checks
whether we have enough runs based on already-accumulated results.

Since we process each output immediately, we do not need Measurement class any
longer and its ConsumeOutput is now integrated directly into TraceConfig.
Similarly AccumulateResults is replaced with RunnableConfig.ProcessOutput as we
do not accumulate results any longer.

R=machenbach@chromium.org

No-Try: true
No-Tree-Checks: true
Bug: chromium:880724
Change-Id: I0fc4846024c43258c10ba8d568312aa4746d746f
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1584325
Commit-Queue: Sergiy Belozorov <sergiyb@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#61048}
2019-04-26 13:33:51 +00:00
Sergiy Belozorov
631c4d4f6b [tools] Refactor Results class into ResultsTracker class based on dicts
This is part of the refactoring needed to implement a feature that allows
re-running benchmarks until needed confidence is reached.

R=machenbach@chromium.org

No-Try: true
No-Tree-Checks: true
Bug: chromium:880724
Change-Id: I45f584a3503ecf567f4c2661a302a74fc5e516af
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1581605
Commit-Queue: Sergiy Belozorov <sergiyb@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#61037}
2019-04-26 09:57:50 +00:00
Michael Achenbach
078d65d6c4 [test] Allow overriding run count in run_perf
This will allow running the integration tests with minimal runs.

NOTRY=true

Bug: chromium:775123
Change-Id: I3f5b975609c6af6f29fc5cf657d9aa5c2d0a7db6
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1583719
Reviewed-by: Sergiy Belozorov <sergiyb@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#61030}
2019-04-26 07:04:38 +00:00
Sergiy Belozorov
2caf6d0a5e [tools] Migrate to argparse module and remove few deprecated arguments
The --run-count-multiplier flag was used to run an experiment and is not needed
anymore. It will be superceeded by another parameter that will run benchmarks as
many times as needed until needed confidence is achieved.

R=machenbach@chromium.org

No-Try: true
No-Tree-Checks: true
Bug: chromium:880724
Change-Id: Iedd3d0f900a838f97cfa3339f582f6d02cce3e93
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1578504
Commit-Queue: Sergiy Belozorov <sergiyb@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#60980}
2019-04-24 12:58:48 +00:00
Sergiy Belozorov
f8374ee4ae [tools] Export runtime duration and timeout for each runnable
R=machenbach@chromium.org, tmrts@chromium.org

Bug: chromium:841700
Change-Id: I9852ccc573eda6b1bab7a7db295f7fd6e7410581
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1570010
Commit-Queue: Sergiy Belozorov <sergiyb@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#60967}
2019-04-24 09:25:37 +00:00
Sergiy Belozorov
b2942ae973 [tools] Remove RunnableGenericConfig and its corresponding tests
This is dead code as no benchmarks seem to be using it and removing this
additional logic will reduce maintenance cost for the perf infrastructure.

R=machenbach@chromium.org, tmrts@chromium.org

No-Try: true
No-Tree-Checks: true
Change-Id: I952af069d4d18d63624510b4c461fa1891703db9
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1570008
Commit-Queue: Sergiy Belozorov <sergiyb@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#60947}
2019-04-23 13:00:01 +00:00
Sergiy Belozorov
c8b5feec0f [tools] Remove TestFailedError and generate Output object for all cases
This is part of the refactoring to allow exporting more information about test
execution to the recipes and upload this information to ChromePerf.

This fixes running secondary even after primary run fails, which will allow us
to differentiate between test and infra failures as latter ones will also affect
refbuilds and re-runs without patch.

R=machenbach@chromium.org, tmrts@chromium.org

Bug: chromium:841700
Change-Id: I29ce49d2f8c5e73158f1d41a73c51f2b35929f36
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1570006
Commit-Queue: Sergiy Belozorov <sergiyb@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#60946}
2019-04-23 12:46:31 +00:00
Sergiy Belozorov
64d7300d8d [tools] Refactor code to use Output objects instead of stdout in most places
This is part of the refactoring to allow exporting more information about
test execution to the recipes and upload this information to ChromePerf.

R=machenbach@chromium.org,tmrts@chromium.org

No-Try: true
No-Tree-Checks: true
Bug: chromium:841700
Change-Id: Iab400e8922231d8eac91a6fa22ce8f45053f7ac6
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1569442
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Sergiy Belozorov <sergiyb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#60944}
2019-04-23 12:27:31 +00:00
Sergiy Belozorov
3295fb7350 [tools] Add timestamps to each log line
We've originally added them in https://crrev.com/c/1159361 and then removed
again in https://crrev.com/c/1291370 since they were implemented in LogDog
Viewer. However, looks like logs from the swarming tasks are all dumped into
step stdout at the same time and thus logdog assigns idential timestamp to all
lines making it fairly useless.

TBR=machenbach@chromium.org

No-Try: true
No-Tree-Checks: true
Bug: chromium:841700
Change-Id: I49bfb23f8007e4a0facdfea6a09cde911f8e5a51
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1569432
Reviewed-by: Sergiy Belozorov <sergiyb@chromium.org>
Commit-Queue: Sergiy Belozorov <sergiyb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#60875}
2019-04-16 15:09:40 +00:00
Sergiy Belozorov
168c186c4b [tools] Add a log line when retrying runnables
TBR=machenbach@chromium.org, tmrts@chromium.org

No-Try: true
No-Tree-Checks: true
Bug: chromium:872257
Change-Id: I12a61a7e09ab240a9ee0f3e4a06f1190583075c9
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1565894
Commit-Queue: Sergiy Belozorov <sergiyb@chromium.org>
Reviewed-by: Sergiy Belozorov <sergiyb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#60824}
2019-04-12 15:46:57 +00:00
Sergiy Belozorov
3478f053d4 [tools] Implement retry mechanism for failures and use it for some flaky tests
This is not fixing the root cause of the flakiness, but prevents us from loosing
data in the short to medium term as investigation proved to be difficult and
will likely take even more time.

R=machenbach@chromium.org, tmrts@chromium.org

Bug: chromium:872257
Change-Id: Id5fbd0a00058f8612089ee4d6a858193924bd868
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1564204
Commit-Queue: Sergiy Belozorov <sergiyb@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#60814}
2019-04-12 12:25:44 +00:00
Sergiy Belozorov
f7afe0087e [tools] Refactor run_perf.py and run_perf_test.py
- Remove all relative imports from mock and os
- Fix mocking in a few tests to prevent cross-test side-effects
- Add run_perf_test.py to v8_presubmit.py
- The vpython config was not added since root .vpython already includes
  coverage and mock libraries
- Convert all double-quoted strings to single-quoted (PS8->PS9)

R=sergiyb@chromium.org

Bug: chromium:123456
Change-Id: I7b3a08dc5d950b0f51cc7a5eb3a012ea953ca824
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1564206
Commit-Queue: Sergiy Belozorov <sergiyb@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#60810}
2019-04-12 11:34:54 +00:00
Sergiy Belozorov
11a2abee3d [tools] Report timeouts and near-timeouts to be displayed on builds
R=machenbach@chromium.org, tmrts@chromium.org

No-Try: true
Bug: chromium:841700
Change-Id: I18552e928223dd8ea03ac518f6b1b2460d10135d
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1559851
Commit-Queue: Sergiy Belozorov <sergiyb@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#60751}
2019-04-10 14:25:27 +00:00
Vadim Gorbachev (bmsdave)
7315d7b3d7 Preparing v8 to use with python3 /tools
There are now less that 400 days until the end of life
of Python 2(aka _legacy_ Python) https://pythonclock.org/ .
The code compatibility check for python2 and python3
used the following tools: futurize, flake8
You can see the reports here: https://travis-ci.com/bmsdave/v8/builds

This CL was uploaded by git cl split.

Bug: v8:8594
Change-Id: I661c52a70527e8ddde841fee6d4dcba282b4a938
Reviewed-on: https://chromium-review.googlesource.com/c/1470123
Commit-Queue: Sergiy Belozorov <sergiyb@chromium.org>
Reviewed-by: Sergiy Belozorov <sergiyb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#59675}
2019-02-19 09:12:07 +00:00
Sergiy Belozorov
f71dd204bf [tools] Add -d/--device options to all test launchers
This allows to select device when multiple devices are connected.

R=machenbach@chromium.org, tmrts@chromium.org

Bug: chromium:893593
Change-Id: I3dfd8b98251f613f5c93d29acd5035b236731ea6
Reviewed-on: https://chromium-review.googlesource.com/c/1452441
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Sergiy Belozorov <sergiyb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#59388}
2019-02-06 09:10:09 +00:00
Michael Achenbach
090687700a Revert "Reland "[tools] Push files using high-level device.PushChangedFiles method""
This reverts commit 6e03d7ee42.

Reason for revert: This breaks the Android bot:
https://ci.chromium.org/p/v8/builders/luci.v8.ci/V8%20Android%20Arm64%20-%20N5X/2933
It's quite hidden due to https://crbug.com/v8/8731 but all shards
time out. It looks like to to this change, testing takes
dramatically longer, maybe the pushing takes now much longer than
before. If we want decide for this, the builder needs to get
many more shards.

Original change's description:
> Reland "[tools] Push files using high-level device.PushChangedFiles method"
> 
> This is a reland of d045f66682
> 
> Original change's description:
> > [tools] Push files using high-level device.PushChangedFiles method
> >
> > R=machenbach@chromium.org
> >
> > No-Try: true
> > Bug: chromium:893593
> > Change-Id: I11cce7694eb7755ccee42c9a342fc1aa22663d85
> > Reviewed-on: https://chromium-review.googlesource.com/c/1382468
> > Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> > Commit-Queue: Sergiy Belozorov <sergiyb@chromium.org>
> > Cr-Commit-Position: refs/heads/master@{#58407}
> 
> Bug: chromium:893593
> Change-Id: I88a7143b3f31d87d266b89221f81efe831ea3823
> Reviewed-on: https://chromium-review.googlesource.com/c/1443055
> Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
> Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#59221}

TBR=machenbach@chromium.org,tandrii@chromium.org,sergiyb@chromium.org,bpastene@chromium.org,jbudorick@chromium.org

# Not skipping CQ checks because original CL landed > 1 day ago.

Bug: chromium:893593
Change-Id: Ifea307b5de8f39b660966fc6bef54601df91d841
Reviewed-on: https://chromium-review.googlesource.com/c/1450119
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Sergiy Belozorov <sergiyb@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#59305}
2019-02-01 20:42:11 +00:00
Sergiy Belozorov
6e03d7ee42 Reland "[tools] Push files using high-level device.PushChangedFiles method"
This is a reland of d045f66682

Original change's description:
> [tools] Push files using high-level device.PushChangedFiles method
>
> R=machenbach@chromium.org
>
> No-Try: true
> Bug: chromium:893593
> Change-Id: I11cce7694eb7755ccee42c9a342fc1aa22663d85
> Reviewed-on: https://chromium-review.googlesource.com/c/1382468
> Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> Commit-Queue: Sergiy Belozorov <sergiyb@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#58407}

Bug: chromium:893593
Change-Id: I88a7143b3f31d87d266b89221f81efe831ea3823
Reviewed-on: https://chromium-review.googlesource.com/c/1443055
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#59221}
2019-01-31 02:40:51 +00:00
Maya Lekova
6e5fb0041a Revert "[tools] Push files using high-level device.PushChangedFiles method"
This reverts commit d045f66682.

Reason for revert: Probably causing a timeout on Android bots - https://ci.chromium.org/p/v8/builders/luci.v8.ci/V8%20Android%20Arm64%20-%20N5X/2393

Original change's description:
> [tools] Push files using high-level device.PushChangedFiles method
> 
> R=​machenbach@chromium.org
> 
> No-Try: true
> Bug: chromium:893593
> Change-Id: I11cce7694eb7755ccee42c9a342fc1aa22663d85
> Reviewed-on: https://chromium-review.googlesource.com/c/1382468
> Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> Commit-Queue: Sergiy Belozorov <sergiyb@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#58407}

TBR=machenbach@chromium.org,sergiyb@chromium.org,bpastene@chromium.org,jbudorick@chromium.org

Change-Id: I23b0a5ef9fb6c1cca56e4b4bfe44fdfe4b80a390
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: chromium:893593
Reviewed-on: https://chromium-review.googlesource.com/c/1387489
Reviewed-by: Maya Lekova <mslekova@chromium.org>
Commit-Queue: Maya Lekova <mslekova@chromium.org>
Cr-Commit-Position: refs/heads/master@{#58422}
2018-12-21 10:13:55 +00:00
Sergiy Belozorov
d045f66682 [tools] Push files using high-level device.PushChangedFiles method
R=machenbach@chromium.org

No-Try: true
Bug: chromium:893593
Change-Id: I11cce7694eb7755ccee42c9a342fc1aa22663d85
Reviewed-on: https://chromium-review.googlesource.com/c/1382468
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Sergiy Belozorov <sergiyb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#58407}
2018-12-20 15:05:43 +00:00
Sergiy Byelozyorov
af120db4af [tools] Correctly identify and report test crashes and infra failures
We define a TestFailedError exception and raise it when we can reliably detect
that a test has crashed. All other exceptions are treated as infra failures and
are captured by the try-catch clause in MainWrapper function.

This also fixes all tests in run_perf_test.py, run_tests_test.py and makes sure
that both are run on any changes in tools directory.

R=machenbach@chromium.org

Bug: chromium:899028
Change-Id: I283bc87b31c814be476bebe9fdda414975494183
Reviewed-on: https://chromium-review.googlesource.com/c/1303293
Commit-Queue: Sergiy Byelozyorov <sergiyb@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#57134}
2018-10-30 15:05:40 +00:00
Sergiy Byelozyorov
4423c06151 [tools] Remove timestamp and print log levels at fixed width
Originally timestamps were added in crrev.com/c/1159361 to ease debugging. Since
then, however, timestamps were introduce to the LogDog viewer and are rendered
in a separate column, making it easier to scan text for important messages
starting with '>>>'. We do keep the log level, but make sure that it always
takes a fixed width such that the actual messages are aligned vertically.

R=machenbach@chromium.org

No-Try: true
Change-Id: I3a741069824aa25e674aca79da03a725017cb956
Reviewed-on: https://chromium-review.googlesource.com/c/1291370
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Sergiy Byelozyorov <sergiyb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#56834}
2018-10-20 07:53:34 +00:00
Sergiy Byelozyorov
94850f5cc2 [tools] Print stdout header even when the output is empty
This will allow us to determine whether tests are not printing anything or
whether our infrastructure is failing to retrieve the stdout.

TBR=machenbach@google.com

No-Try: true
Bug: chromium:891314
Change-Id: I0786e77b4db9f247e02a25924acd6d6773c0db9f
Reviewed-on: https://chromium-review.googlesource.com/c/1282962
Commit-Queue: Sergiy Byelozyorov <sergiyb@chromium.org>
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#56701}
2018-10-16 14:06:16 +00:00
Sergiy Byelozyorov
96fba5f37f Reland "[tools] Report infra failure on incorrect usage or uncaptured exceptions"
This is a reland of 2bac24fe2c

Original change's description:
> [tools] Report infra failure on incorrect usage or uncaptured exceptions
>
> TBR=machenbach@chromium.org
>
> No-Try: true
> Bug: chromium:893464
> Change-Id: If7d9f839a715468ded293a488e7fa12fc4ef3347
> Reviewed-on: https://chromium-review.googlesource.com/c/1270995
> Commit-Queue: Sergiy Byelozyorov <sergiyb@chromium.org>
> Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#56485}

TBR=machenbach@google.com

Test: ran run_perf_test.py locally (it reports same errors as before this CL)
No-Try: true
Bug: chromium:893464
Change-Id: I1bb2fae6fe2e2b7350695e491d7c3d52ff06db14
Reviewed-on: https://chromium-review.googlesource.com/c/1270965
Commit-Queue: Sergiy Byelozyorov <sergiyb@chromium.org>
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#56495}
2018-10-09 15:48:08 +00:00
Sergiy Byelozyorov
69876bccb5 Revert "[tools] Report infra failure on incorrect usage or uncaptured exceptions"
This reverts commit 2bac24fe2c.

Reason for revert: broke perf tests

Original change's description:
> [tools] Report infra failure on incorrect usage or uncaptured exceptions
> 
> TBR=machenbach@chromium.org
> 
> No-Try: true
> Bug: chromium:893464
> Change-Id: If7d9f839a715468ded293a488e7fa12fc4ef3347
> Reviewed-on: https://chromium-review.googlesource.com/c/1270995
> Commit-Queue: Sergiy Byelozyorov <sergiyb@chromium.org>
> Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#56485}

TBR=machenbach@chromium.org,sergiyb@chromium.org

Change-Id: I0f8f9d9d4053669b285c51703fad5223b3d831c3
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: chromium:893464
Reviewed-on: https://chromium-review.googlesource.com/c/1270964
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Commit-Queue: Sergiy Byelozyorov <sergiyb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#56492}
2018-10-09 15:29:40 +00:00
Sergiy Byelozyorov
2bac24fe2c [tools] Report infra failure on incorrect usage or uncaptured exceptions
TBR=machenbach@chromium.org

No-Try: true
Bug: chromium:893464
Change-Id: If7d9f839a715468ded293a488e7fa12fc4ef3347
Reviewed-on: https://chromium-review.googlesource.com/c/1270995
Commit-Queue: Sergiy Byelozyorov <sergiyb@chromium.org>
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#56485}
2018-10-09 14:06:32 +00:00
Sergiy Byelozyorov
1a236fb899 [tools] Print command output on timeout
R=machenbach@chromium.org

Bug: chromium:872257
Change-Id: I6eb1f70e0d44c76e18eb4f6f936c7f2cf0380444
Reviewed-on: https://chromium-review.googlesource.com/1217942
Commit-Queue: Sergiy Byelozyorov <sergiyb@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#55834}
2018-09-12 15:38:39 +00:00
Sergiy Byelozyorov
0e0157a42c [tools] Upload adb logcat to logdog when running tests
R=machenbach@chromium.org

No-Try: true
Bug: chromium:872257
Change-Id: I4de1a9dfccbb0d123c1c42c31a1f697e628623fa
Reviewed-on: https://chromium-review.googlesource.com/1171224
Commit-Queue: Sergiy Byelozyorov <sergiyb@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#55067}
2018-08-11 09:08:14 +00:00
Michael Achenbach
b2537f219d Reland "[test] Add logic to run tests on Android"
This is a reland of 4c0943424c

Original change's description:
> [test] Add logic to run tests on Android
> 
> This adds a new command abstraction for running commands on Android
> using dockered devices on swarming.
> 
> The new abstraction handles pushing all required files to the device.
> The logic used for pushing and running is reused from the perf runner.
> 
> This adds only the mjsunit test suite. Others will be handled in
> follow up CLs. The suite logic is enhanced with auto-detection of files
> to be pushed to devices, for e.g. load or import statements.
> 
> Some test cases need an extra resource section for specifying required
> files.
> 
> Remaining failing tests are marked in the status files for later
> triage.
> 
> Bug: chromium:866862
> Change-Id: I2b957559f07fdcd8c1bd2f7034f5ba7754a31fb7
> Reviewed-on: https://chromium-review.googlesource.com/1150153
> Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
> Commit-Queue: Michael Achenbach <machenbach@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#55041}

Bug: chromium:866862
Change-Id: Icf7e04c75d4abeab7254d10ba21240e46b0022ae
Reviewed-on: https://chromium-review.googlesource.com/1170643
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#55059}
2018-08-10 17:56:12 +00:00
Sergiy Byelozyorov
01c289aa59 [tools] Allow to increase number of times a perf test is retried
R=machenbach@chromium.org

No-Try: true
Bug: chromium:775123
Change-Id: I8883be60cc4860fad572ba644bbc722972551eb3
Reviewed-on: https://chromium-review.googlesource.com/1168483
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Sergiy Byelozyorov <sergiyb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#54999}
2018-08-09 07:49:01 +00:00
Michael Achenbach
e51782d33b [test] Log output on error in perf runner
NOTRY=true
TBR=sergiyb@chromium.org

Bug: chromium:866862
Change-Id: I52737a644c36f78b1803a443b89600bb7a114082
Reviewed-on: https://chromium-review.googlesource.com/1168485
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#54997}
2018-08-09 06:57:49 +00:00
Michael Achenbach
de80e94c9a [test] Refactoring - extract Android driver logic from perf runner
This prepares for reusing the driver logic for correctness testing.

NOTRY=true

Bug: chromium:866862
Change-Id: If901ca8552cf48f29e240ed5b52ea7e722e0e1e4
Reviewed-on: https://chromium-review.googlesource.com/1163608
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#54961}
2018-08-08 09:12:18 +00:00
Sergiy Byelozyorov
8dc41a581c [tools] Refactor run_perf.py to use logging module for all status messages
All status messages will be printed into STDERR and only the results will
continue to be written to STDOUT as documentation suggests. Additionally the
following changes are made:
 - on Buildbot each log entry will have a timestamp to make it easy to measure
   time between status messages printed by test runner, users will continue to
   see just the message itself
 - when devil (or any other module) logs error, they will not be interleaved
   with our own output since logging module is thread-safe
 - critical logs replaced with warning since they do not immediately stop
   the test runner
 - logging.exception is used for all exceptions to get formatted stack trace

TBR=machenbach@chromium.org

No-Try: true
Bug: chromium:838864
Change-Id: I197fcc3bc5890130ce8b35c3e21237f9614fc468
Reviewed-on: https://chromium-review.googlesource.com/1159361
Commit-Queue: Sergiy Byelozyorov <sergiyb@chromium.org>
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#54851}
2018-08-01 23:12:15 +00:00
Sergiy Byelozyorov
6cd560b265 Revert "[tools] Add an option to checkout v8-perf repo before running perf tests"
This reverts commit ba02337157.

Reason for revert: decided to use CIPD package instead, see reasoning here: http://shortn/_MgZHcjC60Q.

Original change's description:
> [tools] Add an option to checkout v8-perf repo before running perf tests
>
> R=​machenbach@chromium.org
>
> No-Try: true
> Bug: chromium:838864
> Change-Id: I02e38778ec4c33a20f4b2e93fb22615baa316883
> Reviewed-on: https://chromium-review.googlesource.com/1151295
> Commit-Queue: Sergiy Byelozyorov <sergiyb@chromium.org>
> Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#54723}

TBR=machenbach@chromium.org,sergiyb@chromium.org

# Not skipping CQ checks because original CL landed > 1 day ago.

No-Try: true
No-Tree-Checks: true
Bug: chromium:838864
Change-Id: I65a5b075c6fede61b59528ba92f6803011f187c7
Reviewed-on: https://chromium-review.googlesource.com/1153687
Commit-Queue: Sergiy Byelozyorov <sergiyb@chromium.org>
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#54759}
2018-07-27 21:28:58 +00:00
Sergiy Byelozyorov
ba02337157 [tools] Add an option to checkout v8-perf repo before running perf tests
R=machenbach@chromium.org

No-Try: true
Bug: chromium:838864
Change-Id: I02e38778ec4c33a20f4b2e93fb22615baa316883
Reviewed-on: https://chromium-review.googlesource.com/1151295
Commit-Queue: Sergiy Byelozyorov <sergiyb@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#54723}
2018-07-26 13:13:56 +00:00
Sergiy Byelozyorov
a82ea31afb [tools] Detect location of android build tools automatically
R=machenbach@chromium.org

No-Try: true
Bug: chromium:838864
Change-Id: Ifbdd9fe986e84197fd1ff6441c1565fed44c2c6f
Reviewed-on: https://chromium-review.googlesource.com/1149869
Commit-Queue: Sergiy Byelozyorov <sergiyb@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#54689}
2018-07-25 11:23:55 +00:00
Sergiy Byelozyorov
9fc3551a7b Reland "[tools] Add benchmark owners to the config"
This is a reland of 989285b7a0

Original change's description:
> [tools] Add benchmark owners to the config
>
> R=machenbach@chromium.org
>
> No-Try: true
> Bug: chromium:826280
> Change-Id: Ic34d13170dfecdd9e791974a34c33ba0248c7a38
> Reviewed-on: https://chromium-review.googlesource.com/1053809
> Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> Commit-Queue: Sergiy Byelozyorov <sergiyb@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#53138}

TBR=machenbach@chromium.org

No-Try: true
Bug: chromium:826280
Change-Id: I169788ff40dd88a7017b46a15b292568b907f38e
Reviewed-on: https://chromium-review.googlesource.com/1061697
Commit-Queue: Sergiy Byelozyorov <sergiyb@chromium.org>
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#53215}
2018-05-16 13:53:49 +00:00
Sergiy Byelozyorov
4b6263be32 Reland "[tools] Add benchmark owners to the config"
This is a reland of 989285b7a0

Original change's description:
> [tools] Add benchmark owners to the config
>
> R=machenbach@chromium.org
>
> No-Try: true
> Bug: chromium:826280
> Change-Id: Ic34d13170dfecdd9e791974a34c33ba0248c7a38
> Reviewed-on: https://chromium-review.googlesource.com/1053809
> Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> Commit-Queue: Sergiy Byelozyorov <sergiyb@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#53138}

R=machenbach@chromium.org

No-Try: true
Bug: chromium:826280
Change-Id: Ie61892c80d5ce004703d3e769abd58c4f2af1eec
Reviewed-on: https://chromium-review.googlesource.com/1057092
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Sergiy Byelozyorov <sergiyb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#53212}
2018-05-16 13:39:34 +00:00
Sergiy Byelozyorov
0a71347353 Revert "[tools] Add benchmark owners to the config"
This reverts commit 989285b7a0.

Reason for revert: broke internal bots

Original change's description:
> [tools] Add benchmark owners to the config
> 
> R=​machenbach@chromium.org
> 
> No-Try: true
> Bug: chromium:826280
> Change-Id: Ic34d13170dfecdd9e791974a34c33ba0248c7a38
> Reviewed-on: https://chromium-review.googlesource.com/1053809
> Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> Commit-Queue: Sergiy Byelozyorov <sergiyb@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#53138}

TBR=machenbach@chromium.org,sergiyb@chromium.org

Change-Id: Iec3f8fa8eda77b1bcfb00274b28a12e4d233d6c4
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: chromium:826280
Reviewed-on: https://chromium-review.googlesource.com/1057091
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Commit-Queue: Sergiy Byelozyorov <sergiyb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#53140}
2018-05-14 09:52:33 +00:00
Sergiy Byelozyorov
989285b7a0 [tools] Add benchmark owners to the config
R=machenbach@chromium.org

No-Try: true
Bug: chromium:826280
Change-Id: Ic34d13170dfecdd9e791974a34c33ba0248c7a38
Reviewed-on: https://chromium-review.googlesource.com/1053809
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Sergiy Byelozyorov <sergiyb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#53138}
2018-05-14 09:27:32 +00:00
Michael Achenbach
f0940a6390 Reland "[build] Make separate snapshot for trusted variant"
This is a reland of 40d66d8bf8

The fix disambiguates duplicate symbols in the generated embedded
builtins file.

Original change's description:
> [build] Make separate snapshot for trusted variant
>
> This enables side-by-side snapshots with and without untrusted-code
> mitigations. It'll be the default in all V8 stand-alone builds
> with external startup data. Internal snapshots are not supported.
>
> The files snapshot_blob.bin and snapshot_blob_trusted.bin will be
> bundled with V8 on swarming and the correct file is loaded dependent
> on the --untrusted-code-mitigations runtime flag.
>
> Likewise we embed two snapshots for builtins.
>
> Side-by-side snapshots won't be supported in Chromium.
>
> Bug: v8:7441
> Change-Id: I2949ddfd5773649946b1c8e74751d48ad1d9c524
> Reviewed-on: https://chromium-review.googlesource.com/960004
> Commit-Queue: Michael Achenbach <machenbach@chromium.org>
> Reviewed-by: Jakob Gruber <jgruber@chromium.org>
> Reviewed-by: Yang Guo <yangguo@chromium.org>
> Reviewed-by: Ross McIlroy <rmcilroy@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#52028}

Bug: v8:7441
Change-Id: I626171d4e07389f0453b4d0a698e2772fd37e8c5
Reviewed-on: https://chromium-review.googlesource.com/968623
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Jakob Gruber <jgruber@chromium.org>
Cr-Commit-Position: refs/heads/master@{#52055}
2018-03-20 10:59:24 +00:00
Michael Achenbach
7e7c04b677 Revert "[build] Make separate snapshot for trusted variant"
This reverts commit 40d66d8bf8.

Reason for revert:
https://build.chromium.org/p/client.v8/builders/V8%20Mac64/builds/20910

Original change's description:
> [build] Make separate snapshot for trusted variant
> 
> This enables side-by-side snapshots with and without untrusted-code
> mitigations. It'll be the default in all V8 stand-alone builds
> with external startup data. Internal snapshots are not supported.
> 
> The files snapshot_blob.bin and snapshot_blob_trusted.bin will be
> bundled with V8 on swarming and the correct file is loaded dependent
> on the --untrusted-code-mitigations runtime flag.
> 
> Likewise we embed two snapshots for builtins.
> 
> Side-by-side snapshots won't be supported in Chromium.
> 
> Bug: v8:7441
> Change-Id: I2949ddfd5773649946b1c8e74751d48ad1d9c524
> Reviewed-on: https://chromium-review.googlesource.com/960004
> Commit-Queue: Michael Achenbach <machenbach@chromium.org>
> Reviewed-by: Jakob Gruber <jgruber@chromium.org>
> Reviewed-by: Yang Guo <yangguo@chromium.org>
> Reviewed-by: Ross McIlroy <rmcilroy@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#52028}

TBR=rmcilroy@chromium.org,machenbach@chromium.org,yangguo@chromium.org,mstarzinger@chromium.org,jgruber@chromium.org

Change-Id: I5b18f7aff7c05a17842fc68d785bb617eeca4a41
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: v8:7441
Reviewed-on: https://chromium-review.googlesource.com/968622
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#52029}
2018-03-19 13:56:35 +00:00
Michael Achenbach
40d66d8bf8 [build] Make separate snapshot for trusted variant
This enables side-by-side snapshots with and without untrusted-code
mitigations. It'll be the default in all V8 stand-alone builds
with external startup data. Internal snapshots are not supported.

The files snapshot_blob.bin and snapshot_blob_trusted.bin will be
bundled with V8 on swarming and the correct file is loaded dependent
on the --untrusted-code-mitigations runtime flag.

Likewise we embed two snapshots for builtins.

Side-by-side snapshots won't be supported in Chromium.

Bug: v8:7441
Change-Id: I2949ddfd5773649946b1c8e74751d48ad1d9c524
Reviewed-on: https://chromium-review.googlesource.com/960004
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Jakob Gruber <jgruber@chromium.org>
Reviewed-by: Yang Guo <yangguo@chromium.org>
Reviewed-by: Ross McIlroy <rmcilroy@chromium.org>
Cr-Commit-Position: refs/heads/master@{#52028}
2018-03-19 13:34:54 +00:00
Dan Elphick
544c970534 [js-perf-tests] Create tests for Array.from
Modify "run_perf.py --filter Array/" so that it will let match "Array" but
not "ArrayLiteralSpread". Previously --filter Array/ would match nothing.

Change-Id: I1f9c677e3558fc2256a71306af5fc1a8394ac698
Reviewed-on: https://chromium-review.googlesource.com/895453
Commit-Queue: Dan Elphick <delphick@chromium.org>
Reviewed-by: Ross McIlroy <rmcilroy@chromium.org>
Cr-Commit-Position: refs/heads/master@{#51098}
2018-02-05 16:23:19 +00:00
Michal Majewski
68c5d01ad6 Reland "Reland "[test] Creating command before execution phase.""
This is a reland of 3b06511052
Original change's description:
> Reland "[test] Creating command before execution phase."
> 
> This is a reland of 98cc9e862f
> Original change's description:
> > [test] Creating command before execution phase.
> > 
> > Immutable command class with shell, flags and
> > environment.
> > 
> > Command creation moved from worker to the main
> > process. Because of that there is no need to send
> > test cases beyond process boundaries and load test
> > suites in worker processes.
> > 
> > Bug: v8:6917
> > Change-Id: Ib6a44278095b4f7141eb9b96802fe3e8117678a6
> > Reviewed-on: https://chromium-review.googlesource.com/791710
> > Commit-Queue: Michał Majewski <majeski@google.com>
> > Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> > Cr-Commit-Position: refs/heads/master@{#49746}
> 
> Bug: v8:6917
> Change-Id: I49c29a8db813c47909f2cc45070ac7721a447c7a
> Reviewed-on: https://chromium-review.googlesource.com/800370
> Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> Commit-Queue: Michał Majewski <majeski@google.com>
> Cr-Commit-Position: refs/heads/master@{#49756}

Bug: v8:6917
Change-Id: Ia39010a0a0f63537ad12490dfab17897d70d4930
Reviewed-on: https://chromium-review.googlesource.com/806034
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michał Majewski <majeski@google.com>
Cr-Commit-Position: refs/heads/master@{#49830}
2017-12-04 13:40:29 +00:00
Michael Achenbach
d3a8dec257 Revert "Reland "[test] Creating command before execution phase.""
This reverts commit 3b06511052.

Reason for revert: Broke fuzzers:
https://build.chromium.org/p/client.v8.clusterfuzz/builders/V8%20Deopt%20Fuzzer/builds/384

Original change's description:
> Reland "[test] Creating command before execution phase."
> 
> This is a reland of 98cc9e862f
> Original change's description:
> > [test] Creating command before execution phase.
> > 
> > Immutable command class with shell, flags and
> > environment.
> > 
> > Command creation moved from worker to the main
> > process. Because of that there is no need to send
> > test cases beyond process boundaries and load test
> > suites in worker processes.
> > 
> > Bug: v8:6917
> > Change-Id: Ib6a44278095b4f7141eb9b96802fe3e8117678a6
> > Reviewed-on: https://chromium-review.googlesource.com/791710
> > Commit-Queue: Michał Majewski <majeski@google.com>
> > Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> > Cr-Commit-Position: refs/heads/master@{#49746}
> 
> Bug: v8:6917
> Change-Id: I49c29a8db813c47909f2cc45070ac7721a447c7a
> Reviewed-on: https://chromium-review.googlesource.com/800370
> Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> Commit-Queue: Michał Majewski <majeski@google.com>
> Cr-Commit-Position: refs/heads/master@{#49756}

TBR=machenbach@chromium.org,sergiyb@chromium.org,majeski@google.com

# Not skipping CQ checks because original CL landed > 1 day ago.

Bug: v8:6917
Change-Id: I4938642c4396366be1e13daf6998c4b8538b688b
Reviewed-on: https://chromium-review.googlesource.com/804254
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#49805}
2017-12-01 20:44:50 +00:00
Michal Majewski
3b06511052 Reland "[test] Creating command before execution phase."
This is a reland of 98cc9e862f
Original change's description:
> [test] Creating command before execution phase.
> 
> Immutable command class with shell, flags and
> environment.
> 
> Command creation moved from worker to the main
> process. Because of that there is no need to send
> test cases beyond process boundaries and load test
> suites in worker processes.
> 
> Bug: v8:6917
> Change-Id: Ib6a44278095b4f7141eb9b96802fe3e8117678a6
> Reviewed-on: https://chromium-review.googlesource.com/791710
> Commit-Queue: Michał Majewski <majeski@google.com>
> Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#49746}

Bug: v8:6917
Change-Id: I49c29a8db813c47909f2cc45070ac7721a447c7a
Reviewed-on: https://chromium-review.googlesource.com/800370
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michał Majewski <majeski@google.com>
Cr-Commit-Position: refs/heads/master@{#49756}
2017-11-30 13:47:40 +00:00
Michael Achenbach
d8b369d2be Revert "[test] Creating command before execution phase."
This reverts commit 98cc9e862f.

Reason for revert: Breaks test isolation:
https://build.chromium.org/p/client.v8/builders/V8%20Linux%20-%20builder/builds/29746

Original change's description:
> [test] Creating command before execution phase.
> 
> Immutable command class with shell, flags and
> environment.
> 
> Command creation moved from worker to the main
> process. Because of that there is no need to send
> test cases beyond process boundaries and load test
> suites in worker processes.
> 
> Bug: v8:6917
> Change-Id: Ib6a44278095b4f7141eb9b96802fe3e8117678a6
> Reviewed-on: https://chromium-review.googlesource.com/791710
> Commit-Queue: Michał Majewski <majeski@google.com>
> Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#49746}

TBR=machenbach@chromium.org,sergiyb@chromium.org,majeski@google.com

Change-Id: I44b99468d18fd093833f4185dad067a9eeaf2bc1
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: v8:6917
Reviewed-on: https://chromium-review.googlesource.com/800292
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#49747}
2017-11-30 12:32:31 +00:00
Michal Majewski
98cc9e862f [test] Creating command before execution phase.
Immutable command class with shell, flags and
environment.

Command creation moved from worker to the main
process. Because of that there is no need to send
test cases beyond process boundaries and load test
suites in worker processes.

Bug: v8:6917
Change-Id: Ib6a44278095b4f7141eb9b96802fe3e8117678a6
Reviewed-on: https://chromium-review.googlesource.com/791710
Commit-Queue: Michał Majewski <majeski@google.com>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#49746}
2017-11-30 12:30:09 +00:00