This extends the ExpectedOutProc runner to print a diff between the
expected and actual output. The behavior of other runners is unchanged.
Change-Id: If2b89d39cf98b8d257b1a209b5471a79ec3868ff
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/3771641
Reviewed-by: Jakob Kummerow <jkummerow@chromium.org>
Reviewed-by: Liviu Rau <liviurau@chromium.org>
Commit-Queue: Clemens Backes <clemensb@chromium.org>
Auto-Submit: Matthias Liedtke <mliedtke@google.com>
Reviewed-by: Clemens Backes <clemensb@chromium.org>
Cr-Commit-Position: refs/heads/main@{#81966}
While writing a new test I realized that the test did not fail if
running into a CHECK or UNREACHABLE *after* printing the last expected
line.
That is because both stderr and the exit status are ignored. This CL
fixes that.
This will uncover a lot of memory leaks, which I plan to address in
follow-up CLs.
R=machenbach@chromium.org
CC=szuend@chromium.org
Bug: chromium:1142437, v8:11107
Cq-Include-Trybots: luci.v8.try:v8_linux64_asan_rel_ng
Cq-Include-Trybots: luci.v8.try:v8_mac64_asan_rel_ng
Cq-Include-Trybots: luci.v8.try:v8_win64_asan_rel_ng
Change-Id: I65f325abf102e063bb4f449353c47e94d84de652
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2519567
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Clemens Backes <clemensb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#71006}
Example can be inspector tests.
Bug: v8:10264
Change-Id: I996bb68d0f36920568a04f93cd8c1256a4f41a96
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2070912
Commit-Queue: Z Nguyen-Huu <duongn@microsoft.com>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#66572}
Timeouts on num-fuzzer only lead to false positives, as tests might
take unpredictably longer.
This CL forgives timeouts through a global override mechanism of the
expected outcomes. This allows to remove already existing scattered
code that allowed timeouts in some test suites only.
Bug: v8:6917
Change-Id: Ib131765d360e335789c1952bc6793ed051e016ea
Reviewed-on: https://chromium-review.googlesource.com/908454
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#51188}
A timeout result of randomly combined tests is not a useful indicator.
Therefore we ignore those now. This also prevents timeouts from being
re-run.
TBR=sergiyb@chromium.org
NOTRY=true
Bug: v8:6917
Change-Id: I8e5eb93e2f13067ca5270fb152c3dc1eda1bb3c2
Reviewed-on: https://chromium-review.googlesource.com/888559
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#50879}
Since we're not winning anything by changing the result between
processors on the main process, reduce is noop there and result is
immutable.
Bug: v8:6917
Change-Id: Ieb282e7abd4ab31162aee6b52493a6e1b6a25109
Cq-Include-Trybots: luci.v8.try:v8_linux64_fyi_rel_ng
Reviewed-on: https://chromium-review.googlesource.com/878239
Commit-Queue: Michał Majewski <majeski@google.com>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#50784}
I added additional exception logging in the execution.py since
errors in processing results were really difficult to debug.
There is a problem on Windows with class serialization when
it comes from dynamically loaded module. To fix it I moved all
output processors to the tools/testrunner/outproc/ and import
them in test/*/testcfg.py.
Bug: v8:6917
Change-Id: Ida604641d659b006e91faf1d56a37769ec47f5f3
Reviewed-on: https://chromium-review.googlesource.com/842784
Commit-Queue: Michał Majewski <majeski@google.com>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#50361}
