tldr: This adds an on-demand comparison with x64 when a difference to
a non-x64 build is detected.
Normally foozzie compares the baseline build (just ignition), with
two secondary builds. One, the default, always uses the shipping
configuration, the second passes additional flags. Both can use a
different architecture than the baseline build as well.
Differences between ignition and turbofan are then often detected
independent of the architectures used, but reported several times
(for each compared architecture).
This makes the reporting more specific, by running another build on
demand that uses the baseline architecture, but otherwise the same
configuration that showed a difference. If it shows the difference as
well, the baseline architecture is used for the report.
As a result only pure architecture differences will be reported with
an architecture other than x64.
This also adds some minor refactorings to reduce the code complexity
when looping over comparisons.
For testing this, the fake-d8s are extended with different behavior
for different flags passed. We add two test cases for testing:
x64 vs. ia32 with difference in x64 and ia32
x64 vs. ia32 with difference only in ia32
Bug: chromium:1196633
No-Try: true
Test: tools/clusterfuzz/v8_foozzie_test.py
Change-Id: Ic470ae8f0b37fb1628b32e4fafc0c39377e16f8c
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2897099
Reviewed-by: Liviu Rau <liviurau@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#74612}
Fuzz tests could mess with some library methods used by stubs for
NaN-pattern problems in typed arrays. This change makes the stubs
more robust.
Bug: chromium:1197627
Change-Id: I84975f798d616fd5e82fd9ab84ad01fc35336a04
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2820968
Auto-Submit: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Maya Lekova <mslekova@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#73896}
Generic wasm wrappers are enabled by default now (since
https://crrev.com/c/2562241), so we can remove the flag from the
nooptimization variant. Instead, we should have a configuration which
tests --no-wasm-generic-wrapper to find bugs in the compiled wrappers
earlier.
Also add an entry for contradictory flags, and reformat that list to
respect the 80 columns limit.
R=machenbach@chromium.orgCC=ahaas@chromium.org
Bug: v8:10982
Change-Id: I780aaedbbf7fe761c39a41af1ff2db58c7447a76
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2773057
Reviewed-by: Andreas Haas <ahaas@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Clemens Backes <clemensb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#73560}
Robustify %CompileBaseline against fuzzing, and allowlist it in the
fuzzer.
Bug: v8:11420
Change-Id: I44947014c8c9362d80ea98636dbbaa5d07d6a177
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2739643
Auto-Submit: Leszek Swirski <leszeks@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#73325}
This is a reland of 4ad08c82f7
The reland organizes the different error types in separate functions
for separate call stacks. Error simulation is also guarded by
a minimum file size to prevent Clusterfuzz from getting stuck with
its bad-build check.
Original change's description:
> Enable simulating errors to test fuzzer reliability
>
> This adds a d8 flag --simulate-errors, which on shutdown will cause
> certain errors. This enables testing the reliability of sanitizers.
>
> This will cause a fatal error, a dcheck (if available) or a
> violation that can be detected with one of the following sanitizers:
> ASAN, UBSAN, MSAN, CFI.
>
> The same flag used in differential fuzzing will cause an error
> subsumed with the error state "fake_difference".
>
> Bug: chromium:1152412
> Change-Id: I4b36c6fe716797004d634263617d22ca67b05600
> Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2554999
> Commit-Queue: Michael Achenbach <machenbach@chromium.org>
> Reviewed-by: Clemens Backes <clemensb@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#71430}
Bug: chromium:1152412
Change-Id: I604258b4c1ebd215c26b1de6b2822663f857bf64
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2565125
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Clemens Backes <clemensb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#71538}
When one comparison run crashes, we cap the outputs to compare to the
shorter one. If one of those, however, contains ignored lines, the
comparison get's skewed.
This makes the main source of ignored lines more robust (the line
printed for unknown flags), by not printing it in the first place in
the context of differential fuzzing.
Bug: chromium:1153871
Change-Id: If2e534959779be14a686be5e43630cbf66e215a0
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2567692
Auto-Submit: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Clemens Backes <clemensb@chromium.org>
Commit-Queue: Clemens Backes <clemensb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#71522}
This reverts commit 4ad08c82f7.
Reason for revert: Need to retry with bugs in separate stack
frames.
Original change's description:
> Enable simulating errors to test fuzzer reliability
>
> This adds a d8 flag --simulate-errors, which on shutdown will cause
> certain errors. This enables testing the reliability of sanitizers.
>
> This will cause a fatal error, a dcheck (if available) or a
> violation that can be detected with one of the following sanitizers:
> ASAN, UBSAN, MSAN, CFI.
>
> The same flag used in differential fuzzing will cause an error
> subsumed with the error state "fake_difference".
>
> Bug: chromium:1152412
> Change-Id: I4b36c6fe716797004d634263617d22ca67b05600
> Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2554999
> Commit-Queue: Michael Achenbach <machenbach@chromium.org>
> Reviewed-by: Clemens Backes <clemensb@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#71430}
TBR=machenbach@chromium.org,clemensb@chromium.org
# Not skipping CQ checks because original CL landed > 1 day ago.
Bug: chromium:1152412
Change-Id: If20cdef372b0e7e92e7080687f446539a587a815
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2565120
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#71475}
The error simulation works and crashes during the smoke test are
detected. But now this masks real crashes (https://crbug.com/1153200),
therefore this change subsumes smoke-test crashes with
--simulate-errors under a separate crash state.
Now Clusterfuzz will detect the case "unexpected crash" as fixed, but
create a new bug for "simulated crash". Which we will keep open
forever.
No-Try: true
Bug: chromium:1152412
Change-Id: I511af167d33430d9b89692151cb8abaf3a90c715
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2563270
Reviewed-by: Clemens Backes <clemensb@chromium.org>
Commit-Queue: Clemens Backes <clemensb@chromium.org>
Auto-Submit: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#71452}
This passes the --simulate-errors flag for differential fuzzing with
a probability of 0.01%.
No-Try: true
Bug: chromium:1152412
Change-Id: I3146c0ea5706be8b5ec13ee8740e1d185e008075
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2562248
Reviewed-by: Clemens Backes <clemensb@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#71432}
This adds a d8 flag --simulate-errors, which on shutdown will cause
certain errors. This enables testing the reliability of sanitizers.
This will cause a fatal error, a dcheck (if available) or a
violation that can be detected with one of the following sanitizers:
ASAN, UBSAN, MSAN, CFI.
The same flag used in differential fuzzing will cause an error
subsumed with the error state "fake_difference".
Bug: chromium:1152412
Change-Id: I4b36c6fe716797004d634263617d22ca67b05600
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2554999
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Clemens Backes <clemensb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#71430}
Don't pass the correctness-fuzzing suppressions to normal fuzzing as
they turn stack overflows and invalid string length checks into
crashes.
This became first now a problem after the flag was passed in an mjsunit
test case.
No-Try: true
Bug: chromium:1151600,chromium:1151599
Change-Id: I5d29900a4b155762cae447fc102055eab1916309
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2551112
Auto-Submit: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Clemens Backes <clemensb@chromium.org>
Reviewed-by: Clemens Backes <clemensb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#71333}
For people not too familiar with cloud storage, the direct URL makes it
easier to download the zip archive.
R=machenbach@chromium.org
Bug: chromium:1142437, v8:10891
Change-Id: I6b29ebaac37489a73c4f75473d07e04e7200f2c3
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2543395
Commit-Queue: Clemens Backes <clemensb@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#71276}
It took me a while to figure out how to set up everything to run the
fuzzer locally. These improved error messages would have helped to find
errors earlier.
R=machenbach@chromium.org
Bug: chromium:1142437
Change-Id: I414de0a8448cd81fdbf999cc7c3c8cf9394354f6
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2543397
Commit-Queue: Clemens Backes <clemensb@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#71239}
This is a JavaScript fuzzer originally authored by Oliver Chang. It
is a mutation based fuzzer using Babel code transformations. For more
information see the included README.md.
The original code was altered:
- Add new V8 copyright headers.
- Make the test expectation generator aware of the headers.
- Fix file endings for presubmit checks.
- Fix `npm test` on fresh checkout with a new fake DB.
- Make test skipping work with new v8/tools location.
- OWNERS file.
- New title section in README.md.
No-Try: true
Bug: chromium:1109770
Change-Id: Ie71752c0a37491a50500c49060a3c526716ef933
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2320330
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Maya Lekova <mslekova@chromium.org>
Cr-Commit-Position: refs/heads/master@{#69164}
This is another case of the known .caller difference that's now added
to the mapping of known issues.
No-Try: true
Bug: chromium:1101870
Change-Id: I6cfca6887362564f625648ba34820cb92a77efb6
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2280087
Reviewed-by: Clemens Backes <clemensb@chromium.org>
Reviewed-by: Maya Lekova <mslekova@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#68674}
Mostly gc-related flags didn't flush out any bugs yet, but often reduce
test performance and lead to timeouts.
No-Try: true
Bug: chromium:1044942
Change-Id: I2a7b55f78bfa3d597de1a5674658829e0812d01a
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2273861
Reviewed-by: Maya Lekova <mslekova@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#68608}
Previously we ran baseline (e.g. ignition) and one random secondary
comparison configuration (e.g. turbofan) from the list of experiments.
But Clusterfuzz imposes limitations on the total amount of fuzz tests.
Therefore this change enables more throughput by always running the
default configuration (ignition_turbofan like V8 is shipped)
additionally to the baseline and the secondary configuration.
This, hence, doubles the number of comparisons we run, with less than
50% additional runtime, since the slow baseline configuration is only
run once.
The experiments table is updated accordingly. Explicit entries running
ignition_turbofan are removed (as it always runs now), instead some
of the other configurations are increased in their relative
percentage. We also get a few new configurations that didn't run
before (e.g. forcing the slow path on x86).
No-Try: true
Bug: chromium:1100114
Change-Id: I69b2a41d78c06e556b309743a2aace1053c22f91
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2270307
Reviewed-by: Liviu Rau <liviurau@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#68607}
This shares the code of running the comparisons on the fuzz test
and on the smoke tests. While this change retains behavior for running
the fuzz test, the smoke test has slight changes in favor of code
readability:
1. Smoke tests can now bail out on output suppressions (There are
none now. And if we'd ever add any we'd need to write them in a way,
such that they ignore the output of the smoke tests, which don't
have much output anyways).
2. Crashes in smoke tests are now a hard failure. This is a desired
feature anyways. It's unlikely that the smoke test crashes as then
nothing would work.
No-Try: true
Bug: chromium:1100114
Change-Id: Ice3c6d18b549727c486a70756a72038c8b2029ac
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2273125
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Liviu Rau <liviurau@chromium.org>
Cr-Commit-Position: refs/heads/master@{#68586}
This adds a class representing all values needed for an execution.
It simplifies the parameters passed to formatting functions and
lifts the restriction of a first/second config pair in the options.
This will allow to run more than two configs in a follow up.
No-Try: true
Bug: chromium:1100114
Change-Id: I568253b8ca3220870b8f5af615108140587680a8
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2270550
Reviewed-by: Tamer Tas <tmrts@chromium.org>
Reviewed-by: Liviu Rau <liviurau@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#68583}
This makes output and test-case suppressions independent of the used
comparison configs and architecture. Such fine-grained suppressions
were only needed during the inception of differential fuzzing, but
by now, most remaining suppressions are implemented in d8 behind
a flag.
This prepares for running with more than two comparison configs in a
follow up.
No-Try: true
Bug: chromium:1100114
Change-Id: I072769adb3ef7c6e6c43459aa23ac906f461b307
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2270095
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Liviu Rau <liviurau@chromium.org>
Reviewed-by: Tamer Tas <tmrts@chromium.org>
Cr-Commit-Position: refs/heads/master@{#68579}
It is obsolete to filter out error-message differences since the
time we pass --correctness-fuzzer-suppressions to d8, which already
stubs all messages:
https://cs.chromium.org/chromium/src/v8/src/execution/messages.cc?l=1031
No-Try: true
Bug: chromium:1100114
Change-Id: Iac42a8e2a32f9bae4034f79eaff429bf3ee41724
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2270024
Reviewed-by: Maya Lekova <mslekova@chromium.org>
Reviewed-by: Tamer Tas <tmrts@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#68577}
This simplifies the lengthy main method by extracting some code and
by replacing the scattered returns with exceptions.
We introduce two exceptions for early bail-out. This enables helper
methods on multiple layers. The early bail-out on time-out is
moved to the point where it is detected.
Previously on timeout and crash we also printed out the step number.
Clusterfuzz doesn't parse this, it was only for statistical purposes,
and the latest version of the experimental workbench only parses
crashes and timeouts, not the step in which they happened. Hence,
this CL removes those step numbers.
Except the change described in the last paragraph, this CL doesn't
intend to change behavior.
No-Try: true
Bug: chromium:1100114
Change-Id: Ie8c18f183e4fc538577f3eb49aaf6df1acd1e4e1
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2270547
Reviewed-by: Liviu Rau <liviurau@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#68576}
If we pass flags that make runs very slow, also the validity checks
might time out. Previously this wasn't checked and output was just
cut off.
This also tightens the timeout on validity checks as they are
expected to run very fast.
No-Try: true
Bug: chromium:1098646
Change-Id: Iea9a932be86e84040b72a2311aaa1d44100b3378
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2262915
Reviewed-by: Maya Lekova <mslekova@chromium.org>
Reviewed-by: Tamer Tas <tmrts@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#68510}
This fixes a bug when an encoded character appears in the difference
string. Python3 doesn't require any encoding.
TBR=tmrts@chromium.org
No-Try: true
Bug: chromium:1095964
Change-Id: I49c66b5b9c105ad64d3a7839d0eb5df97ff5f404
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2249660
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#68384}
This prepares using ochang_js_fuzzer with foozzie. The fuzzer uses
tests from CrashTests in the corpus. This leads to a loop when
used with differential fuzzing, as foozzie dedupes failures based
on the original file path. Foozzie finds a new failure for the
existing failure in CrashTests, for which clusterfuzz creates a new
crash test and so on.
This subsumes all failures from CrashTests under the same key.
Once such a failure is reported, a developer can add it to a
mapping in foozzie.py, after which the global key can be used
again by clusterfuzz to report another failure.
No-Try: true
Bug: chromium:1044942
Change-Id: I801a23faeb0c672d6ad64b4100c463f53e36cbc2
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2214837
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Clemens Backes <clemensb@chromium.org>
Reviewed-by: Maya Lekova <mslekova@chromium.org>
Cr-Commit-Position: refs/heads/master@{#68053}
This prevents bug flooding based on differences from calling
%GetOptimizationStatus in correctness tests. It is supposed to
be suppressed with --allow-natives-for-differential-fuzzing.
This ensures early bail-out in case the flag is forgotten at
some point. The v8_sanity_checks.js file is executed before
each correctness test case for this purpose.
NOTRY=true
Bug: chromium:1044942
Change-Id: I74a836a82562604b35e94e5e123a2a8bff939423
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2184294
Reviewed-by: Maya Lekova <mslekova@chromium.org>
Reviewed-by: Georg Neis <neis@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#67618}
A new realm doesn't contain the mocks and suppressions on the global
object for correctness fuzzing. We replace Realm.eval with eval to
keep exercising the code.
Bug: chromium:1071133
Change-Id: Iffe82d37bf08829fc5937c17c2089277403e71dc
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2153206
Reviewed-by: Ulan Degenbaev <ulan@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#67187}
Yet another corner case how non-deterministic timestamps slipped into
the tests.
Bug: chromium:1064900
Change-Id: I33e8b4c8141b3854b7eca5d7ad9b45b6f5130d9e
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2120584
Reviewed-by: Mathias Bynens <mathias@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#66881}
This adds tests for the mock logic used in differential fuzzing. The
tests uncovered a couple of issues in the mock files that are also
fixed.
This also does some minor code clean up in the mock code.
Bug: chromium:1044942
Change-Id: I5b67f70f8b104bb681548f742ab863395a88360f
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2043843
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Mathias Bynens <mathias@chromium.org>
Cr-Commit-Position: refs/heads/master@{#66304}
This reduces the pointer-compression experiment by 3% as it is in
production since a while.
This also uses turbofan when comparing across architectures as tests
run faster and likely find the same or more relevant bugs.
No-Try: true
Change-Id: I35ec97188a101a7cb43454903f7bf9765e57893c
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2059995
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Igor Sheludko <ishell@chromium.org>
Cr-Commit-Position: refs/heads/master@{#66290}
Pass --no-lazy-feedback-allocation in all second runs depending
on a probability. Also combine with --interrupt-budget=100.
This also allows adding several extra flags behind one probability.
The tests are improved to ensure valid flags and configs.
No-Try: true
Bug: v8:10215
Change-Id: I2766ef5044cd8c7096f6b76f39b60b568f550bde
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2059991
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Tamer Tas <tmrts@chromium.org>
Reviewed-by: Jakob Gruber <jgruber@chromium.org>
Reviewed-by: Mythri Alle <mythria@chromium.org>
Cr-Commit-Position: refs/heads/master@{#66289}
