This is a reland of 4ad08c82f7
The reland organizes the different error types in separate functions
for separate call stacks. Error simulation is also guarded by
a minimum file size to prevent Clusterfuzz from getting stuck with
its bad-build check.
Original change's description:
> Enable simulating errors to test fuzzer reliability
>
> This adds a d8 flag --simulate-errors, which on shutdown will cause
> certain errors. This enables testing the reliability of sanitizers.
>
> This will cause a fatal error, a dcheck (if available) or a
> violation that can be detected with one of the following sanitizers:
> ASAN, UBSAN, MSAN, CFI.
>
> The same flag used in differential fuzzing will cause an error
> subsumed with the error state "fake_difference".
>
> Bug: chromium:1152412
> Change-Id: I4b36c6fe716797004d634263617d22ca67b05600
> Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2554999
> Commit-Queue: Michael Achenbach <machenbach@chromium.org>
> Reviewed-by: Clemens Backes <clemensb@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#71430}
Bug: chromium:1152412
Change-Id: I604258b4c1ebd215c26b1de6b2822663f857bf64
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2565125
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Clemens Backes <clemensb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#71538}
When one comparison run crashes, we cap the outputs to compare to the
shorter one. If one of those, however, contains ignored lines, the
comparison get's skewed.
This makes the main source of ignored lines more robust (the line
printed for unknown flags), by not printing it in the first place in
the context of differential fuzzing.
Bug: chromium:1153871
Change-Id: If2e534959779be14a686be5e43630cbf66e215a0
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2567692
Auto-Submit: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Clemens Backes <clemensb@chromium.org>
Commit-Queue: Clemens Backes <clemensb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#71522}
This reverts commit 4ad08c82f7.
Reason for revert: Need to retry with bugs in separate stack
frames.
Original change's description:
> Enable simulating errors to test fuzzer reliability
>
> This adds a d8 flag --simulate-errors, which on shutdown will cause
> certain errors. This enables testing the reliability of sanitizers.
>
> This will cause a fatal error, a dcheck (if available) or a
> violation that can be detected with one of the following sanitizers:
> ASAN, UBSAN, MSAN, CFI.
>
> The same flag used in differential fuzzing will cause an error
> subsumed with the error state "fake_difference".
>
> Bug: chromium:1152412
> Change-Id: I4b36c6fe716797004d634263617d22ca67b05600
> Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2554999
> Commit-Queue: Michael Achenbach <machenbach@chromium.org>
> Reviewed-by: Clemens Backes <clemensb@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#71430}
TBR=machenbach@chromium.org,clemensb@chromium.org
# Not skipping CQ checks because original CL landed > 1 day ago.
Bug: chromium:1152412
Change-Id: If20cdef372b0e7e92e7080687f446539a587a815
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2565120
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#71475}
This adds a d8 flag --simulate-errors, which on shutdown will cause
certain errors. This enables testing the reliability of sanitizers.
This will cause a fatal error, a dcheck (if available) or a
violation that can be detected with one of the following sanitizers:
ASAN, UBSAN, MSAN, CFI.
The same flag used in differential fuzzing will cause an error
subsumed with the error state "fake_difference".
Bug: chromium:1152412
Change-Id: I4b36c6fe716797004d634263617d22ca67b05600
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2554999
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Clemens Backes <clemensb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#71430}
This makes output and test-case suppressions independent of the used
comparison configs and architecture. Such fine-grained suppressions
were only needed during the inception of differential fuzzing, but
by now, most remaining suppressions are implemented in d8 behind
a flag.
This prepares for running with more than two comparison configs in a
follow up.
No-Try: true
Bug: chromium:1100114
Change-Id: I072769adb3ef7c6e6c43459aa23ac906f461b307
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2270095
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Liviu Rau <liviurau@chromium.org>
Reviewed-by: Tamer Tas <tmrts@chromium.org>
Cr-Commit-Position: refs/heads/master@{#68579}
It is obsolete to filter out error-message differences since the
time we pass --correctness-fuzzer-suppressions to d8, which already
stubs all messages:
https://cs.chromium.org/chromium/src/v8/src/execution/messages.cc?l=1031
No-Try: true
Bug: chromium:1100114
Change-Id: Iac42a8e2a32f9bae4034f79eaff429bf3ee41724
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2270024
Reviewed-by: Maya Lekova <mslekova@chromium.org>
Reviewed-by: Tamer Tas <tmrts@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#68577}
We ensure now that fuzzers don't produce the f.arguments pattern and
instead replace it with a random variable.
No-Try: true
Bug: chromium:1044942, chromium:1020573
Change-Id: I899985f1a238f36d10fb22f76d93b7d5f444eab1
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2051944
Reviewed-by: Georg Neis <neis@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#66238}
Un-ignore some files that only failed in obsolete comparisons.
We don't compare eager anymore and console functions are mocked out
in d8 by now. The exponentiation operator bug doesn't repro anymore.
No-Try: true
Bug: chromium:1044942
Change-Id: I0a572836bc9a4a6aa4736447f638a522ff8e8168
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2050400
Reviewed-by: Georg Neis <neis@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#66233}
The assumtion that V8 has no output differences within a single line
before a stack overflow, didn't hold. The prefix of e.g. console.info
can lead to a difference in a recursive call.
This change makes foozzie's output capping before a crash work on the
level of characters instead of lines to fix this.
No-Try: true
Bug: chromium:1050942
Change-Id: I13f747caf4f5848d40c31bd4232811285bab3c17
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2049844
Reviewed-by: Liviu Rau <liviurau@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#66217}
This will allow uploading repro test cases to clusterfuzz for
already suppressed known issues. This will allow tracking if those
issues still reproduce and that suppressions don't become stale.
No-Try: true
Bug: chromium:1044942
Change-Id: I997f11293c51836b97d143b0fea992055b39955e
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2036083
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Liviu Rau <liviurau@chromium.org>
Reviewed-by: Tamer Tas <tmrts@chromium.org>
Cr-Commit-Position: refs/heads/master@{#66114}
Respective differences were fixed and the suppressions didn't fire
anymore according to the statistics.
No-Try: true
Bug: chromium:664068, chromium:667678
Change-Id: I5e1aa802e0b0288a73d0f236c154289ab45314a0
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2035878
Reviewed-by: Clemens Backes <clemensb@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#66100}
Crashes in the presence of RangeError happen often during differential
fuzzing. Until now we have ignored such cases completely.
After this change we compare as much output as possible when one or
both runs have crashed, dramatically increasing the coverage.
No-Try: true
Bug: chromium:1048099
Change-Id: I923c10e9064b5dc6cae1e39a254e221d2867e0e7
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2030914
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Tamer Tas <tmrts@chromium.org>
Cr-Commit-Position: refs/heads/master@{#66085}
This also ignores the line number for type errors.
NOTRY=true
Bug: chromium:995075
Change-Id: I9d9a91842383eba60db858c2c2e02df7dda1a9e1
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1778287
Reviewed-by: Tamer Tas <tmrts@chromium.org>
Reviewed-by: Michael Starzinger <mstarzinger@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#63494}
We'll soon also host other configurations for general fuzzing, not only
correctness fuzzing in the new tools/clusterfuzz folder.
TBR=yangguo@chromium.org
Bug: chromium:813833
Change-Id: Icd966bfec91cc547522bad5d1a842500b554754f
Reviewed-on: https://chromium-review.googlesource.com/930331
Reviewed-by: Yang Guo <yangguo@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#51480}
