With streaming compilation we delay the generation of errors until after
all bytes are received, so that potentially better error messages get
generated. With this CL we also delay the generation of errors in the
combination of lazy compilation and streaming compilation.
In particular, this CL does the following:
* It avoids the creation of a `DecodeFail` task in
`FinishAsyncCompileJobWithError`, which would create an error immediately before a potential name section arrived.
* It calls `CompilationStateImpl::SetError()` so that an error is
created once the stream finishes.
* It removes the return value of `ProcessFunctionBody` so that wire
bytes continue to be received even after a validation error.
* It adds an early exit to `ProcessFunctionBody` if
`CompilationStateImpl::failed()` is true, so that we don't continue
validation after the first detected error.
R=clemensb@chromium.org
Bug: v8:12852
Change-Id: Ie8c6be243a257ef62cbb29fea6b8e0c205060680
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/3802691
Reviewed-by: Clemens Backes <clemensb@chromium.org>
Commit-Queue: Andreas Haas <ahaas@chromium.org>
Cr-Commit-Position: refs/heads/main@{#82181}
This adds a bunch of tracing hooks to the module decoder and uses
them to support "annotated hexdump" output for full modules in wami:
$ out/x64.release/wami my_module.wasm --full-hexdump
Change-Id: I5821d940b5ec236df9708eecd0124172d8893ffd
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/3754741
Reviewed-by: Manos Koukoutos <manoskouk@chromium.org>
Commit-Queue: Jakob Kummerow <jkummerow@chromium.org>
Cr-Commit-Position: refs/heads/main@{#81791}
Add a missing check to reject illegal sections.
The test is added in three forms, to give fuzzers more food: A fuzzer
regression test for the streaming fuzzer, a unit test for the streaming
decoder, and an mjsunit test for streaming compilation.
Drive-by: Remove a redundant line in the synchronous decoder (this is
already handled by the following statement.
R=ahaas@chromium.org
Bug: chromium:1335023
Change-Id: Ic8c3b301f1b58981c7d68eafcffc89531ed2c64c
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/3698549
Reviewed-by: Andreas Haas <ahaas@chromium.org>
Commit-Queue: Clemens Backes <clemensb@chromium.org>
Cr-Commit-Position: refs/heads/main@{#81071}
The adding of base:: was mostly prepared using git grep and sed:
git grep -l <pattern> | grep -v base/vector.h | \
xargs sed -i 's/\b<pattern>\b/base::<pattern>/
with lots of manual clean-ups due to the resulting
v8::internal::base::Vectors.
#includes were fixed using:
git grep -l "src/utils/vector.h" | \
axargs sed -i 's!src/utils/vector.h!src/base/vector.h!'
Bug: v8:11879
Change-Id: I3e6d622987fee4478089c40539724c19735bd625
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2968412
Reviewed-by: Clemens Backes <clemensb@chromium.org>
Reviewed-by: Hannes Payer <hpayer@chromium.org>
Commit-Queue: Dan Elphick <delphick@chromium.org>
Cr-Commit-Position: refs/heads/master@{#75243}
The streaming decoder computed the code section start from the passed
"offset". That offset is computed from the module offset *after* the
number of functions has been read. Hence 1 is subtracted, with the
comment:
// The offset passed to {ProcessCodeSectionHeader} is an error offset and
// not the start offset of a buffer. Therefore we need the -1 here.
That subtraction of 1 worked when the number of functions was encoded in
a 1-byte LEB, otherwise it was off.
This CL fixes the immediate issue of passing the right code offset. The
usage of the previously existing offset also seems wrong, and I will try
to clean that up in a follow-up CL.
R=ahaas@chromium.orgCC=szuend@chromium.org
Bug: chromium:1150303
Change-Id: I64bb2ececeb4749b7ba2096cd148ccb4079eca4f
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2562383
Commit-Queue: Clemens Backes <clemensb@chromium.org>
Reviewed-by: Andreas Haas <ahaas@chromium.org>
Cr-Commit-Position: refs/heads/master@{#71503}
The implementation of the StreamingDecoder depends on async compilation.
However, when the --single-threaded flag is set, async compilation is
not available. Therefore V8 does not support streaming compilation at
the moment if the --single-threaded flag is set.
This CL is the first step to support streaming compilation in
--single-threaded mode. This CL makes the StreamingDecoder an abstract
class, and the current implementation a sub-class called
AsyncStreamingDecoder. A follow-up CL will provided a second sub-class
implementation for streaming compilation in --single-threaded mode.
Bug: v8:10548
Change-Id: Ice5c01340d3df18f836a4a05d30571207ca8ccf6
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2208869
Commit-Queue: Andreas Haas <ahaas@chromium.org>
Reviewed-by: Thibaud Michaud <thibaudm@chromium.org>
Cr-Commit-Position: refs/heads/master@{#67931}
In streaming compilation, we were computing a way too low code size
estimate, since all {WasmFunction::code} fields were still zero when we
were calling {EstimateNativeModuleCodeSize}. This lead to many separate
code spaces being created during compilation, creating significant
performance and memory overhead.
This CL fixes this by passing the code section length when creating the
{NativeModule}. From this, we can compute the code size estimate just as
before.
Drive-by: Rename "functions_count" to "num_functions" in
{ProcessCodeSectionHeader} to be consistent with the declaration.
R=ahaas@chromium.org
Bug: v8:9950
Change-Id: I30a54c01ed24d0dfecb8a4b6d123015f1803ddeb
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1903439
Commit-Queue: Clemens Backes <clemensb@chromium.org>
Reviewed-by: Andreas Haas <ahaas@chromium.org>
Cr-Commit-Position: refs/heads/master@{#64849}
This fixes the streaming decoder to report the correct error position
for repeating code sections (i.e. only one code section per module).
R=clemensb@chromium.org
Change-Id: Ie02d704d74b4e051fa9b00288dd6d1e46e2418a5
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1890094
Commit-Queue: Michael Starzinger <mstarzinger@chromium.org>
Reviewed-by: Clemens Backes <clemensb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#64654}
Since we switched to C++14 now, we can use {std::make_unique} instead
of our own {base::make_unique} from {template-utils.h}.
R=mstarzinger@chromium.org, yangguo@chromium.org
Bug: v8:9687
No-Try: true
Change-Id: I660eb30038bbb079cee93c7861cd87ccd134f01b
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1789300
Commit-Queue: Clemens Hammacher <clemensh@chromium.org>
Reviewed-by: Yang Guo <yangguo@chromium.org>
Reviewed-by: Michael Starzinger <mstarzinger@chromium.org>
Cr-Commit-Position: refs/heads/master@{#63642}
This fixes a bug in the offset computation when instantiating the
decoder to decode a VarInt32.
It also extends the streaming decoder test to check the error location.
R=ahaas@chromium.org
Bug: v8:8814
Change-Id: Id8ce31ce7e494cce14231febbb5b0c7d91a26e01
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1505453
Reviewed-by: Andreas Haas <ahaas@chromium.org>
Commit-Queue: Clemens Hammacher <clemensh@chromium.org>
Cr-Commit-Position: refs/heads/master@{#60067}
We sometimes store function indexes and number of functions as {size_t}
and sometimes as {int}. Unify a few places to be {int}.
R=ahaas@chromium.org
Change-Id: I1d204cbd9388245f97f291a469b32743457ab2c0
Reviewed-on: https://chromium-review.googlesource.com/c/1491607
Reviewed-by: Andreas Haas <ahaas@chromium.org>
Commit-Queue: Clemens Hammacher <clemensh@chromium.org>
Cr-Commit-Position: refs/heads/master@{#59946}
This removes an outdated section order check from {CreateNewBuffer} and
relies solely on the checks done in {ProcessSection}. Those checks are
more comprehensive and will remain coherent with synchronous decoding.
R=ahaas@chromium.org
TEST=mjsunit/regress/wasm/regress-8846
BUG=v8:8846
Change-Id: Id0cdc3bf3ad78f7970c9fceff66a17ab20f4666b
Reviewed-on: https://chromium-review.googlesource.com/c/1477211
Reviewed-by: Andreas Haas <ahaas@chromium.org>
Commit-Queue: Michael Starzinger <mstarzinger@chromium.org>
Cr-Commit-Position: refs/heads/master@{#59702}
In the wasm streaming decoder error position test, do also check the
error messages generated. This revealed messages that were not quite
fitting and some that were formatted differently than the majority.
R=ahaas@chromium.org
Bug: v8:8814
Change-Id: If157f1083a104413bf14797ac56e756baac98c17
Reviewed-on: https://chromium-review.googlesource.com/c/1463780
Commit-Queue: Clemens Hammacher <clemensh@chromium.org>
Reviewed-by: Andreas Haas <ahaas@chromium.org>
Cr-Commit-Position: refs/heads/master@{#59604}
For macros expanding to function definitions, I removed the spurious ; after
macro invocations. For macros expandign to function declarations, I made the ;
required and consistently inserted it.
No behavior change.
Bug: chromium:926235
Change-Id: Ib8085d85d913d74307e3481f7fee4b7dc78c7549
Reviewed-on: https://chromium-review.googlesource.com/c/1467545
Reviewed-by: Jakob Gruber <jgruber@chromium.org>
Reviewed-by: Ross McIlroy <rmcilroy@chromium.org>
Reviewed-by: Toon Verwaest <verwaest@chromium.org>
Reviewed-by: Michael Starzinger <mstarzinger@chromium.org>
Reviewed-by: Ulan Degenbaev <ulan@chromium.org>
Reviewed-by: Yang Guo <yangguo@chromium.org>
Commit-Queue: Nico Weber <thakis@chromium.org>
Cr-Commit-Position: refs/heads/master@{#59558}
We often use ResultBase or VoidResult to store or pass wasm errors
(errors with locations). This CL extracts a WasmError class which can
store an error (can also be empty), and Result<T> which stores an error
or a T (exactly one of them).
R=titzer@chromium.org
Bug: v8:8689
Change-Id: I3f5203559984a0ae8757e0130a9184957fa28df5
Reviewed-on: https://chromium-review.googlesource.com/c/1409365
Commit-Queue: Clemens Hammacher <clemensh@chromium.org>
Reviewed-by: Ben Titzer <titzer@chromium.org>
Cr-Commit-Position: refs/heads/master@{#58827}
After the first decoder error, the streaming processor should not be
called again. To enforce this, reset the {processor_} field. This also
makes the {ok_} field redundant.
Note that this refactoring is also necessary for a future CL which
makes the {StreamingProcessor} keep the {AsyncCompileJob} alive. By
resetting the processor, we also remove that link.
R=ahaas@chromium.org
Bug: v8:7921
Change-Id: I42f5ed26a8f26c3dc8db5676557a0d82021e132e
Reviewed-on: https://chromium-review.googlesource.com/c/1329179
Commit-Queue: Clemens Hammacher <clemensh@chromium.org>
Reviewed-by: Andreas Haas <ahaas@chromium.org>
Cr-Commit-Position: refs/heads/master@{#57435}
Compilation units currently contain pointers into allocated space that
contains the code of the respective function. This requires us to keep
the StreamingDecoder alive as long as compilation is still running
(including tiering).
This CL refactors this by having an additional redirection
(WireBytesStorage) which can point to either the StreamingDecoder or
the NativeModule. We only keep the code section buffer alive as long as
the StreamingWireBytesStorage is still in use.
I will further refactor memory ownership in a follow-up CL to not make
the AsyncCompileJob keep the StreamingDecoder alive.
R=ahaas@chromium.org
Bug: v8:8343,v8:7921,v8:8050
Change-Id: I780582c3217abf64000454f2c9c108b9ac9fbff1
Reviewed-on: https://chromium-review.googlesource.com/c/1319588
Reviewed-by: Andreas Haas <ahaas@chromium.org>
Commit-Queue: Clemens Hammacher <clemensh@chromium.org>
Cr-Commit-Position: refs/heads/master@{#57317}
- Adds embedder callback to notify fully tiered compilation is finished,
returning a WasmCompiledModule for serialization.
- Adds function to pass previously compiled bytes into WASM streaming
compilation, for deserialization.
- Plumbs this API through StreamingDecoder.
Bug: chromium:719172
Cq-Include-Trybots: luci.chromium.try:linux_chromium_rel_ng
Change-Id: Ibe376f3a8ccfa90fda730ef4ff6628a1532da45c
Reviewed-on: https://chromium-review.googlesource.com/c/1252884
Reviewed-by: Adam Klein <adamk@chromium.org>
Reviewed-by: Andreas Haas <ahaas@chromium.org>
Reviewed-by: Clemens Hammacher <clemensh@chromium.org>
Commit-Queue: Bill Budge <bbudge@chromium.org>
Cr-Commit-Position: refs/heads/master@{#56617}
Another pair of {std::unique_ptr} and {size_t} that can be stored as
one {OwnedVector}, which allows to pass it as one thing.
R=mstarzinger@chromium.org
Bug: v8:7754
Change-Id: Ideac0dbd390ba8147b6620daa86f0d3da6c3b609
Reviewed-on: https://chromium-review.googlesource.com/1118236
Commit-Queue: Clemens Hammacher <clemensh@chromium.org>
Reviewed-by: Michael Starzinger <mstarzinger@chromium.org>
Cr-Commit-Position: refs/heads/master@{#54091}
This patch normalizes the casing of hexadecimal digits in escape
sequences of the form `\xNN` and integer literals of the form
`0xNNNN`.
Previously, the V8 code base used an inconsistent mixture of uppercase
and lowercase.
Google’s C++ style guide uses uppercase in its examples:
https://google.github.io/styleguide/cppguide.html#Non-ASCII_Characters
Moreover, uppercase letters more clearly stand out from the lowercase
`x` (or `u`) characters at the start, as well as lowercase letters
elsewhere in strings.
BUG=v8:7109
TBR=marja@chromium.org,titzer@chromium.org,mtrofin@chromium.org,mstarzinger@chromium.org,rossberg@chromium.org,yangguo@chromium.org,mlippautz@chromium.org
NOPRESUBMIT=true
Cq-Include-Trybots: master.tryserver.blink:linux_trusty_blink_rel;master.tryserver.chromium.linux:linux_chromium_rel_ng
Change-Id: I790e21c25d96ad5d95c8229724eb45d2aa9e22d6
Reviewed-on: https://chromium-review.googlesource.com/804294
Commit-Queue: Mathias Bynens <mathias@chromium.org>
Reviewed-by: Jakob Kummerow <jkummerow@chromium.org>
Cr-Commit-Position: refs/heads/master@{#49810}
The streaming decoder allocates the whole section buffer of the code
section when it reads the section length of the code section. Therefore
we have to check that the different parts of the code section actually
use all the bytes, and that the different parts of the code section do
not need more bytes than available. The check that all bytes are used
was missing in the case where the code section contained zero functions.
In addition, this CL adds some tracing to the streaming decoder which
may be useful in future debugging.
R=clemensh@chromium.org
Bug: chromium:783595
Change-Id: Icf056c25a3000b4a08a791939dab0ccde9fc3f80
Reviewed-on: https://chromium-review.googlesource.com/768788
Commit-Queue: Andreas Haas <ahaas@chromium.org>
Reviewed-by: Clemens Hammacher <clemensh@chromium.org>
Cr-Commit-Position: refs/heads/master@{#49380}
At the moment we check only in the module-decoder if the sections in a
module appear at most once. The code section, however, we process
already before this check. With this CL we check that there is at most
one code section before we start processing it.
R=clemensh@chromium.org
TEST=WasmStreamingDecoderTest.TwoCodeSections
Bug: chromium:771916
Change-Id: Icc79d5a87ab39f450a35c688f74ea5e67cae4b3c
Reviewed-on: https://chromium-review.googlesource.com/702379
Reviewed-by: Clemens Hammacher <clemensh@chromium.org>
Commit-Queue: Andreas Haas <ahaas@chromium.org>
Cr-Commit-Position: refs/heads/master@{#48314}
In this CL I implement streaming compilation for WebAssembly,
as described in the design doc I have sent out already.
In this implementation the decoding of sections other than the
code section is done immediately on the foreground thread.
Eventually all decoding should happen in the background. I
think it is acceptable to do the decoding on the foreground
thread for now because I have finished it already, and
decoding in the background would add even more complexity to
this CL.
Bug:v8:6785
Cq-Include-Trybots: master.tryserver.chromium.linux:linux_chromium_rel_ng
Change-Id: I285e1e5e1a5a243113c92571b25ee9bae551d0ed
Reviewed-on: https://chromium-review.googlesource.com/631721
Reviewed-by: Mircea Trofin <mtrofin@chromium.org>
Reviewed-by: Clemens Hammacher <clemensh@chromium.org>
Reviewed-by: Andreas Haas <ahaas@chromium.org>
Commit-Queue: Andreas Haas <ahaas@chromium.org>
Cr-Commit-Position: refs/heads/master@{#48022}
This CL implements a streaming decoder which takes the bytes
of a wasm module as an input, potentially split into multiple
chunks, and decodes them into segments. Each segment either
contains the payload of a whole section, or the code of a
single function. The goal is that the streaming decoder is
used for streaming compilation. That's where the interface
comes from, see
(https://cs.chromium.org/chromium/src/v8/include/v8.h?q=OnBytesReceived&sq=package:chromium&l=4060)
Error positions are not reported correctly at the moment. I
plan to do this in a separate CL.
Change-Id: I6e3df6a91945c7baec2dc4f5de2e5f47636083df
Reviewed-on: https://chromium-review.googlesource.com/471350
Commit-Queue: Andreas Haas <ahaas@chromium.org>
Reviewed-by: Clemens Hammacher <clemensh@chromium.org>
Reviewed-by: Mircea Trofin <mtrofin@chromium.org>
Cr-Commit-Position: refs/heads/master@{#45250}
