Change the calling convention for the slow write barrier call to allow
callers to avoid saving caller-saved registers.
Bug: chromium:1406464
Change-Id: I314bdacb235727e0e78a192ed7cbff09e9bc6b4b
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/4156476
Commit-Queue: Michael Lippautz <mlippautz@chromium.org>
Reviewed-by: Anton Bikineev <bikineev@chromium.org>
Cr-Commit-Position: refs/heads/main@{#85254}
Introduces a slim write barrier for Oilpan behind
`cppgc_enable_slim_write_barrier` that is enabled by default. The slim
write barrier only performs a single approximate global check for
whether the write barrier is needed and delegates all other checks to
a slow path call. This is beneficial in configurations that do not need
many checks for the barrier overall, i.e., configurations without
young generation. Young generation is off by default which is why this
approach is beneficial.
On Speedometer the write barrier is hit 75M times with a fast bailout
of 99.3%. Progression on Speedometer2 is somewhere around 0.2-0.5%.
The resulting code embedded in another function is only 34 bytes
compared to 128 bytes before. See attached bug for detailed assembly
snippet.
Change-Id: I6869513186e7a26104c46f1f2ac2cfa855689f64
Bug: chromium:1406464
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/4152488
Reviewed-by: Anton Bikineev <bikineev@chromium.org>
Commit-Queue: Michael Lippautz <mlippautz@chromium.org>
Cr-Commit-Position: refs/heads/main@{#85232}
Now that we have all useful flags on the API side, use to them.
Bug: chromium:1056170
Change-Id: Ia849b0925a2b2c10ace30b6c2b6871bd3572da31
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/3899306
Commit-Queue: Anton Bikineev <bikineev@chromium.org>
Reviewed-by: Michael Lippautz <mlippautz@chromium.org>
Cr-Commit-Position: refs/heads/main@{#83238}
The split is rudimental and now is not needed at all:
- as part of the shared-cage effort we added HeapHandle pointer to the
BasePageHandle class (on the API side);
- for the value-full barrier we get HeapHandle from bitmasking the
value;
- for the value-less barrier we get it from the callback provided by the
caller.
The CL entirely removes the split and uses the single
BoundedPageAllocator. A minor note: the conservative stack scanning can
become sligthly more expensive.
Bug: chromium:1361582, chromium:1325007
Change-Id: I2a8aded3dd12037998f36341c68af8e23b0dcd88
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/3899320
Reviewed-by: Michael Lippautz <mlippautz@chromium.org>
Commit-Queue: Anton Bikineev <bikineev@chromium.org>
Cr-Commit-Position: refs/heads/main@{#83232}
Thread through compressed pointer into write barrier to allow to delay
compression after checking whether a write barrier is actually needed.
Change-Id: If7e6cbb69a57cc9aeeb551c11f685bace4e56c4c
Bug: chromium:1325007
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/3769826
Commit-Queue: Michael Lippautz <mlippautz@chromium.org>
Reviewed-by: Anton Bikineev <bikineev@chromium.org>
Cr-Commit-Position: refs/heads/main@{#81816}
Currently, young generation assumes that all the on-heap pointers are
compressed, which is currently not the case due to the backing-store
pointers in Blink. Fixing collections to have Member<> is not easy,
because the inlined collections may have the semantics that
backing_pointer_ == inlined_buffer_;
where the inlined_buffer_ can be off-heap (e.g. on stack).
The CL introduces another type of barrier specifically for uncompressed
pointers. The followup is to use that barrier from Blink.
Bug: chromium:1029379
Change-Id: If0f519220658268dbdf915235c2e5afd7887dc0c
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/3695358
Reviewed-by: Michael Lippautz <mlippautz@chromium.org>
Commit-Queue: Anton Bikineev <bikineev@chromium.org>
Cr-Commit-Position: refs/heads/main@{#81105}
The CL does the following:
1) Globalizes CagedHeap for all HeapBases;
2) Adds the global variable representing the cage base;
3) Changes all write-barriers to use this global variable for value/slot
checks;
4) Removes no longer needed functionality introduced in previous CLs.
Bug: v8:12231
Change-Id: I281a7b0bf67e349c988486fc2d43ec6d703fd292
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/3688050
Commit-Queue: Anton Bikineev <bikineev@chromium.org>
Reviewed-by: Michael Lippautz <mlippautz@chromium.org>
Cr-Commit-Position: refs/heads/main@{#81027}
The CL is a prerequisite for the shared cage. Instead of storing
state variables (is_incremental_marking_in_progress,
is_young_generation_enabled) in the cage metadata, the CL moves them to
HeapHandle. The HeapHandle pointer is now retrieved from page-headers.
To make sure that the write-barrier code is better optimized, the
HeapHandle definition is moved to internal/ headers. The part of
BasePage that contains HeapBase (i.e. HeapHandle) pointer is also
extracted and moved to the headers.
Bug: v8:12231
Change-Id: I44bf65d99a621d9548e4250386cf87476ca186ac
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/3689730
Commit-Queue: Anton Bikineev <bikineev@chromium.org>
Reviewed-by: Michael Lippautz <mlippautz@chromium.org>
Cr-Commit-Position: refs/heads/main@{#81005}
The CL introduces a new option --cppgc-young-generation. This option
can't be enabled statically, because V8 options are parsed after heap
initialization. The CL changes minor GC so that it can be enabled
dynamically. The way it works is as follows:
- the user calls YoungGenerationEnabler::Enable();
- a heap checks in the next atomic pause whether the flag was enabled;
- if so, the heap enables young generation for itself.
To avoid barrier regressions without young-generation enabled, the CL changes the meaning of the global flag is-any-incremental-or-concurrent-marking to is-barrier-enabled.
The runtime option would enable us to test young generation on try-
and performance-bots.
Bug: chromium:1029379
Change-Id: I664cccdcd208225ffcbf9901f1284b56d088c5c3
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/3607993
Commit-Queue: Anton Bikineev <bikineev@chromium.org>
Reviewed-by: Michael Lippautz <mlippautz@chromium.org>
Cr-Commit-Position: refs/heads/main@{#80274}
If the following conditions hold:
1) value is kSentinel,
2) slot is on stack,
3) stack is allocated below 4GB,
then the generational barrier would be erroneously triggered for the
stack object object. This CL fixes it. At the same time, it aims to
simplify the code and potentially optimizes it (by having 'and' instead
of 'sub').
Bug: chromium:1029379
Change-Id: Iafd91d50b0a1c3d97647f7bf3643dfcc7e9fb48f
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/3608629
Reviewed-by: Michael Lippautz <mlippautz@chromium.org>
Commit-Queue: Anton Bikineev <bikineev@chromium.org>
Cr-Commit-Position: refs/heads/main@{#80202}
This reduces card granularity from 4096 to 512 bytes with the goal to
improve write barrier filtering.
Bug: chromium:1029379
Change-Id: I22e2a9c61ef4c36c3db65404370213d0a8048e08
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/3582393
Reviewed-by: Michael Lippautz <mlippautz@chromium.org>
Commit-Queue: Anton Bikineev <bikineev@chromium.org>
Cr-Commit-Position: refs/heads/main@{#79960}
The generational barrier for source objects records the entire source
object to be processed later during remembered set visitation. It's
planned to be used for Blink backing stores when an inlined object (or a
range thereof) is added (HeapAllocator::NotifyNewObject(s)).
An alternative approach would be to eagerly process the inlined objects
using a custom callback. However, this requires changing Visitors to
bring slots into the context. This approach should better work for
scenarios where small ranges or single elements are added, to avoid
processing potentially large backing stores. The followup CL implements
this idea.
Bug: chromium:1029379
Change-Id: Iacb59e4b10a66354526ed293d7f43f14d8761a8f
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/3460402
Reviewed-by: Michael Lippautz <mlippautz@chromium.org>
Commit-Queue: Anton Bikineev <bikineev@chromium.org>
Cr-Commit-Position: refs/heads/main@{#79073}
This bailout applies only when young generation is disabled.
Otherwise, getting the caged heap is required and the global load
bailout becomes redundant.
Bug: chromium:1056170, chromium:1239030
Change-Id: I826b355f4356a5c20812d9c8bf6ebc7ea0997b8a
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/3173685
Commit-Queue: Omer Katz <omerkatz@chromium.org>
Reviewed-by: Michael Lippautz <mlippautz@chromium.org>
Cr-Commit-Position: refs/heads/main@{#77006}
GetWriteBarrierType used to consider the slot so that a barrier is not
triggered for on-stack slots. For strongifying weak collections we want
the write barrier to trigger even if the backing store is only reachable
from stack.
Blink counterpart: crrev.com/c/3162170
Bug: chromium:1244057
Change-Id: I75b1ca62ad5de7bae3d2f4c1a9acce839f3ccdc1
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/3162127
Commit-Queue: Omer Katz <omerkatz@chromium.org>
Reviewed-by: Michael Lippautz <mlippautz@chromium.org>
Cr-Commit-Position: refs/heads/main@{#76872}
The assumption doesn't necessarily hold on linux and Android either.
Bug: chromium:1056170, chromium:1239287
Change-Id: Ibb0d8f5f814580bff4e8a7dce9a3397df1385896
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/3097273
Reviewed-by: Michael Lippautz <mlippautz@chromium.org>
Reviewed-by: Anton Bikineev <bikineev@chromium.org>
Commit-Queue: Omer Katz <omerkatz@chromium.org>
Cr-Commit-Position: refs/heads/master@{#76299}
On OSes other than Windows and Fuchsia the write barrier assumes that
the caged heap is allocated below the stack.
Add CHECK that the assumption holds.
Bug: chromium:1056170
Change-Id: I64c790e61b4cfa2adb8274ed74111f0433e9aefb
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/3080570
Auto-Submit: Omer Katz <omerkatz@chromium.org>
Commit-Queue: Omer Katz <omerkatz@chromium.org>
Reviewed-by: Michael Lippautz <mlippautz@chromium.org>
Cr-Commit-Position: refs/heads/master@{#76173}
crrev.com/c/3069146 fixed a write barrier issue leading to a null
dereference on Windows that was triggered by having the stack allocated
at address below 4GB.
Turns out the same can happen on Fuchsia.
Bug: chromium:1230763, chromium:1056170
Change-Id: I74ba0b465c3230b4274f2c23d279c4f73183eddb
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/3071402
Commit-Queue: Omer Katz <omerkatz@chromium.org>
Commit-Queue: Anton Bikineev <bikineev@chromium.org>
Auto-Submit: Omer Katz <omerkatz@chromium.org>
Reviewed-by: Anton Bikineev <bikineev@chromium.org>
Cr-Commit-Position: refs/heads/master@{#76094}
Windows can allocate the stack at low addresses. A low-address on-stack
slot (e.g. backing store reference for Blink's on-heap collections) with
a null value would make TryGetCagedHeap falsely think that the slot
resides in a caged heap that starts at a null address.
We will still crash for low-address on-stack slots with non-null
on-stack value, since these cases are not considered valid and should
not happen.
The null value check is added only to Windows. It is not an issue on
other OSes where the stack always resides at high addresses and we
prefer to keep the write barrier as cheap as possible.
Bug: chromium:1230794, chromium:1056170
Change-Id: I07e2d178cd95edba57015d6bc6eb127a443b0589
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/3069146
Commit-Queue: Omer Katz <omerkatz@chromium.org>
Reviewed-by: Anton Bikineev <bikineev@chromium.org>
Cr-Commit-Position: refs/heads/master@{#76069}
Used include-what-you-use [1] to clean up cppgc API headers. The tool
does somewhat work but requires some cleanup afterwards as it cannot
nicely deal with `v8config.h` and the defines it generates.
[1] https://github.com/include-what-you-use/include-what-you-use/
Bug: chromium:1056170
Change-Id: I7b03797c615f8e033510fc959bbdb792d8b7a4ee
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2907612
Reviewed-by: Omer Katz <omerkatz@chromium.org>
Reviewed-by: Anton Bikineev <bikineev@chromium.org>
Commit-Queue: Michael Lippautz <mlippautz@chromium.org>
Cr-Commit-Position: refs/heads/master@{#74693}
Process global accessors for total physical size and used size are
needed for blink. These are implemented via an allocation observer that
atomically updates static counters.
The public api only provides getters for the counters. An internal class
is in charge of updating the counters as needed. A similar split is also
applied to IsAnyIncrementalOrConcurrentMarking().
Drive-by: ProcessHeap is merged into cppgc::internal::WriteBarrier.
Bug: chromium:1056170
Change-Id: Iaedebd1ac9d49238ce6bdd52ffa5d1ef4d28203d
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2695394
Commit-Queue: Omer Katz <omerkatz@chromium.org>
Reviewed-by: Michael Lippautz <mlippautz@chromium.org>
Cr-Commit-Position: refs/heads/master@{#72832}
This change avoid dispatching a write barrier during the atomic pause.
The dispatch can generally be triggered through pre-finalizers.
In future, further checks may be added to avoid mis-use of
pre-finalizers.
Bug: chromium:1056170, chromium:1175560
Change-Id: I119e18372633b2375f60e17b4c881f68bb20bf66
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2679685
Reviewed-by: Omer Katz <omerkatz@chromium.org>
Commit-Queue: Michael Lippautz <mlippautz@chromium.org>
Cr-Commit-Position: refs/heads/master@{#72560}
Some types of supported low-level write barrier only requires passing
a slot, which may not be even part of a heap object but stack.
This complicates the situation, as even with caged heap, there's no
way to distinguish a stack and heap slot.
Solve this by passing an optional callback that can lazy be used to
get the heap. This can be used by the embedder to retrieve the heap
from e.g. TLS if needed. This aligns the barrier with Oilpan in
Blink.
Bug: chromium:1056170
Change-Id: I1e5d022ab17a2614a67b6ef39ed12691bcbd0ac6
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2675924
Reviewed-by: Ulan Degenbaev <ulan@chromium.org>
Reviewed-by: Omer Katz <omerkatz@chromium.org>
Commit-Queue: Michael Lippautz <mlippautz@chromium.org>
Cr-Commit-Position: refs/heads/master@{#72550}
Provide a way to trigger a write barrier when updating the embedder
fields. In future, such a mechanism should be encapsulated into V8.
Bug: chromium:1056170
Change-Id: I4e43362993c3e58d5bebdd58a7d46a39c0aa4f06
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2640419
Reviewed-by: Omer Katz <omerkatz@chromium.org>
Reviewed-by: Ulan Degenbaev <ulan@chromium.org>
Commit-Queue: Michael Lippautz <mlippautz@chromium.org>
Cr-Commit-Position: refs/heads/master@{#72227}
Refactor write barriers and split calls, as e.g. DijkstraWriteBarrier
also contained logic for recording slots (cards) for the young
generation.
The new API exposes the following:
- GetWriteBarrierType(): Retrieving the type of barrier that must be
emitted;
- DijkstraWriteBarrier(), DijkstraWriteBarrierRange(): Dijkstra-style
write barriers;
- SteeleWriteBarrier(): Steele-style write barrier;
- GenerationalBarrier(): Barrier for recording slots when using
multiple generations;
Compilers running with -O3 optimize the DijkstraWriteBarrierPolicy
down to the same instructions as before the split.
Change-Id: If68839cc6357b2f568986c9ce8ca753b1e96a70a
Bug: chromium:1056170
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2557514
Commit-Queue: Michael Lippautz <mlippautz@chromium.org>
Reviewed-by: Omer Katz <omerkatz@chromium.org>
Reviewed-by: Ulan Degenbaev <ulan@chromium.org>
Cr-Commit-Position: refs/heads/master@{#71407}
Exposes an opaque handle for uniformly (cppgc and V8) referring to an
instance of a heap.
Exposes a set of raw write barriers for advances embedders through
subtle::HeapConsistency which is a mirror into write barrier internals.
The following barriers are exposed:
- DijkstraWriteBarrier: Regular Dijkstra-style write barrier (add to
wavefront);
- DijkstraWriteBarrierRange: Same as DijkstraWriteBarrier but
operating on a range of slots that are composite (inlined) objects;
- SteeleWriteBarrier: Regular Steele-style write barrier (retreating
wavefront);
Change-Id: Ib5ac280204686bf887690f72df1cdb506ea6ef70
Bug: chromium:1056170
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2554601
Reviewed-by: Omer Katz <omerkatz@chromium.org>
Reviewed-by: Anton Bikineev <bikineev@chromium.org>
Reviewed-by: Ulan Degenbaev <ulan@chromium.org>
Commit-Queue: Michael Lippautz <mlippautz@chromium.org>
Cr-Commit-Position: refs/heads/master@{#71381}
This adds the following things:
- age table for 4K regions;
- generational barrier for mixed 4K regions;
- unmarking for major collections;
- young generation flags.
Bug: chromium:1029379
Change-Id: Ief1229f0dac5f90c5f06d3168c8ffb4b7d1f1b53
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2246566
Commit-Queue: Anton Bikineev <bikineev@chromium.org>
Reviewed-by: Omer Katz <omerkatz@chromium.org>
Reviewed-by: Ulan Degenbaev <ulan@chromium.org>
Reviewed-by: Michael Lippautz <mlippautz@chromium.org>
Cr-Commit-Position: refs/heads/master@{#68379}
