The big change here is having the C++ toolchain use
Bazel platforms instead of the C++ specific flags/setup.
In Bazel, platforms are a general purpose way to define
things like os, cpu architecture, etc. We were not using
platforms previously, because the best documentation at
the time focused on the old ways.
However, the old ways were clumsy/difficult when trying
to manage cross-compilation, specifically when trying
to have a Mac host trigger a build on our Linux RBE
system targeting a Linux x64 system. Thus, rather than
keep investing in the legacy system, this CL migrates
us to using platforms where possible.
Suggested background reading to better understand this CL:
- https://bazel.build/concepts/platforms-intro
- https://bazel.build/docs/platforms
- https://bazel.build/docs/toolchains#registering-building-toolchains
The hermetic toolchain itself is not changing in this CL
(and likely does not need to), only how we tell Bazel
about it (i.e. registering it) and how Bazel decides
to use it (i.e. resolving toolchains).
Here is my understanding of how platforms and toolchains
interact (supported by some evidence from [1][2])
- Bazel needs to resolve platforms for the Host, Execution,
and Target.
- If not specified via flags, these are the machine from
which Bazel is invoked, aka "@local_config_platform//:host".
- With this CL, the Host could be a Mac laptop, the Execution
platform is our Linux RBE pool, and the Target is "a Linux
system with a x64 CPU"
- To specify the Host, that is, describe to Bazel the
capabilities of the system it is running on, one can
set --host_platform [3] with a label pointing to a platform()
containing the appropriate settings. Tip: have this
platform inherit from @local_config_platform//:host
so it can add to any of the constraint_settings and
constraint_values that Bazel deduces automatically.
- To specify the Target platform(s), that is, the system
on which a final output resides and can execute, one
can set the --platforms flag with a label referencing
a platform().
- Bazel will then choose an execution platform to fulfill
that request. Bazel will look through a list of available
platforms, which can be augmented* with the
--extra_execution_platforms. Platforms specified by this
flag will be considered higher than the default platforms!
- Having selected the appropriate platforms, Bazel now
needs to select a toolchain to actually run the actions
of the appropriate type.
- Bazel looks through the list of available toolchains
and finds one that "matches" the Execution and the Target
platform. This means, the toolchain's exec_compatible_with
is a strict subset of the Execution platform and
the toolchain's target_compatible_with is a strict subset
of the Target platform. To register toolchains* (i.e. add
them to the resolution list), we use --extra_toolchains.
Once Bazel finds a match, it stops looking.
Using --toolchain_resolution_debug=".*" makes Bazel log
how it is resolving these toolchains and what execution
platform it picked.
* We can also register execution platforms and toolchains in
WORKSPACE.bazel [4], but the flags come with higher priority
and that made resolution a bit tricky. Also, when we want
to conditionally add them (e.g. --config=linux_rbe), we
cannot remove them conditionally in the WORKSPACE.bazel file.
The above resolution flow directly necessitated the changes
in this CL.
Example usage of the new configs and platforms:
# Can be run on a x64 Linux host and uses the hermetic toolchain.
bazel build //:skia_public
# Can be run on Mac or Linux and uses the Linux RBE system along
# with the hermetic toolchain to compile a binary for Linux x64.
bazel build //:skia_public --config=linux_rbe --config=for_linux_x64
# Shorthand for above
bazel build //:skia_public --config=for_linux_x64_with_rbe
Notice we don't have to type out --config=clang_linux anymore!
That was due to me reading the Bazel docs more carefully and
realizing we can set options for *all* Bazel build commands.
Current Limitations:
- Targets which require a py_binary (e.g. Dawn's genrules)
will not work on RBE when cross compiling because the
python runtime we download is for the host machine, not
the executor. This means //example:hello_world_dawn does
not work on Mac when cross-compiling via linux_rbe.
- Mac M1 linking not quite working with SkOpts settings.
Probably need to set -target [5]
Suggested Review order:
- toolchain/BUILD.bazel Notice how we do away with
cc_toolchain_suite for toolchain. These have the same
role: giving Bazel the information about where a toolchain
can run. The platforms one is more expressive (IMO), allowing
us to say both where to run the toolchain and what it can
make. In order to more easily force the use of our hermetic
toolchain, but also allow the hermetic toolchain to be used
on RBE, we specify "use_hermetic_toolchain" only on the target,
because the RBE image does not have the hermetic toolchain
on it by default (but can certainly run it).
- bazel/platform/BUILD.bazel to see the custom constraint_setting
and corresponding constraint_value. The names for both of these
are completely arbitrary - they do not need to have any deeper
meaning or relation to any file or Docker image or system or
any other constraints. Think of the constraint_setting as
an Enum and the constraint_value being the one and only member.
We need to pass around a constant value, not a type, so we
need to provide the constraint_value (e.g. in toolchain/BUILD.bazel)
but not a constraint_setting. However we need a
constraint_setting declared so we can make a constraint_value
of that "type".
Notice the platform declared here - it allows us to force
Bazel to use the hermetic toolchain because of the extra
constraint_value.
- .bazelrc I set a few flags that will be on for all
bazel build commands. Importantly, this causes the C++
build logic to use platforms and not the old, bespoke way.
I also found a way to avoid using the local toolchain on
the host, which will hopefully lead to clearer errors
if platforms are mis-specified instead of odd compile
errors because the host toolchain is too old or something.
There are also a few RBE settings tweaked to be a bit
more modern, as well the new shorthands for specifying
target platforms (e.g. for_linux_x64).
- bazel/buildrc where we have to turn off the platforms
logic for emscripten https://github.com/emscripten-core/emsdk/issues/984
- bazel/rbe/BUILD.bazel for a fix in the platform description
that makes it work on Mac.
- Notice that _m1 has been removed from the mac-related toolchain
files because the same toolchain should work on both
architectures.
- All other changes in any order.
[1] https://bazel.build/docs/toolchains#debugging-toolchains
[2] https://bazel.build/docs/toolchains#toolchain-resolution
[3] https://bazel.build/reference/command-line-reference
[4] https://bazel.build/docs/toolchains#registering-building-toolchains
[5] 17dc3f16fc/gn/skia/BUILD.gn (L258-L271)
Change-Id: I515c114099d659639a808f74e47d489a68b7af62
Bug: skia:12541
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/549737
Reviewed-by: Erik Rose <erikrose@google.com>
Reviewed-by: Jorge Betancourt <jmbetancourt@google.com>
These four directories had IWYU enforced previously and
that enforcement was turned back on:
- src/sksl/
- src/utils/
- src/svg/
- tools/debugger/
It was discovered that src/sksl/ir had been missed with the
previous IWYU enforcement, so many files needed updating
(see https://skia-review.googlesource.com/c/skia/+/547256).
We do not currently include src/svg/ or tools/debugger/ in
any Bazel builds, so that enforcement has not been tested
with the new system. When we add in builds that use those
packages, we may need to update includes.
Suggested Review order:
- clang_trampoline_linux.sh to see list expanded
- bazel/Makefile to see convenient target for testing this
locally (follow-up CL will have a CI job for this).
Change-Id: Ifef1659ccd1a0e6c862b82102576a06296a6b42e
Bug: skia:12541 skia:13052
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/546608
Reviewed-by: Ben Wagner <bungeman@google.com>
Organization v3.5, if we are keeping track :)
This splits the "srcs" filegroup into "srcs" and "private_hdrs",
and renames "hdrs" to "public_hdrs".
To assist with the split, I created the macro split_srcs_and_hdrs.
Rather than keep two separate lists of header and source files,
I figured it would be easiest, at least for the common case,
to keep one list of files and then have a for loop split them
apart. I've tried to be consistent with having the list
of files be named with a _FILES suffix - maybe we can use this
as a marker to generate .gni files in the future?
Suggested review order:
- //bazel/macros.bzl. Note this needs a corresponding
G3 change (http://cl/452279799) as well. The exports_files_legacy
change is the better approach to something I manually
handled yesterday when fixing the G3 roll.
- //BUILD.bazel to see the new target skia_internal and
the previous skia_core renamed to skia_public.
- //src/core/BUILD.bazel to see a typical usage of
split_srcs_and_hdrs.
- //include/... to see the change to public_hdrs and
private_hdrs
- //src/... to see many more usages of split_srcs_and_hdrs
- //tools/... to see changes to skia_internal where
appropriate.
- Everything else. Note that //modules/... might also need
to be built with skia_internal instead of skia_public,
but we can fix that up later, if necessary.
Change-Id: Ie1cc969455d97b029b2d77faa222c4a9bad70671
Bug: skia:12541
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/545716
Reviewed-by: Ben Wagner <bungeman@google.com>
Reviewed-by: Leandro Lovisolo <lovisolo@google.com>
This may look like a lot, but //modules/canvaskit/BUILD.bazel
is nearly identical to how it was with gazelle:
162dfca340/modules/canvaskit/BUILD.bazel
I removed the "wasm_gm_tests" targets from it, because they
had bitrotted slightly and fixing them is its own task.
CanvasKit depends on Skottie and Particles, which depend on
the SkParagraph, SkShaper, SkUnicode, and SkResources modules.
I've structured the BUILD.bazel files in the //modules directory
in a similar fashion as the "hierarchical filegroup"
introduced in https://skia-review.googlesource.com/c/skia/+/543977
Suggested Review Order
- //modules/skottie/...
- //modules/skparagraph/...
- all other modules.
- Note that modules/canvaskit/go/gold_test_env/BUILD.bazel is
generated from gazelle, because we like how gazelle handles
golang files and deps.
- All other files in any order.
Change-Id: I0aa9e6f81dba2c00f15cae7b19fe49a2027dcf1d
Bug: skia:12541
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/544676
Reviewed-by: Leandro Lovisolo <lovisolo@google.com>
This adds or fixes rules to build three binaries that previously
had Bazel support.
How we build skslc with Bazel differs from GN in a large way:
GN has a small set of C++ files [1] it compiles in, but with
Bazel that was too hard/clumsy to pipe through (and get the
headers to work well). So, I just had skslc depend on
//:skia_core for simplicity. skslc did need to include something
in //src, so I made a special filegroup for it. For more
complex executables that need more headers from //src, we
should probably make a "src_hdrs" filegroup or something
to expose those. That or a skia_for_tests cc_library that
has all headers as "public".
[1] https://github.com/google/skia/blob/main/gn/sksl.gni#L235
Change-Id: Ie1382e982228059369886f4bfef4947f686b11b5
Bug: skia:12541
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/544637
Reviewed-by: Ben Wagner <bungeman@google.com>
This adds targets which test our Dawn, GL, and Vulkan backends.
It follows the hierarchical filegroup pattern, as
outlined in https://skia-review.googlesource.com/c/skia/+/543977
Suggested Review order:
- tools/sk_app/BUILD.bazel. For many things in tools, I anticipate
they will depend on //:skia_core and other //tools targets.
sk_app shows this off, as well how to make the target
specific to a given platform and pull in the proper native code.
I'm trying out setting test_only = True, to see if we can
partition Skia's tests and helpers from the actual Skia library.
- other changes to //tools/, especially looking at sk_app's
dependencies.
- //example/BUILD.bazel. This uses the cc_binary_with_flags which
existed previously [1] to make it so people don't have to
specify all the flags for a given binary and can build it as is.
These targets nows how up in //bazel/Makefile
- //include/... and //src/..., where some typos from previous
CLs were fixed and rules expanded.
- Misc changes to .cpp files to remove unnecessary includes
that were assuming the GL backend was being compiled in.
- All other changes
[1] 162dfca340/bazel/cc_binary_with_flags.bzl
Change-Id: Ieacec464d44368cad0da0890c7dc85a6c0b900c9
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/544317
Reviewed-by: Ben Wagner <bungeman@google.com>
Reviewed-by: Leandro Lovisolo <lovisolo@google.com>
The primary goal of this organization structure is to keep
our top level BUILD.bazel file short, with as little logic
as feasible. The logic required to control which files to
include, which third_party deps are needed, what system libraries
should be linked again, etc, should be in the BUILD.bazel
file best should be as close to the affected files as feasible.
In essence, we use filegroup() rules to bubble up the files
needed to build Skia (all as one big cc_library call) and
cc_library rules to bubble up the other components needed to build.
For example, //src/ports/SkFontHost_FreeType.cpp needs FreeType,
but only if we are compiling Skia with that type of font
support. With the new organization structure in this CL,
//src/ports/BUILD.bazel should have the logic that determines
if the cpp file should be included in the build of Skia and
if it is, that the Skia build should depend on //third_party:freetype2
Another example is //src/gpu/ganesh/BUILD.bazel, which
chooses which of the dawn, gl, vulkan, etc backend sources,
and the associated dependencies to include in the build.
It does not specify what those are, but delegates to the
BUILD.bazel files in the subdirectories housing the
backend-specific code.
The structure guidelines for BUILD.bazel files are as follows:
- Have a filegroup() called "hdrs" (for public headers) or
"srcs" (for private headers and all .cpp files) that is
visible to the parent directory. This should list the
files from the containing directory to include in the
build.
See //include/core/BUILD.bazel and //src/effects/BUILD.bazel
as examples.
- filegroup() rules can list a child directory's "hdrs"
or "srcs" in their "srcs" attributes, but should not contain
select statements pertaining to child directory files.
See //include/gpu/BUILD.bazel and //src/gpu/ganesh/BUILD.bazel
as examples.
- May have a cc_library() called "deps". This can specify
dependencies, cc_opts, and linkopts, but not srcs or hdrs. [1]
See //src/codec/BUILD.bazel as an example. These should
be visible to the parent directory.
- "hdrs", "srcs", and "deps" for the primary Skia build
(currently called "skia_core") should bubble up through
//include/BUILD.bazel and //src/BUILD.bazel, one directory
at a time.
This CL demonstrates a very basic build of Skia with many features
turned off (CPU only, no fonts, no codecs). Follow-on CLs will
add to these rules as more targets are supported. See bazel/Makefile
for the builds that work with just this CL.
Suggested Review Order:
- //BUILD.bazel to see the very small skia_core rule which
delegates all the logic down stack. Note that it has a
dependency on //bazel:defines_from_flags which will set
all the defines listed there when compiling all the
.cpp and .h files in skia_core *and* anything that depends
on skia_core, but *not* //src:deps.
- //include/BUILD.bazel and other BUILD.bazel files in the
subdirectories of that folder. Note that the filegroups in
//include/private/... are called "srcs" to be similar to
how Bazel wants "private headers" to be in the "srcs" of
cc_library, cc_binary, etc. and only public headers are
to be in "hdrs" [2].
- //src/BUILD.bazel and other BUILD.bazel files in the
subdirectories of that folder. //src/gpu/ganesh/...
will be filled in for dawn, vulkan, and GL in the next CL.
- //PRESUBMIT.py, which adds a check that runs buildifier [3]
on modified BUILD.bazel files to make sure they stay
consistently formatted.
- //bazel/... to see the new option I added to make sksl
opt-in or opt-out, so one could build Skia with sksl,
but not with a gpu backend.
- Misc .h and .cpp files, whose includes were removed if
unnecessary or #ifdef'd out to make the minimal build
work without GPU or SkSL includes.
- //bazel/Makefile to see the builds that work with this CL.
[1] Setting srcs or hdrs is error-prone at best, because those
files will be compiled with a different set of defines than
the rest of skia_core, because they wouldn't depend on
//bazel:defines_from_flags.
[2] https://bazel.build/reference/be/c-cpp#cc_library.hdrs
[3] https://github.com/bazelbuild/buildtools/releases
Change-Id: I5e0e3ae01ad42d672506d5aad1239f2512188191
Bug: skia:12541
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/543977
Reviewed-by: Leandro Lovisolo <lovisolo@google.com>
Reviewed-by: Ben Wagner <bungeman@google.com>
gazelle ended up being more liability than asset for our C++ rules.
It required devs to manually run the command frequently (and was
easy to forget until the CQ failed). The fact that we still had to
edit the source files (e.g. the "srcs" cc_libraries) meant that
the mixture between generated and hand-written caused some
tension (see include/third_party/vulkan for a good example).
The combination of gazelle and our IWYU enforcement added several
bits of churn without any real benefit. The generated rules
also didn't help identify cases where we were not keeping tight
boundaries (e.g. non-gpu code and gpu code).
Identifying third_party deps automatically ended up being trickier
than anticipated (see the deleted //third_party/file_map_for_bazel.json)
Using the "maximum set of dependencies" worked ok, but ended up
increasing build time unnecessarily. For example, compiling
CanvasKit for WebGL always needed to compile Dawn because
SkSLCompiler.cpp sometimes needs to include tint/tint.h.
Follow-up CLs will rebuild the BUILD.bazel rules without gazelle.
Note to Reviewers:
- The only file worth manually reviewing here is bazel/Makefile.
Change-Id: I36d6fc3747487fabaf699690780c95f1f6765770
Bug: skia:12541
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/543976
Reviewed-by: Leandro Lovisolo <lovisolo@google.com>
Reviewed-by: Ben Wagner <bungeman@google.com>
The `make -C bazel generate` command doesn't function properly on
Windows builders because Windows uses backslashes as the path separator.
Fortunately, adding quotes around the path is enough to concince
Windows to run the executable, allowing the `generate` target to
complete successfully, and doing so doesn't impact Linux builders.
Bug: skia:13366
Change-Id: I50c461635b70cc59cd6e79bf17a6945c80df1986
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/544756
Reviewed-by: Kevin Lubick <kjlubick@google.com>
Commit-Queue: Kevin Lubick <kjlubick@google.com>
This reverts commit 5c6bf4f692.
Reason for revert: git diff does not work well with deleted files
Original change's description:
> [bazel] Run gazelle only on the files that changed
>
> With this change, make generate takes 1.8 seconds instead of
> 7.9 seconds.
>
> We still have generate_force to run on everything.
>
> Change-Id: I6d57031adbe38a7f25a59570baea89970eea024f
> Bug: skia:12541
> Reviewed-on: https://skia-review.googlesource.com/c/skia/+/540740
> Reviewed-by: Michael Ludwig <michaelludwig@google.com>
Bug: skia:12541
Change-Id: I47c23adf09bbc6324817e166f7ab33eb16f4bf61
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/540743
Reviewed-by: John Stiles <johnstiles@google.com>
With this change, make generate takes 1.8 seconds instead of
7.9 seconds.
We still have generate_force to run on everything.
Change-Id: I6d57031adbe38a7f25a59570baea89970eea024f
Bug: skia:12541
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/540740
Reviewed-by: Michael Ludwig <michaelludwig@google.com>
Rather than having a monolithic third_party/BUILD.bazel, this moves
our Dawn rules to their own subdirectory and makes it callable via
@dawn instead of //third_party/dawn.
This will help with the G3 roll and make our rules more organized in
general.
This also rolls Dawn
Roll Dawn from ab9757036bd6 to e831fb61046b (22 revisions)
https://dawn.googlesource.com/dawn.git/+log/ab9757036bd6..e831fb61046b
Suggested Review Order:
- WORKSPACE.bazel, where we define @dawn and its deps
(@vulkan_headers and @vulkan_tools). I initially thought
I needed to define all of Dawn's deps in the workspace_file_content
for new_local_repository, but that WORKSPACE file is
ignored when building Skia rules.
- third_party/dawn/BUILD.bazel, the contents of which were copied
from //third_party/BUILD.bazel and modified largely via
find-and-replace to point to files relative to
//third_party/externals/dawn. One exception is the cpu_wasm
config_setting because @dawn isn't able to see Skia's
//bazel/macros.bzl.
- All other files
Change-Id: Ib2d7bc972ef00b6b68370ce5c2839ffb70ed9a2f
Bug: skia:12541, skia:13211
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/538638
Reviewed-by: Ben Wagner <bungeman@google.com>
Commit-Queue: Kevin Lubick <kjlubick@google.com>
For additional context, see "Codifying Certain Build Options"
and "Building on the CI" in the design doc go/skia-bazel
Suggested review order:
- builder_name_schema.json to see the three required and
one optional part of BazelBuild jobs.
- jobs.json to see one new BazelBuild job added. In an
ideal world, this job would have been named
BazelBuild-//modules/canvaskit:canvaskit_wasm-debug-linux_x64
but Buildbucket (?) requires jobs match the regex
^[a-zA-Z0-9\\-_.\\(\\) ]{1,128}$
so we use spaces instead of slashes or colons.
- gen_tasks_logic.go; noting the makeBazelLabel function
expands most of the spaces to / and the last one to a
colon to make a single-target label. If there are three
dots, then it is a multi-target label, and we do not
need to add a colon.
- bazel_build.go; This is a very simple task driver, and
I do not anticipate getting too much more complex.
The place where we decide which args to augment
a build with depend on the host platform and thus
should be set in gen_tasks_logic.go.
- bazel/buildrc to see some initial configurations set,
one of which, "debug", is used by the new job.
The "release" version of CanvasKit probably works on
3.1.10 which had a bugfix, but we are still on
3.1.9
- .bazelrc to see a rename of the linux-rbe config to
linux_rbe (our configs should have no dashes if
we want to specify them verbatim in our Job names).
It also imports the Skia-specified build configs
from //bazel/buildrc and supports the user-specified
//bazel/user/buildrc file if it exists.
- All other files in any order.
Change-Id: Ib954dd6045100eadcbbf4ffee0888f6fbce65fa7
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/537797
Reviewed-by: Eric Boren <borenet@google.com>
Reviewed-by: Jorge Betancourt <jmbetancourt@google.com>
This is a reland of commit ae5e846047
Original change's description:
> [graphite] Move Graphite into Skia base directories.
>
> Change-Id: Ie0fb74f3766a8b33387c145bd1151344c25808cb
> Reviewed-on: https://skia-review.googlesource.com/c/skia/+/528708
> Reviewed-by: Kevin Lubick <kjlubick@google.com>
> Reviewed-by: Greg Daniel <egdaniel@google.com>
> Commit-Queue: Jim Van Verth <jvanverth@google.com>
Change-Id: Ia575fd49206ad0b665a6a9153317e738bb321446
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/529059
Reviewed-by: Kevin Lubick <kjlubick@google.com>
Reviewed-by: Greg Daniel <egdaniel@google.com>
Commit-Queue: Jim Van Verth <jvanverth@google.com>
Change-Id: Ie0fb74f3766a8b33387c145bd1151344c25808cb
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/528708
Reviewed-by: Kevin Lubick <kjlubick@google.com>
Reviewed-by: Greg Daniel <egdaniel@google.com>
Commit-Queue: Jim Van Verth <jvanverth@google.com>
This ports the GN skia_executable [1] and adds third_party
Bazel rules for spirv_cross, translated from [2]. spirv_cross,
unlike spirv_tools, did not have pre-made Bazel rules.
This binary compiles and links with
bazelisk build //tools/skslc --config=clang
[1] ad324a31e6/BUILD.gn (L712)
[2] ad324a31e6/third_party/spirv-cross/BUILD.gn (L10)
Change-Id: I4abd51889552153fc7e64a5f7f3d9f0f597524e7
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/528044
Reviewed-by: John Stiles <johnstiles@google.com>
Commit-Queue: Kevin Lubick <kjlubick@google.com>
IWYU doesn't always understand that we want defines to come from
certain files, so we add a pragma to force it.
This also adds an extra entry to known_good_builds so I don't miss
this type of thing again when building locally.
Change-Id: I2321ea95edfc6a4506d51a011983965eb9bdf1c0
Bug: skia:13052
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/528164
Reviewed-by: John Stiles <johnstiles@google.com>
Owners-Override: Kevin Lubick <kjlubick@google.com>
This moves the Build-Debian10-BazelClang-x86_64-Release-IWYU
job from experimental to on when a file in one of the
folders that we enforce IWYU is modified (currently
for svg, sksl, and now, debugger).
Change-Id: Ia6fe1e7b30fc486db3eb081b6a64bc4c250cbf0b
Bug: skia:13052
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/525796
Reviewed-by: Derek Sollenberger <djsollen@google.com>
Commit-Queue: Kevin Lubick <kjlubick@google.com>
A new RBE worker-pool called gce_linux was created in
conjunction with this CL. See
https://docs.google.com/document/d/14xMZCKews69SSTfULhE8HDUzT5XvPwZ4CvRufEvcZ74/edit#
for some details on that.
Note: everything under bazel/rbe/gce_linux was autogenerated
and can be ignored from manual review. It basically specifies
what files are on the RBE image that are necessary for running
Bazel.
Testing it out can be done by authenticating for RBE
gcloud auth application-default login --no-browser
Then, run make -C bazel rbe_known_good_builds
to test it out.
On my 4 core laptop with an empty local cache, but a
warm remote cache, the build took <2 min instead of the
10+ minutes it would have [1].
The folder structure in //bazel/rbe is meant to let us
have multiple remote configurations there, e.g.
//bazel/rbe/gce_windows.
Suggested Review Order:
- bazel/rbe/README.md
- bazel/rbe/gce_linux_container/Dockerfile to see the
bare-bones RBE image.
- bazel/rbe/BUILD.bazel to see a custom platform defined.
It is nearly identical to the autogenerated one
in bazel/rbe/gce_linux/config/BUILD, with one extra
field to force the gce_linux pool to be used.
- .bazelrc to see the settings needed to make
--config=linux-rbe work. The naming convention was
inspired by SkCMS's setup [2], and allows us to have
some common RBE settings (i.e. config:remote) and
some specialized ones for the given host machine
(e.g. config:linux-rbe) A very important, but subtle
configuration, is on line 86 of .bazelrc where we say
to use our hermetic toolchain and not whatever C++
compiler and headers are on the host machine (aka
the RBE container).
- toolchain/build_toolchain.bzl to see some additional
dependencies needed in the toolchain (to run IWYU) which
I had installed locally but didn't realize were important.
- third_party/BUILD.bazel to see an example of how failing
to specify all files can result in something that works
locally, but fails remotely.
--execution_log_json_file=/tmp/execlog.json helped debug
these issues.
- All other files.
[1] http://go/scrcast/NjM1ODE4MDI0NzM3MTc3Nnw3ODViZmFkMi1iOA
[2] https://skia.googlesource.com/skcms/+/30c8e303800c256febb03a09fdcda7f75d119b1b/.bazelrc#20
Change-Id: Ia0a9e6a06c1a13071949ab402dc5d897df6b12e1
Bug: skia:12541
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/524359
Reviewed-by: Leandro Lovisolo <lovisolo@google.com>
This regenerates the files and fixes the harfbuzz rule so CanvasKit
compiles.
Change-Id: I2db2bddaabf793f360e8a4fa1a6a2b96222dfdf8
Bug: skia:12541
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/522816
Reviewed-by: Ben Wagner <bungeman@google.com>
PS1 regenerates the Bazel files. Use it as the base change when
comparing patchsets.
IWYU seems to do a good job of working with MyFile.cpp and
MyFile.h, but if there is just a MyHeader.h, it doesn't always
seem to throw errors if the includes aren't correct. This was
observed with include/sksl/DSL.h This might be due to the fact
that headers are not compiled on their own, so they are never
sent directly to the IWYU binary.
This change sets enforce_iwyu_on_package() on the all sksl
packages and then fixes the includes until all those checks
are happy. There were a few files that needed fixes outside
of the sksl folder. Examples include:
- src/gpu/effects/GrConvexPolyEffect.cpp
- tests/SkSLDSLTest.cpp
To really enforce this, we need to add a CI/CQ job that runs
bazel build //example:hello_world_gl --config=clang \
--sandbox_base=/dev/shm --features skia_enforce_iwyu
If that failed, a dev could make the changes described in
the logs and/or run the command locally to see those
prescribed fixes.
I had to add several entries to toolchain/IWYU_mapping.imp
in order to fix some private includes and other atypical
choices. I tried adding a rule there to allow inclusion of
SkTypes.h to make sure defines like SK_SUPPORT_GPU, but
could not get it to work for all cases, so I deferred to
using the IWYU pragma: keep (e.g. SkSLPipelineStageCodeGenerator.h)
Change-Id: I4c3e536d8e69ff7ff2d26fe61a525a6c2e80db06
Bug: skia:13052
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/522256
Reviewed-by: John Stiles <johnstiles@google.com>
Reviewed-by: Greg Daniel <egdaniel@google.com>
This will use the recently added Bazel toolchain feature
to enforce proper includes for all files in //src/svg/...
In the future, I envision a CI/CQ job that will run
bazel build with a few different configurations and the
--feature skia_enforce_iwyu on to make sure we don't
regress.
Change-Id: Ibb9f816ab626415c11bd2b9b74c503297c4b0723
Bug: skia:13052
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/521036
Reviewed-by: Ben Wagner <bungeman@google.com>
Reviewed-by: Brian Salomon <bsalomon@google.com>
Commit-Queue: Kevin Lubick <kjlubick@google.com>
PS1 regenerates the Bazel files.
It is recommended to review this CL with a diff from PS1.
Example output when a file does not pass the test:
tools/sk_app/CommandSet.h should add these lines:
#include "include/core/SkTypes.h"
#include "include/private/SkTArray.h"
#include "tools/skui/InputState.h"
#include "tools/skui/Key.h"
#include "tools/skui/ModifierKey.h"
namespace sk_app { class Window; }
tools/sk_app/CommandSet.h should remove these lines:
- #include "tools/sk_app/Window.h"
The full include-list for tools/sk_app/CommandSet.h:
#include "include/core/SkString.h"
#include "include/core/SkTypes.h"
#include "include/private/SkTArray.h"
#include "tools/skui/InputState.h"
#include "tools/skui/Key.h"
#include "tools/skui/ModifierKey.h"
#include <functional>
#include <vector>
class SkCanvas;
namespace sk_app { class Window; }
---
This makes use of Bazel's toolchain features
https://bazel.build/docs/cc-toolchain-config-reference#features
to allow us to configure compiler flags when compiling
individual files. This analysis is off by default, and can
be turned on with --features skia_enforce_iwyu. When enabled,
it will only be run for files that have opted in.
Example:
bazelisk build //example:hello_world_gl --config=clang \
--sandbox_base=/dev/shm --features skia_enforce_iwyu
There are two ways to opt files in:
- Add enforce_iwyu = True to a generated_cc_atom rule
- Add enforce_iwyu_on_package() to a BUILD.bazel file
(which enforces IWYU for all rules in that file)
Note that Bazel does not propagate features to dependencies
or dependents, so trying to enable the feature on cc_library
or cc_executable targets will only impact any files listed in
srcs or hdrs, not deps. This may be counter-intuitive when
compared to things like defines.
IWYU supports a mapping file, which we supply to help properly
handle things system headers (//toolchain/IWYU_mapping.imp)
Suggested Review Order:
- toolchain/build_toolchain.bzl to see how we get the IWYU
binaries into the toolchain
- toolchain/BUILD.bazel and toolchain/IWYU_mapping.imp
to see how the mapping file is made available for
all compile steps
- toolchain/clang_toolchain_config.bzl, where we define the
skia_enforce_iwyu feature to turn on any verification at
all and skia_opt_file_into_iwyu to enable the check for
specific files using a define.
- toolchain/clang_trampoline.sh, which is the toolchain is
configured to call instead of clang directly (see line 83
of clang_toolchain_config.bzl). This bash script used to
just forward all arguments directly onto clang. Now it
inspects them and either calls clang directly (if
it does not find the define in the arguments or we are
linking [bazel sometimes links with clang instead of ld])
or calls clang and then include-what-you-use. In all cases,
the trampoline sends the arguments to clang and IWYU
unchanged).
- //tools/sk_app/... to see enforcement enabled (and fixed)
for select files, as an example of that method.
- //experimental/bazel_test/... to see enforcement enabled
for all rules in a BUILD.bazel file.
- all other files.
Change-Id: I60a2ea9d5dc9955b6a8f166bd449de9e2b81a233
Bug: skia:13052
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/519776
Reviewed-by: Ben Wagner <bungeman@google.com>
Reviewed-by: Leandro Lovisolo <lovisolo@google.com>
Commit-Queue: Kevin Lubick <kjlubick@google.com>
PS1 regenerates BUILD.bazel files
I suggest reviewing the deltas between PS1 and the latest
PS to focus on the interesting bits.
The changes here allow for a Vulkan-only build of HelloWorld
based on sk_app. The toughest change was properly fetching
the VisualID after removing the gl calls that used to
fill that in.
There are a few changes that fix resolution of Dawn
header files, but those won't actually be built until
a follow-on CL.
Change-Id: I54fb58b5dd7ecd4313562aed401759b3eaed53c0
Bug: skia:12541
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/516999
Reviewed-by: Greg Daniel <egdaniel@google.com>
Commit-Queue: Kevin Lubick <kjlubick@google.com>
One very important, but agonizing to discover, change was
to go_repositories.bzl. Without it, we see cryptic errors like:
external/org_chromium_go_luci/cipd/api/cipd/v1/BUILD.bazel:22:17: no such package '@org_chromium_go_luci//go.chromium.org/luci/cipd/api/cipd/v1': BUILD file not found in directory 'go.chromium.org/luci/cipd/api/cipd/v1' of external repository @org_chromium_go_luci. Add a BUILD file to a directory to mark it as a package. and referenced by '@org_chromium_go_luci//cipd/api/cipd/v1:api_go_proto'
The rest of these changes are very similar to
https://skia-review.googlesource.com/c/buildbot/+/514074
which also has justification for the use of task drivers,
even in a Bazel-driven world.
All the BUILD.bazel files under infra/bots/task_drivers were
generated by Gazelle.
Note that the infra/bots/BUILD.bazel can happily build and
package up the task drivers from the infra repo. The old
build_task_drivers tasks did this too, because we have some
task drivers that are used in both repos.
Change-Id: I13c46c62bc7a6a4bfe7935b28efbfb34caabb6f2
Bug: skia:12541
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/515296
Reviewed-by: Eric Boren <borenet@google.com>
Commit-Queue: Kevin Lubick <kjlubick@google.com>
In order to extract the PNG files produced by our CanvasKit gms,
we need our JS tests to POST them to a server which can write to
disk. The easiest way to do this is to use the test_on_env
rule defined in the Skia Infra repo for exactly this purpose.
This required https://skia-review.googlesource.com/c/buildbot/+/510717
to be able to configure the binary correctly and
https://skia-review.googlesource.com/c/buildbot/+/511862, for nicer
debugging so the skia-infra dep was updated via the following commands:
$ go get go.skia.org/infra@d8a552a29e
$ go mod download
$ make -C infra/bots train
$ make -C bazel gazelle_update_repo
This caused many automated changes to infra/bots/tasks.json
The flow is:
1. User types bazelisk test :hello_world_test_with_env
2. The test_on_env rule starts gold_test_env and waits
for the file defined in $ENV_READY_FILE to be created.
3. gold_test_env starts a web server on a random port. It
writes this port number to $ENV_DIR/port. Then, it
creates $ENV_READY_FILE to signal ready.
4. test_on_env sees the ready file and then starts the
karma_test rule. (Reminder: this is a bash script
which starts karma using the Bazel-bundled chromium).
5. The karma_test rule runs the karma.bazel.js file (which
has been injected with some JS code to fill in Bazel
paths and settings) using Bazel-bundled node. This reads
in the port file and sets up a Karma proxy to redirect
/gold_rpc/report to http://localhost:PORT/report
6. The JS tests run via Karma (and do assertions via Jasmine).
Some tests, the gms, make POST requests to the proxy.
7. gold_test_env gets these POST requests writes the images
to a special Bazel folder on disk as defined by
$TEST_UNDECLARED_OUTPUTS_DIR.
8. test_on_env identifies that the tests finish (because the
karma_test script returns 0). It sends SIGINT to gold_test_env.
9. gold_test_env stops the webserver. The special Bazel folder
will zip up anything inside it and make it available for
future rules (e.g. a rule that will upload to Gold via goldctl).
Suggested Review Order:
- bazel/karma_test.bzl to see the test_on_env rule bundled into
the karma_test macro. I chose to put it there because it might
be confusing to have to define both a karma_test and test_on_env
rule in the same package but not be able to call one because it
will fail to talk to the server.
- gold_test_env.go to see how the appropriate files are written
to signal the environment is ready and the handlers are set up.
- karma.bazel.js to see how we make our own proxy given the
port from the env binary. The fact that we could not create
our own proxy with the existing karma_test rule was why the
chain ending in https://skia-review.googlesource.com/c/skia/+/508797
had to be abandoned.
- tests/*.js to see how the environment is probed via /healthz
and then used to make POST requests with data.
- Everything else.
Change-Id: I32a90def41796ca94cf187d640cfff8e262f85f6
BUG: skia:12541
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/510737
Reviewed-by: Leandro Lovisolo <lovisolo@google.com>
PS 1 is re-generating existing BUILD.bazel files
PS 2 is generating BUILD.bazel files for tests/gms
PS 3+ makes modifications to build all of the gms and tests.
It is recommended to view this CL with just a diff between
PS 2 and the end, due to the large amount of generated changes
in PS 1 and 2.
We make a filegroup for the gms and tests because they need
to be compiled as one large blob in order for the registries
to work. Maybe in the future we will break these up, but at least
for WASM/JS, the overhead of starting a browser for each new
test would likely grind things to a halt, so we just group them
all together for now. It's also the most similar to what we
currently do.
In gm/BUILD.bazel and tests/BUILD.bazel, we add a cc_library
that encapsulates all of the deps of the tests, so we can
easily include that the build. These were discovered via
trial and error, not anything automatic or systematic.
The is_skia_dev_build config_setting is very similar to the
GN equivalent from which it was based.
The list of gms and tests to skip (e.g. which are incompatible
with WASM) was determined by building the wasm bundle:
modules/canvaskit$ make bazel_gms_release
tools/run-wasm-gm-tests$ make run_local_debug
# Don't forget to click the button on the screen after the
# browser loads
This way of invoking the tests will be replace soon with
`bazel test <something>`. As such, I didn't bother fully
documenting the current way.
Suggested review order:
- modules/canvaskit/BUILD.bazel taking note that we always
use profiling-funcs to make the stacktraces human readable.
- gm/BUILD.bazel and tests/BUILD.bazel to see the lists of
gms/tests. Notice the tests are roughly partitioned because
we don't support things like vulkan/PDF in the wasm build
and we will want a way to not build certain tests for
certain configurations
- tools/* noting some of the cc_libraries added to make
dependencies easier to add when needed.
- All other files.
Change-Id: I43059cd93c28af1c4c12b93d6ebd9c46a12d381f
Bug: skia:12541
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/506256
Reviewed-by: Ben Wagner <bungeman@google.com>
Commit-Queue: Kevin Lubick <kjlubick@google.com>
PS 1 adds particles to the build
PS 2+ ports many of the options from //modules/canvaskit/compile.sh
With this CL, all the CanvasKit tests pass with both the
debug and release build.
Change-Id: Id70f0c16a087109c56949417f940849f2e3b5200
Bug: skia:12541
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/504537
Reviewed-by: Brian Osman <brianosman@google.com>
Reviewed-by: Ben Wagner <bungeman@google.com>
PS 1 regenerates existing Bazel files
PS 2 adds generated Bazel files to skottie and its dependencies,
as well as incorporating it into CanvasKit.
This changes the version of Bazel we use to 5.0.0 (recently
released).We had been using a pre-release of 6.0 because we
wanted the new features in one of the 5.0 release candidates,
but not the regression that was there (and reverted before the
full 5.0 release). I'd like to stick to the latest stable Bazel
release where possible.
Suggested Review Order:
- //modules/skottie/BUILD.bazel (this was hand written
to encapsulate the skottie library). The files in the
deps are based on skottie.gni.
- //modules/skresources/BUILD.bazel and //modules/sksg/BUILD.bazel
which expose all sources
- //third_party/file_map_for_bazel.json which ignores the
ffmpeg libraries (we won't actually build the SkVideoDecoder
stuff because HAVE_VIDEO_DECODER is not set).
- //modules/canvaskit/BUILD.bazel which makes use of the skottie
library and includes the interface skottie.js file.
- .bazelversion which changes the Bazel version used (e.g. by
Bazelisk).
- All other changes should be auto-generated or related to
deleted files.
Change-Id: Ic26f9a9dea5310f2cbd9cda7d701847924a39a22
Bug: skia:12541
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/503828
Reviewed-by: Ben Wagner <bungeman@google.com>
Reviewed-by: Jorge Betancourt <jmbetancourt@google.com>
This documents the various factory settings (I kept getting
confused as to what each was doing).
Additionally, this makes setting the factory flag bring in
the dependent code as well (like our current GN rules do).
Change-Id: I93437651b078baac04433c14c573a95982b7bc15
Bug: skia:12541
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/493396
Reviewed-by: Ben Wagner <bungeman@google.com>
bazel run //example:hello_world --config=clang
causes a window to open and draws a circle and a square.
Text to follow in a future CL.
To make this work, I had to get rid of musl and use glibc.
All the shared libraries (.so files) that were pre-built
and available for download (e.g. from https://packages.debian.org/bullseye/amd64/libgl1/download)
were compiled against glibc. When I tried to run a
program statically linked with musl and dynamically linked
against things using glibc, I got a segmentation fault
on things like calloc().
Initial attempts to use glibc had failed because it was thought
that the libc.so.6 file could only be referred to by absolute
path (and thus Bazel would not be happy about it). As it turns out,
that was simply a misconfiguration of the builtin_sysroot
parameter to cc_common.create_cc_toolchain_config_info
(see //toolchain/clang_toolchain_config.bzl). By setting that
to `external/clang_linux_amd64` and not
`external/clang_linux_amd64/usr`, the libc binary which had
been extracted to `external/clang_linux_amd64/lib/x86_64-linux-gnu`
was perfectly reachable from
`external/clang_linux_amd64/usr/usr/lib/x86_64-linux-gnu/libc.so`
To bring in the shared libraries to link against (e.g. X11, GL)
I made build_toolchain.bzl easier to modify in that we simply need
to add a debian download url and sha256 hash to a list (rather than
having to plumb this through via arguments).
Recommended Review Order:
- example/BUILD.bazel (not sure if we always want to set bare
link arguments like that or if we want to use "features" to
pass those along to the toolchain).
- tools/sk_app/BUILD.bazel to see initial cc_library for
wrapping sk_app code.
- toolchain/build_toolchain.bzl to see removal of musl and
new list of debs.
- toolchain/clang_toolchain_config.bzl (where use of the
no-canonical-prefixes was key to compilation success).
Notice also that we statically linked libc++ (I did not
have any shared libraries for it locally, so I guessed
a typical developer might not either).
- Rest of toolchain/ for trivial renames.
- bazel/Makefile to see extra docs on those targets and
a new target that compiles all the exes so far for a
quick way to test the build.
- third_party/BUILD.bazel and src/gpu/BUILD.bazel which have
non-generated changes. (all other BUILD.bazel files do).
- go.mod, which needed to update the infra repo version in
order to pick up http://review.skia.org/491736).
Change-Id: I8687bd227353040eca2dffa9465798d8bd395027
Bug: skia:12541
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/492117
Reviewed-by: Ben Wagner <bungeman@google.com>
Reviewed-by: Leandro Lovisolo <lovisolo@google.com>
Commit-Queue: Kevin Lubick <kjlubick@google.com>
- Use latest emscripten toolchain (3.1.0)
- Autogenerate the atoms and manually fix some of the file lists.
- Add a known_good_builds target to bazel/Makefile to help
check the things we expect to work with Bazel.
Change-Id: Ia5f51e7b9eb5c108386820ad59180c8f862f5a70
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/491438
Reviewed-by: Ben Wagner <bungeman@google.com>
As a follow-up to https://skia-review.googlesource.com/c/skia/+/476219,
this sketches out how we can maybe use cc_library for the things
in //modules to make sure something in //src doesn't depend on
anything in //modules, for example.
The following succeeds:
bazel build //modules/skparagraph:skparagraph --config=clang \
--shaper_backend=harfbuzz_shaper --with_icu
As does `make bazel_canvaskit_debug` in //modules/canvaskit
Suggested Review Order:
- third_party/BUILD.bazel for ICU and harfbuzz rules. Pay
special attention to the genrules used to call the python
script for turning the icu .dat file into .S or .cpp.
- bazelrc and bazel/ for new flags and defines that control
use of ICU and harfbuzz. Unlike GN, with the public_defines
that get added in automatically if icu or harfbuzz is
depended upon, we need to set the defines at the top level.
This necessity might go away if we change the atoms to
depend on //modules/skshaper, which could define that flag.
- Top level BUILD.bazel files in //modules/skparagraph,
//modules/skshaper, //modules/skunicode, //modules/canvaskit
- All other .bazel file changes are automatic.
Bug: skia:12541
Change-Id: I38a9e0a9261d7e142eeb271c2ddb23f362f91473
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/478116
Reviewed-by: Ben Wagner <bungeman@google.com>
Reviewed-by: Leandro Lovisolo <lovisolo@google.com>
To make the atomic rules a bit easier to work with, in many
of the folders, this adds in cc_library rules to group
together the sources from that folder (and subfolders
where prudent). We only needs sources because those atoms
should have their headers as deps.
One issue that was pointed out is that there is currently
no way to restrict the inclusion of certain packages,
a la, `gn check`. For example, there is no mechanism from
stopping a dev from adding
#include "modules/canvaskit/WasmCommon.h"
to something in //src/core (except circular dependencies).
We can probably address that using Bazel's visibility
rules as needed:
https://docs.bazel.build/versions/main/visibility.htmlhttps://docs.bazel.build/versions/main/be/functions.html#package_group
It is recommended to look at this CL patchset by patchset.
PS1: Update gazelle command to generate rules in more folders.
PS2: A few changes to make generation work better.
PS3: The result of running make generate in //bazel
PS4: Adding the rules to build sksllex, the simplest binary I
could find in the Skia repo.
PS5: Adding the rules to build skdiff, a more complex binary.
I tried a few approaches, but ended up gravitating back
towards the layout where we have each folder/package
group up the sources. I imagine at some point, we'll have
skdiff depend on skia_core or something, which will
have things like //src/core, //src/codecs, //src/pathops
all bundled together.
PS7: Added in the groupings of sources, similar to what we had
earlier. I liked these for readability. These helped fix
up the //:skia_core build, and by extension, the CanvasKit
build.
Change-Id: I3faa7c4e821c876b243617aacf0246efa524cbde
Bug: skia:12541
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/476219
Reviewed-by: Ben Wagner <bungeman@google.com>
Reviewed-by: Leandro Lovisolo <lovisolo@google.com>
This adds a simple go program to test the installed go
toolchain, and a Make rule to codify the arguments to
our gazelle binary, built with extensions.
I could not figure out how to get the .json file to work
with the gazelle() Bazel rule, but this works ok for now.
Bug: skia:12541
Change-Id: I5067b15c7518951aeb69559d3871799d3b5745f4
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/475716
Reviewed-by: Leandro Lovisolo <lovisolo@google.com>
