A new RBE worker-pool called gce_linux was created in
conjunction with this CL. See
https://docs.google.com/document/d/14xMZCKews69SSTfULhE8HDUzT5XvPwZ4CvRufEvcZ74/edit#
for some details on that.
Note: everything under bazel/rbe/gce_linux was autogenerated
and can be ignored from manual review. It basically specifies
what files are on the RBE image that are necessary for running
Bazel.
Testing it out can be done by authenticating for RBE
gcloud auth application-default login --no-browser
Then, run make -C bazel rbe_known_good_builds
to test it out.
On my 4 core laptop with an empty local cache, but a
warm remote cache, the build took <2 min instead of the
10+ minutes it would have [1].
The folder structure in //bazel/rbe is meant to let us
have multiple remote configurations there, e.g.
//bazel/rbe/gce_windows.
Suggested Review Order:
- bazel/rbe/README.md
- bazel/rbe/gce_linux_container/Dockerfile to see the
bare-bones RBE image.
- bazel/rbe/BUILD.bazel to see a custom platform defined.
It is nearly identical to the autogenerated one
in bazel/rbe/gce_linux/config/BUILD, with one extra
field to force the gce_linux pool to be used.
- .bazelrc to see the settings needed to make
--config=linux-rbe work. The naming convention was
inspired by SkCMS's setup [2], and allows us to have
some common RBE settings (i.e. config:remote) and
some specialized ones for the given host machine
(e.g. config:linux-rbe) A very important, but subtle
configuration, is on line 86 of .bazelrc where we say
to use our hermetic toolchain and not whatever C++
compiler and headers are on the host machine (aka
the RBE container).
- toolchain/build_toolchain.bzl to see some additional
dependencies needed in the toolchain (to run IWYU) which
I had installed locally but didn't realize were important.
- third_party/BUILD.bazel to see an example of how failing
to specify all files can result in something that works
locally, but fails remotely.
--execution_log_json_file=/tmp/execlog.json helped debug
these issues.
- All other files.
[1] http://go/scrcast/NjM1ODE4MDI0NzM3MTc3Nnw3ODViZmFkMi1iOA
[2] https://skia.googlesource.com/skcms/+/30c8e303800c256febb03a09fdcda7f75d119b1b/.bazelrc#20
Change-Id: Ia0a9e6a06c1a13071949ab402dc5d897df6b12e1
Bug: skia:12541
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/524359
Reviewed-by: Leandro Lovisolo <lovisolo@google.com>
The file generation logic that dawn [1] uses to make some
source files requires jinja2, which also requires MarkupSafe.
The GN build handles this by specifying those repos in
DEPS, checking them out at a certain git hash, and then
providing them via a command line arg [2].
We do not have to do it this way in Bazel to have reproducible
builds. This CL specifies an exact version (verified by sha256)
of those two deps and then uses a hermetic version of
Python 3.9 to run all py_binary commands.
Previously, we would rely on the system Python (and installed
libraries). That happened to work on my machine, but not on
other machines without jinja2 and MarkupSafe installed. After
this CL, it should work on machines that do not have python
even installed.
I chose the same jinja2 version used by Dawn [3], which was
2.11.3. Then I chose the newest version of MarkupSafe that
was compatible with jinja2 (2.0.1).
If we have other python scripts that need external deps, we
should be able to specify them in the py_binary that needs
them and in requirements.txt. Then, the pip_install() step
in WORKSPACE.bazel will download them and make them available.
[1] https://dawn.googlesource.com/dawn.git/+/refs/heads/main/docs/dawn/overview.md
[2] https://dawn.googlesource.com/dawn.git/+/e45ff6a4b3c2f06dade68ec0f01ddc3bfd70c282/generator/generator_lib.gni#77
[3] ee69aa00ee
Change-Id: I3d0074f3003de179400e239e00107c34f35f4901
Bug: skia:12541
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/524217
Reviewed-by: Ben Wagner <bungeman@google.com>
Reviewed-by: Leandro Lovisolo <lovisolo@google.com>
sk_app has existing support for Dawn on top of Vulkan, and
this adds support to build //example:hello_world_dawn and
get this to run on Linux.
Dawn depends on Tint and abseil-cpp. Tint further depends on
spirv_tools and spirv_headers (for writing to the SPIR-V format).
Dawn and Tint only have GN and CMake support, so we need to make
our Bazel rules for them (see //third_party/BUILD.bazel).
abseil-cpp and the SPIR-V libraries have Bazel support, so we
can just include them (see //WORKSPACE.bazel). It is important
that @spirv_headers be called that exactly because @spirv_tools
depends on it by that name.
The hand-crafted cc_library rules for Dawn and Tint were produced
by reading the appropriate GN files and using the parts necessary
for a supporting Vulkan+Linux. If we use Dawn for other backends
(e.g. WebGPU), we will need to expand the Bazel rules. One day,
we might contribute the Bazel rules to Dawn and Tint so they
can support them and avoid breaking us if new files are added.
Suggested Review Order
- bazel/common_config_settings/BUILD.bazel to see introduction
of new select-able option "has_gpu_backend" which cleans up
some of our code that is enabled for any GPU backend.
- src/*/BUILD.bazel to see has_gpu_backend rolled out.
- WORKSPACE.bazel to see DEPS declared there (using the files
in third_party/externals, which are brought in via
tools/git-sync-deps).
- third_party/BUILD.bazel which adds Dawn and Tint rules.
It may be helpful to look in third_party/externals for
the Dawn [1] and Tint [2] GN files. Especially interesting
are the Python scripts [3] Dawn uses to generate some
header and source files.
- All other files.
[1] https://dawn.googlesource.com/dawn/+/d9f22ce0346b222759d5510be3d1cd93caa5ab86/src/dawn/native/BUILD.gn#183
[2] https://dawn.googlesource.com/tint/+/453d5ae84ec30ab51ac592c13d472412ae8b5fc9/src/tint/BUILD.gn#174
[3] https://dawn.googlesource.com/dawn/+/d9f22ce0346b222759d5510be3d1cd93caa5ab86/generator/dawn_json_generator.py#774
Change-Id: Ied5b162045d8e841b9666457f0158457e2b078d4
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/516996
Reviewed-by: Ben Wagner <bungeman@google.com>
Reviewed-by: Greg Daniel <egdaniel@google.com>
Commit-Queue: Kevin Lubick <kjlubick@google.com>
One very important, but agonizing to discover, change was
to go_repositories.bzl. Without it, we see cryptic errors like:
external/org_chromium_go_luci/cipd/api/cipd/v1/BUILD.bazel:22:17: no such package '@org_chromium_go_luci//go.chromium.org/luci/cipd/api/cipd/v1': BUILD file not found in directory 'go.chromium.org/luci/cipd/api/cipd/v1' of external repository @org_chromium_go_luci. Add a BUILD file to a directory to mark it as a package. and referenced by '@org_chromium_go_luci//cipd/api/cipd/v1:api_go_proto'
The rest of these changes are very similar to
https://skia-review.googlesource.com/c/buildbot/+/514074
which also has justification for the use of task drivers,
even in a Bazel-driven world.
All the BUILD.bazel files under infra/bots/task_drivers were
generated by Gazelle.
Note that the infra/bots/BUILD.bazel can happily build and
package up the task drivers from the infra repo. The old
build_task_drivers tasks did this too, because we have some
task drivers that are used in both repos.
Change-Id: I13c46c62bc7a6a4bfe7935b28efbfb34caabb6f2
Bug: skia:12541
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/515296
Reviewed-by: Eric Boren <borenet@google.com>
Commit-Queue: Kevin Lubick <kjlubick@google.com>
Run the tests in headless mode and output the logs
bazel test :hello_world --test_output=all
Start up a visible web browser with the karma test driver
(need to go to Debug tab to actually run tests)
bazel run :hello_world
Suggested review order
- package.json to see the karma dependencies to run
jasmine tests on chrome and firefox.
- WORKSPACE.bazel to see how the packages listed in
package.json and package-lock.json are downloaded
into the Bazel sandbox/cache via the npm_install rule.
As mentioned in the package.json comment, the version
of build_bazel_rules_nodejs which emscripten uses [1]
is 4.4.1 and if we tried to install it ourselves, that
installation will be ignored. We also bring in hermetic
browsers via io_bazel_rules_webtesting.
- bazel/karma_test.bzl which defines a new rule _karma_test
and a macro karma_test which joins the new rule with
an existing web_test rule to run it on a hermetic browser
which Bazel downloads. This rule takes heavy inspiration
from @bazel/concatjs [2], but is much simpler and lets us
configure more things (e.g. proxies, so we can work with
test_on_env).
- karma.bazel.js, which is a pretty ordinary looking karma
configuration file [2] with effectively a JS macro
BAZEL_APPLY_SETTINGS. JS doesn't have a preprocessor or
actual macros, but this string will be replaced by the
JS code in karma_test.bzl which will set correct filepaths
for Bazel content.
- All other files.
[1] c33c7be17f/bazel/deps.bzl (L10)
[2] 700b7a3c5f/packages/concatjs/web_test/karma_web_test.bzl (L318)
[3] http://karma-runner.github.io/6.3/config/configuration-file.html
Change-Id: Id64c0a86d6be37d627762cef0beaaf23ad390ac1
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/509717
Reviewed-by: Leandro Lovisolo <lovisolo@google.com>
PS1 regenerates the BUILD.bazel files
This allows us to use closure to minify the JS in canvaskit.js
Change-Id: Ib8326d2e3a19cd2168b740b6946f9165a2810133
Bug: skia:12541
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/509177
Reviewed-by: Ben Wagner <bungeman@google.com>
- We always download from https
- All external assets have a primary and a mirror URL.
- We prioritize the sources as follows:
bazel mirror, github/original source, our mirror
- There is a way (see build_toolchain) to test the sources
from the mirrors (done before CL submission).
This adds a utility to upload files to the mirror in a
consistent, scripted way. It includes a way to copy in
parts of our bazel files (e.g. debs_to_install from
toolchain/build_toolchain.bzl) to update many things
at once.
Our Bazel mirror (gs://skia-world-readable/bazel)
is a Content Addressable Storage system, where the
file name is based on the sha256sum of the contents
(the same hash that Bazel uses). All files in it should
be publicly accessible.
Change-Id: Ida8b8e07d27a0a557bc49467ebbc86c806cabbd3
Bug: skia:12541
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/494478
Reviewed-by: Joe Gregorio <jcgregorio@google.com>
bazel run //example:hello_world --config=clang
causes a window to open and draws a circle and a square.
Text to follow in a future CL.
To make this work, I had to get rid of musl and use glibc.
All the shared libraries (.so files) that were pre-built
and available for download (e.g. from https://packages.debian.org/bullseye/amd64/libgl1/download)
were compiled against glibc. When I tried to run a
program statically linked with musl and dynamically linked
against things using glibc, I got a segmentation fault
on things like calloc().
Initial attempts to use glibc had failed because it was thought
that the libc.so.6 file could only be referred to by absolute
path (and thus Bazel would not be happy about it). As it turns out,
that was simply a misconfiguration of the builtin_sysroot
parameter to cc_common.create_cc_toolchain_config_info
(see //toolchain/clang_toolchain_config.bzl). By setting that
to `external/clang_linux_amd64` and not
`external/clang_linux_amd64/usr`, the libc binary which had
been extracted to `external/clang_linux_amd64/lib/x86_64-linux-gnu`
was perfectly reachable from
`external/clang_linux_amd64/usr/usr/lib/x86_64-linux-gnu/libc.so`
To bring in the shared libraries to link against (e.g. X11, GL)
I made build_toolchain.bzl easier to modify in that we simply need
to add a debian download url and sha256 hash to a list (rather than
having to plumb this through via arguments).
Recommended Review Order:
- example/BUILD.bazel (not sure if we always want to set bare
link arguments like that or if we want to use "features" to
pass those along to the toolchain).
- tools/sk_app/BUILD.bazel to see initial cc_library for
wrapping sk_app code.
- toolchain/build_toolchain.bzl to see removal of musl and
new list of debs.
- toolchain/clang_toolchain_config.bzl (where use of the
no-canonical-prefixes was key to compilation success).
Notice also that we statically linked libc++ (I did not
have any shared libraries for it locally, so I guessed
a typical developer might not either).
- Rest of toolchain/ for trivial renames.
- bazel/Makefile to see extra docs on those targets and
a new target that compiles all the exes so far for a
quick way to test the build.
- third_party/BUILD.bazel and src/gpu/BUILD.bazel which have
non-generated changes. (all other BUILD.bazel files do).
- go.mod, which needed to update the infra repo version in
order to pick up http://review.skia.org/491736).
Change-Id: I8687bd227353040eca2dffa9465798d8bd395027
Bug: skia:12541
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/492117
Reviewed-by: Ben Wagner <bungeman@google.com>
Reviewed-by: Leandro Lovisolo <lovisolo@google.com>
Commit-Queue: Kevin Lubick <kjlubick@google.com>
- Use latest emscripten toolchain (3.1.0)
- Autogenerate the atoms and manually fix some of the file lists.
- Add a known_good_builds target to bazel/Makefile to help
check the things we expect to work with Bazel.
Change-Id: Ia5f51e7b9eb5c108386820ad59180c8f862f5a70
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/491438
Reviewed-by: Ben Wagner <bungeman@google.com>
This adds a simple go program to test the installed go
toolchain, and a Make rule to codify the arguments to
our gazelle binary, built with extensions.
I could not figure out how to get the .json file to work
with the gazelle() Bazel rule, but this works ok for now.
Bug: skia:12541
Change-Id: I5067b15c7518951aeb69559d3871799d3b5745f4
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/475716
Reviewed-by: Leandro Lovisolo <lovisolo@google.com>
This re-works src/ports/BUILD.bazel to work like our other
BUILD files, i.e. one rule "srcs" that brings in the necessary
private filegroups.
To work around an abort with LLVM [1], we have to go back to an
earlier version of emscripten (temporarily?).
Future work should look at using transitions [2] to allow various
executables (e.g. CanvasKit, DM) to set their own set of Bazel
flags, w/o the build invokers having to specify them.
These transitions might be able to handle more complex cases
that we currently use if statements in GN to deal with.
The Freetype build rule was created by taking the BUILD.gn
rule, adding in all the sources listed there and then playing
compile-whack-a-mole to add in all the headers and included
.c files.
Suggested Review Order:
- third_party/BUILD.bazel to see freetype build rules
- bazel/common_config_settings/ to see treatment of fontmgr
like codecs (many possible) and fontmgr_factory (only one).
- src/ports/BUILD.bazel
- BUILD.bazel
- modules/canvaskit/BUILD.bazel. Take note of the gen_rule that
calls tools/embed_resources.py to produce the .cpp file
containing the embedded font data.
- Everything else.
[1] https://github.com/emscripten-core/emscripten/issues/15528
[2] https://github.com/bazelbuild/examples/tree/main/rules/starlark_configurations/cc_binary_selectable_copts
Bug: skia:12541
Change-Id: I08dab82a901d80507007b354ca20cbfad2c2388f
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/471636
Reviewed-by: Ben Wagner <bungeman@google.com>
Reviewed-by: Leandro Lovisolo <lovisolo@google.com>
These rules can be used to build our GMs on WASM+WebGL and
libskia.a with just the CPU backend (and most other features
turned off).
This can be done with the following commands:
- bazel build //modules/canvaskit:gm-bindings-wasm --gpu_backend=gl_backend --with_gl_standard=webgl_standard
- bazel build :skia-core --config clang
This pivots slightly from http://review.skia.org/463517
by using config_settings [1] instead of platforms for
the optional features that we control. This pivot was
suggested in [2]
We have BUILD.bazel files in many of the subdirectories
that specify filegroups for the appropriate files. In
an effort to make //BUILD.bazel more readable, it is
the responsibility of these subfolders to deal with
conditionally including certain .h or .cpp files.
This is done using select statements and config_settings
or platform constraints as necessary.
For example, src/gpu/BUILD.bazel will different private
filegroups for each of the supported gpu backends [3]
and a more-visible filegroup called "srcs" that has
the right selection of the private files to be used
for compilation.
An effort has been made to avoid using glob() in our
BUILD.bazel files. These file lists were made by using
`ls -1` and some regex to add in quotes. We might want
to make a helper script to assist with that, if necessary.
To specify which options we have, the settings in
//bazel/common_config_settings/BUILD.bazel have been
redesigned. They make use of a macro `string_flag_with_values`
that removes the boilerplate. Patchset 36 shows what the
file looks like w/o the macro.
The top level BUILD.bazel file will still need to use
some logic to handle defines, because local_defines is
a list of strings, not a list of labels [4].
Suggested Review Order:
- WORKSPACE.bazel to see the new dependencies on the
emsdk toolchain and bazel_skylib
- bazel/common_config_settings/* to see the few settings
defined (we have more to define, see BUILD.gn and
//gn/skia.gni for ideas)
- BUILD.bazel to see the "skia-core" cc_library rule.
See also "gms" and "tests"
- modules/canvaskit/BUILD.bazel to see the use of
the emscripten "wasm_cc_binary" rule, which depends
on the "skia-core", "gms", and "tests" rule. Note that
it only builds some of the gms as a proof of concept.
- The other BUILD.bazel files. Some of these are not
platform or feature dependent (e.g. pathops). Others
are (e.g. gpu).
- All other files.
[1] https://docs.bazel.build/versions/4.2.1/skylark/config.html#user-defined-build-settings
[2] https://github.com/emscripten-core/emsdk/pull/920
[3] In this CL, that's just the webgl one.
[4] https://docs.bazel.build/versions/main/be/c-cpp.html#cc_library.local_defines
Change-Id: Ieecf9c106d5e3a6ae97d13d66be06b4b3c207089
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/458637
Reviewed-by: Ben Wagner <bungeman@google.com>
Reviewed-by: Leandro Lovisolo <lovisolo@google.com>
Owners-Override: Kevin Lubick <kjlubick@google.com>
This can successfully build a C library:
bazel build --config=clang //third_party:libpng
This can build and run a statically-linked executable:
bazel test --config=clang //:bazel_test
For more verbose compile and linking output, add the
`--features diagnostic`
flag to a Bazel command (see _make_diagnostic_flags() in
toolchain/clang_toolchain_config.bzl. Similarly, a
`--features print_search_dirs` can be used to show where
clang is looking for libraries etc to link against.
These features are made available for easier debugging.
Suggested review order:
- Read https://docs.bazel.build/versions/4.2.1/tutorial/cc-toolchain-config.html
if unfamiliar with setting up C++ toolchains in Bazel
- .bazelrc and WORKSPACE.bazel that configure use and download
of the toolchain (Clang 13, musl 1.2.2)
- toolchain/build_toolchain.bzl which downloads and assembles
the toolchain (w/o installing anything on the host machine)
- toolchain/BUILD.bazel and toolchain/*trampoline.sh to see
the setup of the toolchain rules.
- toolchain/clang_toolchain_config.bzl to see the configuration
of the toolchain. Pay special attention to the various
command line flags that are set.
- See that tools/bazel_test.cc has made a new home in
experimental/bazel_test/bazel_test.cpp, with a companion
BUILD.bazel. Note the addition of some function calls
that test use of the C++ standard library.
The number being used to test the PNG library is the latest
and greatest that verifies we are compiling the one brought
in via DEPS (and not a local one).
- third_party/* to see how png (and its dependent zlib) have
been built. Pay special attention to the musl_compat hack
to fix static linking (any idea what the real cause is?)
- //BUILD.bazel to see definition of the bazel_test executable.
Change-Id: I7b0922d0d45cb9be8df2fd5fa5a1f48492654d5f
Bug: skia:12541
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/461178
Reviewed-by: Ben Wagner <bungeman@google.com>
Reviewed-by: Leandro Lovisolo <lovisolo@google.com>
A few first steps toward a Bazel build.
To try it out, I think just
$ bazel test ...
I added third_party to .bazelignore to prevent Bazel from looking there.
It can handle external dependencies itself, so no need to poke into what
we sync from DEPS. Some of those have Bazel configs and we don't want
to be building them yet.
I've started by with libpng using new_git_repository(), mostly because
it's small, with a mildly complex build, and needs dependencies of its
own, zlib. Mysteriously zlib is built-in to Bazel, so that was easy.
Next up is probably a dependency that does support Bazel, using
git_repository(). That should make sure we can handle the full mix.
Change-Id: I5775a1b254d341b9a90630aa1cc433a24167f2fd
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/316636
Commit-Queue: Mike Klein <mtklein@google.com>
Reviewed-by: Joe Gregorio <jcgregorio@google.com>
