In order to extract the PNG files produced by our CanvasKit gms,
we need our JS tests to POST them to a server which can write to
disk. The easiest way to do this is to use the test_on_env
rule defined in the Skia Infra repo for exactly this purpose.
This required https://skia-review.googlesource.com/c/buildbot/+/510717
to be able to configure the binary correctly and
https://skia-review.googlesource.com/c/buildbot/+/511862, for nicer
debugging so the skia-infra dep was updated via the following commands:
$ go get go.skia.org/infra@d8a552a29e
$ go mod download
$ make -C infra/bots train
$ make -C bazel gazelle_update_repo
This caused many automated changes to infra/bots/tasks.json
The flow is:
1. User types bazelisk test :hello_world_test_with_env
2. The test_on_env rule starts gold_test_env and waits
for the file defined in $ENV_READY_FILE to be created.
3. gold_test_env starts a web server on a random port. It
writes this port number to $ENV_DIR/port. Then, it
creates $ENV_READY_FILE to signal ready.
4. test_on_env sees the ready file and then starts the
karma_test rule. (Reminder: this is a bash script
which starts karma using the Bazel-bundled chromium).
5. The karma_test rule runs the karma.bazel.js file (which
has been injected with some JS code to fill in Bazel
paths and settings) using Bazel-bundled node. This reads
in the port file and sets up a Karma proxy to redirect
/gold_rpc/report to http://localhost:PORT/report
6. The JS tests run via Karma (and do assertions via Jasmine).
Some tests, the gms, make POST requests to the proxy.
7. gold_test_env gets these POST requests writes the images
to a special Bazel folder on disk as defined by
$TEST_UNDECLARED_OUTPUTS_DIR.
8. test_on_env identifies that the tests finish (because the
karma_test script returns 0). It sends SIGINT to gold_test_env.
9. gold_test_env stops the webserver. The special Bazel folder
will zip up anything inside it and make it available for
future rules (e.g. a rule that will upload to Gold via goldctl).
Suggested Review Order:
- bazel/karma_test.bzl to see the test_on_env rule bundled into
the karma_test macro. I chose to put it there because it might
be confusing to have to define both a karma_test and test_on_env
rule in the same package but not be able to call one because it
will fail to talk to the server.
- gold_test_env.go to see how the appropriate files are written
to signal the environment is ready and the handlers are set up.
- karma.bazel.js to see how we make our own proxy given the
port from the env binary. The fact that we could not create
our own proxy with the existing karma_test rule was why the
chain ending in https://skia-review.googlesource.com/c/skia/+/508797
had to be abandoned.
- tests/*.js to see how the environment is probed via /healthz
and then used to make POST requests with data.
- Everything else.
Change-Id: I32a90def41796ca94cf187d640cfff8e262f85f6
BUG: skia:12541
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/510737
Reviewed-by: Leandro Lovisolo <lovisolo@google.com>
PS1 is the automatic regenerated changes.
PS2-3 adds an #ifdef guard to the addToKey method on shaders.
The SkShaderCodeDictionary class helps generate SkSL and is
only necessary when we are building with SkSL (gpu builds and
cpu builds with SkVM).
Suggested Review order:
- Use Gerrit to diff PS 1 and the last PS
- src/core/BUILD.bazel adds some sources to the "only
necessary if sksl is enabled" bucket
- All the .cpp and .h files to see the #ifdef is added
correctly.
Change-Id: I4d4ce61a4957ef1e0840204acff08ce7e616f9cb
Bug: skia:12541
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/512157
Reviewed-by: Michael Ludwig <michaelludwig@google.com>
Commit-Queue: Kevin Lubick <kjlubick@google.com>
PS 1 is re-generating existing BUILD.bazel files
PS 2 is generating BUILD.bazel files for tests/gms
PS 3+ makes modifications to build all of the gms and tests.
It is recommended to view this CL with just a diff between
PS 2 and the end, due to the large amount of generated changes
in PS 1 and 2.
We make a filegroup for the gms and tests because they need
to be compiled as one large blob in order for the registries
to work. Maybe in the future we will break these up, but at least
for WASM/JS, the overhead of starting a browser for each new
test would likely grind things to a halt, so we just group them
all together for now. It's also the most similar to what we
currently do.
In gm/BUILD.bazel and tests/BUILD.bazel, we add a cc_library
that encapsulates all of the deps of the tests, so we can
easily include that the build. These were discovered via
trial and error, not anything automatic or systematic.
The is_skia_dev_build config_setting is very similar to the
GN equivalent from which it was based.
The list of gms and tests to skip (e.g. which are incompatible
with WASM) was determined by building the wasm bundle:
modules/canvaskit$ make bazel_gms_release
tools/run-wasm-gm-tests$ make run_local_debug
# Don't forget to click the button on the screen after the
# browser loads
This way of invoking the tests will be replace soon with
`bazel test <something>`. As such, I didn't bother fully
documenting the current way.
Suggested review order:
- modules/canvaskit/BUILD.bazel taking note that we always
use profiling-funcs to make the stacktraces human readable.
- gm/BUILD.bazel and tests/BUILD.bazel to see the lists of
gms/tests. Notice the tests are roughly partitioned because
we don't support things like vulkan/PDF in the wasm build
and we will want a way to not build certain tests for
certain configurations
- tools/* noting some of the cc_libraries added to make
dependencies easier to add when needed.
- All other files.
Change-Id: I43059cd93c28af1c4c12b93d6ebd9c46a12d381f
Bug: skia:12541
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/506256
Reviewed-by: Ben Wagner <bungeman@google.com>
Commit-Queue: Kevin Lubick <kjlubick@google.com>
PS 1 regenerates existing Bazel files
PS 2 adds generated Bazel files to skottie and its dependencies,
as well as incorporating it into CanvasKit.
This changes the version of Bazel we use to 5.0.0 (recently
released).We had been using a pre-release of 6.0 because we
wanted the new features in one of the 5.0 release candidates,
but not the regression that was there (and reverted before the
full 5.0 release). I'd like to stick to the latest stable Bazel
release where possible.
Suggested Review Order:
- //modules/skottie/BUILD.bazel (this was hand written
to encapsulate the skottie library). The files in the
deps are based on skottie.gni.
- //modules/skresources/BUILD.bazel and //modules/sksg/BUILD.bazel
which expose all sources
- //third_party/file_map_for_bazel.json which ignores the
ffmpeg libraries (we won't actually build the SkVideoDecoder
stuff because HAVE_VIDEO_DECODER is not set).
- //modules/canvaskit/BUILD.bazel which makes use of the skottie
library and includes the interface skottie.js file.
- .bazelversion which changes the Bazel version used (e.g. by
Bazelisk).
- All other changes should be auto-generated or related to
deleted files.
Change-Id: Ic26f9a9dea5310f2cbd9cda7d701847924a39a22
Bug: skia:12541
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/503828
Reviewed-by: Ben Wagner <bungeman@google.com>
Reviewed-by: Jorge Betancourt <jmbetancourt@google.com>
See toolchain/clang_toolchain_config.bzl for the c++17 switch.
Most of the other changes were automatically generated
(with the exception of //third_party).
Change-Id: I8c0f4b29b5967da3f48b17eb298a7e92156277ac
Bug: skia:12541
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/502407
Reviewed-by: John Stiles <johnstiles@google.com>
Reviewed-by: Ben Wagner <bungeman@google.com>
- We always download from https
- All external assets have a primary and a mirror URL.
- We prioritize the sources as follows:
bazel mirror, github/original source, our mirror
- There is a way (see build_toolchain) to test the sources
from the mirrors (done before CL submission).
This adds a utility to upload files to the mirror in a
consistent, scripted way. It includes a way to copy in
parts of our bazel files (e.g. debs_to_install from
toolchain/build_toolchain.bzl) to update many things
at once.
Our Bazel mirror (gs://skia-world-readable/bazel)
is a Content Addressable Storage system, where the
file name is based on the sha256sum of the contents
(the same hash that Bazel uses). All files in it should
be publicly accessible.
Change-Id: Ida8b8e07d27a0a557bc49467ebbc86c806cabbd3
Bug: skia:12541
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/494478
Reviewed-by: Joe Gregorio <jcgregorio@google.com>
- Use latest emscripten toolchain (3.1.0)
- Autogenerate the atoms and manually fix some of the file lists.
- Add a known_good_builds target to bazel/Makefile to help
check the things we expect to work with Bazel.
Change-Id: Ia5f51e7b9eb5c108386820ad59180c8f862f5a70
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/491438
Reviewed-by: Ben Wagner <bungeman@google.com>
This uses the gazelle extension from
https://skia-review.googlesource.com/c/buildbot/+/473357
Review Tips:
- Ignore any changes to .h or .cpp files. Those have been
pulled out into their own CLs.
- Start with bazel/macros.bzl.
- Read the CL with the generation code, if you haven't already.
- Look at third_party/file_map_for_bazel.json.
- See experimental/bazel_test for an idea of how a cc_binary
would be made.
- Spot check one or two of the BUILD.bazel files.
This CL generates the "atomic" rules for src/, include/ and
modules/skshaper, as a starting point.
`bazel build --config clang //include/...` works
`bazel build --config clang //src/...` starts compiling,
(which verifies that the BUILD.bazel files are all valid),
but runs into errors because not all third_party deps have
been resolved, and there are some files missing from the
toolchain still (e.g. EGL headers).
Change-Id: Ib7e0fb0efdb9f08655f06cbc56e9bb4cf416294b
Bug: skia:12541
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/474240
Reviewed-by: Ben Wagner <bungeman@google.com>
Reviewed-by: Leandro Lovisolo <lovisolo@google.com>
This ports the third_party BUILD.gn files related to codecs
(with a best-effort on arm/SIMD stuff). This includes:
- libpng
- libjpeg-turbo
- libwebp
- wuffs (gif)
- libgifcodec
- dng_sdk and piex (raw codec)
This expands the string_flag_with_values macro to allow
multiple values to be set at once. This was added in Bazel 5.0.0,
however the latest pre-release version of that has a bug [1]
which slows down compilation dramatically. This was fixed at
ToT, but not released. As a result, I started using the Bazel
6 pre-release (via bazelisk).
The macro select_multi makes writing select() where multiple
elements could be on possible/easier.
One can try compiling certain codecs by running:
bazel build :skia-core --config clang --include_codec=raw_codec --include_codec=png_codec
Suggested Review Order:
- bazel/macros.bzl
- bazel/common_config_settings/defs.bzl and its BUILD.bazel
to see how the codec options are defined.
- BUILD.bazel to see how the codec settings are used.
- src/codec/BUILD.bazel to see the inclusion of Skia files to
deal with specific codecs.
- third_party/BUILD.bazel (while referencing the corresponding
BUILD.gn files, such as third_party/libwebp/BUILD.gn)
- Everything else.
Change-Id: I797375a35fa345d9835e7b2a2ab23371c45953c3
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/469456
Reviewed-by: Leandro Lovisolo <lovisolo@google.com>
Reviewed-by: Ben Wagner <bungeman@google.com>
Reviewed-by: Leon Scroggins <scroggo@google.com>
These rules can be used to build our GMs on WASM+WebGL and
libskia.a with just the CPU backend (and most other features
turned off).
This can be done with the following commands:
- bazel build //modules/canvaskit:gm-bindings-wasm --gpu_backend=gl_backend --with_gl_standard=webgl_standard
- bazel build :skia-core --config clang
This pivots slightly from http://review.skia.org/463517
by using config_settings [1] instead of platforms for
the optional features that we control. This pivot was
suggested in [2]
We have BUILD.bazel files in many of the subdirectories
that specify filegroups for the appropriate files. In
an effort to make //BUILD.bazel more readable, it is
the responsibility of these subfolders to deal with
conditionally including certain .h or .cpp files.
This is done using select statements and config_settings
or platform constraints as necessary.
For example, src/gpu/BUILD.bazel will different private
filegroups for each of the supported gpu backends [3]
and a more-visible filegroup called "srcs" that has
the right selection of the private files to be used
for compilation.
An effort has been made to avoid using glob() in our
BUILD.bazel files. These file lists were made by using
`ls -1` and some regex to add in quotes. We might want
to make a helper script to assist with that, if necessary.
To specify which options we have, the settings in
//bazel/common_config_settings/BUILD.bazel have been
redesigned. They make use of a macro `string_flag_with_values`
that removes the boilerplate. Patchset 36 shows what the
file looks like w/o the macro.
The top level BUILD.bazel file will still need to use
some logic to handle defines, because local_defines is
a list of strings, not a list of labels [4].
Suggested Review Order:
- WORKSPACE.bazel to see the new dependencies on the
emsdk toolchain and bazel_skylib
- bazel/common_config_settings/* to see the few settings
defined (we have more to define, see BUILD.gn and
//gn/skia.gni for ideas)
- BUILD.bazel to see the "skia-core" cc_library rule.
See also "gms" and "tests"
- modules/canvaskit/BUILD.bazel to see the use of
the emscripten "wasm_cc_binary" rule, which depends
on the "skia-core", "gms", and "tests" rule. Note that
it only builds some of the gms as a proof of concept.
- The other BUILD.bazel files. Some of these are not
platform or feature dependent (e.g. pathops). Others
are (e.g. gpu).
- All other files.
[1] https://docs.bazel.build/versions/4.2.1/skylark/config.html#user-defined-build-settings
[2] https://github.com/emscripten-core/emsdk/pull/920
[3] In this CL, that's just the webgl one.
[4] https://docs.bazel.build/versions/main/be/c-cpp.html#cc_library.local_defines
Change-Id: Ieecf9c106d5e3a6ae97d13d66be06b4b3c207089
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/458637
Reviewed-by: Ben Wagner <bungeman@google.com>
Reviewed-by: Leandro Lovisolo <lovisolo@google.com>
Owners-Override: Kevin Lubick <kjlubick@google.com>
