This moves the Build-Debian10-BazelClang-x86_64-Release-IWYU
job from experimental to on when a file in one of the
folders that we enforce IWYU is modified (currently
for svg, sksl, and now, debugger).
Change-Id: Ia6fe1e7b30fc486db3eb081b6a64bc4c250cbf0b
Bug: skia:13052
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/525796
Reviewed-by: Derek Sollenberger <djsollen@google.com>
Commit-Queue: Kevin Lubick <kjlubick@google.com>
Because these tasks use RBE, the machines they run on do not
need to be as powerful. In practice, we are seeing a lot of
the build steps be a hit on the remote-cache, so we don't need
the number of Bazel jobs to be as high, so I've arbitrarily
set it to be 100. We can revisit this later if we notice
things are slow.
To facilitate this change, I had to add cloud-platform scope
to all our GCE VMs. There is a script in the infra repo [1]
that helped with this:
go run ./scripts/add_gce_scopes/add_gce_scopes.go \
--zone us-central1-c --project skia-swarming-bots \
--scope https://www.googleapis.com/auth/cloud-platform \
--instance skia-e-gce-100,skia-e-gce-101,...
[1] b103ea24f5/scripts/add_gce_scopes/add_gce_scopes.go
No-Try: true
Change-Id: I7f1e7b1e9e4a22f5383cf9ce1c8c0350e62b5283
Bug: skia:12541
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/525577
Reviewed-by: Joe Gregorio <jcgregorio@google.com>
Commit-Queue: Kevin Lubick <kjlubick@google.com>
The root disk on our GCE VMs typically only have 15GB
and have a much larger disk attached to them.
We want the Bazel cache to be on this larger disk so
we don't run out of disk space as often.
BuildTaskDrivers was not doing that, but the task
drivers which use Bazel are.
Change-Id: I0f797188576707341972a1db7418e8916633333c
Bug: skia:12541
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/525456
Reviewed-by: Ravi Mistry <rmistry@google.com>
The toolchain now uses extract_ar from
https://skia-review.googlesource.com/c/buildbot/+/524764
which is a static executable to extract the .deb files.
This was necessary because the llvm-ar that had previously
been used requires glibc 2.31+ to run, but our Debian10
machines on Swarming have an older version (2.28).
A longer-term fix is to have Bazel support .ar files,
which I plan to attempt to contribute this week.
The RBE task will be added as an experimental CQ job, to
see how it handles the load of running often. With the
remote execution cache, I hope it performs well, once
the toolchains are cached on both the Swarming
machines and in the RBE workers.
Note: We had to add several files to the CAS spec
(see compile_cas.go) which are required for Bazel to work.
Change-Id: Ie70c70d5f33768c957760f9eeb7835025109b487
Bug: skia:12541
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/524759
Reviewed-by: Joe Gregorio <jcgregorio@google.com>
One very important, but agonizing to discover, change was
to go_repositories.bzl. Without it, we see cryptic errors like:
external/org_chromium_go_luci/cipd/api/cipd/v1/BUILD.bazel:22:17: no such package '@org_chromium_go_luci//go.chromium.org/luci/cipd/api/cipd/v1': BUILD file not found in directory 'go.chromium.org/luci/cipd/api/cipd/v1' of external repository @org_chromium_go_luci. Add a BUILD file to a directory to mark it as a package. and referenced by '@org_chromium_go_luci//cipd/api/cipd/v1:api_go_proto'
The rest of these changes are very similar to
https://skia-review.googlesource.com/c/buildbot/+/514074
which also has justification for the use of task drivers,
even in a Bazel-driven world.
All the BUILD.bazel files under infra/bots/task_drivers were
generated by Gazelle.
Note that the infra/bots/BUILD.bazel can happily build and
package up the task drivers from the infra repo. The old
build_task_drivers tasks did this too, because we have some
task drivers that are used in both repos.
Change-Id: I13c46c62bc7a6a4bfe7935b28efbfb34caabb6f2
Bug: skia:12541
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/515296
Reviewed-by: Eric Boren <borenet@google.com>
Commit-Queue: Kevin Lubick <kjlubick@google.com>
Change-Id: I1bef9f9c8db0147d8cfb60354aa519cc3ca61b43
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/507319
Reviewed-by: Michael Ludwig <michaelludwig@google.com>
Commit-Queue: Brian Osman <brianosman@google.com>
This CL adds the task driver and plumbing necessary to run CI tasks that upload build statistics to https://codesize.skia.org.
For now, a single CodeSize-dm-Debian10-Clang-x86_64-Release task is defined, which as the name suggests:
- Depends on Build-Debian10-Clang-x86_64-Release.
- Takes the "dm" binary built by the aforementioned job.
- Runs Bloaty against said binary.
- Uploads the Bloaty output, along with a JSON file with metadata, to the GCS bucket where https://codesize.skia.org gets its data from.
If one wishes to upload codesize statistics for another binary, the steps are as follows:
- Identify the target binary, for example "fm".
- Identify a compile task that builds said binary with the desired settings, for example "Build-Debian10-Clang-x86_64-Debug".
- Add a new job in //infra/bots/jobs.json named "CodeSize-<binary>-<compile task>", where <compile task> is the name of the compile task without the "Build-" prefix, for example "CodeSize-fm-Debian10-Clang-x86_64-Debug".
- Run "make train" from the //infra/bots directory.
Eventually the codesize.skia.org webserver will automatically pick up the output of any such CodeSize-* task and show it in the UI, with no additional steps needed.
One caveat is that the binary file name, and therefore the "<binary>" part of the task name, cannot contain dashes. This is due to how //infra/bots/gen_tasks_logic.go works. See comments in said Go file for ideas on how to work around this if we ever need to.
Bug: skia:12151
Change-Id: If406944ca7660c4dd15c8e6b8f34e48c65cbbe2f
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/502788
Reviewed-by: Kevin Lubick <kjlubick@google.com>
Reviewed-by: Eric Boren <borenet@google.com>
Commit-Queue: Leandro Lovisolo <lovisolo@google.com>
This required an update to emscripten, due to there being
a few bug fixes regarding the non-dynamic code and the
Closure compiler/minifier.
Change-Id: Icc922bd98cdd52a6923a9367da3747dac2b897b3
Bug: skia:12795
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/492916
Reviewed-by: Nathaniel Nifong <nifong@google.com>
* Using docker.New instead of docker.Login which needed configDir passed around.
* Also cleaned up unneeded references to infraCommonEnv and infraCommonBuildArgs.
Bug: skia:12812
Change-Id: Ie60ad77d75a5a17f9350e0028afb8c54bf4c63cb
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/494581
Reviewed-by: Kevin Lubick <kjlubick@google.com>
Commit-Queue: Ravi Mistry <rmistry@google.com>
* Add support for fiddler's bazel build in the existing push_apps_from_skia_image task driver. After api is migrated to bazel it will be cleaned up.
* Cleanup push_bazel_apps_from_wasm_image.go by using utils from buildbot.git
* Make the two autodeployment bots be triggered only on the main branch.
Bug: skia:12788
Change-Id: Iaf2adaa5ba58a1992691bbc024558a0d7392f41d
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/493037
Reviewed-by: Kevin Lubick <kjlubick@google.com>
Commit-Queue: Ravi Mistry <rmistry@google.com>
Failures in this bot are easy to miss since it runs infrequently
(nightly/weekly).
This change reports build and asset creation successes/failures to new
metrics. Alerts will be created from these metrics.
Bug: skia:12759
Change-Id: I07db58dcd5ee1929e16717cc7be66062f30d3670
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/487976
Reviewed-by: Joe Gregorio <jcgregorio@google.com>
Commit-Queue: Ravi Mistry <rmistry@google.com>
Using prometheus's pushgateway since the metric will be reported from a task driver (a short-lived service-level job).
Updated go.mod/go.sum to get pushgateway libraries with:
$ go get go.skia.org/infra@2dce552b7a
$ go mod download
$ make -C infra/bots train
Bug: skia:12521
Change-Id: Ic63487b08d06163ef92556999c4920e5f178e285
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/487227
Reviewed-by: Joe Gregorio <jcgregorio@google.com>
Commit-Queue: Ravi Mistry <rmistry@google.com>
AFAIK, they aren't being used from the infra repo, so
putting them here will make it easier to work on them
(e.g. as we continue migrating the apps to Bazel).
Change-Id: I983a85a4934c8d849f2b59055554689216983b4c
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/482077
Reviewed-by: Eric Boren <borenet@google.com>
AFAIK, there was no need for it to be in the infra repo
and that made it difficult to iterate on.
It successfully checks out the skia-infra repo at ToT
and runs make bazel_release_ci in //jsfiddle.
This uploads the docker image, so a followup will
pipe it into the Pub/Sub flow.
Change-Id: I8c8fd9755003162b4ff38a7ffea97d1ea64cd493
Bug: skia:10614
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/481236
Reviewed-by: Eric Boren <borenet@google.com>
Bug: skia:12657
Change-Id: I432dc004ac2c8a1f9b0fd0874b8690dcb7e16dbf
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/473146
Reviewed-by: Eric Boren <borenet@google.com>
Commit-Queue: Ravi Mistry <rmistry@google.com>
Bug: skia:12630
Change-Id: Ie4d4d9a62b4152b8ce3ebb5edebfe9aab88a2a25
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/469858
Reviewed-by: Eric Boren <borenet@google.com>
Commit-Queue: Ravi Mistry <rmistry@google.com>
Change-Id: Ia5da00913d0b96e9edc47f8b2274bd26c35c251c
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/469457
Commit-Queue: Kevin Lubick <kjlubick@google.com>
Reviewed-by: Eric Boren <borenet@google.com>
Bug: chromium:1244124
Change-Id: If44330139ddecc37da40ad7ad4ef90dbf45ddf6a
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/447586
Reviewed-by: Erik Rose <erikrose@google.com>
Commit-Queue: Eric Boren <borenet@google.com>
Don't have the cycles to maintain the build.
Change-Id: Ia06cd78f5920091fcad250391a4d91b039e97f5d
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/432777
Reviewed-by: Kevin Lubick <kjlubick@google.com>
Change-Id: I13ae7034cda4291743f433406ee2e8267dd48d3e
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/416576
Commit-Queue: Eric Boren <borenet@google.com>
Reviewed-by: Ravi Mistry <rmistry@google.com>
The make_lotties_with_assets is a handy tool to take a
directory of lottie files and move them into their own
folders. This will make it easier to add assets (e.g. fonts,
images) if we want to get more accurate benchmarks.
This also makes the test not stop on the first failure, but complete
all of them, so we can see more easily if multiple things have broken.
Change-Id: Idb2ece71376a11e7524cd513a2798884a44bb59f
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/421927
Reviewed-by: Florin Malita <fmalita@google.com>
A previous change to goldctl removed the special-casing for
Skia, so we need to specify it ourselves.
Change-Id: If4d122daa4ee4bb865b628b7c6ee1cbe5d44d670
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/409396
Reviewed-by: Ravi Mistry <rmistry@google.com>
This skp started causing
Perf-Debian10-EMCC-GCE-CPU-AVX2-wasm-Release-All-Puppeteer_RenderSKP
to timeout.
Change-Id: I248b4be8f2287ecb16969f71f985acd794bd8d11
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/390298
Reviewed-by: Ravi Mistry <rmistry@google.com>
I got to thinking that seeing flags like,
out/fm --nonativeFonts -b cpu --nativeFonts -s ...
is kind of confusing, and I'm also trying to figure out
how to identify these runs to Gold. I think the answer
to both might be to track a map[string]string for flags,
allowing overrides rather than just appending, and then
that flag map ends up being the identifying properties.
Change-Id: Ie5f80ee8b145c205edc768ae871eb70a3e1bc5b7
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/378355
Reviewed-by: Eric Boren <borenet@google.com>
Try to uncover races by running parallel replicas.
The default --race 0 should keep FM working as before, but now with
--race ≥2 we'll actively try to race replicas, syncing between tests.
--race 1 is almost pointless, just changing the thread tests run on but
without any interesting concurrency.
Rearrange a bit how fm_driver decides what flags to pass to which
invocations of FM, so individual runs can easily override defaults (e.g.
--nativeFonts overriding the usual --nonativeFonts). Use that here to
set --race 0 for unit tests; many unit tests are not reentrant.
Change-Id: Ida451626c093793b0805d3036beb185e7d54f27e
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/376761
Reviewed-by: Brian Osman <brianosman@google.com>
Reviewed-by: Eric Boren <borenet@google.com>
Commit-Queue: Mike Klein <mtklein@google.com>
tabl_mozilla.skp is uselessly large without a clip.
Change-Id: I6e8ab8c31e790b6629be01e6eeb2e8d60c6ff56f
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/378360
Reviewed-by: Brian Osman <brianosman@google.com>
Change-Id: Ibe4e929b7d160b3dbf0e5b6ae093b4bab419817d
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/375198
Commit-Queue: Eric Boren <borenet@google.com>
Reviewed-by: Ravi Mistry <rmistry@google.com>
On batch failure we're rerunning every source in the batch, while we
really only need to rerun sources that we don't know succeeded.
If for example we run sources "foo", "bar", and "baz", and foo produces
a known hash, then bar crashes, we only need to rerun bar and baz. The
batch run was enough to demonstrate foo's good.
Change-Id: I17634a6095906bcc2ad0bd33bb78eba000654b5e
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/369456
Reviewed-by: Eric Boren <borenet@google.com>
Move definition of Work struct until just before it's used,
and show one of the sources as an example at kickoff-level step.
These are just cosmetic/refactors.
Change-Id: Ib23b9379683b9867e097c8d68ef8736013719cee
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/369356
Reviewed-by: Eric Boren <borenet@google.com>
If we track how many pending batches a kickoff()
has in flight, we can endStep() it properly when
that number hits zero.
This double sync.WaitGroup trick is pretty neat.
Now we're thinking with portals...
Added some comments to prevent myself falling in
the trap of assuming we'll have runtime.NumCPU()
batches... rounding the batch size up means we'll
sometimes have fewer.
Change-Id: If50615c204485862462c240b9bbdfd4ddbad43b2
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/366142
Reviewed-by: Eric Boren <borenet@google.com>
It's nice to see it in the task log, and to be able to see
it's not there when we're not working with Gold (*SAN) bots.
(One trybot of each kind here.)
Change-Id: Ibb4aa20badf95ef603f3890e1c8248cad675507f
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/366143
Reviewed-by: Eric Boren <borenet@google.com>
Group batches from a single kickoff() into another mid-level step:
Top-Level
kickoff --some flags
batch sources...
batch (exec)
batch other sources ...
batch (exec)
rerun (exec)
rerun (exec)
batch yet other sources ...
batch (exec)
rerun (exec)
kickoff --some other flags
...
Big question: is it okay for the kickoff steps to td.EndStep() while its
kids are still running (or haven't even started) on other goroutines?
Change-Id: I77ad2274e35cea0151be0cca6c690eafc4f8983e
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/366140
Reviewed-by: Eric Boren <borenet@google.com>
