skia2/infra/bots
recipe-roller be9aa64ff9 Roll recipe dependencies (trivial).
This is an automated CL created by the recipe roller. This CL rolls
recipe changes from upstream projects (depot_tools, recipe_engine) into this repository.

depot_tools:
09f358bae3~..b39f6bbf82e3a9bfbed3d24b561726c247d7a48d
  09f358b (ehmaldonado@google.com)
      metrics: Don't collect metrics on DEPOT_TOOLS_METRICS=0
  534f505 (songfangzhen@bytedance.com)
      Update format of `git cl split`
  3ac55bc (nodir@chromium.org)
      Revert "[dirmd] Roll dirmd"
  9f6aa1b (raphael.kubo.da.costa@intel.com)
      gerrit_util: Make GceAuthenticator also consider proxy errors i...
  6f0df68 (dpranke@google.com)
      Add use_python3 to codereview.settings to set default for PRESU...
  a73eec2 (fdegans@google.com)
      Add a new argument to skip shebang checks on python3
  2a6dffa (tikuta@chromium.org)
      Reland "use python3 for goma related scripts"
  4577360 (sokcevic@google.com)
      Reland "Reland "Update gsutil to use gsutil version 4.61, pytho...
  fc3fb20 (brucedawson@chromium.org)
      Revert "Reland "use python3 for goma related scripts""
  a806594 (brucedawson@chromium.org)
      Reland "Reland "use python3 for goma related scripts""
  db632b6 (sokcevic@google.com)
      Use main as default branch in documentation
  a50bd44 (vadimsh@chromium.org)
      [cipd] Update CIPD client to 2.6.0.
  965a05b (kyleju@chromium.org)
      Fix py3 presubmit error. Use unicode strings explicitly.
  1bfda8e (sigurds@chromium.org)
      Respect git config cl.date-order in `git cl st`
  6b08432 (sokcevic@google.com)
      Use main as default branch for git recipe module
  866be0f (wnwen@chromium.org)
      Add an option to specify pylint version
  f0d7ed8 (sokcevic@google.com)
      Allow custom filters for json validation
  0aabd63 (victorsamun@yandex-team.ru)
      Fix python2/3 compatibility in DownloadUsingHttp function
  b3d52d3 (sokcevic@google.com)
      Revert "Use main as default branch for git recipe module"
  c079895 (vadimsh@chromium.org)
      [cipd] Update CIPD client and vpython.
  26adc2a (sokcevic@google.com)
      Fix generating man pages
  eb140ae (yyanagisawa@chromium.org)
      Roll out new goma client binary VERSION=225
  b39f6bb (bryner@google.com)
      Roll new vpython to depot_tools.

recipe_engine:
fe59faf05e~..af228e7f3c4fe44fe10fd65d866dccaedff4a219
  fe59faf (tikuta@chromium.org)
      cas: get cipd package revision from file
  fd22f03 (tikuta@chromium.org)
      cas: simplify _version a bit
  f6ae625 (chromium-autoroll@skia-public.iam.gserviceaccount.com)
      Roll CAS Client from 34b847e00d95 to bcbbe2d4b506
  b19a5af (yuanjunh@google.com)
      [python3] Make recipes doc command py3-compatible
  7b628b8 (iannucci@chromium.org)
      [python3] Add indicator for python3 compat at recipe/module level.
  d731cbd (iannucci@chromium.org)
      Add ability to read + simulate GLOBAL_SHUTDOWN.
  0032604 (tandrii@google.com)
      Make autoroller owner of CAS Client pin.
  f62b45d (chromium-autoroll@skia-public.iam.gserviceaccount.com)
      Roll CAS Client from 78702ac003f3 to 76ddc72fae59
  010e03b (chromium-autoroll@skia-public.iam.gserviceaccount.com)
      Roll CAS Client from c277f3162d43 to 1403fab7e195
  1b6e184 (chromium-autoroll@skia-public.iam.gserviceaccount.com)
      Roll CAS Client from f8fee322da09 to 68a9d9f74279
  b14cfa0 (chromium-autoroll@skia-public.iam.gserviceaccount.com)
      Roll CAS Client from 8dce0cb59e09 to 9464003f0708
  bc01d1b (qyearsley@chromium.org)
      [python3 compatibility] Switch to iteritems in recipe engine un...
  cd13c55 (qyearsley@chromium.org)
      recipe engine: Spellcheck and minor formatting
  78cc13b (chromium-autoroll@skia-public.iam.gserviceaccount.com)
      Roll CAS Client from 9464003f0708 to 8dd195b8fed9
  fad480e (qyearsley@chromium.org)
      [python3 compatibility] Run some futurize "stage 2" fixers on r...
  89cf3c8 (chromium-autoroll@skia-public.iam.gserviceaccount.com)
      Roll CAS Client from 8dd195b8fed9 to e14bb15691db
  4abe5ab (chromium-autoroll@skia-public.iam.gserviceaccount.com)
      Roll CAS Client from e14bb15691db to 13eb6a325aaa
  4543597 (martiniss@google.com)
      Improve cas archive default
  e16a16a (chromium-autoroll@skia-public.iam.gserviceaccount.com)
      Roll CAS Client from 102bb894249d to 40001c4c5150
  e88454d (chromium-autoroll@skia-public.iam.gserviceaccount.com)
      Roll CAS Client from eba70e5d7b53 to c85e8f0ecb90
  284f8eb (chromium-autoroll@skia-public.iam.gserviceaccount.com)
      Roll CAS Client from c85e8f0ecb90 to 01ae064f288d
  434b4f0 (chromium-autoroll@skia-public.iam.gserviceaccount.com)
      Roll CAS Client from 6fa1788bbb0e to 713757bf8a41
  8238109 (chromium-autoroll@skia-public.iam.gserviceaccount.com)
      Roll CAS Client from 8b378ab01c51 to 40f945205c86
  f0f25eb (chromium-autoroll@skia-public.iam.gserviceaccount.com)
      Roll CAS Client from f50bf9c6481d to 63c49027cfa9
  18e4134 (chromium-autoroll@skia-public.iam.gserviceaccount.com)
      Roll CAS Client from 63c49027cfa9 to 89085d1a792a
  442820a (tikuta@chromium.org)
      tricium: show comment even if it exceeds 50
  d44e046 (chromium-autoroll@skia-public.iam.gserviceaccount.com)
      Roll CAS Client from da9c74b88f55 to d425349380f6
  92a7948 (chromium-autoroll@skia-public.iam.gserviceaccount.com)
      Roll CAS Client from d425349380f6 to 13d1aa30c73d
  56686d5 (chromium-autoroll@skia-public.iam.gserviceaccount.com)
      Roll CAS Client from 13d1aa30c73d to d5505d31f87f
  443bf98 (chromium-autoroll@skia-public.iam.gserviceaccount.com)
      Roll CAS Client from 4f05b72dc873 to dddfb00ee624
  30dbfb4 (chromium-autoroll@skia-public.iam.gserviceaccount.com)
      Roll CAS Client from dddfb00ee624 to d60b89d9dffa
  1951b83 (chromium-autoroll@skia-public.iam.gserviceaccount.com)
      Roll CAS Client from d60b89d9dffa to e29eacaa51b1
  3606812 (chromium-autoroll@skia-public.iam.gserviceaccount.com)
      Roll CAS Client from 02e723041f2e to af2dc3835e4f
  46491c0 (iannucci@chromium.org)
      Add RECIPES_DEBUG_SLEEP to have engine do sleep for led.
  bb91c77 (chromium-autoroll@skia-public.iam.gserviceaccount.com)
      Roll CAS Client from af2dc3835e4f to 9a1c1dd351da
  8a92136 (iannucci@chromium.org)
      Add function to compute effective python compatibility.
  2ad13e4 (chromium-autoroll@skia-public.iam.gserviceaccount.com)
      Roll CAS Client from 9a1c1dd351da to 5e2c34b93cdf
  e47c6d4 (chromium-autoroll@skia-public.iam.gserviceaccount.com)
      Roll CAS Client from 5e2c34b93cdf to aca8d68f3040
  1c85729 (chromium-autoroll@skia-public.iam.gserviceaccount.com)
      Roll CAS Client from b53f44dad74f to f483cf4004ef
  e4db17b (chromium-autoroll@skia-public.iam.gserviceaccount.com)
      Roll CAS Client from 0862ebb13f0c to 61d5434b44c8
  8d05dc8 (tikuta@chromium.org)
      isolated: remove on_path
  c4b8357 (chromium-autoroll@skia-public.iam.gserviceaccount.com)
      Roll CAS Client from 9f7beee9fe44 to 91cf324fb7a0
  478f380 (chromium-autoroll@skia-public.iam.gserviceaccount.com)
      Roll CAS Client from 3fadd3e58f93 to 74b60d9d714f
  af228e7 (chromium-autoroll@skia-public.iam.gserviceaccount.com)
      Roll CAS Client from 74b60d9d714f to 6808332cfd84

More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug.

R=borenet@google.com

Recipe-Tryjob-Bypass-Reason: Autoroller
Bugdroid-Send-Email: False
Change-Id: I43f528d39adae46c34f695fc92f65b939b0f790d
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/425144
Commit-Queue: Recipe Roller <recipe-mega-autoroller@chops-service-accounts.iam.gserviceaccount.com>
Bot-Commit: Recipe Roller <recipe-mega-autoroller@chops-service-accounts.iam.gserviceaccount.com>
2021-07-07 15:15:41 +00:00
..
analysis A set of scripts that analyzes jobs.json to look for possible holes in our 2021-06-02 16:11:43 +00:00
assets Reland "[infra] Roll Infra dep to update CIPD packages" with fix 2021-06-22 17:31:55 +00:00
buildstats Fix Python3 compatibility 2021-03-25 14:15:35 +00:00
gen_tasks_logic Add Spin513 bots to the tree. 2021-07-01 21:30:19 +00:00
recipe_modules [infra] Remove Pixel (sailfish) jobs 2021-06-28 16:52:04 +00:00
recipes [infra] Remove Pixel (sailfish) jobs 2021-06-28 16:52:04 +00:00
task_drivers [skottie] Update lotties_with_assets to include all lotties from lottie-samples 2021-06-28 14:59:24 +00:00
tools/luci-go [Isolate] Update clients to hit /_ah/api instead of /api 2018-06-04 22:14:49 +00:00
build_task_drivers.sh add Mac/Win FM bots 2021-02-01 17:35:52 +00:00
bundle_recipes.sh Reland "[infra] Run recipes through Kitchen" 2018-04-19 11:27:46 +00:00
cfg.json Remove all traces of OpenCL 2021-05-27 19:17:17 +00:00
check_deps.py [sk] Add DEPS entry for "sk" tool 2021-04-09 14:21:37 +00:00
gen_tasks.go Reland "[infra] Make most builds idempotent" 2019-07-19 12:11:27 +00:00
git_utils.py Update references to master branch 2021-05-21 15:18:16 +00:00
infra_tests.py Fix Python3 compatibility 2021-03-25 14:15:35 +00:00
jobs.json Add Spin513 bots to the tree. 2021-07-01 21:30:19 +00:00
Makefile Add infra/bots/Makefile to simplify test/train 2017-04-13 17:01:17 +00:00
OWNERS [infra] Remove stephana@ from OWNERS 2019-06-21 12:07:54 +00:00
README.md Update references to master branch 2021-05-21 15:18:16 +00:00
README.recipes.md Roll recipe dependencies (trivial). 2021-07-07 15:15:41 +00:00
recipes.py Roll recipe dependencies (trivial). 2021-07-07 15:15:41 +00:00
run_recipe.py Fix Python3 compatibility 2021-03-25 14:15:35 +00:00
tasks.json Add Spin513 bots to the tree. 2021-07-01 21:30:19 +00:00
test_utils.py Fix Python3 compatibility 2021-03-25 14:15:35 +00:00
upload_skps.py [Android compile bots] Add comment to CL when both nopatch and withpatch builds fail 2019-10-28 14:55:01 +00:00
utils.py Update references to master branch 2021-05-21 15:18:16 +00:00
zip_utils_test.py Fix Python3 compatibility 2021-03-25 14:15:35 +00:00
zip_utils.py Fix Python3 compatibility 2021-03-25 14:15:35 +00:00

Skia Infrastructure

This directory contains infrastructure elements.

Tasks and Jobs

Files in this directory define a DAG of tasks which run at every Skia commit. A task is a small, self-contained unit which runs via Swarming on a machine in the pool. Tasks may be chained together, eg. one task to compile test binaries and another to actually run them.

Jobs are collections of related tasks which help define sub-sections of the DAG, for example, to be used as try jobs. Each job is defined as an entry point into the DAG.

The tasks.json file in this directory is the list of tasks and jobs for the repo. Note that tasks.json is NEVER edited by hand but generated via gen_task.go and the input files enumerated below. The Task Scheduler reads the tasks.json file at each commit to determine which jobs to run. For convenience, gen_tasks.go is provided to generate tasks.json and also to test it for correct syntax and detecting cycles and orphaned tasks. Always edit gen_tasks.go or one of the following input JSON files, rather than tasks.json itself:

  • cfg.json - Basic configuration information for gen_tasks.go.
  • jobs.json - The list of all jobs to run. Edit this to add or remove bots.

Whenever gen_tasks.go, any of the above JSON files, or assets are changed, you need to run gen_tasks.go to regenerate tasks.json:

$ go run infra/bots/gen_tasks.go

Or:

$ cd infra/bots; make train

There is also a test mode which performs consistency checks and verifies that tasks.json is unchanged:

$ go run infra/bots/gen_tasks.go --test

Or:

$ cd infra/bots; make test

Recipes

Recipes are the framework used by Skia's infrastructure to perform work inside of Swarming tasks. The main elements are:

  • recipes.py - Used for running and testing recipes.
  • recipes - These are the entry points for each type of task, eg. compiling or running tests.
  • recipe_modules - Shared modules which are used by recipes.
  • .recipe_deps - Recipes and modules may depend on modules from other repos. The recipes.py script automatically syncs those dependencies in this directory.

Isolate Files

These files determine which parts of the repository are transferred to the bot when a Swarming task is triggered. The Isolate tool hashes each file and will upload any new/changed files. Bots maintain a cache so that they can efficiently download only the files they don't have.

Assets

Artifacts used by the infrastructure are versioned here, along with scripts for recreating/uploading/downloading them. See the README in that directory for more information. Any time an asset used by the bots changes, you need to re-run gen_tasks.go.

Tools

Assorted other infrastructure-related tools, eg. isolate and CIPD binaries.

CT

Helpers for running Skia tasks in Cluster Telemetry.