Update Buildbot doc for Task Scheduler
BUG=skia:6118 NOTRY=true DOCS_PREVIEW= https://skia.org/?cl=7023 Change-Id: I398f4fd98959bb37bcb5499d38e517542e0df90f Reviewed-on: https://skia-review.googlesource.com/7023 Commit-Queue: Ben Wagner <benjaminwagner@google.com> Reviewed-by: Ben Wagner <benjaminwagner@google.com>
This commit is contained in:
Eric Boren
Skia Commit-Bot
parent
7cae170c1d
commit
6d4d6cce5e
90
site/dev/testing/automated_testing.md
Normal file
90
site/dev/testing/automated_testing.md
Normal file
@ -0,0 +1,90 @@
|
||||
Skia Automated Testing
|
||||
======================
|
||||
|
||||
Overview
|
||||
--------
|
||||
|
||||
Skia uses [Swarming](https://github.com/luci/luci-py/blob/master/appengine/swarming/doc/Design.md)
|
||||
to do the heavy lifting for our automated testing. It farms out tasks, which may
|
||||
consist of compiling code, running tests, or any number of other things, to our
|
||||
bots, which are virtual or real machines living in our local lab, Chrome Infra's
|
||||
lab, or in GCE.
|
||||
|
||||
The [Skia Task Scheduler](http://go/skia-task-scheduler) determines what tasks
|
||||
should run on what bots at what time. See the link for a detailed explanation of
|
||||
how relative task priorities are derived. A *task* corresponds to a single
|
||||
Swarming task. A *job* is composed of a directed acyclic graph of one or more
|
||||
*tasks*. The job is complete when all of its component tasks have succeeded
|
||||
or is considered a failure when any of its component tasks fails. The scheduler
|
||||
may automatically retry tasks within its set limits. Jobs are not retried.
|
||||
Multiple jobs may share the same task, for example, tests on two different
|
||||
Android devices which use the same compiled code.
|
||||
|
||||
Each Skia repository has an infra/bots/tasks.json file which defines the jobs
|
||||
and tasks for the repo. Most jobs will run at every commit, but it is possible
|
||||
to specify nightly and weekly jobs as well. For convenience, most repos also
|
||||
have a gen_tasks.go which will generate tasks.json. You will need to
|
||||
[install Go](https://golang.org/doc/install). From the repository root:
|
||||
|
||||
$ go get -u go.skia.org/infra/...
|
||||
$ go run infra/bots/gen_tasks.go
|
||||
|
||||
It is necessary to run gen_tasks.go every time it is changed or every time an
|
||||
[asset](https://skia.googlesource.com/skia/+/master/infra/bots/assets/README.md)
|
||||
has changed. There is also a test mode which simply verifies that the tasks.json
|
||||
file is up to date:
|
||||
|
||||
$ go run infra/bots/gen_tasks.go --test
|
||||
|
||||
|
||||
|
||||
Try Jobs
|
||||
--------
|
||||
|
||||
It is useful to know how your change will perform before it is submitted. After
|
||||
uploading your CL to [Gerrit](https://skia-review.googlesource.com/), you may
|
||||
trigger a try job for any job listed in tasks.json:
|
||||
|
||||
$ git cl try -B <bucket name> -b <job name>
|
||||
|
||||
The bucket name refers to the [Buildbucket](https://chromium.googlesource.com/infra/infra/+/master/appengine/cr-buildbucket/README.md)
|
||||
bucket to which the request will be submitted. Most public Skia repos use the
|
||||
"skia.primary" bucket, and most private Skia repos use the "skia.internal"
|
||||
bucket.
|
||||
|
||||
|
||||
Status View
|
||||
------------
|
||||
|
||||
The status view shows a table with tasks, grouped by test type and platform,
|
||||
on the X-axis and commits on the Y-axis. The cells are colored according to
|
||||
the status of the task for each commit:
|
||||
|
||||
* green: success
|
||||
* orange: failure
|
||||
* purple: exception (infrastructure issue)
|
||||
* black border, no fill: task in progress
|
||||
* blank: no task has started yet for a given revision
|
||||
|
||||
Commits are listed by author, and the branch on which the commit was made is
|
||||
shown on the very left. A purple result will override an orange result.
|
||||
|
||||
For more detail, you can click on an individual cell to get a summary of the
|
||||
task. You can also click one of the white bars at the top of each column to see
|
||||
a summary of recent tasks with the same name.
|
||||
|
||||
The status page has several filters which can be used to show only a subset of
|
||||
task specs:
|
||||
|
||||
* Interesting: Task specs which have both successes and failures within the
|
||||
visible commit window.
|
||||
* Failures: Task specs which have failures within the visible commit window.
|
||||
* Comments: Task specs which have comments.
|
||||
* Failing w/o comment: task specs which have failures within the visible commit
|
||||
window but have no comments.
|
||||
* All: Display all tasks.
|
||||
* Search: Enter a search string. Substrings and regular expressions may be
|
||||
used, per the Javascript String Match() rules:
|
||||
http://www.w3schools.com/jsref/jsref_match.asp
|
||||
|
||||
|
||||
@ -1,102 +0,0 @@
|
||||
Skia Buildbots
|
||||
==============
|
||||
|
||||
Overview
|
||||
--------
|
||||
|
||||
Like the Chromium team, the Skia team uses [buildbot](http://trac.buildbot.net/)
|
||||
to run continuous builds and tests.
|
||||
|
||||
Here is a link to our main status page: https://status.skia.org/
|
||||
|
||||
There are also buildbot console pages for a detailed view of those results:
|
||||
|
||||
Externally-facing:
|
||||
|
||||
* http://build.chromium.org/p/client.skia/console
|
||||
* http://build.chromium.org/p/client.skia.android/console
|
||||
* http://build.chromium.org/p/client.skia.compile/console
|
||||
* http://build.chromium.org/p/client.skia.fyi/console
|
||||
|
||||
Internally-facing:
|
||||
|
||||
* http://uberchromegw.corp.google.com/i/client.skia/console
|
||||
* http://uberchromegw.corp.google.com/i/client.skia.android/console
|
||||
* http://uberchromegw.corp.google.com/i/client.skia.compile/console
|
||||
* http://uberchromegw.corp.google.com/i/client.skia.fyi/console
|
||||
* http://uberchromegw.corp.google.com/i/client.skia.internal/console
|
||||
|
||||
|
||||
Architecture
|
||||
------------
|
||||
|
||||
The buildbot system consists of these elements: \(see
|
||||
http://buildbot.net/buildbot/docs/current/manual/introduction.html#system-architecture
|
||||
for more detail\)
|
||||
|
||||
* Buildbot Master
|
||||
|
||||
* Watches for new commits to land in the Skia repository
|
||||
\(https://skia.googlesource.com/skia\)
|
||||
* Whenever a new commit lands, it triggers a **Build** on each **Builder**
|
||||
to test the new revision.
|
||||
* Serves up status pages whenever anybody requests them
|
||||
|
||||
* Build
|
||||
|
||||
* One run of a particular **Builder**, at a particular code revision.
|
||||
* "Build" is sort of a misnomer; it's just a list of steps (typically shell
|
||||
commands) which are run by the **Buildslave** process on the host
|
||||
machine, and those may include compiling and running code as well as
|
||||
arbitrary other commands.
|
||||
|
||||
* Builder
|
||||
|
||||
* One repeatable build and/or test configuration on a given platform. The
|
||||
Builder is basically a blueprint which provides logic to determine which
|
||||
steps to run within a Build.
|
||||
|
||||
* Buildslave \(or "buildbot slave"\)
|
||||
|
||||
* A process running on a host machine that builds and runs code as directed
|
||||
by the Buildbot Master.
|
||||
* One or more Builders may run on a given Buildslave, but only one runs at
|
||||
a time.
|
||||
* One or more Buildslaves may run on a given host machine.
|
||||
|
||||
|
||||
Status View
|
||||
------------
|
||||
|
||||
The status view shows a table with builders, grouped by test type and platform,
|
||||
on the X-axis and commits on the Y-axis. The cells are colored according to
|
||||
the status of the build for each commit:
|
||||
|
||||
* green: success
|
||||
* orange: failure
|
||||
* purple: exception (infrastructure issue)
|
||||
* black border, no fill: build in progress
|
||||
* blank: no build has started yet for a given revision
|
||||
|
||||
Commits are listed by author, and the branch on which the commit was made is
|
||||
shown on the very left. A purple result will override an orange result.
|
||||
|
||||
For more detail, you can click on an individual cell to get a summary of the
|
||||
steps which ran for that build. You can also click one of the white bars at
|
||||
the top of each column to see a summary of recent builds for a given builder.
|
||||
|
||||
The status page has several filters which can be used to show only a subset of
|
||||
bots:
|
||||
|
||||
* Interesting: Bots which have both successes and failures within the visible
|
||||
commit window.
|
||||
* Failures: Bots which have failures within the visible commit window.
|
||||
* Comments: Bots which have comments.
|
||||
* Failing w/o comment: Bots which have failures within the visible commit window
|
||||
but have no comments.
|
||||
* All: Display all bots.
|
||||
* Search: Enter a search string. Substrings and regular expressions may be
|
||||
used, per the Javascript String Match() rules:
|
||||
http://www.w3schools.com/jsref/jsref_match.asp
|
||||
|
||||
|
||||
Loading…
Reference in New Issue
Block a user