v8/infra/testing
Jakob Gruber c76bae7db6 Add a test variant for concurrent_inlining
.. and enable it on fyi bots. Skip one test in this variant due to
incompatible flags.

Bug: v8:7790
Change-Id: I5b8fdd8572435c6f4474c505464ff1a22c830757
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2797287
Commit-Queue: Jakob Gruber <jgruber@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Georg Neis <neis@chromium.org>
Cr-Commit-Position: refs/heads/master@{#73770}
2021-04-01 12:38:14 +00:00
..
builders.pyl Add a test variant for concurrent_inlining 2021-04-01 12:38:14 +00:00
PRESUBMIT.py [test] Prepare removing gpu defaults for tester bots 2021-02-04 14:58:23 +00:00
README.md Fix broken link to infra-side specifications 2021-01-25 15:21:23 +00:00

Src-side test specifications

Src-side test specifications enable developers to quickly add tests running on specific bots on V8's continuous infrastructure (CI) or tryserver. Features to be tested must live behind runtime flags, which are mapped to named testing variants specified here. Changes to src-side test specifications go through CQ like any other CL and require tests added for specific trybots to pass.

The test specifications are defined in a V8-side python-literal file infra/testing/builders.pyl.

The structure of the file is:

{
  <buildername>: {
    'tests': [
      {
        'name': <test-spec name>,
        'suffix': <step suffix>,
        'variant': <variant name>,
        'shards': <number of shards>,
        'test_args': <list of flags>,
        'swarming_task_attrs': {...},
        'swarming_dimensions': {...},
      },
      ...
    ],
    'swarming_task_attrs': {...},
    'swarming_dimensions': {...},
  },
  ...
}

The <buildername> is a string name of the builder to execute the tests. <test-spec name> is a label defining a test specification matching the infra-side. The optional suffix will be appended to test-step names for disambiguation. The optional variant is a testing variant specified here. The optional shards (default 1) can be provided to increase the swarming shards for long-running tests. The optional test_args is a list of string flags that will be passed to the V8 test driver. The optional swarming_task_attrs is a dict allowing to override the defaults for priority, expiration and hard_timeout. The optional swarming_dimensions is a dict allowing to override the defaults for cpu, cores and os. Both swarming_task_attrs and swarming_dimensions can be defined per builder and per test, whereas the latter takes precedence.

Example:

{
  'v8_linux64_rel_ng_triggered': {
    'tests': [
      {
        'name': 'v8testing',
        'suffix': 'stress',
        'variant': 'nooptimization',
        'shards': 2,
        'test_args': ['--gc-stress'],
        'swarming_dimensions': {'os': 'Ubuntu-14.4'},
      },
    ],
    'swarming_properties': {'priority': 35},
    'swarming_dimensions': {'os': 'Ubuntu'},
  },
}

Guidelines

Please keep trybots and continuous bots in sync. E.g. add the same configuration for the release and debug CI bots and the corresponding trybot (where applicable). E.g.

tryserver.v8:
  v8_linux64_rel_ng_triggered
client.v8:
  V8 Linux64
  V8 Linux64 - debug

Please only add tests that are expected to pass, or skip failing tests via status file for the selected testing variants only. If you want to add FYI tests (i.e. not closing the tree and not blocking CQ) you can do so for the following set of bots:

tryserver.v8:
  v8_linux64_fyi_rel_ng_triggered
client.v8:
  V8 Linux64 - fyi
  V8 Linux64 - debug - fyi