skia2/site/dev/testing/skialab.md

SkiaLab
=======

Overview
--------

Skia's buildbots are hosted in three places:

* Google Compute Engine. This is the preferred location for bots which don't
  need to run on physical hardware, ie. anything that doesn't require a GPU,
  stable performance numbers, or a specific hardware configuration. Most of our
  compile bots live here, along with some non-GPU test bots on Linux and
  Windows.
* Chrome Golo. This is the preferred location for bots which require specific
  hardware or OS configurations that are not supported by GCE. We have several
  Mac, Linux, and Windows bots in the Golo.
* The local SkiaLab in Chapel Hill. Anything we can't get in GCE or the Golo
  lives here. This includes newer or uncommon GPUs and all Android, ChromeOS,
  and iOS devices.

This page covers the local SkiaLab in Chapel Hill.


Layout
------

The SkiaLab consists of three wireframe racks which hold machines connected to
two KVM switches. Each KVM switch has a monitor, mouse, and keyboard and is the
primary mode of access to the lab machines. In general, the machines are on the
same rack as the KVM switch used to access them. The switch nearest the door
(labeled "DOOR"), is connected to machines on its own rack as well as a smaller
rack closer to the door.

Each machine is labeled with its hostname and the number or letter used to
access it on the KVM switch. Android devices are located on the rack nearest
the interior of the office (the KVM switch is labeled "OFFICE"). They are
labeled with their serial number and the name of the buildslave they are
associated with. Each device connects to a host machine, either directly or
by way of a powered USB hub.

**Disclaimer: Please ONLY make changes on a lab machine as a last resort, as it
is disruptive to the running bots and can leave the machines in a dirty state.
If you must make changes, such as cloning a copy of Skia to run tests and debug
failures, be sure to clean up after yourself. If a permanent change needs to be
made on the machine (such as a driver update), please contact an infra team
member.**


Common Tasks
------------

### Locating the host machine for a failing bot

Sometimes failures can only be reproduced on a particular hardware
configuration. In these cases, it is sometimes necessary to log into the host
machine where a failing bot is running in order to debug the failure.

From the [Status](https://status.skia.org/) page:

1. Click on the box associated with a failed build.
2. A popup will appear with some information about the build, including the
   builder and buildslave. Click the "Lookup" link next to "Host machine". This
   will bring you to the [SkiaLab Hosts](https://status.skia.org/hosts) page,
   which contains information about the machines in the lab, pre-filtered to
   select the machine which runs the buildslave in question.
3. The information box will display the hostname of the machine as well as the
   KVM switch and number used to access the machine, if the machine is in the
   SkiaLab.
4. Walk over to the lab. While standing at the KVM switch indicated by the host
   information page, double tap \<ctrl\> and then press the number or letter from
   the information page. It may be necessary to move or click the mouse to wake
   the machine up.
5. Log in to the machine if necessary. The password is stored in
   [Valentine](https://valentine/) as "Chapel Hill buildbot slave password".

### Rebooting a problematic Android device

Follow the same process as above, with some slight changes:

1. On the [Status](https://status.skia.org/) page, click the box for the failed
   build.
2. Click the "Lookup" link for the host machine. Remember the name of the
   buildslave which ran the build.
3. The hosts page will display the information used to access the host machine
   for the device as well as the serial number for the device next to the name
   of its buildsave.
4. Walk over to the lab and find the Android device with the serial number from
   the hosts page. Hold the power and volume-up buttons until the device
   reboots.
5. Access the host machine for the device, per the above instructions. Use the
   `which_devices.py` script to verify that the device has re-attached. From
   the home directory:

        $ python buildbot/scripts/which_devices.py


Maintenance Tasks
-----------------

### Bringing up a new buildbot host machine

This assumes that we're just adding a host machine for a new buildbot slave,
and doesn't cover how to make changes to the buildbot code to change the
behavior of the builder itself.

1. Obtain the machine itself and place it on the racks in the lab. Connect
   power, ethernet, and KVM cables.
2. If we already have a disk image appropriate for this machine, follow the
   instructions for flashing a disk image to a machine below. Otherwise, follow
   the instructions for bringing up a new machine from scratch.
3. Power on the machine. Be sure to kill any buildbot processes that start up,
   eg. `killall python` on Linux and Mac, and just close any cmd instances which
   pop up on Windows.
4. Set the hostname for the machine.
5. Ensure that the machine is labeled with its hostname and KVM number.
6. Add the new slave to the slaves.cfg file on the appropriate master, eg.
   https://chromium.googlesource.com/chromium/tools/build/+/master/masters/master.client.skia/slaves.cfg,
   and upload the change for code review.
7. Add an entry for the new host machine to the slave_hosts_cfg.py file in the
   Skia infra repo: https://skia.googlesource.com/buildbot/+/master/site_config/slave_hosts_cfg.py,
   and upload it for review.
8. Commit the change to add the slave to the master. Once it lands, commit the
   slave_hosts_cfg.py change immediately afterward.
9. Restart the build master. Either ask borenet@ to do this or file a
   [ticket](https://code.google.com/p/chromium/issues/entry?template=Build%20Infrastructure&labels=Infra-Labs,Restrict-View-Google,Infra-Troopers&summary=Restart%20request%20for%20[%20name%20]&comment=Please%20provide%20the%20reason%20for%20restart.%0A%0ASet%20to%20Pri-0%20if%20immediate%20restarted%20is%20required,%20otherwise%20please%20set%20to%20Pri-1%20and%20the%20restart%20will%20happen%20when%20the%20trooper%20gets%20a%20free%20moment.) for a trooper to do it.
10. Reboot the machine and monitor the build master to ensure that it connects.
    This can take some time, since the bot needs to sync Chrome.


### Bringing up a new Android bot

1. Locate or add a host machine. We generally want to keep the number of
   devices attached to each host below 5 or so. If a new host machine is
   required, follow the above instructions for bringing up a new buildbot
   host machine, with the exception that the slave corresponds to the Android
   device, not the host machine itself.
2. Ensure that the buildslave is not yet running:

        $ killall python

3. Disable MTP and PTP on the device.  Some devices require one or the other to
   be enabled; in that case, select PTP and choose to 'do nothing' when
   attaching to the host machine.
4. Connect the device to the host machine, either through a powered USB hub or
   directly to the machine.
5. Make sure that the device is in developer mode and that USB debugging is
   enabled.
6. Authorize the device for USB debugging on the host machine by checking the
   "always allow" box on dialog box which appears on the Android device after
   plugging it into the host.
7. Ensure that the device appears as "connected" when you run the
   `which_devices.py` script:

        $ python buildbot/scripts/which_devices.py

8. Reboot the machine to start the buildslave.


### Bringing up a new machine from scratch

TODO(borenet): Migrate from Google Docs.

OS-specific instructions are available in a
[Google Doc](https://docs.google.com/document/d/1X7Hvsj33AlBmj-KEWfFbmdCArUJJAICLkB7ipDcxRV8/edit)


### Flashing a disk image to a machine

1. Find the USB key labeled, "Clonezilla" in the SkiaLab and insert it into the
   machine.
2. Turn on the machine and load the boot menu. For Shuttle machines, press
   \<del\> or \<esc\>. Mac machines require that you plug in the Mac keyboard and
   press the \<option\> key at boot. Boot from the USB key. It's typically UEFI
   and named something like "FlashBlu" or "Kanguru".
3. At the Clonezilla menu, choose the "to RAM" option.
4. Choose your preferred language.
5. "Don't touch keymap".
6. "Start Clonezilla".
7. "device-image".
8. "local_dev".
9. Unplug the flash drive and plug in the external hard drive labeled, "Disk
   images." Wait for the "Attached Enclosure device" message to appear, then
   hit \<enter\>.
10. Select the external drive to use for /home/partimag, something like,
    "1000GB_ntfs_My_Passport".
11. Select the bot_img directory.
12. Hit \<enter\> to continue.
13. "Beginner"
14. "restoredisk"
15. Select the image to use. Make sure that it's compatible with this machine.
16. Choose the hard drive in the machine. It should be the only option.
17. "y" and "y"
18. Choose "reboot" after flashing the image to the machine.
19. Set the hostname of the machine so that it doesn't conflict with any
    existing machines.

### Capturing a disk image

1. Make sure that the machine is in a clean state: no pre-existing buildslave
   checkouts, extra software, etc.
2. Find the USB key labeled, "Clonezilla" in the SkiaLab and insert it into the
   machine.
3. Turn on the machine and load the boot menu. For Shuttle machines, press
   \<del\> or \<esc\>. Mac machines require that you plug in the Mac keyboard and
   press the \<option\> key at boot. Boot from the USB key. It's typically UEFI
   and named something like "FlashBlu" or "Kanguru".
4. At the Clonezilla menu, choose the "to RAM" option.
5. Choose your preferred language.
6. "Don't touch keymap".
7. "Start Clonezilla".
8. "device-image".
9. "local_dev"
10. Unplug the flash drive and plug in the external hard drive labeled, "Disk
    images." Wait for the "Attached Enclosure device" message to appear, then
    hit \<enter\>.
11. Select the external drive to use for /home/partimag, something like,
    "1000GB_ntfs_My_Passport".
12. Select the bot_img directory.
13. "Beginner"
14. "savedisk"
15. Choose a name for the disk image. The convention is:
    `skiabot-<hardware type>-<OS>-<disk image revision #>`
12. Choose the hard drive in the machine. It should be the only option.
13. "y"
14. Choose "reboot" or "shut down" when finished.