Apparently, the tarfile Python module spends a lot of time in
grp.getgrid for retrieving a piece information (the name of the
primary group) which we don't need anyway. There is no
proper way to disable these slow calls, but there's a workaround
which relies on the way in which grp (and pwd) is used.
In fact, pwd and grp are imported in this fashion:
try:
import grp, pwd
except ImportError:
grp = pwd = None
and then used with the following pattern [2]:
if grp:
try:
tarinfo.gname = grp.getgrgid(tarinfo.gid)[0]
except KeyError:
pass
By setting grp and pwd to None, thus skipping the calls, I was
able to achieve a 35x speedup on my workstation.
The user and group names are set to test262 when building the tar.
The downside to this approach is that we are relying on an
implementation detail, which is not in the public API.
However, the blamelist shows that the relevant bits of the module
have not been updated since 2003 [3], so we might as well assume
that the workaround will keep working, on cPython 2.x at least.
---
[1] https://hg.python.org/cpython/file/2.7/Lib/tarfile.py#l56
[2] https://hg.python.org/cpython/file/2.7/Lib/tarfile.py#l1933
[3] https://hg.python.org/cpython/rev/f9a5ed092660
BUG=chromium:535160
LOG=N
Review URL: https://codereview.chromium.org/1727773002
Cr-Commit-Position: refs/heads/master@{#34245}
This experimentally implements taring/untaring the test data
for test262 on the v8-side before test isolation and when
running the tests.
It archives on demand only if the tar is outdated compared
to the contained files. This comes with a cost of ~1s extra
to run gyp on linux and ~6s extra on windows. Ninja is
lightning fast afterwards in detecting changes. Also, we
archive only when test_isolation_mode is set and when
the test262_run target is required.
The archiving itself costs ~30s on all platforms. But as the
files will change seldom this shouldn't have a big impact.
Extraction on the test runner side is below 2s on mac and
linux. The speedup is enormous. Around 5 minutes were spent
on download on swarming slaves before, which is now only
a few seconds. So total test time for release (no variants),
e.g. goes from 8 to 3 minutes.
BUG=chromium:535160
LOG=n
Review URL: https://codereview.chromium.org/1713993002
Cr-Commit-Position: refs/heads/master@{#34155}
