disable buidling/deployment of python wheels (#583)

* [appveyor] remove 'deploy' stage; only test python 2.7 and 3.6

all the other python versions are being built and tested on
https://github.com/google/brotli-wheels/blob/d571d63/appveyor.yml

* remove terrify submodule as not needed any more

* [travis] just test py2.7 and 3.6 on linux; remove extra osx python builds

All the other python versions for OSX are being built/tested on:
https://github.com/google/brotli-wheels/blob/d571d63/.travis.yml

Also, there's no need to build and deploy wheels here, as that's done
in the separate repository.

* [setup.py] only rebuild if dependency are newer; fix typo in list of 'depends'

https://github.com/python/cpython/blob/v3.6.2/Lib/distutils/command/build_ext.py#L485-L500

* [ci] only run 'python setup.py test'

if we run 'python setup.py built test', the setuptools 'test' command will
forcibly re-run the build_ext subcommand because it wants to pass the --inplace
option (it ignores whether it's up to date, just re-runs it all the time).

with this we go from running built_ext twice, to running it only once per build

* [Makefile] run 'build_ext --inplace' instead of 'develop' as default target

The 'develop' command is like 'install' in the sense that it
modifies the user's python environment.
The default make target should be less intrusive, i.e. just building
the extension module in-place without modify anything in the user's
environment.

We don't need to tell make about the dependency between 'test' and
'build' target as that is baked in the `python setup.py test` command.

* [Makefile] add 'develop' target; remove unnecessary 'tests' target

`make test` is good enough

* [Makefile] `setup.py test` requires setuptools; run `python -m unittest`

This will work even if setuptools is not installed, which is unlikely
nowadays but still our `setup.py` works with plain distutils, so
we may well have our tests work without setuptools.

* [python/README.md] add ref to 'develop' target; remove 'tests', just 'make test'

* [setup.py] import modules as per nicksay's comment

https://github.com/google/brotli/pull/583#discussion_r131981049

* [Makefile] add 'develop' to .PHONY targets

remove 'tests' from .PHONY

* [appveyor] remove unused setup scripts

We don't need to install custom python versions, we are
using the pre-installed ones on Appveyor.

* [appveyor] remove unneeded setup code

.gitmodules

@@ -1,6 +1,3 @@
-[submodule "terryfy"]
-	path = scripts/terryfy
-	url = https://github.com/MacPython/terryfy.git
 [submodule "research/esaxx"]
 	path = research/esaxx
 	url = https://github.com/hillbig/esaxx

.travis.yml

@@ -125,9 +125,22 @@ matrix:
       env: BUILD_SYSTEM=cmake C_COMPILER=pgcc CXX_COMPILER=pgc++
 
     ###
-    ## Python build on Linux
+    ## Python 2.7 and 3.6 builds on Linux
     ###
     - os: linux
+      language: python
+      python: 2.7
+      env: BUILD_SYSTEM=python C_COMPILER=gcc-6 CXX_COMPILER=g++-6
+      addons:
+        apt:
+          sources:
+          - ubuntu-toolchain-r-test
+          packages:
+          - gcc-6
+          - g++-6
+    - os: linux
+      language: python
+      python: 3.6
       env: BUILD_SYSTEM=python C_COMPILER=gcc-6 CXX_COMPILER=g++-6
       addons:
         apt:
@@ -163,14 +176,10 @@ matrix:
       env: BUILD_SYSTEM=cmake C_COMPILER=gcc-4.4 CXX_COMPILER=g++-4.4
 
     ###
-    ## Python OS X builds
+    ## Python 2.7 OS X build (using the system /usr/bin/python)
     ###
     - os: osx
-      env: BUILD_SYSTEM=python INSTALL_TYPE=macpython PYTHON_VERSION=2.7.12 C_COMPILER=gcc CXX_COMPILER=g++
+      env: BUILD_SYSTEM=python C_COMPILER=gcc CXX_COMPILER=g++
-    - os: osx
-      env: BUILD_SYSTEM=python INSTALL_TYPE=macpython PYTHON_VERSION=3.4.4 C_COMPILER=gcc CXX_COMPILER=g++
-    - os: osx
-      env: BUILD_SYSTEM=python INSTALL_TYPE=macpython PYTHON_VERSION=3.5.2 C_COMPILER=gcc CXX_COMPILER=g++
 
     ###
     ## Sanitizers
@@ -262,19 +271,9 @@ after_success:
 - scripts/.travis.sh after_success
 
 before_deploy:
-- if [ "${BUILD_SYSTEM}" = "python" ] && [ "${TRAVIS_OS_NAME}" = "osx" ]; then export WHEELS=$(ls ./dist/*.whl); fi
 - scripts/.travis.sh before_deploy
 
 deploy:
-- provider: releases
-  api_key:
-    secure: YcCBi6W/w4dtKCa59Wfm8L5lGWvK7KxaFNDr3yh1Hz5aStXXf758pEMHGewnlbfbwuj5a3SjBb1nLp1M69OQJfxm442uXBaBKo52PM9PPbD7NjvbNIso73pqcSODXQXKuZxDFpEhfuDTVq3hUkUqiwhChWhrFucJsSL51i7qSss=
-  file: "${WHEELS}"
-  skip_cleanup: true
-  on:
-    repo: "google/brotli"
-    tags: true
-    condition: "${BUILD_SYSTEM} = python && ${TRAVIS_OS_NAME} = osx"
 - provider: bintray
   file: "scripts/.bintray.json"
   user: "eustas"

python/Makefile

@@ -8,10 +8,12 @@
 .PHONY: all
 # Build
 .PHONY: build
+# Develop
+.PHONY: develop
 # Install
 .PHONY: install
 # Test
-.PHONY: test tests
+.PHONY: test
 # Clean
 .PHONY: clean
 # Format
@@ -31,16 +33,17 @@ all: build
 build: $(EXTENSIONS)
 
 $(EXTENSIONS): $(EXT_SOURCES)
+	@cd .. && $(PYTHON) setup.py build_ext --inplace
+
+develop:
 	@cd .. && $(PYTHON) setup.py develop
 
 install:
 	@cd .. && $(PYTHON) setup.py install
 
-test: tests
+test: build
-
-tests: build
 	@echo 'running tests'
-	@$(PYTHON) -m unittest discover -p '*_test.py'
+	@$(PYTHON) -m unittest discover -v -p '*_test.py'
 
 clean:
 	@cd .. && $(PYTHON) setup.py clean

python/README.md

@@ -17,19 +17,20 @@ following command from this directory:
 
 ### Development
 
-For development, reinstalling the module with every change is time
+You may run the following commands from this directory:
-consuming.  Instead, we recommend using the `setuptools`
-"[development mode][]" to make the module available while still being
-able to edit the source files.
 
-For convenience, you may run the following commands from this directory:
+    $ make          # Build the module in-place
    
-    $ make          # Deploy the module in "development mode"
+    $ make test     # Test the module
-   
-    $ make tests    # Test the module
 
     $ make clean    # Remove all temporary files and build output
 
+If you wish to make the module available while still being
+able to edit the source files, you can use the `setuptools`
+"[development mode][]":
+
+    $ make develop  # Install the module in "development mode"
+
 ### Code Style
 
 Brotli's code follows the [Google Python Style Guide][].  To

scripts/.travis.sh

@@ -16,11 +16,6 @@ case "$1" in
 		esac
 
 		case "${BUILD_SYSTEM}" in
-		    "python")
-			source scripts/terryfy/travis_tools.sh
-			get_python_environment $INSTALL_TYPE $PYTHON_VERSION venv
-			pip install --upgrade wheel
-			;;
 		    "bazel")
 			brew install bazel
 			;;
@@ -50,10 +45,7 @@ case "$1" in
 		ctest -V
 		;;
 	    "python")
-		if [ "${TRAVIS_OS_NAME}" = "osx" ]; then
+		python setup.py test
-			source venv/bin/activate
-		fi
-		python setup.py build test
 		;;
 	    "maven")
 		cd java/org/brotli
@@ -65,16 +57,6 @@ case "$1" in
 	esac
 	;;
     "after_success")
-	case "${BUILD_SYSTEM}" in
-	    "python")
-		case "${TRAVIS_OS_NAME}" in
-		    "osx")
-			source venv/bin/activate
-			pip wheel -w dist .
-			;;
-		esac
-		;;
-	esac
 	;;
     "before_deploy")
 	case "${BUILD_SYSTEM}" in

scripts/appveyor.yml

@@ -1,11 +1,4 @@
 environment:
-
-  global:
-    # SDK v7.0 MSVC Express 2008's SetEnv.cmd script will fail if the
-    # /E:ON and /V:ON options are not enabled in the batch script intepreter
-    # See: http://stackoverflow.com/a/13751649/163740
-    WITH_COMPILER: "cmd /E:ON /V:ON /C .\\scripts\\run_with_compiler.cmd"
-
   matrix:
     - BUILD_SYSTEM: CMake
       GENERATOR: Visual Studio 14 2015 Win64
@@ -25,13 +18,8 @@ environment:
       PYTHON_ARCH: "32"
 
     - BUILD_SYSTEM: Python
-      PYTHON: "C:\\Python34"
+      PYTHON: "C:\\Python36"
-      PYTHON_VERSION: "3.4.x"
+      PYTHON_VERSION: "3.6.x"
-      PYTHON_ARCH: "32"
-
-    - BUILD_SYSTEM: Python
-      PYTHON: "C:\\Python35"
-      PYTHON_VERSION: "3.5.0"
       PYTHON_ARCH: "32"
 
     - BUILD_SYSTEM: Python
@@ -40,29 +28,21 @@ environment:
       PYTHON_ARCH: "64"
 
     - BUILD_SYSTEM: Python
-      PYTHON: "C:\\Python34-x64"
+      PYTHON: "C:\\Python36-x64"
-      PYTHON_VERSION: "3.4.x"
+      PYTHON_VERSION: "3.6.x"
-      PYTHON_ARCH: "64"
-
-    - BUILD_SYSTEM: Python
-      PYTHON: "C:\\Python35-x64"
-      PYTHON_VERSION: "3.5.0"
       PYTHON_ARCH: "64"
 
 install:
   - ps: |
       if ($env:BUILD_SYSTEM -eq "Python") {
-        # install Python and pip when not already installed
+        # prepend Python to the PATH
-        if (-not(Test-Path($env:PYTHON))) { & scripts\install_py.ps1 }
-
-        # prepend newly installed Python to the PATH
         $env:Path = $env:PYTHON + ';' + $env:PYTHON + '\\Scripts;' + $env:PATH
 
         # upgrade pip to avoid out-of-date warnings
         pip install --disable-pip-version-check --user --upgrade pip
 
-        # install/upgrade setuptools and wheel to build packages
+        # install/upgrade setuptools to build packages
-        pip install --upgrade setuptools six wheel
+        pip install --upgrade setuptools
       }
 
 before_build:
@@ -82,26 +62,4 @@ build_script:
 
 test_script:
   - if "%BUILD_SYSTEM%" == "CMake" ( ctest --output-on-failure --interactive-debug-mode 0 -C Debug )
-  - if "%BUILD_SYSTEM%" == "Python" ( python setup.py build test )
+  - if "%BUILD_SYSTEM%" == "Python" ( python setup.py test )
-
-after_test:
-  - if "%BUILD_SYSTEM%" == "Python" ( pip wheel -w dist . )
-  - if "%BUILD_SYSTEM%" == "Python" ( python setup.py sdist --formats=gztar,zip )
-
-artifacts:
-  # archive the generated packages in the ci.appveyor.com build report
-  - path: dist\*.whl
-  - path: dist\*.zip
-  - path: dist\*.tar.gz
-
-# For info, see: http://www.appveyor.com/docs/deployment/github
-
-deploy:
-  - provider: GitHub
-    auth_token:
-      secure: dfL56DgbwuGJNNE5GzKi/pAgBQnJ37Du+AnCtnsTnIYxpis8ah3fPmA/G+bn4NJ3
-    artifact:
-    draft: false
-    prerelease: false
-    on:
-      appveyor_repo_tag: true

scripts/install_py.ps1

@@ -1,177 +0,0 @@
-# Sample script to install Python and pip under Windows
-# Authors: Olivier Grisel, Jonathan Helmus, Kyle Kastner, and Alex Willmer
-# License: CC0 1.0 Universal: http://creativecommons.org/publicdomain/zero/1.0/
-# Source: https://github.com/ogrisel/python-appveyor-demo/blob/master/appveyor/install.ps1
-
-$BASE_URL = "https://www.python.org/ftp/python/"
-$GET_PIP_URL = "https://bootstrap.pypa.io/get-pip.py"
-$GET_PIP_PATH = "C:\get-pip.py"
-
-$PYTHON_PRERELEASE_REGEX = @"
-(?x)
-(?<major>\d+)
-\.
-(?<minor>\d+)
-\.
-(?<micro>\d+)
-(?<prerelease>[a-z]{1,2}\d+)
-"@
-
-
-function Download ($filename, $url) {
-    $webclient = New-Object System.Net.WebClient
-
-    $basedir = $pwd.Path + "\"
-    $filepath = $basedir + $filename
-    if (Test-Path $filename) {
-        Write-Host "Reusing" $filepath
-        return $filepath
-    }
-
-    # Download and retry up to 3 times in case of network transient errors.
-    Write-Host "Downloading" $filename "from" $url
-    $retry_attempts = 2
-    for ($i = 0; $i -lt $retry_attempts; $i++) {
-        try {
-            $webclient.DownloadFile($url, $filepath)
-            break
-        }
-        Catch [Exception]{
-            Start-Sleep 1
-        }
-    }
-    if (Test-Path $filepath) {
-        Write-Host "File saved at" $filepath
-    } else {
-        # Retry once to get the error message if any at the last try
-        $webclient.DownloadFile($url, $filepath)
-    }
-    return $filepath
-}
-
-
-function ParsePythonVersion ($python_version) {
-    if ($python_version -match $PYTHON_PRERELEASE_REGEX) {
-        return ([int]$matches.major, [int]$matches.minor, [int]$matches.micro,
-                $matches.prerelease)
-    }
-    $version_obj = [version]$python_version
-    return ($version_obj.major, $version_obj.minor, $version_obj.build, "")
-}
-
-
-function DownloadPython ($python_version, $platform_suffix) {
-    $major, $minor, $micro, $prerelease = ParsePythonVersion $python_version
-
-    if (($major -le 2 -and $micro -eq 0) `
-        -or ($major -eq 3 -and $minor -le 2 -and $micro -eq 0) `
-        ) {
-        $dir = "$major.$minor"
-        $python_version = "$major.$minor$prerelease"
-    } else {
-        $dir = "$major.$minor.$micro"
-    }
-
-    if ($prerelease) {
-        if (($major -le 2) `
-            -or ($major -eq 3 -and $minor -eq 1) `
-            -or ($major -eq 3 -and $minor -eq 2) `
-            -or ($major -eq 3 -and $minor -eq 3) `
-            ) {
-            $dir = "$dir/prev"
-        }
-    }
-
-    if (($major -le 2) -or ($major -le 3 -and $minor -le 4)) {
-        $ext = "msi"
-        if ($platform_suffix) {
-            $platform_suffix = ".$platform_suffix"
-        }
-    } else {
-        $ext = "exe"
-        if ($platform_suffix) {
-            $platform_suffix = "-$platform_suffix"
-        }
-    }
-
-    $filename = "python-$python_version$platform_suffix.$ext"
-    $url = "$BASE_URL$dir/$filename"
-    $filepath = Download $filename $url
-    return $filepath
-}
-
-
-function InstallPython ($python_version, $architecture, $python_home) {
-    Write-Host "Installing Python" $python_version "for" $architecture "bit architecture to" $python_home
-    if (Test-Path $python_home) {
-        Write-Host $python_home "already exists, skipping."
-        return $false
-    }
-    if ($architecture -eq "32") {
-        $platform_suffix = ""
-    } else {
-        $platform_suffix = "amd64"
-    }
-    $installer_path = DownloadPython $python_version $platform_suffix
-    $installer_ext = [System.IO.Path]::GetExtension($installer_path)
-    Write-Host "Installing $installer_path to $python_home"
-    $install_log = $python_home + ".log"
-    if ($installer_ext -eq '.msi') {
-        InstallPythonMSI $installer_path $python_home $install_log
-    } else {
-        InstallPythonEXE $installer_path $python_home $install_log
-    }
-    if (Test-Path $python_home) {
-        Write-Host "Python $python_version ($architecture) installation complete"
-    } else {
-        Write-Host "Failed to install Python in $python_home"
-        Get-Content -Path $install_log
-        Exit 1
-    }
-}
-
-
-function InstallPythonEXE ($exepath, $python_home, $install_log) {
-    $install_args = "/quiet InstallAllUsers=1 TargetDir=$python_home"
-    RunCommand $exepath $install_args
-}
-
-
-function InstallPythonMSI ($msipath, $python_home, $install_log) {
-    $install_args = "/qn /log $install_log /i $msipath TARGETDIR=$python_home"
-    $uninstall_args = "/qn /x $msipath"
-    RunCommand "msiexec.exe" $install_args
-    if (-not(Test-Path $python_home)) {
-        Write-Host "Python seems to be installed else-where, reinstalling."
-        RunCommand "msiexec.exe" $uninstall_args
-        RunCommand "msiexec.exe" $install_args
-    }
-}
-
-function RunCommand ($command, $command_args) {
-    Write-Host $command $command_args
-    Start-Process -FilePath $command -ArgumentList $command_args -Wait -Passthru
-}
-
-
-function InstallPip ($python_home) {
-    $pip_path = $python_home + "\Scripts\pip.exe"
-    $python_path = $python_home + "\python.exe"
-    if (-not(Test-Path $pip_path)) {
-        Write-Host "Installing pip..."
-        $webclient = New-Object System.Net.WebClient
-        $webclient.DownloadFile($GET_PIP_URL, $GET_PIP_PATH)
-        Write-Host "Executing:" $python_path $GET_PIP_PATH
-        & $python_path $GET_PIP_PATH
-    } else {
-        Write-Host "pip already installed."
-    }
-}
-
-
-function main () {
-    InstallPython $env:PYTHON_VERSION $env:PYTHON_ARCH $env:PYTHON
-    InstallPip $env:PYTHON
-}
-
-main

scripts/run_with_compiler.cmd

@@ -1,80 +0,0 @@
-:: To build extensions for 64 bit Python 3, we need to configure environment
-:: variables to use the MSVC 2010 C++ compilers from GRMSDKX_EN_DVD.iso of:
-:: MS Windows SDK for Windows 7 and .NET Framework 4 (SDK v7.1)
-::
-:: To build extensions for 64 bit Python 2, we need to configure environment
-:: variables to use the MSVC 2008 C++ compilers from GRMSDKX_EN_DVD.iso of:
-:: MS Windows SDK for Windows 7 and .NET Framework 3.5 (SDK v7.0)
-::
-:: 32 bit builds, and 64-bit builds for 3.5 and beyond, do not require specific
-:: environment configurations.
-::
-:: Note: this script needs to be run with the /E:ON and /V:ON flags for the
-:: cmd interpreter, at least for (SDK v7.0)
-::
-:: More details at:
-:: https://github.com/cython/cython/wiki/64BitCythonExtensionsOnWindows
-:: http://stackoverflow.com/a/13751649/163740
-::
-:: Original source:
-:: https://github.com/ogrisel/python-appveyor-demo/blob/master/appveyor/run_with_env.cmd
-::
-:: Author: Olivier Grisel
-:: License: CC0 1.0 Universal: http://creativecommons.org/publicdomain/zero/1.0/
-@ECHO OFF
-
-SET COMMAND_TO_RUN=%*
-SET WIN_SDK_ROOT=C:\Program Files\Microsoft SDKs\Windows
-SET WIN_WDK=c:\Program Files (x86)\Windows Kits\10\Include\wdf
-
-:: Extract the major and minor versions, and allow for the minor version to be
-:: more than 9.  This requires the version number to have two dots in it.
-SET MAJOR_PYTHON_VERSION=%PYTHON_VERSION:~0,1%
-IF "%PYTHON_VERSION:~3,1%" == "." (
-    SET MINOR_PYTHON_VERSION=%PYTHON_VERSION:~2,1%
-) ELSE (
-    SET MINOR_PYTHON_VERSION=%PYTHON_VERSION:~2,2%
-)
-
-:: Based on the Python version, determine what SDK version to use, and whether
-:: to set the SDK for 64-bit.
-IF %MAJOR_PYTHON_VERSION% == 2 (
-    SET WINDOWS_SDK_VERSION="v7.0"
-    SET SET_SDK_64=Y
-) ELSE (
-    IF %MAJOR_PYTHON_VERSION% == 3 (
-        SET WINDOWS_SDK_VERSION="v7.1"
-        IF %MINOR_PYTHON_VERSION% LEQ 4 (
-            SET SET_SDK_64=Y
-        ) ELSE (
-            SET SET_SDK_64=N
-            IF EXIST "%WIN_WDK%" (
-                :: See: https://connect.microsoft.com/VisualStudio/feedback/details/1610302/
-                REN "%WIN_WDK%" 0wdf
-            )
-        )
-    ) ELSE (
-        ECHO Unsupported Python version: "%MAJOR_PYTHON_VERSION%"
-        EXIT 1
-    )
-)
-
-IF %PYTHON_ARCH% == 64 (
-    IF %SET_SDK_64% == Y (
-        ECHO Configuring Windows SDK %WINDOWS_SDK_VERSION% for Python %MAJOR_PYTHON_VERSION% on a 64 bit architecture
-        SET DISTUTILS_USE_SDK=1
-        SET MSSdk=1
-        "%WIN_SDK_ROOT%\%WINDOWS_SDK_VERSION%\Setup\WindowsSdkVer.exe" -q -version:%WINDOWS_SDK_VERSION%
-        "%WIN_SDK_ROOT%\%WINDOWS_SDK_VERSION%\Bin\SetEnv.cmd" /x64 /release
-        ECHO Executing: %COMMAND_TO_RUN%
-        call %COMMAND_TO_RUN% || EXIT 1
-    ) ELSE (
-        ECHO Using default MSVC build environment for 64 bit architecture
-        ECHO Executing: %COMMAND_TO_RUN%
-        call %COMMAND_TO_RUN% || EXIT 1
-    )
-) ELSE (
-    ECHO Using default MSVC build environment for 32 bit architecture
-    ECHO Executing: %COMMAND_TO_RUN%
-    call %COMMAND_TO_RUN% || EXIT 1
-)

scripts/terryfy

@@ -1 +0,0 @@
-Subproject commit 8bb673f4410819df06920fdcfd24e18d235d84f7

setup.py

@@ -15,6 +15,9 @@ except:
     from distutils.core import Extension
     from distutils.core import setup
 from distutils.command.build_ext import build_ext
+from distutils import errors
+from distutils import dep_util
+from distutils import log
 
 
 CURR_DIR = os.path.abspath(os.path.dirname(os.path.realpath(__file__)))
@@ -53,6 +56,20 @@ class BuildExt(build_ext):
         return filenames
 
     def build_extension(self, ext):
+        if ext.sources is None or not isinstance(ext.sources, (list, tuple)):
+            raise errors.DistutilsSetupError(
+                "in 'ext_modules' option (extension '%s'), "
+                "'sources' must be present and must be "
+                "a list of source filenames" % ext.name)
+
+        ext_path = self.get_ext_fullpath(ext.name)
+        depends = ext.sources + ext.depends
+        if not (self.force or dep_util.newer_group(depends, ext_path, 'newer')):
+            log.debug("skipping '%s' extension (up-to-date)", ext.name)
+            return
+        else:
+            log.info("building '%s' extension", ext.name)
+
         c_sources = []
         cxx_sources = []
         for source in ext.sources:
@@ -211,7 +228,7 @@ EXT_MODULES = [
             'c/enc/cluster_inc.h',
             'c/enc/command.h',
             'c/enc/compress_fragment.h',
-            'c/enc/compress_fragment_two_pass.h'
+            'c/enc/compress_fragment_two_pass.h',
             'c/enc/context.h',
             'c/enc/dictionary_hash.h',
             'c/enc/entropy_encode.h',