Commit Graph

150 Commits

Author SHA1 Message Date
David L. Jones
ff06e23d23
Add a setuptools extension for generating Python protobufs (#7783)
This extension allows Python code to be generated from setup.py files, so they
are created as part of a normal Python build. The extension uses an already-existing
protoc binary, which can be explicitly specified if needed.
2020-08-31 10:40:57 -07:00
Paul Yang
d4ca92962c
Port php c extension to php8 (#7793)
* Only ported c extension to php8.
* Didn't fixed the issue of throwing warnings for missing arginfo in bundled files.
* Tests not fixed, because syntax of phpunit (<7 vs >9.3) are not compatible.
* In next release, needs to drop php5 and php7.0 support (in order to use phpunit > 7)
2020-08-11 19:30:46 -07:00
David L. Jones
0f9ae07fe0 Override pip and virtualenv installation for macOS. (#7704)
This commit cleans up our environment setup for macOS builds.

-  Always use `python -m {pip | virtualenv}` form of commands.
-  Only upgrade in-place, and never use `--ignore-installed`.
-  Use latest tox on macos, and test 3.6 on macos presubmit.

These changes ensure that multibuilder won't break the system-installed Python on the Kokoro macOS builder, and thus won't resolve the ancient system version. For Tox, make sure we test with Python 3.6 (the version of Python in current macOS).
2020-08-07 14:44:49 -07:00
Joshua Haberman
093faebcdb
New PHP C Extension based on upb_msg (#7614)
* Added code for the new PHP extension.

* Removed a bunch of unused variables shown by compiler warnings.

* Test protobuf2 in the tests.

* Updated upb version to fix a goldenfile test.

* Added pure-PHP changes. Extension now passes all tests.

* Enabled protobuf2 for all C extension tests.

* Fixed pure=PHP lib: full names no longer start with '.'.

* Added files for new extension to Makefile.am.

* Downgraded make-preload.py to python 2, since python3 isn't available in the php_all Kokoro image.

* Disable tests of new C extension with PHP 5.x.

* Also do not compile the extension for PHP5.

* Accept version 5.*.*, and use /usr/bin/python.

* Addressed PR comments.

* Addressed PR comments.

* Added "const" to a parameter and fixed a memory leak seen in Valgrind.

* Stop testing the C extension for PHP5.

The next release of protobuf will deprecate the C extension
for PHP5, see:
https://github.com/protocolbuffers/protobuf/issues/7525

* Made the PHP5.6 Mac test only exercise pure-PHP.

* Build protoc for PHP 5.6 test.

* Rewrote bundling script in PHP to avoid dependency on Python.

* A few more fixes.

* Fixed int32/int64 behavior for 32-bit builds.

* Match more PHP versions in testing script.

* Use phpize --clean before building extension.

* Force-delete configure.in to avoid phpize problems cross-version.

* Delete both configure.ac and configure.in.
2020-06-25 12:21:01 -07:00
Joshua Haberman
f1ce8663ac
De-duplicated code to generate PHP protos and install phpunit. (#7605)
* De-duplicated code to generate PHP protos and install phpunit.

* Removed all references to generate_php_test_proto.

* Replaced with internal_build_cpp.

* Make Timestamp::__construct() static to avoid conflicts with MongoDB.

* Replicated PHPUnit versions and added new script to Makefile.am.

* Fixed filename in Makefile.am.

* Disabled test that SEGV's on macOS.

* Removed extraneous "set -e".

* Make sure generate_protos.sh happens on every test path.

* Removed stray '$' chars.

* Added proper support for aggregate_metadata tests. But now I get a stack overflow.

Stack overflow:
/Users/haberman/code/protobuf/php/tests/generated/GPBMetadata/Proto/TestDescriptors.php:16
/Users/haberman/code/protobuf/php/tests/generated/GPBMetadata/Proto/TestDescriptors.php:16
/Users/haberman/code/protobuf/php/tests/generated/GPBMetadata/Proto/TestDescriptors.php:16
/Users/haberman/code/protobuf/php/tests/generated/GPBMetadata/Proto/TestDescriptors.php:16
/Users/haberman/code/protobuf/php/tests/generated/GPBMetadata/Proto/TestDescriptors.php:16
/Users/haberman/code/protobuf/php/tests/generated/GPBMetadata/Proto/TestDescriptors.php:16
/Users/haberman/code/protobuf/php/tests/generated/GPBMetadata/Proto/TestDescriptors.php:16
/Users/haberman/code/protobuf/php/tests/generated/GPBMetadata/Proto/TestDescriptors.php:16

namespace GPBMetadata\Proto;

class TestDescriptors
{
    public static $is_initialized = false;

    public static function initOnce() {
        $pool = \Google\Protobuf\Internal\DescriptorPool::getGeneratedPool();
        if (static::$is_initialized == true) {
          return;
        }
        \GPBMetadata\Proto\TestDescriptors::initOnce();
        $pool->internalAddGeneratedFile(hex2bin(
            ""
        ), true);
        static::$is_initialized = true;
    }
}

* Fixed and verified metadata aggregation testing.
2020-06-09 15:40:29 -07:00
Thomas Van Lenten
ce8b7af46b Update the brew workflow
- Remove the `brew install`; the kokoro image already has it, just
  update/upgrade instead.
- Remove `prune`, logs had:
    """
    Error: Unknown command: prune
    """
- Remove `uninstall`, logs had:
    """
    Error: Refusing to uninstall /usr/local/Cellar/cmake/3.16.4 and /usr/local/Cellar/icu4c/64.2
    because they are required by ceres-solver, ffmpeg, harfbuzz, libass and opencv, which are currently installed.
    You can override this and force removal with:
      brew uninstall --ignore-dependencies node icu4c cmake wget
    """
- Skip installing some things since they are already in the base image, logs
  had:
    """
    Warning: gflags 2.2.2 is already installed and up-to-date
    To reinstall 2.2.2, run `brew reinstall gflags`
    Warning: openssl@1.1 1.1.1g is already installed and up-to-date
    To reinstall 1.1.1g, run `brew reinstall openssl@1.1`
    Warning: pcre 8.44 is already installed and up-to-date
    To reinstall 8.44, run `brew reinstall pcre`
    """
- Don't install gpg gpg2 as gnupg is already installed, also use gpg instead of
  gpg2 for commands (and update the commands), logs had:
    """
    kokoro/macos/prepare_build_macos_rc: line 44: gpg2: command not found
    kokoro/macos/prepare_build_macos_rc: line 45: gpg2: command not found
    """
- Add env guards to control all the option installs and only request them be
  installed in the cases that need it.  This avoids having to install/update
  the things like ruby when some other tool only needed in some configs is
  install differently and could have conflicts.
- Switch to brew for cocoapods to avoid compat issues on the supporting
  libraries.
2020-06-09 09:57:17 -04:00
Joshua Haberman
2cc68efd56
Simplified PHP testing scripts. (#7574)
* Simplified PHP testing scripts.

- allow them to be run from any directory.
- remove "VERSION" arg, we can get PHP from $PATH.

* Added a few places I missed.

* Removed redundant "cd `dirname $0`".

Also replaced all backticks with $(), for consistency.
2020-05-28 19:24:08 -07:00
Joshua Haberman
1688ea8d60
PHP: use php-config instead of explicitly setting include paths. (#7572) 2020-05-28 12:10:53 -07:00
Joshua Haberman
79b1c3bac3
Fixed JavaScript Kokoro tests by installing NodeJS. (#7504)
* Fix JavaScript tests.

* Added missing popd.

* Added missing 'export' for $PATH.

* Fixed path to include 'bin'.
2020-05-13 16:16:27 -07:00
Joshua Haberman
27ba0c5bb0
Fixed the PHP Kokoro build. (#7503)
* Fixed the PHP Kokoro build.

* Renamed all PHP7.4 Mac tests -> PHP7.3.
2020-05-13 15:36:21 -07:00
Joe Tsai
654edd8e74 Fix integration tests for Go examples
A proper integration test should be hermetic such that its operation
is agnostic to whatever happens outside of its controlled environment.
The script is not so since it uses a fixed (and very old) version of Go,
but continues to build protoc-gen-go from head, leading to an eventual
breakage when protoc-gen-go at head no longer operates on such an old
version of Go.

Fix the script to pin to an older version of protoc-gen-go (i.e., v1.3.5).
2020-05-12 14:47:54 -07:00
Masaki Hara
f494cb222d
Test in Ruby 2.7 (#7386)
* Test in Ruby 2.7

* Turn off GC.stress for 'require' in Ruby 2.7

* Backport 2c8364b to compatibility_tests/v3.0.0
2020-04-24 09:50:27 -07:00
Eric Walker
2c8364bcd1
Rails2.7 segfaults (#7091)
* Add a test suite for ruby 2.7

* Call BigDecimal() instead of BigDecimal.new()

BigDecimal.new was deprecated in ruby 2.6

* Switch FrozenError expectation to a matcher

The error message for FrozenError changed to include more information
about the mutated object. Switch from an exact match to an aproximate
match (equal => match). This does not change the prefix.

* We can safely ignore newest array methods from ruby 2.7
2020-04-17 10:20:38 -07:00
David L. Jones
f130827371 Don't expect to include Python ReadTheDocs metadata in the distribution. 2020-03-06 17:34:07 -08:00
Rafi Kamal
0eb476b696 Merge branch 3.11.x to master 2020-02-18 16:42:47 -08:00
Tim Swast
29c83baecc
python: add sphinx docs (#6525)
* python: generate documentation with Sphinx and Read the Docs

Background:

Formerly, the Python protobuf reference documentation was built with
[Epydoc](http://epydoc.sourceforge.net/). This package has not been
updated since 2008, and it has inconsistent formatting (see internal
issue 131415575) with most Python documentation. Sphinx is used for the
official docs.python.org docs as well as most other Python packages,
including the Google client libraries and related packages, such as
https://googleapis.dev/python/google-api-core/latest/

To build the docs with Sphinx:

1. Install the needed packages (`sphinx`, `sphinxcontrib-napoleon` for
Google-style docstring support). I've created a conda environment file
to make this easier:

```
conda env create -f python/docs/environment.yml
```

2. (Optional) Generate reference docs files and regenerate index:

```
cd python
python generate_docs.py
cd ..
```

3. Run Sphinx.

```
cd python/docs
make html
```

About this change:

The script at `python/generate_docs.py` creates a ReStructured Text file
for each public module in the protobuf Python package. The script also
updates the table of contents in `python/docs/index.rst` to point to
these module references.

Future work:

Testing the docs build on PRs requires contributors to actually do some
setup work to configure builds on their fork. It'd be better if CI had a
docs build session to verify that the Sphinx docs generation at least
runs.

There are many warnings due to not-quite-correct docstrings in the
actual Python code itself. I'm choosing to ignore these errors to keep
the PR small, but I recommend you fix these and then enable "fail on
warnings" in the docs build on CI.

* add docs to EXTRA_DIST

* add instructions to build documentation to generate_docs.py

* exclude python/odcs from cpp_distcheck
2020-02-11 11:40:17 -08:00
Paul Yang
85219578fa
Add continuous test for php7.4 on mac (#7153) 2020-01-30 12:45:47 -08:00
Rafi Kamal
ed8688de72
Merge pull request #7064 from rafi-kamal/3.11.x-202001071136
Merge branch 3.11.x to master
2020-01-08 12:24:20 -08:00
Brian Wignall
a104dffcb6 Fix typos (#7050)
Uses https://en.wikipedia.org/wiki/Wikipedia:Lists_of_common_misspellings/For_machines to find likely typos, with https://github.com/bwignall/typochecker to help automate the checking.
2020-01-08 10:18:20 -08:00
Rafi Kamal
4500f31a6a Merge branch 3.11.x to master 2020-01-07 11:52:29 -08:00
Jie Luo
7faab5eeeb add python38 (#7009)
* add python38

* add python3.8

* fix tox
2019-12-13 15:49:18 -08:00
Paul Yang
fe1790ca0d
Fix Multiple Request for PHP (#7008)
* Add scripts to test multirequest

* chmod ug+x multirequest.sh

* Add continuous test

* Compile c extension

* Class entry is obsolete in the second request
1) Needes to use class name in persistent map
2) Invalidate class entry stored in descriptor

* Add new files to dist

* Fix compile_extension

* Cleanup outputs for phpize
2019-12-12 13:59:51 -08:00
Paul Yang
a66423f0fd
Make c extension portable for php 7.4 (#6968)
* Make c extension portable for php 7.4

* Fix conformance tests

* Fix comments

* Fix 32-bit

* Update conformance failure list

* Fix compiler warnings

* Cleanup configure created by phpize
The file created in php 7.4 is not recognizable by previous versions

* Fix conformance tests for 64-bit php

* Fix conformance test

* Fix compile warning

* Fix compile warnings
2019-12-09 17:02:10 -08:00
Tomo Suzuki
6396a47d5d Simplifying linkage monitor test setup (#6855) 2019-12-04 09:49:51 -08:00
Rafi Kamal
9b0fa3e7fb Remove java/lite/proguard.pgcfg from EXTRA_DIST since it's not part of
protoc.
2019-11-12 13:38:58 -08:00
Tomo Suzuki
ca25badcbe Installing BOM before running Java test 2019-10-22 13:42:35 -04:00
Tomo Suzuki
c1a046528c MVN 2019-09-06 09:56:55 -04:00
Tomo Suzuki
f460c178fe Local install before changing version 2019-09-06 09:53:44 -04:00
Jie Luo
d1eeb852fc
compatibility test between last released and current for java python php (#6441)
* compatibility tests between last released and the current version
2019-07-31 17:49:26 -07:00
Jie Luo
e399308ef0 Add csharp compatibility tests between last released and the current version (#6407)
* Add csharp compatibility tests for 3.7.0

* Add compatibility test between last major version with current for C#

* Update last version number

* compatibility tests between last released and the current version

* fix typo
2019-07-25 14:31:00 -07:00
Tomo Suzuki
bedef1ede2 Kokoro: run Linkage Monitor on presubmit (#6318)
* adding mvn install with snapshot version

* Added linkage monitor

* comment

* space

* Added comment

* Maven to use $HOME/.m2

* setting MVN
2019-07-01 14:03:04 -07:00
Tomo Suzuki
7051725941 To setup JDK8 2019-06-26 15:28:49 -04:00
Adam Cozzette
8645d893ab Merge branch '3.7.x' into merge-3-7-x 2019-03-26 14:32:20 -07:00
David Supplee
d65b8647f2 Add terminating character to zend_lookup_class call (#5871) (#5885)
* add terminating character

* adjust name length input

* add test cases

* correct script name

* Misc fixes

* add new script to EXTRA_DIST list
2019-03-13 13:31:55 -07:00
Jan Tattermusch
db1d4dc4c1
Merge pull request #5876 from jtattermusch/csharp_win_tests
C# testing improvements
2019-03-13 07:27:29 +01:00
David Supplee
93533f780b Add terminating character to zend_lookup_class call (#5871)
* add terminating character

* adjust name length input

* add test cases

* correct script name

* Misc fixes

* add new script to EXTRA_DIST list
2019-03-12 21:30:06 -07:00
Adam Cozzette
45aba801ed Avoid passing -Og flag to C++ compiler
It appears that some versions of Clang do not support this flag, and
this came up as a problem for Homebrew builds in issue #5815.
2019-03-12 10:15:50 -07:00
Jan Tattermusch
67fee07308 C#: set some dotnet options when testing 2019-03-12 05:16:52 -04:00
Hao Nguyen
f85c823c90
Create test to verify that the dist artifact can be installed. (#5854)
* Add a new test for verifying that the dist package can be installed.
2019-03-07 14:34:28 -08:00
Joshua Haberman
f5e8ee41dd
PHP updates for new upb APIs (#5604) 2019-03-06 12:04:17 -08:00
Bo Yang
cb6fa92ee8 Merge branch '3.7.x' 2019-03-04 10:18:18 -08:00
Yilun Chong
400845ae17 fix macos build for High Sierra (#5780)
* fix macos build by changing xcode version

* add comments

* fix benchmark build for mac

* fix for gem option and install npm

* add test for node and npm

* fix node

* fix for node

* another attempts to fix node

* another attempts

* clean up
2019-02-26 15:51:04 -08:00
Yilun Chong
89b914f81d
fix macos build for High Sierra (#5780)
* fix macos build by changing xcode version

* add comments

* fix benchmark build for mac

* fix for gem option and install npm

* add test for node and npm

* fix node

* fix for node

* another attempts to fix node

* another attempts

* clean up
2019-02-26 12:52:06 -08:00
Bo Yang
8400e29044 Divide commands into separate lines 2019-01-28 20:58:58 +00:00
Bo Yang
71e5de01d3 phpunit-5.6.10 link is broken
Use 5.6.8 instead
2019-01-28 20:57:51 +00:00
Adam Cozzette
d135f07bba
Do "strict" heapcheck instead of "draconian" (#5635)
The draconian heapcheck turns up memory leaks outside our control, such
as ones in libc and in the heapchecker itself. I think for an automated
test it makes more sense for us to use the "strict" heapcheck, since
that already passes and should only report leaks that are within our
control.
2019-01-25 10:28:52 -08:00
Hao Nguyen
4f8a635679
Create test for protoc with tcmalloc (#5612)
* Add TC malloc kokoro test

* Update tcmalloc test
2019-01-23 10:02:51 -08:00
Thomas Van Lenten
bd00671b92 ObjC: Add a Xcode project for tvOS.
The CocoaPod (and source) have always support tvOS, but adding the project
makes it easier to run the tests and debug if there ever were an issue.

(not adding a watchOS project at the moment because Apple doesn't provide
 XCTest, so all it could do is build the library. We still support that,
 just can't "test" it.)
2019-01-08 08:22:12 -05:00
Paul Yang
de9e1a04a6
Add ruby 2.6 test (#5528)
* Add ruby 2.6 test

* Fix broken tests

* Fix compatibility test

* Fix bundler version

* Use new docker image

* Fix broken test

* Use new docker image in ruby26 test

* Install ruby for mac
2019-01-03 14:25:50 -08:00
Yilun Chong
152c8301cf
Fix python benchmark throughput; Change back to not using docker for benchmar (#5503) 2018-12-20 17:15:51 -08:00