Commit Graph

157 Commits

Author SHA1 Message Date
Paul Yang
525be941e0
Add test for php5 for pure php implementation (#8251)
* Add test for php5 for pure php implementation

* Add php5.6 to docker image

* Add php5.6 to php 32-bit docker image
2021-02-04 09:50:28 -08:00
David L. Jones
ac63e80bbb
Enhance explanation in kokoro/README.md. (#8253) 2021-02-02 16:49:58 -08:00
gerben-s
10ecb080cb
Merge pull request #8121 from busunkim96/add-39
add python 3.9
2021-02-01 19:34:02 -08:00
gerben-s
fe47e99a4f
Add cmake to the path to fix the build on kokoro (#8246) 2021-02-01 15:48:50 -08:00
Jan Tattermusch
94c3916dcf
fix problem when restoring C# dependencies on win2016 (#8243) 2021-02-01 09:25:56 -08:00
Joshua Haberman
9abf6e2ab0
Ported Ruby extension to upb_msg (#8184)
* WIP.

* WIP.

* WIP.

* WIP.

* WIP.

* WIP.

* Added some missing files.

* WIP.

* WIP.

* Updated upb.

* Extension loads, but crashes immediately.

* Gets through the test suite without SEGV!

Still a lot of bugs to fix, but it is a major step!

214 tests, 378 assertions, 37 failures, 147 errors, 0 pendings, 0 omissions, 0 notifications
14.0187% passed

* Test and build for Ruby 3.0

* Fixed a few more bugs, efficient #inspect is almost done.

214 tests, 134243 assertions, 30 failures, 144 errors, 0 pendings, 0 omissions, 0 notifications
18.6916% passed

* Fixed message hash initialization and encode depth checking.

214 tests, 124651 assertions, 53 failures, 70 errors, 0 pendings, 0 omissions, 0 notifications
42.5234% passed

* A bunch of fixes to failing tests, now 70% passing.

214 tests, 202091 assertions, 41 failures, 23 errors, 0 pendings, 0 omissions, 0 notifications
70.0935% passed

* More than 80% of tests are passing now.

214 tests, 322331 assertions, 30 failures, 9 errors, 0 pendings, 0 omissions, 0 notifications
81.7757% passed

Unfortunately there is also a sporadic bug/segfault hanging around
that appears to be GC-related.

* Add linux/ruby30 and macos/ruby30

* Use rvm master for 3.0.0-preview2

* Over 90% of tests are passing!

214 tests, 349898 assertions, 15 failures, 1 errors, 0 pendings, 0 omissions, 0 notifications
92.5234% passed

* Passes all tests!

214 tests, 369388 assertions, 0 failures, 0 errors, 0 pendings, 0 omissions, 0 notifications
100% passed

* A bunch of cleanup.

1. Removed a bunch of internal-only symbols from headers.
2. Required a frozen check to get a non-const pointer to a map or array.
3. De-duplicated the code to get a type argument for Map/RepeatedField.

* Removed a bunch more stuff from protobuf.h.  There is an intermittent assert failure.

Intermittent failure:

ruby: ../../../../ext/google/protobuf_c/protobuf.c:263: ObjectCache_Add: Assertion `rb_funcall(obj_cache2, (__builtin_constant_p("[]") ? __extension__ ({ static ID rb_intern_id_cache; if (!rb_intern_id_cache) rb_intern_id_cache = rb_intern2((("[]")
), (long)strlen(("[]"))); (ID) rb_intern_id_cache; }) : rb_intern("[]")), 1, key_rb) == val' failed

* Removed a few more things from protobuf.h.

* Ruby 3.0.0-preview2 to 3.0.0

* Require rake-compiler-dock >= 1.1.0

* More progress, fighting with the object cache.

* Passes on all Ruby versions!

* Updated and clarified comment regarding WeakMap.

* Fixed the wyhash compile.

* Fixed conformance tests for Ruby.

Conformance results now look like:

RUBYLIB=../ruby/lib:. ./conformance-test-runner --enforce_recommended --failure_list failure_list_ruby.txt --text_format_failure_list text_format_failure_list_ruby.txt ./conformance_ruby.rb

CONFORMANCE TEST BEGIN ====================================

CONFORMANCE SUITE PASSED: 1955 successes, 0 skipped, 58 expected failures, 0 unexpected failures.

CONFORMANCE TEST BEGIN ====================================

CONFORMANCE SUITE PASSED: 0 successes, 111 skipped, 8 expected failures, 0 unexpected failures.

Fixes include:

- Changed Ruby compiler to no longer reject proto2 maps.
- Changed Ruby compiler to emit a warning when proto2 extensions are
  present instead of rejecting the .proto file completely.
- Fixed conformance tests to allow proto2 and look up message by name
  instead of hardcoding a specific list of messages.
- Fixed conformance test to support the "ignore unknown" option for
  JSON.
- Fixed conformance test to properly report serialization errors.

* Removed debug printf and fixed #inspect for floats.

* Fixed compatibility test to have proper semantics for #to_json.

* Updated Makefile.am with new file list.

* Don't try to copy wyhash when inside Docker.

* Fixed bug where we would forget that a sub-object is frozen in Ruby >=2.7.

* Avoid exporting unneeded symbols and refactored a bit of code.

* Some more refactoring.

* Simplified and added more comments.

* Some more comments and simplification. Added a missing license block.

Co-authored-by: Masaki Hara <hara@wantedly.com>
2021-01-13 12:16:25 -08:00
Adam Cozzette
635156e406 Fix Mac tests by making an unshallow copy of Homebrew repos
It seems that updating shallow Git clones is expensive, and as a result
Homebrew recently started refusing to update shallow clones (see
https://github.com/Homebrew/discussions/discussions/226). This commit
tries to fix the problem by making these repos into full clones before
running "brew update".

I also came across another error about there being a conflicting version
of pip3 in /usr/local. I suspect that is related to the other Python
binaries that the script has to delete, so I added pip3 to the list and
that seemed to solve the problem.
2020-12-23 11:52:19 -08:00
Paul Yang
823f351448
Port for php8 (#8105)
* Port for php8

* Port php c extension for php8

* Update composer.json

* Drop php7.0 support

* Update phpunit for php7.1 in c extension test

* Add back support for php7.0

* Add badge for php8 continuous build
2020-12-04 13:46:34 -08:00
Bu Sun Kim
51c80ce1d5 add python 3.9 2020-12-04 21:16:47 +00:00
Paul Yang
70531697bc
Update docker file for php 32bit tests (#8116) 2020-12-02 16:44:56 -08:00
Paul Yang
e8906e4ecd
Update docker file for php7 (#8112) 2020-12-02 10:10:18 -08:00
Paul Yang
9432c99f37
Update docker image for php8 (#8103) 2020-11-30 10:19:50 -08:00
Paul Yang
4eea2b00a3
Remove php 5 tests on mac (#7956) 2020-10-09 15:53:12 -07:00
David L. Jones
b774da0d42
Change Python kokoro configs to test only their specified version. (#7927)
Using non-versioned scripts for `build_file` means every Python version is
tested (under Tox), and this is multiplied by each Python version running Tox.

For example, when the `python38` config is used to run Tox, the
Tox layer runs on Python 3.8. This then tests all of the Python versions
individually, including, for example, python27 tests.

This change fixes the `build_file` paths to point to the same-pathed build
script. For example, `kokoro/linux/python27/presubmit.cfg` now uses
`kokoro/linux/python27/build.sh`.

Some additional fixes:

*  Use `python -m tox` in tests.sh instead of just `tox`. This helps non-site
   installations of tox, where the `tox` script may not be on `$PATH`.
*  Ensure tox (and other Python build-related packages) are available in
   Python testing images. (New images have been pushed.)
*  Disable `--warnings_as_errors` due to deprecated function.
*  Remove apt lists per [Docker best practices][1].

[1]: https://docs.docker.com/develop/develop-images/dockerfile_best-practices/
2020-09-30 15:56:17 -07:00
Adam Cozzette
58f4260e8e Merged 3.13.x into master 2020-08-17 11:06:11 -07:00
Paul Yang
d4ca92962c
Port php c extension to php8 (#7793)
* Only ported c extension to php8.
* Didn't fixed the issue of throwing warnings for missing arginfo in bundled files.
* Tests not fixed, because syntax of phpunit (<7 vs >9.3) are not compatible.
* In next release, needs to drop php5 and php7.0 support (in order to use phpunit > 7)
2020-08-11 19:30:46 -07:00
Bo Yang
8947701be5 Add docker file for php8.0 2020-08-10 13:51:06 -07:00
Peter Newman
e2cc2de304
Fix lots of spelling errors (#7751)
* Fix a typo

* Fix lots of spelling errors

* Fix a few more spelling mistakes

* s/parsable/parseable/

* Don't touch the third party files

* Cloneable is the preferred C# term

* Copyable is the preferred C++ term

* Revert "s/parsable/parseable/"

This reverts commit 534ecf7675.

* Revert unparseable->unparsable corrections
2020-08-10 11:08:25 -07:00
David L. Jones
0f9ae07fe0 Override pip and virtualenv installation for macOS. (#7704)
This commit cleans up our environment setup for macOS builds.

-  Always use `python -m {pip | virtualenv}` form of commands.
-  Only upgrade in-place, and never use `--ignore-installed`.
-  Use latest tox on macos, and test 3.6 on macos presubmit.

These changes ensure that multibuilder won't break the system-installed Python on the Kokoro macOS builder, and thus won't resolve the ancient system version. For Tox, make sure we test with Python 3.6 (the version of Python in current macOS).
2020-08-07 14:44:49 -07:00
Jan Tattermusch
fed7e18193 upgrade dotnet SDK in the CI to 3.1 2020-06-30 08:53:31 -04:00
Thomas Van Lenten
ce8b7af46b Update the brew workflow
- Remove the `brew install`; the kokoro image already has it, just
  update/upgrade instead.
- Remove `prune`, logs had:
    """
    Error: Unknown command: prune
    """
- Remove `uninstall`, logs had:
    """
    Error: Refusing to uninstall /usr/local/Cellar/cmake/3.16.4 and /usr/local/Cellar/icu4c/64.2
    because they are required by ceres-solver, ffmpeg, harfbuzz, libass and opencv, which are currently installed.
    You can override this and force removal with:
      brew uninstall --ignore-dependencies node icu4c cmake wget
    """
- Skip installing some things since they are already in the base image, logs
  had:
    """
    Warning: gflags 2.2.2 is already installed and up-to-date
    To reinstall 2.2.2, run `brew reinstall gflags`
    Warning: openssl@1.1 1.1.1g is already installed and up-to-date
    To reinstall 1.1.1g, run `brew reinstall openssl@1.1`
    Warning: pcre 8.44 is already installed and up-to-date
    To reinstall 8.44, run `brew reinstall pcre`
    """
- Don't install gpg gpg2 as gnupg is already installed, also use gpg instead of
  gpg2 for commands (and update the commands), logs had:
    """
    kokoro/macos/prepare_build_macos_rc: line 44: gpg2: command not found
    kokoro/macos/prepare_build_macos_rc: line 45: gpg2: command not found
    """
- Add env guards to control all the option installs and only request them be
  installed in the cases that need it.  This avoids having to install/update
  the things like ruby when some other tool only needed in some configs is
  install differently and could have conflicts.
- Switch to brew for cocoapods to avoid compat issues on the supporting
  libraries.
2020-06-09 09:57:17 -04:00
Joshua Haberman
c077d949d8 Merge branch '3.12.x' into merge-312 2020-05-26 16:45:43 -07:00
Stanley Cheung
ef22086980
Fix :protobuf_objc bazel target (#7538)
* Potential 3.12 ObjC bazel patch

* Add test targets to bazel test

* Remove :protobuf_objc test target
2020-05-21 12:46:35 -07:00
Daniel Azuma
cf81f38b5d
Restore binary builds of Ruby 2.3 and 2.4 (#7529) (#7531) 2020-05-19 15:52:59 -07:00
Daniel Azuma
ba1ce70744
Restore binary builds of Ruby 2.3 and 2.4 (#7529) 2020-05-19 15:28:57 -07:00
Joshua Haberman
27ba0c5bb0
Fixed the PHP Kokoro build. (#7503)
* Fixed the PHP Kokoro build.

* Renamed all PHP7.4 Mac tests -> PHP7.3.
2020-05-13 15:36:21 -07:00
Joshua Haberman
e19bfc0e28
Stop building binary gems for ruby <2.5. (#7453)
It is no longer possible to "bundle install" these versions:

$ bundle install
Fetching gem metadata from https://rubygems.org/.................
reline-0.1.3 requires ruby version >= 2.5, which is incompatible with the current version, ruby 2.4.9p362
2020-05-01 12:35:09 -07:00
Masaki Hara
f494cb222d
Test in Ruby 2.7 (#7386)
* Test in Ruby 2.7

* Turn off GC.stress for 'require' in Ruby 2.7

* Backport 2c8364b to compatibility_tests/v3.0.0
2020-04-24 09:50:27 -07:00
Paul Yang
f15ac4091f
Drop 3.3, 3.4 and use single version docker images for all python tests (#7396)
* Create a new docker image for python 3.6

The previous one cannot build, because python3.6 was removed from
stretch.

* Drop support for 3.3 and 3.4

Also update all docker file to be single version
2020-04-18 14:39:52 -07:00
Masaki Hara
64f6c59ccb
Build extensions for Ruby 2.7 (#7027)
* Build extensions for Ruby 2.7

* Try installing bundler 2.x

* Try bumping rake-compiler-dock

* Use standard RCD images

* Avoid 'rake cross native' with rake-compiler-dock

* Use Ruby 2.5 for building Ruby <= 2.6

* Use rake-compiler 1.1.0

* Specify target

* Don't update Ruby test image for now
2020-04-16 23:31:47 -07:00
Joshua Haberman
c649397029
Set execute bit on files if and only if they begin with (#!). (#7347)
* Set execute bit on files if and only if they begin with (#!).

Git only tracks the 'x' (executable) bit on each file. Prior to this
CL, our files were a random mix of executable and non-executable.
This change imposes some order by making files executable if and only
if they have shebang (#!) lines at the beginning.

We don't have any executable binaries checked into the repo, so
we shouldn't need to worry about that case.

* Added fix_permissions.sh script to set +x iff a file begins with (#!).
2020-04-01 15:28:25 -07:00
David L. Jones
42903f87f0
Merge pull request #7154 from tswast/issue4498-googleapis.dev
python: publish sphinx docs to googleapis.dev
2020-02-27 14:30:54 -08:00
Rafi Kamal
0eb476b696 Merge branch 3.11.x to master 2020-02-18 16:42:47 -08:00
Rafi Kamal
df2bce345d
Remove 32-bit Mac protoc reference from collect_all_artifacts (#7228) 2020-02-18 12:55:01 -08:00
Rafi Kamal
d314101531
Remove protoc release for 32-bit Macs (#7209)
* Remove protoc release for 32-bit Macs 

Apple has been removing the support for 32-bit Mac apps: https://support.apple.com/en-us/HT208436

Our release infrastructure no longer supports building for 32-bit architecture.

* Remove protoc artifact for MacOS 32-bit
2020-02-13 14:00:19 -08:00
Yannic
f0cb9cdb95
[bazel] Move Java runtime/toolchains into //java (#7190)
* [bazel] Move Java runtime/toolchains into //java

This change moves `java_library` targets from the top-level BUILD file
into `//java/{core,lite,util}` and declares `alias` targets to point to
their new locations (hence, this is not a breaking change).

This will allow users that don't use Java to stop depending on
`@rules_java` (e.g. as requested in
https://github.com/bazelbuild/rules_scala/pull/989#issuecomment-583405161).

Note that there is no intention to deprecate + remove the top-level
targets in the foreseeable future.

* Add BUILD files in //java to java_EXTRA_DIST
2020-02-13 13:04:14 -08:00
Rafi Kamal
1440569231
Update the Xcode version number in Kokoro (#7202) (#7203)
Currently all of our MacOS tests are failing with the error: "autom4te: need GNU m4 1.4 or later: /usr/bin/m4". This is likely to be happening because Kokoro has updated their Xcode version to 11.3.
2020-02-12 11:34:51 -08:00
Rafi Kamal
a24e594833
Update the Xcode version number in Kokoro (#7202)
Currently all of our MacOS tests are failing with the error: "autom4te: need GNU m4 1.4 or later: /usr/bin/m4". This is likely to be happening because Kokoro has updated their Xcode version to 11.3.
2020-02-12 11:28:38 -08:00
Adam Cozzette
802e1b99b2 Attempt to mitigate "docker pull" failures
Lots of CI tests have been failing recently with error messages like
this from "docker pull":

  Error response from daemon: unexpected EOF

This seems to be an issue affecting other projects as well:
https://forums.docker.com/t/docker-pull-causes-error-response-from-daemon-unexpected-eof/88608

This commit tries to mitigate the problem by periodically retrying
failed Docker pulls.
2020-02-11 12:59:49 -08:00
Tim Swast
d1f4e25e15
python: publish sphinx docs to googleapis.dev
**Background:**

This is a follow-up to the PR that adds sphinx docs.

Googleapis.dev hosts client library documentation for several languages as
well as documentation for core packages such as google-api-core. It is owned
and maintained by the Cloud DevRel Core team.

**About this change:**

The `kokoro/docs/publish-python.sh` script builds the docs with sphinx, then
uses the [googleapis/docuploader](https://github.com/googleapis/docuploader)
tool to publish to googleapis.dev. The publish script is triggered manually
with a Kokoro RELEASE type job. Googlers, see go link
protobuf-docs-python-design for additional internal references.
2020-02-11 13:55:20 -06:00
Rafi Kamal
bc237eb672
Install RubyGem bundler version specified in Gemfile.lock to fix release failure (#7144) (#7156)
Ref: https://bundler.io/blog/2019/05/14/solutions-for-cant-find-gem-bundler-with-executable-bundle.html
2020-01-31 09:43:58 -08:00
Rafi Kamal
4cf5bfee95
Install RubyGem bundler version specified in Gemfile.lock to fix release failure (#7144)
* Update Gem version to fix Ruby release failure

Ref: https://bundler.io/blog/2019/05/14/solutions-for-cant-find-gem-bundler-with-executable-bundle.html
2020-01-30 13:08:31 -08:00
Paul Yang
85219578fa
Add continuous test for php7.4 on mac (#7153) 2020-01-30 12:45:47 -08:00
Yannic
948740bc9d [bazel] Fix blacklisted_protos in cc_toolchain and add test (#7075) 2020-01-15 10:27:35 -08:00
Yannic
29cd005ce1 [bazel] Fix blacklisted_protos in cc_toolchain and add test (#7096) 2020-01-15 10:22:16 -08:00
salamaniibm
c3ffdabed5 correcting the s390x Arch name 2020-01-10 16:07:15 -08:00
Rafi Kamal
ed8688de72
Merge pull request #7064 from rafi-kamal/3.11.x-202001071136
Merge branch 3.11.x to master
2020-01-08 12:24:20 -08:00
Brian Wignall
a104dffcb6 Fix typos (#7050)
Uses https://en.wikipedia.org/wiki/Wikipedia:Lists_of_common_misspellings/For_machines to find likely typos, with https://github.com/bwignall/typochecker to help automate the checking.
2020-01-08 10:18:20 -08:00
Rafi Kamal
4500f31a6a Merge branch 3.11.x to master 2020-01-07 11:52:29 -08:00
Jie Luo
63cfdafacb
Fix python release breakages by set MB_PYTHON_OSX_VER (#7038) 2019-12-19 14:06:29 -08:00