This uses the dockcross manylinux docker image to enable crosscompiling aarch64 python wheels. The wheels built for aarch64 linux are excluded from the release for now, pending testing (pending, in PR#8392).
Some of our Kokoro builds have been failing because Kokoro is unable to
copy root-owned files when the build is complete. This commit fixes the
problem by deleting these files at the end.
The installation of apt-transport-https is failing and causing this
script to exit early. I suspect the package is no longer needed since
recent versions have moved it into the apt package, so this change
updates the script to stop trying to install apt-transport-https.
* WIP.
* WIP.
* WIP.
* WIP.
* WIP.
* WIP.
* Added some missing files.
* WIP.
* WIP.
* Updated upb.
* Extension loads, but crashes immediately.
* Gets through the test suite without SEGV!
Still a lot of bugs to fix, but it is a major step!
214 tests, 378 assertions, 37 failures, 147 errors, 0 pendings, 0 omissions, 0 notifications
14.0187% passed
* Test and build for Ruby 3.0
* Fixed a few more bugs, efficient #inspect is almost done.
214 tests, 134243 assertions, 30 failures, 144 errors, 0 pendings, 0 omissions, 0 notifications
18.6916% passed
* Fixed message hash initialization and encode depth checking.
214 tests, 124651 assertions, 53 failures, 70 errors, 0 pendings, 0 omissions, 0 notifications
42.5234% passed
* A bunch of fixes to failing tests, now 70% passing.
214 tests, 202091 assertions, 41 failures, 23 errors, 0 pendings, 0 omissions, 0 notifications
70.0935% passed
* More than 80% of tests are passing now.
214 tests, 322331 assertions, 30 failures, 9 errors, 0 pendings, 0 omissions, 0 notifications
81.7757% passed
Unfortunately there is also a sporadic bug/segfault hanging around
that appears to be GC-related.
* Add linux/ruby30 and macos/ruby30
* Use rvm master for 3.0.0-preview2
* Over 90% of tests are passing!
214 tests, 349898 assertions, 15 failures, 1 errors, 0 pendings, 0 omissions, 0 notifications
92.5234% passed
* Passes all tests!
214 tests, 369388 assertions, 0 failures, 0 errors, 0 pendings, 0 omissions, 0 notifications
100% passed
* A bunch of cleanup.
1. Removed a bunch of internal-only symbols from headers.
2. Required a frozen check to get a non-const pointer to a map or array.
3. De-duplicated the code to get a type argument for Map/RepeatedField.
* Removed a bunch more stuff from protobuf.h. There is an intermittent assert failure.
Intermittent failure:
ruby: ../../../../ext/google/protobuf_c/protobuf.c:263: ObjectCache_Add: Assertion `rb_funcall(obj_cache2, (__builtin_constant_p("[]") ? __extension__ ({ static ID rb_intern_id_cache; if (!rb_intern_id_cache) rb_intern_id_cache = rb_intern2((("[]")
), (long)strlen(("[]"))); (ID) rb_intern_id_cache; }) : rb_intern("[]")), 1, key_rb) == val' failed
* Removed a few more things from protobuf.h.
* Ruby 3.0.0-preview2 to 3.0.0
* Require rake-compiler-dock >= 1.1.0
* More progress, fighting with the object cache.
* Passes on all Ruby versions!
* Updated and clarified comment regarding WeakMap.
* Fixed the wyhash compile.
* Fixed conformance tests for Ruby.
Conformance results now look like:
RUBYLIB=../ruby/lib:. ./conformance-test-runner --enforce_recommended --failure_list failure_list_ruby.txt --text_format_failure_list text_format_failure_list_ruby.txt ./conformance_ruby.rb
CONFORMANCE TEST BEGIN ====================================
CONFORMANCE SUITE PASSED: 1955 successes, 0 skipped, 58 expected failures, 0 unexpected failures.
CONFORMANCE TEST BEGIN ====================================
CONFORMANCE SUITE PASSED: 0 successes, 111 skipped, 8 expected failures, 0 unexpected failures.
Fixes include:
- Changed Ruby compiler to no longer reject proto2 maps.
- Changed Ruby compiler to emit a warning when proto2 extensions are
present instead of rejecting the .proto file completely.
- Fixed conformance tests to allow proto2 and look up message by name
instead of hardcoding a specific list of messages.
- Fixed conformance test to support the "ignore unknown" option for
JSON.
- Fixed conformance test to properly report serialization errors.
* Removed debug printf and fixed #inspect for floats.
* Fixed compatibility test to have proper semantics for #to_json.
* Updated Makefile.am with new file list.
* Don't try to copy wyhash when inside Docker.
* Fixed bug where we would forget that a sub-object is frozen in Ruby >=2.7.
* Avoid exporting unneeded symbols and refactored a bit of code.
* Some more refactoring.
* Simplified and added more comments.
* Some more comments and simplification. Added a missing license block.
Co-authored-by: Masaki Hara <hara@wantedly.com>
It seems that updating shallow Git clones is expensive, and as a result
Homebrew recently started refusing to update shallow clones (see
https://github.com/Homebrew/discussions/discussions/226). This commit
tries to fix the problem by making these repos into full clones before
running "brew update".
I also came across another error about there being a conflicting version
of pip3 in /usr/local. I suspect that is related to the other Python
binaries that the script has to delete, so I added pip3 to the list and
that seemed to solve the problem.
* Port for php8
* Port php c extension for php8
* Update composer.json
* Drop php7.0 support
* Update phpunit for php7.1 in c extension test
* Add back support for php7.0
* Add badge for php8 continuous build
Using non-versioned scripts for `build_file` means every Python version is
tested (under Tox), and this is multiplied by each Python version running Tox.
For example, when the `python38` config is used to run Tox, the
Tox layer runs on Python 3.8. This then tests all of the Python versions
individually, including, for example, python27 tests.
This change fixes the `build_file` paths to point to the same-pathed build
script. For example, `kokoro/linux/python27/presubmit.cfg` now uses
`kokoro/linux/python27/build.sh`.
Some additional fixes:
* Use `python -m tox` in tests.sh instead of just `tox`. This helps non-site
installations of tox, where the `tox` script may not be on `$PATH`.
* Ensure tox (and other Python build-related packages) are available in
Python testing images. (New images have been pushed.)
* Disable `--warnings_as_errors` due to deprecated function.
* Remove apt lists per [Docker best practices][1].
[1]: https://docs.docker.com/develop/develop-images/dockerfile_best-practices/
* Only ported c extension to php8.
* Didn't fixed the issue of throwing warnings for missing arginfo in bundled files.
* Tests not fixed, because syntax of phpunit (<7 vs >9.3) are not compatible.
* In next release, needs to drop php5 and php7.0 support (in order to use phpunit > 7)
* Fix a typo
* Fix lots of spelling errors
* Fix a few more spelling mistakes
* s/parsable/parseable/
* Don't touch the third party files
* Cloneable is the preferred C# term
* Copyable is the preferred C++ term
* Revert "s/parsable/parseable/"
This reverts commit 534ecf7675.
* Revert unparseable->unparsable corrections
This commit cleans up our environment setup for macOS builds.
- Always use `python -m {pip | virtualenv}` form of commands.
- Only upgrade in-place, and never use `--ignore-installed`.
- Use latest tox on macos, and test 3.6 on macos presubmit.
These changes ensure that multibuilder won't break the system-installed Python on the Kokoro macOS builder, and thus won't resolve the ancient system version. For Tox, make sure we test with Python 3.6 (the version of Python in current macOS).
- Remove the `brew install`; the kokoro image already has it, just
update/upgrade instead.
- Remove `prune`, logs had:
"""
Error: Unknown command: prune
"""
- Remove `uninstall`, logs had:
"""
Error: Refusing to uninstall /usr/local/Cellar/cmake/3.16.4 and /usr/local/Cellar/icu4c/64.2
because they are required by ceres-solver, ffmpeg, harfbuzz, libass and opencv, which are currently installed.
You can override this and force removal with:
brew uninstall --ignore-dependencies node icu4c cmake wget
"""
- Skip installing some things since they are already in the base image, logs
had:
"""
Warning: gflags 2.2.2 is already installed and up-to-date
To reinstall 2.2.2, run `brew reinstall gflags`
Warning: openssl@1.1 1.1.1g is already installed and up-to-date
To reinstall 1.1.1g, run `brew reinstall openssl@1.1`
Warning: pcre 8.44 is already installed and up-to-date
To reinstall 8.44, run `brew reinstall pcre`
"""
- Don't install gpg gpg2 as gnupg is already installed, also use gpg instead of
gpg2 for commands (and update the commands), logs had:
"""
kokoro/macos/prepare_build_macos_rc: line 44: gpg2: command not found
kokoro/macos/prepare_build_macos_rc: line 45: gpg2: command not found
"""
- Add env guards to control all the option installs and only request them be
installed in the cases that need it. This avoids having to install/update
the things like ruby when some other tool only needed in some configs is
install differently and could have conflicts.
- Switch to brew for cocoapods to avoid compat issues on the supporting
libraries.
It is no longer possible to "bundle install" these versions:
$ bundle install
Fetching gem metadata from https://rubygems.org/.................
reline-0.1.3 requires ruby version >= 2.5, which is incompatible with the current version, ruby 2.4.9p362
* Create a new docker image for python 3.6
The previous one cannot build, because python3.6 was removed from
stretch.
* Drop support for 3.3 and 3.4
Also update all docker file to be single version
* Build extensions for Ruby 2.7
* Try installing bundler 2.x
* Try bumping rake-compiler-dock
* Use standard RCD images
* Avoid 'rake cross native' with rake-compiler-dock
* Use Ruby 2.5 for building Ruby <= 2.6
* Use rake-compiler 1.1.0
* Specify target
* Don't update Ruby test image for now
* Set execute bit on files if and only if they begin with (#!).
Git only tracks the 'x' (executable) bit on each file. Prior to this
CL, our files were a random mix of executable and non-executable.
This change imposes some order by making files executable if and only
if they have shebang (#!) lines at the beginning.
We don't have any executable binaries checked into the repo, so
we shouldn't need to worry about that case.
* Added fix_permissions.sh script to set +x iff a file begins with (#!).
* Remove protoc release for 32-bit Macs
Apple has been removing the support for 32-bit Mac apps: https://support.apple.com/en-us/HT208436
Our release infrastructure no longer supports building for 32-bit architecture.
* Remove protoc artifact for MacOS 32-bit
* [bazel] Move Java runtime/toolchains into //java
This change moves `java_library` targets from the top-level BUILD file
into `//java/{core,lite,util}` and declares `alias` targets to point to
their new locations (hence, this is not a breaking change).
This will allow users that don't use Java to stop depending on
`@rules_java` (e.g. as requested in
https://github.com/bazelbuild/rules_scala/pull/989#issuecomment-583405161).
Note that there is no intention to deprecate + remove the top-level
targets in the foreseeable future.
* Add BUILD files in //java to java_EXTRA_DIST
Currently all of our MacOS tests are failing with the error: "autom4te: need GNU m4 1.4 or later: /usr/bin/m4". This is likely to be happening because Kokoro has updated their Xcode version to 11.3.
Currently all of our MacOS tests are failing with the error: "autom4te: need GNU m4 1.4 or later: /usr/bin/m4". This is likely to be happening because Kokoro has updated their Xcode version to 11.3.
Lots of CI tests have been failing recently with error messages like
this from "docker pull":
Error response from daemon: unexpected EOF
This seems to be an issue affecting other projects as well:
https://forums.docker.com/t/docker-pull-causes-error-response-from-daemon-unexpected-eof/88608
This commit tries to mitigate the problem by periodically retrying
failed Docker pulls.
**Background:**
This is a follow-up to the PR that adds sphinx docs.
Googleapis.dev hosts client library documentation for several languages as
well as documentation for core packages such as google-api-core. It is owned
and maintained by the Cloud DevRel Core team.
**About this change:**
The `kokoro/docs/publish-python.sh` script builds the docs with sphinx, then
uses the [googleapis/docuploader](https://github.com/googleapis/docuploader)
tool to publish to googleapis.dev. The publish script is triggered manually
with a Kokoro RELEASE type job. Googlers, see go link
protobuf-docs-python-design for additional internal references.
I'm not exactly sure why, but this fixes the failing Bazel presubmit
test. Using the most recent version seems like a good idea anyway so
that we can make sure we're compatible with any new Bazel changes.
* Bump target frameworks from netcoreapp1.0 to netcoreapp2.2.
Move global.json up to root of repo, change SDK ver to 2.2.100
Change .net core sdk in dockerfile for kokoro to ver 2.2.100
* Re-add curl install
* Change all exe target to 2.1
* Fix incorrect versions in global.json and Dockerfile
* Downgrade version to 2.1 to match exe targets
* introduce separate testing Dockerfile for C#
* revert changes to the shared Dockerfile
* use netcoreapp2.1 for C# conformance tests
* use language specific dockerfile for testing C#
* Edit compatibility tests script to use parameters instead of file copies
* install dotnet SDK on windows before running the tests
* update csharp_EXTRA_DIST
* fix macos build by changing xcode version
* add comments
* fix benchmark build for mac
* fix for gem option and install npm
* add test for node and npm
* fix node
* fix for node
* another attempts to fix node
* another attempts
* clean up
* fix macos build by changing xcode version
* add comments
* fix benchmark build for mac
* fix for gem option and install npm
* add test for node and npm
* fix node
* fix for node
* another attempts to fix node
* another attempts
* clean up
* Add several fixes for python toolchain
* Fix versin regex
* Make script exit on error
* Fix version regex
* Fix version regex
* Fix version regex
* Fix version regex
* Make test run on the current commit
* Fix test
* Fix test
* Use git to retrieve current commit
* Fix tests
* Fix tests
* Also make linux and mac work on the current commit
* Fix test
* Use gnu for ruby build because strptime is provided by posix
* Move option to extconf.rb
* Remove unused code in Rakefile
* Add config files for kokoro test
* Use gnu 11
* Define _XOPEN_SOURCE
* Add gnu11 option
* Remove XOPEN
* Try base_cc_flags
* Try config_options
* Move time.h to top
* Try -D_XOPEN_SOURCE=700
* Define XOPEN_SOURCE directly
* Try extconf.rb
* Try CFLAGS
* Try ext.cross_config_options
* Make mac ruby release job fail on error
* Try again
* Try define _XOPEN_SOURCE in extconf.rb
* Try again
* Define __USE_XOPEN
* Remove 2.6.0
* No mingw
* Throw error on mingw
* Remove XOPEN_SOURCE in upb
* Add back mingw
* Remove comment
* Fix python release build.
* Separate config.sh by platform
* Use KOKORO_JOR_NAME to determine the BUILD_COMMIT.
* Use -std=c++14 on MacOS so -latomic can be used.
* Fix python release build.
* Separate config.sh by platform
* Use KOKORO_JOR_NAME to determine the BUILD_COMMIT.
* Use -std=c++14 on MacOS so -latomic can be used.
* Add ruby 2.6 gem build for linux
* Add docker hub organization
* Add ruby2.6 in cross compile targets
* Fix tests
* Check mac version
* No need to install php in preparation for mac
* Add ruby 2.6 test
* Fix broken tests
* Fix compatibility test
* Fix bundler version
* Use new docker image
* Fix broken test
* Use new docker image in ruby26 test
* Install ruby for mac
* Rebuild python docker image
* Update docker image
* Change call to build_and_run_docker.sh
* Set up python tests for different versions
* Fix comments
* Inherit from message instead of implement
When regestering class (implmenets other class) during MINIT, zend_class_implements
would call zend_class_entry->interface_gets_implemented(). In PHP-7.3 interface_gets_implemented shares the same location with create_object.
However, during MINIT, the global object storeage hasn't been initialized. And thus, caused segment fault in php 7.3.
* Use zend_string_init where interned string may be the value.
zend_string_dup will keep using the existing interned string.
In php 7.3, interned string cannot be destroyed from user's code.
* Uncommment debug code
* Use latest phpunit for each php versions
* Revert change in Dockerfile
* Update php test to use the new docker image
* Update composer
* Change docker organization
* Update phpunit
* Debug phpunit
* Store phpunit into bin dir in docker image
* Install valgrind to docker
* Fix compatibility test
* Remove generated_service_test from compatibility c extension test
* Update 32bit php test to the new docker image
* Install bison
* Fix build.sh
* Fix DOCKERIMAGE_PREFIX
* Fix basename
* Add comment to build_and_run_docker2.sh
* Remove commented code
* Fix comments
* Add kokoro build for python source package
* Use libc++ for xcode 10 (#5303)
The xcode 10 removes the deprecated libstdc++ library. We could set
"MACOSX_DEPLOYMENT_TARGET" to "10.9" to use libc++ instead.
* Add python 3.7 build
* Add build for python 3.7 on linux and windows
* Remove unused source build
* Add comment
* Fix $MACOSX_DEPLOYMENT_TARGET mismatch
* Fix MACOSX_DEPLOYMENT_TARGET mismatch
* Add missing import for sysconfig
* Add missing imports
Changes:
1. Remove stuff no longer needed. Lots of the heavy lifting were there
because we were running our own jenkins cluster and had to manage all
the test logs ourselves. Now they are useless.
2. Change "-j2" to "-j4" to speed up the test a little bit. Kokoro
machines have 4 logic CPUs according to their spec.
This was fairly straightforward using the existing build-protoc.sh
script. The only problem I ran into was that the x86 Docker builds
create output directories owned by root, which caused some permission
issues. Fortunately it was easy to get around that just by doing those
Docker builds last.
* Install rake compiler
* Add kokoro config to build ruby gem on linux
* Rename from linix to linux
* Fix prepare_build.sh name
* Clean up
* Install bundler
* Install bundler
* Use c99 in order to build gem on mingw-32 on ruby 2.0.0
See https://github.com/rake-compiler/rake-compiler-dock/issues/4
* Move c99 config to extcofig.rb
This configuration builds both 32-bit and 64-bit binaries for Mac OS X.
One thing I had to change was to increase our minimum supported version
for 10.7 to 10.9, because 10.9 (Mavericks) appears to be the earliest
version supporting C++11.
This will allow SourceLink as per #4179, and mean that we can use C#
7.0 language features in the library (but not in generated code).
This does not affect which platforms we're *targeting*, so end users
won't see any difference.
It would be nice to update to 2.1.4, but AppVeyor's "Visual Studio
2017" environment is only 2.0.3.
