Should prevent future flakiness if unreleased JRuby features are dependent on unreleased Java features by populating the local Maven repository with the artifacts needed when running `tests.sh jruby`.
googletest uses a Python script in its build which is not compatible
with Python 3. Unfortunately we can't easily upgrade googletest right
now, so this commit works around the problem by putting a python symlink
pointing to Python 2 in the $PATH.
* Fix dist install test by ensuring that we use Python 3
Now that we have dropped Python 2 support, we need to make sure this
install test uses Python 3.
* Update Docker image to install Python 3 version of setuptools
* Run pip3 instead of pip
This is necessary because the Kotlin support requires Java 8 or
higher.
Making this test use Java 8 required switching it to the java_stretch
Docker image and updating that image to include some Python
dependencies.
* Simplified PHP testing setup.
- Consolidated on a single autoloader, created by composer.
- Consolidated on a single phpunit invocation strategy: we run
phpunit on a directory, which will run all tests matching *Test.php
in that directory.
- We now rely on autoloading to import all test protos. require_once()
calls for test protos are removed.
- For now the valgrind tests are removed. A follow-up PR will re-enable
them in a more robust way.
* More improvements to PHP testing.
1. Replace custom PHPUnit-selection logic in test.sh with generic
composer version selection.
2. Optimized both test proto generation and the custom extension
build to avoid unnecessary work when the files are already up
to date.
* Added assertions to verify that the C test doesn't use PHP sources.
* Updated tests.sh for the new PHP testing commands.
* Removed obsolete rules from tests.sh.
* Fixed generate_test_protos.sh for when tmp does not exist.
Also removed undefined_test.php and fixed Makefile.am.
* Added php8.0_all again which is still used.
* Added missing file to Makefile.am.
* Re-added php_all_32 rule which is also still used.
* Updated testing commands for macOS and download composer.
* Use /usr/local/bin on mac instead of /usr/bin, since the latter is not writable.
* WIP.
* WIP.
* WIP.
* WIP.
* WIP.
* WIP.
* Added some missing files.
* WIP.
* WIP.
* Updated upb.
* Extension loads, but crashes immediately.
* Gets through the test suite without SEGV!
Still a lot of bugs to fix, but it is a major step!
214 tests, 378 assertions, 37 failures, 147 errors, 0 pendings, 0 omissions, 0 notifications
14.0187% passed
* Test and build for Ruby 3.0
* Fixed a few more bugs, efficient #inspect is almost done.
214 tests, 134243 assertions, 30 failures, 144 errors, 0 pendings, 0 omissions, 0 notifications
18.6916% passed
* Fixed message hash initialization and encode depth checking.
214 tests, 124651 assertions, 53 failures, 70 errors, 0 pendings, 0 omissions, 0 notifications
42.5234% passed
* A bunch of fixes to failing tests, now 70% passing.
214 tests, 202091 assertions, 41 failures, 23 errors, 0 pendings, 0 omissions, 0 notifications
70.0935% passed
* More than 80% of tests are passing now.
214 tests, 322331 assertions, 30 failures, 9 errors, 0 pendings, 0 omissions, 0 notifications
81.7757% passed
Unfortunately there is also a sporadic bug/segfault hanging around
that appears to be GC-related.
* Add linux/ruby30 and macos/ruby30
* Use rvm master for 3.0.0-preview2
* Over 90% of tests are passing!
214 tests, 349898 assertions, 15 failures, 1 errors, 0 pendings, 0 omissions, 0 notifications
92.5234% passed
* Passes all tests!
214 tests, 369388 assertions, 0 failures, 0 errors, 0 pendings, 0 omissions, 0 notifications
100% passed
* A bunch of cleanup.
1. Removed a bunch of internal-only symbols from headers.
2. Required a frozen check to get a non-const pointer to a map or array.
3. De-duplicated the code to get a type argument for Map/RepeatedField.
* Removed a bunch more stuff from protobuf.h. There is an intermittent assert failure.
Intermittent failure:
ruby: ../../../../ext/google/protobuf_c/protobuf.c:263: ObjectCache_Add: Assertion `rb_funcall(obj_cache2, (__builtin_constant_p("[]") ? __extension__ ({ static ID rb_intern_id_cache; if (!rb_intern_id_cache) rb_intern_id_cache = rb_intern2((("[]")
), (long)strlen(("[]"))); (ID) rb_intern_id_cache; }) : rb_intern("[]")), 1, key_rb) == val' failed
* Removed a few more things from protobuf.h.
* Ruby 3.0.0-preview2 to 3.0.0
* Require rake-compiler-dock >= 1.1.0
* More progress, fighting with the object cache.
* Passes on all Ruby versions!
* Updated and clarified comment regarding WeakMap.
* Fixed the wyhash compile.
* Fixed conformance tests for Ruby.
Conformance results now look like:
RUBYLIB=../ruby/lib:. ./conformance-test-runner --enforce_recommended --failure_list failure_list_ruby.txt --text_format_failure_list text_format_failure_list_ruby.txt ./conformance_ruby.rb
CONFORMANCE TEST BEGIN ====================================
CONFORMANCE SUITE PASSED: 1955 successes, 0 skipped, 58 expected failures, 0 unexpected failures.
CONFORMANCE TEST BEGIN ====================================
CONFORMANCE SUITE PASSED: 0 successes, 111 skipped, 8 expected failures, 0 unexpected failures.
Fixes include:
- Changed Ruby compiler to no longer reject proto2 maps.
- Changed Ruby compiler to emit a warning when proto2 extensions are
present instead of rejecting the .proto file completely.
- Fixed conformance tests to allow proto2 and look up message by name
instead of hardcoding a specific list of messages.
- Fixed conformance test to support the "ignore unknown" option for
JSON.
- Fixed conformance test to properly report serialization errors.
* Removed debug printf and fixed #inspect for floats.
* Fixed compatibility test to have proper semantics for #to_json.
* Updated Makefile.am with new file list.
* Don't try to copy wyhash when inside Docker.
* Fixed bug where we would forget that a sub-object is frozen in Ruby >=2.7.
* Avoid exporting unneeded symbols and refactored a bit of code.
* Some more refactoring.
* Simplified and added more comments.
* Some more comments and simplification. Added a missing license block.
Co-authored-by: Masaki Hara <hara@wantedly.com>
* Port for php8
* Port php c extension for php8
* Update composer.json
* Drop php7.0 support
* Update phpunit for php7.1 in c extension test
* Add back support for php7.0
* Add badge for php8 continuous build
Using non-versioned scripts for `build_file` means every Python version is
tested (under Tox), and this is multiplied by each Python version running Tox.
For example, when the `python38` config is used to run Tox, the
Tox layer runs on Python 3.8. This then tests all of the Python versions
individually, including, for example, python27 tests.
This change fixes the `build_file` paths to point to the same-pathed build
script. For example, `kokoro/linux/python27/presubmit.cfg` now uses
`kokoro/linux/python27/build.sh`.
Some additional fixes:
* Use `python -m tox` in tests.sh instead of just `tox`. This helps non-site
installations of tox, where the `tox` script may not be on `$PATH`.
* Ensure tox (and other Python build-related packages) are available in
Python testing images. (New images have been pushed.)
* Disable `--warnings_as_errors` due to deprecated function.
* Remove apt lists per [Docker best practices][1].
[1]: https://docs.docker.com/develop/develop-images/dockerfile_best-practices/
This extension allows Python code to be generated from setup.py files, so they
are created as part of a normal Python build. The extension uses an already-existing
protoc binary, which can be explicitly specified if needed.
* Only ported c extension to php8.
* Didn't fixed the issue of throwing warnings for missing arginfo in bundled files.
* Tests not fixed, because syntax of phpunit (<7 vs >9.3) are not compatible.
* In next release, needs to drop php5 and php7.0 support (in order to use phpunit > 7)
This commit cleans up our environment setup for macOS builds.
- Always use `python -m {pip | virtualenv}` form of commands.
- Only upgrade in-place, and never use `--ignore-installed`.
- Use latest tox on macos, and test 3.6 on macos presubmit.
These changes ensure that multibuilder won't break the system-installed Python on the Kokoro macOS builder, and thus won't resolve the ancient system version. For Tox, make sure we test with Python 3.6 (the version of Python in current macOS).
* Added code for the new PHP extension.
* Removed a bunch of unused variables shown by compiler warnings.
* Test protobuf2 in the tests.
* Updated upb version to fix a goldenfile test.
* Added pure-PHP changes. Extension now passes all tests.
* Enabled protobuf2 for all C extension tests.
* Fixed pure=PHP lib: full names no longer start with '.'.
* Added files for new extension to Makefile.am.
* Downgraded make-preload.py to python 2, since python3 isn't available in the php_all Kokoro image.
* Disable tests of new C extension with PHP 5.x.
* Also do not compile the extension for PHP5.
* Accept version 5.*.*, and use /usr/bin/python.
* Addressed PR comments.
* Addressed PR comments.
* Added "const" to a parameter and fixed a memory leak seen in Valgrind.
* Stop testing the C extension for PHP5.
The next release of protobuf will deprecate the C extension
for PHP5, see:
https://github.com/protocolbuffers/protobuf/issues/7525
* Made the PHP5.6 Mac test only exercise pure-PHP.
* Build protoc for PHP 5.6 test.
* Rewrote bundling script in PHP to avoid dependency on Python.
* A few more fixes.
* Fixed int32/int64 behavior for 32-bit builds.
* Match more PHP versions in testing script.
* Use phpize --clean before building extension.
* Force-delete configure.in to avoid phpize problems cross-version.
* Delete both configure.ac and configure.in.
* De-duplicated code to generate PHP protos and install phpunit.
* Removed all references to generate_php_test_proto.
* Replaced with internal_build_cpp.
* Make Timestamp::__construct() static to avoid conflicts with MongoDB.
* Replicated PHPUnit versions and added new script to Makefile.am.
* Fixed filename in Makefile.am.
* Disabled test that SEGV's on macOS.
* Removed extraneous "set -e".
* Make sure generate_protos.sh happens on every test path.
* Removed stray '$' chars.
* Added proper support for aggregate_metadata tests. But now I get a stack overflow.
Stack overflow:
/Users/haberman/code/protobuf/php/tests/generated/GPBMetadata/Proto/TestDescriptors.php:16
/Users/haberman/code/protobuf/php/tests/generated/GPBMetadata/Proto/TestDescriptors.php:16
/Users/haberman/code/protobuf/php/tests/generated/GPBMetadata/Proto/TestDescriptors.php:16
/Users/haberman/code/protobuf/php/tests/generated/GPBMetadata/Proto/TestDescriptors.php:16
/Users/haberman/code/protobuf/php/tests/generated/GPBMetadata/Proto/TestDescriptors.php:16
/Users/haberman/code/protobuf/php/tests/generated/GPBMetadata/Proto/TestDescriptors.php:16
/Users/haberman/code/protobuf/php/tests/generated/GPBMetadata/Proto/TestDescriptors.php:16
/Users/haberman/code/protobuf/php/tests/generated/GPBMetadata/Proto/TestDescriptors.php:16
namespace GPBMetadata\Proto;
class TestDescriptors
{
public static $is_initialized = false;
public static function initOnce() {
$pool = \Google\Protobuf\Internal\DescriptorPool::getGeneratedPool();
if (static::$is_initialized == true) {
return;
}
\GPBMetadata\Proto\TestDescriptors::initOnce();
$pool->internalAddGeneratedFile(hex2bin(
""
), true);
static::$is_initialized = true;
}
}
* Fixed and verified metadata aggregation testing.
- Remove the `brew install`; the kokoro image already has it, just
update/upgrade instead.
- Remove `prune`, logs had:
"""
Error: Unknown command: prune
"""
- Remove `uninstall`, logs had:
"""
Error: Refusing to uninstall /usr/local/Cellar/cmake/3.16.4 and /usr/local/Cellar/icu4c/64.2
because they are required by ceres-solver, ffmpeg, harfbuzz, libass and opencv, which are currently installed.
You can override this and force removal with:
brew uninstall --ignore-dependencies node icu4c cmake wget
"""
- Skip installing some things since they are already in the base image, logs
had:
"""
Warning: gflags 2.2.2 is already installed and up-to-date
To reinstall 2.2.2, run `brew reinstall gflags`
Warning: openssl@1.1 1.1.1g is already installed and up-to-date
To reinstall 1.1.1g, run `brew reinstall openssl@1.1`
Warning: pcre 8.44 is already installed and up-to-date
To reinstall 8.44, run `brew reinstall pcre`
"""
- Don't install gpg gpg2 as gnupg is already installed, also use gpg instead of
gpg2 for commands (and update the commands), logs had:
"""
kokoro/macos/prepare_build_macos_rc: line 44: gpg2: command not found
kokoro/macos/prepare_build_macos_rc: line 45: gpg2: command not found
"""
- Add env guards to control all the option installs and only request them be
installed in the cases that need it. This avoids having to install/update
the things like ruby when some other tool only needed in some configs is
install differently and could have conflicts.
- Switch to brew for cocoapods to avoid compat issues on the supporting
libraries.
* Simplified PHP testing scripts.
- allow them to be run from any directory.
- remove "VERSION" arg, we can get PHP from $PATH.
* Added a few places I missed.
* Removed redundant "cd `dirname $0`".
Also replaced all backticks with $(), for consistency.
A proper integration test should be hermetic such that its operation
is agnostic to whatever happens outside of its controlled environment.
The script is not so since it uses a fixed (and very old) version of Go,
but continues to build protoc-gen-go from head, leading to an eventual
breakage when protoc-gen-go at head no longer operates on such an old
version of Go.
Fix the script to pin to an older version of protoc-gen-go (i.e., v1.3.5).
* Add a test suite for ruby 2.7
* Call BigDecimal() instead of BigDecimal.new()
BigDecimal.new was deprecated in ruby 2.6
* Switch FrozenError expectation to a matcher
The error message for FrozenError changed to include more information
about the mutated object. Switch from an exact match to an aproximate
match (equal => match). This does not change the prefix.
* We can safely ignore newest array methods from ruby 2.7
* python: generate documentation with Sphinx and Read the Docs
Background:
Formerly, the Python protobuf reference documentation was built with
[Epydoc](http://epydoc.sourceforge.net/). This package has not been
updated since 2008, and it has inconsistent formatting (see internal
issue 131415575) with most Python documentation. Sphinx is used for the
official docs.python.org docs as well as most other Python packages,
including the Google client libraries and related packages, such as
https://googleapis.dev/python/google-api-core/latest/
To build the docs with Sphinx:
1. Install the needed packages (`sphinx`, `sphinxcontrib-napoleon` for
Google-style docstring support). I've created a conda environment file
to make this easier:
```
conda env create -f python/docs/environment.yml
```
2. (Optional) Generate reference docs files and regenerate index:
```
cd python
python generate_docs.py
cd ..
```
3. Run Sphinx.
```
cd python/docs
make html
```
About this change:
The script at `python/generate_docs.py` creates a ReStructured Text file
for each public module in the protobuf Python package. The script also
updates the table of contents in `python/docs/index.rst` to point to
these module references.
Future work:
Testing the docs build on PRs requires contributors to actually do some
setup work to configure builds on their fork. It'd be better if CI had a
docs build session to verify that the Sphinx docs generation at least
runs.
There are many warnings due to not-quite-correct docstrings in the
actual Python code itself. I'm choosing to ignore these errors to keep
the PR small, but I recommend you fix these and then enable "fail on
warnings" in the docs build on CI.
* add docs to EXTRA_DIST
* add instructions to build documentation to generate_docs.py
* exclude python/odcs from cpp_distcheck
* Add scripts to test multirequest
* chmod ug+x multirequest.sh
* Add continuous test
* Compile c extension
* Class entry is obsolete in the second request
1) Needes to use class name in persistent map
2) Invalidate class entry stored in descriptor
* Add new files to dist
* Fix compile_extension
* Cleanup outputs for phpize
* Make c extension portable for php 7.4
* Fix conformance tests
* Fix comments
* Fix 32-bit
* Update conformance failure list
* Fix compiler warnings
* Cleanup configure created by phpize
The file created in php 7.4 is not recognizable by previous versions
* Fix conformance tests for 64-bit php
* Fix conformance test
* Fix compile warning
* Fix compile warnings
