Commit Graph

512 Commits

Author SHA1 Message Date
Masaki Hara
4111d50488 Merge branch 'master' into arm64-darwin 2021-09-29 11:50:23 +09:00
zhangskz
740c4b082d
Update ruby_generator.cc to allow proto2 imports in proto3 (#9003)
* Update ruby_generator.cc to allow proto2 imports in proto3, with updated unit tests

* Update Makefile.am with new ruby_generated_code_proto2_import.proto

* Fix ruby_generator unit test to use temporary test directory for imported protos

* Add test for imported proto2 to ruby/tests

* Fix proto_path, restore to ../src/protoc, and fix/cleanup unit test.

* Rename Proto2TestMessage to TestImportedMessage for consistency, for ruby compiler tests
2021-09-22 09:16:09 -07:00
Paul Yang
12fb2b7450
Update protobuf version (#8973)
* Update protobuf version

* Update php notes
2021-09-13 12:34:03 -07:00
Joshua Haberman
79ac88bbbb Fixed a memory bug with RepeatedField#+.
We need to fuse the arrays so that the second array's data
stays live.
2021-09-10 17:17:14 -07:00
Joshua Haberman
6d8b05bfd4 Added missing "options" parameter. 2021-08-30 22:58:58 -07:00
Bo Yang
d8f55782b0 Update protobuf version 2021-08-27 14:39:58 -07:00
Paul Yang
7cbc08615f
Update protobuf version (#8887) 2021-08-18 17:20:20 -07:00
Joshua Haberman
3d51eb5d9f Make the Ruby DSL use a unique filename for each implicit file. 2021-08-02 20:43:43 -07:00
Joshua Haberman
9d578a3a9c Ported Ruby to ABSL wyhash. 2021-08-02 15:32:01 -07:00
Joshua Haberman
b7ab6255d6 Addressed PR comments. 2021-08-02 12:59:49 -07:00
Joshua Haberman
d339e1748e Updated Ruby 3.0 test to 3.0.2. 2021-08-01 12:07:15 -07:00
Joshua Haberman
cb649b61e3 Fix for newer Rubies where you cannot #force_encoding on a frozen string. 2021-07-31 13:12:14 -07:00
Joshua Haberman
16e16eb8cd pure-Ruby DSL is passing all tests! 2021-07-31 11:13:08 -07:00
Joshua Haberman
b4b5c89c31 Merge branch 'master' into rubybuilder 2021-07-30 15:04:53 -07:00
Deanna Garcia
367851d124 Update protobuf version 2021-06-04 21:21:39 +00:00
Deanna Garcia
5b8208ecea Update protobuf version 2021-05-25 19:36:05 +00:00
Joshua Haberman
367e4691d2
Fixed memory bug: properly root repeated/map field when assigning. (#8639)
* Fixed memory bug: properly root repeated/map field when assigning.

Previously the protobuf extension would not properly root
memory from a repeated field or map when assigning to a
message field (see the attached test case).  This could cause
crashes if the repeated field is subsequently accessed.

* Add accidentally-deleted Ruby test.
2021-05-21 23:04:09 -07:00
Adam Cozzette
52784ced2f
Update protobuf version (#8633) 2021-05-19 16:43:44 -07:00
Joshua Haberman
0b87475592
Php & Ruby Cherry Picks for 3.17.1 (#8632)
* Some more updates to PHP testing infrastructure (#8576)

* WIP.

* Added build config for all of the tests.

* Use ../src/protoc if it is available, for cases where Bazel isn't available.

* Added test_php.sh.

* Fix for the broken macOS tests.

* Move all jobs to use php80 instead of lots of separate jobs.

* Only pass -t flag if we are running in a terminal.

* Updated php_all job to use new Docker stuff.

* Fixed PHP memory leaks and arginfo errors (#8614)

* Fixed a bunch of incorrect arginfo and a few incorrect error messages.

* Passes mem check test with no leaks!

* WIP.

* Fix build warning that was causing Bazel build to fail.

* Added compatibility code for PHP <8.0.

* Added test_valgrind target and made tests Valgrind-clean.

* Updated Valgrind test to fail if memory leaks are detected.

* Removed intermediate shell script so commands are easier to cut, paste, and modify.

* Passing all Valgrind tests!

* Hoist addref into ObjCache_Get().

* Removed special case of map descriptors by keying object map on upb_msgdef.

* Removed all remaining RETURN_ZVAL() macros.

* Removed all explicit reference add/del operations.

* Added REFCOUNTING.md to Makefile.am.

* Updated upb version and fixed PHP to not get unset message field. (#8621)

* Updated upb version and fixed PHP to not get unset message field.

* Updated changelog.

* Fixed preproc test to handle old versions of Clang withot __has_attribute().

* A second try at fixing __has_attribute().

* Copy __has_attribute() fix to cc file also.

* Updated failure list for PHP for fixed test.

* Updated version of upb for Ruby (#8624)

* Updated upb.

* Preserve legacy behavior for unset messages.

* Updated failure list.

* Updated CHANGES.txt.

* Added erroneously-deleted test file.

* Fixed condition on compatibility code.

* Re-introduced deleted file again, and fixed Rakefile to not delete it.

* Fix generation of test protos.
2021-05-19 15:53:47 -07:00
Adam Cozzette
652d99a8ee
Update protobuf version (#8606) 2021-05-12 16:20:18 -07:00
Adam Cozzette
f807c02226
Update protobuf version to 3.17.0-rc2 (#8586)
* Update protobuf version

* Updated Kotlin versions and updated script to handle them
2021-05-10 15:31:35 -07:00
Adam Cozzette
7025f37cde
Update protobuf version (#8575) 2021-05-07 09:51:15 -07:00
Daniel Azuma
7e3bbeda92
fix(ruby): Fix crash when calculating Message hash values on 64-bit Windows (#8565)
* fix(ruby): Fix crash when calculating Message hash values on 64-bit Windows

* Better mapping for values outside the fixnum range

* Simpler downcasting of hash values

* Fix precedence

* Fix bundle on Ruby 2.4
2021-05-06 09:45:11 -07:00
Daniel Azuma
414aca5f13
fix(ruby): Fix various exceptions in Ruby on 64-bit Windows (#8563)
* fix(ruby): Fix various exceptions in Ruby on 64-bit Windows.

Activates the secondary ObjectCache map on this platform, to prevent weak keys from being garbage collected. This happened on 64-bit Windows because pointers don't necessarily fit in a Fixnum, and were being represented as GC-able Bignums on that platform.

* Removed extraneous code, and used VALUE instead of intptr_t

* Call the C function for new object instance rather than evaling a Ruby string
2021-05-05 15:42:25 -07:00
Chris McClymont
6d847adda7 Add class method from_time to ruby well known types - Timestamp 2021-05-05 13:38:05 +10:00
Joshua Haberman
fecf7e9340
Merge pull request #8356 from lfittl/support-json-name-for-ruby
Ruby: Add support for proto3 json_name in compiler and field definitions
2021-04-20 11:21:35 -07:00
James D
68cb69ea68
Fix source gem compilation (#8471)
* Select appropriate protoc binary for gem build

This should allow gem installation when
`bundle config force_ruby_platform true` has been set.

It has the unfortunate side effect of printing the protoc version during
build, but a quiet cross-platform solution is more complicated.

* Add path to wyhash header for source gem compilation
2021-04-12 10:06:44 -07:00
Adam Cozzette
5679811239 Merge branch '3.15.x' into merge-3-15-x 2021-04-08 10:54:02 -07:00
Adam Cozzette
436bd7880e
Updated version to 3.15.8 (#8463)
* Update protobuf version

* Updated CHANGES.txt and package.xml for 3.15.8
2021-04-07 15:38:38 -07:00
Joshua Haberman
67fee915e0 Fixed memory leak of Ruby arena objects.
In our free() method, we were freeing the memory from the
upb arena but we were failing to free the memory for the
Ruby arena object. This was causing every Ruby arena object
to leak: even though the objects were getting GC'd, the
underlying memory was not getting released.
2021-04-07 10:30:51 -07:00
Lukas Fittl
93c703e816 Add test for json_name functionality 2021-04-03 18:42:54 -07:00
Lukas Fittl
07f263c7ef Address review feedback 2021-04-03 18:42:43 -07:00
Adam Cozzette
87aa9adb86 Merge branch 'master' into 3.15.x 2021-04-02 14:55:02 -07:00
Adam Cozzette
878be3569e Update protobuf version 2021-04-02 11:06:56 -07:00
Joshua Haberman
f3f8707ba5
Merge pull request #8434 from haberman/ruby-message-eq
Fixed message equality in cases where the message type is different.
2021-04-01 12:16:53 -07:00
Joshua Haberman
38e1b59258 Elided the TYPE() and msgdef checks by using CLASS_OF. 2021-04-01 10:22:35 -07:00
Joshua Haberman
d69f482883 Fixed message equality in cases where the message type is different.
If the message types are different, equality comparison must return
false.
2021-03-29 17:37:36 -07:00
Joshua Haberman
a38319b5c4
Merge pull request #8429 from haberman/ruby-gc-secondarymap
Fix unbounded memory growth for Ruby <2.7.
2021-03-29 13:31:28 -07:00
Joshua Haberman
2fe27d8764 Addressed PR comments and fixed a bug.
We now hold the mutex for both map insertions, to protect
against a concurrent GC that removes from the seconary map
before we can insert into the weak map.
2021-03-29 12:30:49 -07:00
Joshua Haberman
e1ac393725 Added some more comments and refactored slightly. 2021-03-25 11:54:50 -07:00
Joshua Haberman
f0d6fcb2da Wrap secondary map mutations in a mutex, to avoid mutation races. 2021-03-25 11:54:50 -07:00
Joshua Haberman
b75a49f9e0 GC secondary map periodically. 2021-03-25 11:54:50 -07:00
Adam Cozzette
9db84d08fb Merge branch '3.15.x' into merge-3-15-x 2021-03-12 15:05:10 -08:00
Adam Cozzette
6aa539bf01 Update protobuf version 2021-03-10 14:53:26 -08:00
Aaron Patterson
addd0615bf
Register mark objects so Protobuf is compaction friendly
This commit removes an unused reference and registers globals with the
GC so that they will never die.  Ruby is getting a compacting GC, and it
means that these references can move.  Registering them with
`rb_gc_register_mark_object` will ensure the constants don't move and
will not be collected.
2021-03-10 13:54:04 -08:00
Joshua Haberman
b2991b8419
Merge pull request #8386 from haberman/ruby-string-compare
[Ruby] Fixed bug in string comparison logic.
2021-03-05 12:36:22 -08:00
Joshua Haberman
bb322c2b39 [Ruby] Fixed bug in string comparison logic. 2021-03-05 08:23:42 -08:00
Adam Cozzette
4fff47a418 Update protobuf version 2021-03-04 13:35:04 -08:00
Joshua Haberman
9b5e357df5 Fixed quadratic memory use in array append for PHP and Ruby. 2021-03-04 09:12:29 -08:00
Adam Cozzette
fd8aabf32d Update protobuf version 2021-03-03 11:37:48 -08:00
Joshua Haberman
cf7d81f88c
Merge pull request #8363 from haberman/ruby-nil-fix
Fixed SEGV when users pass nil messages. This also disallows `nil` in RepeatedField or Map.
2021-03-02 17:26:46 -08:00
Joshua Haberman
7d63b996e1 Removed compatibility test that tries to append "nil" to repeated field. 2021-03-02 15:20:06 -08:00
Joshua Haberman
dfa54577d6 [Ruby] Fixed SEGV when users pass nil messages. 2021-03-02 12:44:16 -08:00
Lukas Fittl
2c354c6626 Ruby: Add support for proto3 json_name in compiler and field definitions 2021-03-01 00:25:18 -08:00
Adam Cozzette
983d115bd9 Update protobuf version 2021-02-25 09:20:20 -08:00
Joshua Haberman
9879f423ff Ruby <2.7now uses WeakMap too, which prevents memory leaks.
Ruby <2.7 does not allow non-finalizable objects to be WeakMap
keys: https://bugs.ruby-lang.org/issues/16035

We work around this by using a secondary map for Ruby <2.7 which
maps the non-finalizable integer to a distinct object.

For now we accept that the entries in the secondary map wil never
be collected.  If this becomes a problem we can perform a GC pass
every so often that looks at the contents of the object cache to
decide what can be deleted from the secondary map.
2021-02-24 16:41:35 -08:00
Adam Cozzette
d7e943b8d2 Update protobuf version 2021-02-23 13:21:42 -08:00
Joshua Haberman
4e3ea74e42 [Ruby] Fix for FieldDescriptor.get(msg).
This fix is similar to the previous bug found in
Message.[]. The fix is the same: we need to handle
arrays and maps properly.

Fixes: https://github.com/protocolbuffers/protobuf/issues/8325
2021-02-22 17:14:46 -08:00
Adam Cozzette
052dc799d2 Update protobuf version 2021-02-19 15:12:24 -08:00
Joshua Haberman
3b3aac95a6 [Ruby] Fix for truncating behavior when converting Float to Duration. 2021-02-19 13:58:19 -08:00
Joshua Haberman
256f1327ea [Ruby] Bugfix for Message.[] for repeated or map fields. 2021-02-19 09:29:06 -08:00
Adam Cozzette
ae50d9b990 Update protobuf version 2021-02-18 11:50:15 -08:00
Adam Cozzette
8260126500 Update protobuf version 2021-02-17 10:51:33 -08:00
Adam Cozzette
e9f2ef308c Update protobuf version 2021-02-05 15:37:10 -08:00
Joshua Haberman
cf28f28347
Updated upb to allow nonzero offset minutes in JSON timestamps. (#8258)
* Updated upb to allow nonzero offset minutes in JSON timestamps.

This is to pick up https://github.com/protocolbuffers/upb/pull/367

* Reverted error message change.

* Fixed quote chars.
2021-02-03 17:59:10 -08:00
Joel Courtney
ef70acbc03 Improvement: better ruby from_* support in well_known_types.rb
* Added capability to support from_* requests properly by adding class methods and returning self for instance methods
    * `Timestamp.from_time`
    * `Value.from_ruby`
2021-02-03 12:49:20 +11:00
Masaki Hara
0a7a9a98c5 Ruby: build extensions for arm64-darwin 2021-01-25 15:04:50 +09:00
Rob Widmer
48234f5f01 Make the new multi-level ruby tests fit the existing structure better and add them to the makefile 2021-01-14 12:26:04 -08:00
Rob Widmer
1d39f78818 Fix jruby support to handle messages nested more than 1 level deep 2021-01-14 12:26:04 -08:00
Joshua Haberman
9abf6e2ab0
Ported Ruby extension to upb_msg (#8184)
* WIP.

* WIP.

* WIP.

* WIP.

* WIP.

* WIP.

* Added some missing files.

* WIP.

* WIP.

* Updated upb.

* Extension loads, but crashes immediately.

* Gets through the test suite without SEGV!

Still a lot of bugs to fix, but it is a major step!

214 tests, 378 assertions, 37 failures, 147 errors, 0 pendings, 0 omissions, 0 notifications
14.0187% passed

* Test and build for Ruby 3.0

* Fixed a few more bugs, efficient #inspect is almost done.

214 tests, 134243 assertions, 30 failures, 144 errors, 0 pendings, 0 omissions, 0 notifications
18.6916% passed

* Fixed message hash initialization and encode depth checking.

214 tests, 124651 assertions, 53 failures, 70 errors, 0 pendings, 0 omissions, 0 notifications
42.5234% passed

* A bunch of fixes to failing tests, now 70% passing.

214 tests, 202091 assertions, 41 failures, 23 errors, 0 pendings, 0 omissions, 0 notifications
70.0935% passed

* More than 80% of tests are passing now.

214 tests, 322331 assertions, 30 failures, 9 errors, 0 pendings, 0 omissions, 0 notifications
81.7757% passed

Unfortunately there is also a sporadic bug/segfault hanging around
that appears to be GC-related.

* Add linux/ruby30 and macos/ruby30

* Use rvm master for 3.0.0-preview2

* Over 90% of tests are passing!

214 tests, 349898 assertions, 15 failures, 1 errors, 0 pendings, 0 omissions, 0 notifications
92.5234% passed

* Passes all tests!

214 tests, 369388 assertions, 0 failures, 0 errors, 0 pendings, 0 omissions, 0 notifications
100% passed

* A bunch of cleanup.

1. Removed a bunch of internal-only symbols from headers.
2. Required a frozen check to get a non-const pointer to a map or array.
3. De-duplicated the code to get a type argument for Map/RepeatedField.

* Removed a bunch more stuff from protobuf.h.  There is an intermittent assert failure.

Intermittent failure:

ruby: ../../../../ext/google/protobuf_c/protobuf.c:263: ObjectCache_Add: Assertion `rb_funcall(obj_cache2, (__builtin_constant_p("[]") ? __extension__ ({ static ID rb_intern_id_cache; if (!rb_intern_id_cache) rb_intern_id_cache = rb_intern2((("[]")
), (long)strlen(("[]"))); (ID) rb_intern_id_cache; }) : rb_intern("[]")), 1, key_rb) == val' failed

* Removed a few more things from protobuf.h.

* Ruby 3.0.0-preview2 to 3.0.0

* Require rake-compiler-dock >= 1.1.0

* More progress, fighting with the object cache.

* Passes on all Ruby versions!

* Updated and clarified comment regarding WeakMap.

* Fixed the wyhash compile.

* Fixed conformance tests for Ruby.

Conformance results now look like:

RUBYLIB=../ruby/lib:. ./conformance-test-runner --enforce_recommended --failure_list failure_list_ruby.txt --text_format_failure_list text_format_failure_list_ruby.txt ./conformance_ruby.rb

CONFORMANCE TEST BEGIN ====================================

CONFORMANCE SUITE PASSED: 1955 successes, 0 skipped, 58 expected failures, 0 unexpected failures.

CONFORMANCE TEST BEGIN ====================================

CONFORMANCE SUITE PASSED: 0 successes, 111 skipped, 8 expected failures, 0 unexpected failures.

Fixes include:

- Changed Ruby compiler to no longer reject proto2 maps.
- Changed Ruby compiler to emit a warning when proto2 extensions are
  present instead of rejecting the .proto file completely.
- Fixed conformance tests to allow proto2 and look up message by name
  instead of hardcoding a specific list of messages.
- Fixed conformance test to support the "ignore unknown" option for
  JSON.
- Fixed conformance test to properly report serialization errors.

* Removed debug printf and fixed #inspect for floats.

* Fixed compatibility test to have proper semantics for #to_json.

* Updated Makefile.am with new file list.

* Don't try to copy wyhash when inside Docker.

* Fixed bug where we would forget that a sub-object is frozen in Ruby >=2.7.

* Avoid exporting unneeded symbols and refactored a bit of code.

* Some more refactoring.

* Simplified and added more comments.

* Some more comments and simplification. Added a missing license block.

Co-authored-by: Masaki Hara <hara@wantedly.com>
2021-01-13 12:16:25 -08:00
root
68f72f3f6c bug fix. in a 32-bit system, sizeof(mem_block) is only 8, _upb_arena_alignup(sizeof(mem_block)) is 16 2021-01-08 19:01:37 +00:00
Rob Widmer
53be4109c0 Fixed spelling error in JRuby Protobuf class 2020-12-11 16:02:41 -08:00
Rob Widmer
24b2094f8a Set up testing configuration for jruby 2020-12-11 16:02:41 -08:00
Rob Widmer
7b1d6abbe4 Fix some more failing tests 2020-12-11 16:02:41 -08:00
Rob Widmer
8313e577e5 Fix ruby gem to work with JRuby 2020-12-11 16:02:41 -08:00
Adam Cozzette
f3192d6128 Update protobuf version 2020-11-13 11:27:32 -08:00
Adam Cozzette
1b851b3fe6 Update protobuf version 2020-11-12 12:44:26 -08:00
Adam Cozzette
a8ab1fade9 Update protobuf version 2020-11-10 17:15:32 -08:00
Adam Cozzette
27100b40fd Update protobuf version 2020-11-05 14:30:53 -08:00
alusco-scratch
00a7ea4de1
Override Map.clone to use Map's dup method
Message and Repeated field override clone so that it uses the internal implementation of dup but Map is missing this and only implements dup.  This can lead to unexpected behavior since two out of three complex types behave correctly.
2020-10-01 21:20:31 -07:00
Adam Cozzette
58f4260e8e Merged 3.13.x into master 2020-08-17 11:06:11 -07:00
Adam Cozzette
fde7cf7358 Update protobuf version 2020-08-14 15:20:53 -07:00
Adam Cozzette
98c2e69f8c Update protobuf version 2020-08-12 14:49:20 -07:00
Peter Newman
e2cc2de304
Fix lots of spelling errors (#7751)
* Fix a typo

* Fix lots of spelling errors

* Fix a few more spelling mistakes

* s/parsable/parseable/

* Don't touch the third party files

* Cloneable is the preferred C# term

* Copyable is the preferred C++ term

* Revert "s/parsable/parseable/"

This reverts commit 534ecf7675.

* Revert unparseable->unparsable corrections
2020-08-10 11:08:25 -07:00
Adam Cozzette
1c587a8059 Updated version to 3.13.0-rc2 2020-08-07 14:44:49 -07:00
Adam Cozzette
8940150706 Updated version to 3.13.0-rc1. 2020-08-07 14:44:49 -07:00
Joshua Haberman
9ce8c330e7
Updated version to 3.12.3 and updated CHANGES.txt. (#7580)
* Updated version to 3.12.3 and updated CHANGES.txt.

* Re-ran generate_descriptor_protos.sh and made it more parallel.
2020-06-01 13:36:50 -07:00
Joshua Haberman
beeb621f50
Update protobuf version (#7557) 2020-05-26 15:46:39 -07:00
Joshua Haberman
6ad138d85a
Update protobuf version (#7535) 2020-05-20 10:54:29 -07:00
Daniel Azuma
cf81f38b5d
Restore binary builds of Ruby 2.3 and 2.4 (#7529) (#7531) 2020-05-19 15:52:59 -07:00
Joshua Haberman
e90b71cdee Update protobuf version 2020-05-15 13:28:07 -07:00
Joshua Haberman
f597a24627 Update protobuf version 2020-05-12 12:49:46 -07:00
Joshua Haberman
e19bfc0e28
Stop building binary gems for ruby <2.5. (#7453)
It is no longer possible to "bundle install" these versions:

$ bundle install
Fetching gem metadata from https://rubygems.org/.................
reline-0.1.3 requires ruby version >= 2.5, which is incompatible with the current version, ruby 2.4.9p362
2020-05-01 12:35:09 -07:00
Joshua Haberman
a9f11d7d64
Updated version to 3.12.0-rc1. (#7449) 2020-04-30 15:34:10 -07:00
Masaki Hara
f494cb222d
Test in Ruby 2.7 (#7386)
* Test in Ruby 2.7

* Turn off GC.stress for 'require' in Ruby 2.7

* Backport 2c8364b to compatibility_tests/v3.0.0
2020-04-24 09:50:27 -07:00
Joshua Haberman
6b759688a1
Implemented proto3 presence for Ruby. (#7406)
* WIP.

* WIP.

* Builds and runs. Tests need to be updated to test presence.

* Ruby: proto3 presence is passing all tests.

* Fixed a bug where empty messages has the wrong oneof count.
2020-04-23 12:54:25 -07:00
David L. Jones
94afb8ab36
Add Ruby tests for oneof cases. (#7385)
In Ruby, the oneof case returns a symbol with the name of the field that is set. These were not previously tested.
2020-04-23 12:49:04 -07:00
Joshua Haberman
18950451c7
Ruby: assigning 'nil' to submessage should clear the field. (#7397)
Previously if you assigned 'nil' to a submessage in proto2
the field would be set to 'nil' but would still have its hasbit
set. This was a clear bug so I'm fixing it outright, even though
it is an observable behavior change.
2020-04-20 15:51:05 -07:00
Joshua Haberman
2adea4c64a Some preliminary work towards a ruby builder. 2020-04-19 11:40:59 -07:00
Eric Walker
2c8364bcd1
Rails2.7 segfaults (#7091)
* Add a test suite for ruby 2.7

* Call BigDecimal() instead of BigDecimal.new()

BigDecimal.new was deprecated in ruby 2.6

* Switch FrozenError expectation to a matcher

The error message for FrozenError changed to include more information
about the mutated object. Switch from an exact match to an aproximate
match (equal => match). This does not change the prefix.

* We can safely ignore newest array methods from ruby 2.7
2020-04-17 10:20:38 -07:00