AuroraMiddleware/protobuf
1
0
Fork 0
Code Issues 1 Pull Requests Projects Releases Wiki Activity

Ported Ruby extension to upb_msg (#8184)

Browse Source 
* WIP.

* WIP.

* WIP.

* WIP.

* WIP.

* WIP.

* Added some missing files.

* WIP.

* WIP.

* Updated upb.

* Extension loads, but crashes immediately.

* Gets through the test suite without SEGV!

Still a lot of bugs to fix, but it is a major step!

214 tests, 378 assertions, 37 failures, 147 errors, 0 pendings, 0 omissions, 0 notifications
14.0187% passed

* Test and build for Ruby 3.0

* Fixed a few more bugs, efficient #inspect is almost done.

214 tests, 134243 assertions, 30 failures, 144 errors, 0 pendings, 0 omissions, 0 notifications
18.6916% passed

* Fixed message hash initialization and encode depth checking.

214 tests, 124651 assertions, 53 failures, 70 errors, 0 pendings, 0 omissions, 0 notifications
42.5234% passed

* A bunch of fixes to failing tests, now 70% passing.

214 tests, 202091 assertions, 41 failures, 23 errors, 0 pendings, 0 omissions, 0 notifications
70.0935% passed

* More than 80% of tests are passing now.

214 tests, 322331 assertions, 30 failures, 9 errors, 0 pendings, 0 omissions, 0 notifications
81.7757% passed

Unfortunately there is also a sporadic bug/segfault hanging around
that appears to be GC-related.

* Add linux/ruby30 and macos/ruby30

* Use rvm master for 3.0.0-preview2

* Over 90% of tests are passing!

214 tests, 349898 assertions, 15 failures, 1 errors, 0 pendings, 0 omissions, 0 notifications
92.5234% passed

* Passes all tests!

214 tests, 369388 assertions, 0 failures, 0 errors, 0 pendings, 0 omissions, 0 notifications
100% passed

* A bunch of cleanup.

1. Removed a bunch of internal-only symbols from headers.
2. Required a frozen check to get a non-const pointer to a map or array.
3. De-duplicated the code to get a type argument for Map/RepeatedField.

* Removed a bunch more stuff from protobuf.h.  There is an intermittent assert failure.

Intermittent failure:

ruby: ../../../../ext/google/protobuf_c/protobuf.c:263: ObjectCache_Add: Assertion `rb_funcall(obj_cache2, (__builtin_constant_p("[]") ? __extension__ ({ static ID rb_intern_id_cache; if (!rb_intern_id_cache) rb_intern_id_cache = rb_intern2((("[]")
), (long)strlen(("[]"))); (ID) rb_intern_id_cache; }) : rb_intern("[]")), 1, key_rb) == val' failed

* Removed a few more things from protobuf.h.

* Ruby 3.0.0-preview2 to 3.0.0

* Require rake-compiler-dock >= 1.1.0

* More progress, fighting with the object cache.

* Passes on all Ruby versions!

* Updated and clarified comment regarding WeakMap.

* Fixed the wyhash compile.

* Fixed conformance tests for Ruby.

Conformance results now look like:

RUBYLIB=../ruby/lib:. ./conformance-test-runner --enforce_recommended --failure_list failure_list_ruby.txt --text_format_failure_list text_format_failure_list_ruby.txt ./conformance_ruby.rb

CONFORMANCE TEST BEGIN ====================================

CONFORMANCE SUITE PASSED: 1955 successes, 0 skipped, 58 expected failures, 0 unexpected failures.

CONFORMANCE TEST BEGIN ====================================

CONFORMANCE SUITE PASSED: 0 successes, 111 skipped, 8 expected failures, 0 unexpected failures.

Fixes include:

- Changed Ruby compiler to no longer reject proto2 maps.
- Changed Ruby compiler to emit a warning when proto2 extensions are
  present instead of rejecting the .proto file completely.
- Fixed conformance tests to allow proto2 and look up message by name
  instead of hardcoding a specific list of messages.
- Fixed conformance test to support the "ignore unknown" option for
  JSON.
- Fixed conformance test to properly report serialization errors.

* Removed debug printf and fixed #inspect for floats.

* Fixed compatibility test to have proper semantics for #to_json.

* Updated Makefile.am with new file list.

* Don't try to copy wyhash when inside Docker.

* Fixed bug where we would forget that a sub-object is frozen in Ruby >=2.7.

* Avoid exporting unneeded symbols and refactored a bit of code.

* Some more refactoring.

* Simplified and added more comments.

* Some more comments and simplification. Added a missing license block.

Co-authored-by: Masaki Hara <hara@wantedly.com>
This commit is contained in:
Joshua Haberman 2021-01-13 12:16:25 -08:00 committed by GitHub
parent 468bc193ec
commit 9abf6e2ab0
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
42 changed files with 14726 additions and 23983 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

12
Makefile.am
View File

@ -1094,17 +1094,21 @@ ruby_EXTRA_DIST= \
ruby/compatibility_tests/v3.0.0/test.sh \
ruby/compatibility_tests/v3.0.0/Rakefile \
ruby/compatibility_tests/v3.0.0/README.md \
ruby/ext/google/protobuf_c/convert.c \
ruby/ext/google/protobuf_c/convert.h \
ruby/ext/google/protobuf_c/defs.c \
ruby/ext/google/protobuf_c/encode_decode.c \
ruby/ext/google/protobuf_c/defs.h \
ruby/ext/google/protobuf_c/extconf.rb \
ruby/ext/google/protobuf_c/map.c \
ruby/ext/google/protobuf_c/map.h \
ruby/ext/google/protobuf_c/message.c \
ruby/ext/google/protobuf_c/message.h \
ruby/ext/google/protobuf_c/protobuf.c \
ruby/ext/google/protobuf_c/protobuf.h \
ruby/ext/google/protobuf_c/repeated_field.c \
ruby/ext/google/protobuf_c/storage.c \
ruby/ext/google/protobuf_c/upb.c \
ruby/ext/google/protobuf_c/upb.h \
ruby/ext/google/protobuf_c/repeated_field.h \
ruby/ext/google/protobuf_c/ruby-upb.c \
ruby/ext/google/protobuf_c/ruby-upb.h \
ruby/ext/google/protobuf_c/wrap_memcpy.c \
ruby/google-protobuf.gemspec \
ruby/lib/google/protobuf/message_exts.rb \

6
conformance/Makefile.am
View File

@ -5,7 +5,7 @@ conformance_protoc_inputs = \
$(top_srcdir)/src/google/protobuf/test_messages_proto3.proto
# proto2 input files, should be separated with proto3, as we
# can't generate proto2 files for ruby, php and objc
# can't generate proto2 files for php.
conformance_proto2_protoc_inputs = \
$(top_srcdir)/src/google/protobuf/test_messages_proto2.proto
@ -261,7 +261,7 @@ if USE_EXTERNAL_PROTOC
# Some implementations include pre-generated versions of well-known types.
protoc_middleman: $(conformance_protoc_inputs) $(conformance_proto2_protoc_inputs) $(well_known_type_protoc_inputs) google-protobuf
$(PROTOC) -I$(srcdir) -I$(top_srcdir) --cpp_out=. --java_out=. --ruby_out=. --objc_out=. --python_out=. --php_out=. --js_out=import_style=commonjs,binary:. $(conformance_protoc_inputs)
$(PROTOC) -I$(srcdir) -I$(top_srcdir) --cpp_out=. --java_out=. --objc_out=. --python_out=. --js_out=import_style=commonjs,binary:. $(conformance_proto2_protoc_inputs)
$(PROTOC) -I$(srcdir) -I$(top_srcdir) --cpp_out=. --java_out=. --ruby_out=. --objc_out=. --python_out=. --js_out=import_style=commonjs,binary:. $(conformance_proto2_protoc_inputs)
$(PROTOC) -I$(srcdir) -I$(top_srcdir) --cpp_out=. --java_out=. --ruby_out=. --python_out=. --js_out=import_style=commonjs,binary:google-protobuf $(well_known_type_protoc_inputs)
## $(PROTOC) -I$(srcdir) -I$(top_srcdir) --java_out=lite:lite $(conformance_protoc_inputs) $(well_known_type_protoc_inputs)
touch protoc_middleman
@ -273,7 +273,7 @@ else
# building out-of-tree.
protoc_middleman: $(top_srcdir)/src/protoc$(EXEEXT) $(conformance_protoc_inputs) $(conformance_proto2_protoc_inputs) $(well_known_type_protoc_inputs) google-protobuf
oldpwd=`pwd` && ( cd $(srcdir) && $$oldpwd/../src/protoc$(EXEEXT) -I. -I$(top_srcdir)/src --cpp_out=$$oldpwd --java_out=$$oldpwd --ruby_out=$$oldpwd --objc_out=$$oldpwd --python_out=$$oldpwd --php_out=$$oldpwd --js_out=import_style=commonjs,binary:$$oldpwd $(conformance_protoc_inputs) )
oldpwd=`pwd` && ( cd $(srcdir) && $$oldpwd/../src/protoc$(EXEEXT) -I. -I$(top_srcdir)/src --cpp_out=$$oldpwd --java_out=$$oldpwd --objc_out=. --python_out=$$oldpwd --js_out=import_style=commonjs,binary:$$oldpwd $(conformance_proto2_protoc_inputs) )
oldpwd=`pwd` && ( cd $(srcdir) && $$oldpwd/../src/protoc$(EXEEXT) -I. -I$(top_srcdir)/src --cpp_out=$$oldpwd --java_out=$$oldpwd --ruby_out=$$oldpwd --objc_out=$$oldpwd --python_out=$$oldpwd --js_out=import_style=commonjs,binary:$$oldpwd $(conformance_proto2_protoc_inputs) )
oldpwd=`pwd` && ( cd $(srcdir) && $$oldpwd/../src/protoc$(EXEEXT) -I. -I$(top_srcdir)/src --cpp_out=$$oldpwd --java_out=$$oldpwd --ruby_out=$$oldpwd --python_out=$$oldpwd --js_out=import_style=commonjs,binary:$$oldpwd/google-protobuf $(well_known_type_protoc_inputs) )
## @mkdir -p lite
## oldpwd=`pwd` && ( cd $(srcdir) && $$oldpwd/../src/protoc$(EXEEXT) -I. -I$(top_srcdir)/src --java_out=lite:$$oldpwd/lite $(conformance_protoc_inputs) $(well_known_type_protoc_inputs) )

61
conformance/conformance_ruby.rb
View File

@ -32,6 +32,7 @@
require 'conformance_pb'
require 'google/protobuf/test_messages_proto3_pb'
require 'google/protobuf/test_messages_proto2_pb'
$test_count = 0
$verbose = false
@ -39,39 +40,39 @@ $verbose = false
def do_test(request)
test_message = ProtobufTestMessages::Proto3::TestAllTypesProto3.new
response = Conformance::ConformanceResponse.new
descriptor = Google::Protobuf::DescriptorPool.generated_pool.lookup(request.message_type)
unless descriptor
response.skipped = "Unknown message type: " + request.message_type
end
begin
case request.payload
when :protobuf_payload
if request.message_type.eql?('protobuf_test_messages.proto3.TestAllTypesProto3')
begin
test_message = ProtobufTestMessages::Proto3::TestAllTypesProto3.decode(
request.protobuf_payload)
rescue Google::Protobuf::ParseError => err
response.parse_error = err.message.encode('utf-8')
return response
end
elsif request.message_type.eql?('protobuf_test_messages.proto2.TestAllTypesProto2')
response.skipped = "Ruby doesn't support proto2"
return response
else
fail "Protobuf request doesn't have specific payload type"
end
when :json_payload
begin
test_message = ProtobufTestMessages::Proto3::TestAllTypesProto3.decode_json(
request.json_payload)
test_message = descriptor.msgclass.decode(request.protobuf_payload)
rescue Google::Protobuf::ParseError => err
response.parse_error = err.message.encode('utf-8')
return response
end
when :text_payload
begin
response.skipped = "Ruby doesn't support proto2"
return response
end
when :json_payload
begin
options = {}
if request.test_category == :JSON_IGNORE_UNKNOWN_PARSING_TEST
options[:ignore_unknown_fields] = true
end
test_message = descriptor.msgclass.decode_json(request.json_payload, options)
rescue Google::Protobuf::ParseError => err
response.parse_error = err.message.encode('utf-8')
return response
end
when :text_payload
begin
response.skipped = "Ruby doesn't support text format"
return response
end
when nil
fail "Request didn't have payload"
@ -82,10 +83,18 @@ def do_test(request)
fail 'Unspecified output format'
when :PROTOBUF
response.protobuf_payload = test_message.to_proto
begin
response.protobuf_payload = test_message.to_proto
rescue Google::Protobuf::ParseError => err
response.serialize_error = err.message.encode('utf-8')
end
when :JSON
response.json_payload = test_message.to_json
begin
response.json_payload = test_message.to_json
rescue Google::Protobuf::ParseError => err
response.serialize_error = err.message.encode('utf-8')
end
when nil
fail "Request didn't have requested output format"

166
conformance/failure_list_ruby.txt
View File

@ -1,114 +1,58 @@
Recommended.FieldMaskNumbersDontRoundTrip.JsonOutput
Recommended.FieldMaskPathsDontRoundTrip.JsonOutput
Recommended.FieldMaskTooManyUnderscore.JsonOutput
Recommended.Proto2.JsonInput.FieldNameExtension.Validator
Recommended.Proto3.JsonInput.BytesFieldBase64Url.JsonOutput
Recommended.Proto3.JsonInput.BytesFieldBase64Url.ProtobufOutput
Recommended.Proto3.JsonInput.DurationHas3FractionalDigits.Validator
Recommended.Proto3.JsonInput.DurationHas6FractionalDigits.Validator
Recommended.Proto3.JsonInput.FieldMaskInvalidCharacter
Recommended.Proto3.JsonInput.MapFieldValueIsNull
Recommended.Proto3.JsonInput.NullValueInNormalMessage.Validator
Recommended.Proto3.JsonInput.NullValueInOtherOneofNewFormat.Validator
Recommended.Proto3.JsonInput.NullValueInOtherOneofOldFormat.Validator
Recommended.Proto3.JsonInput.RepeatedFieldMessageElementIsNull
Recommended.Proto3.JsonInput.RepeatedFieldPrimitiveElementIsNull
Recommended.Proto3.JsonInput.StringEndsWithEscapeChar
Recommended.Proto3.JsonInput.StringFieldSurrogateInWrongOrder
Recommended.Proto3.JsonInput.StringFieldUnpairedHighSurrogate
Recommended.Proto3.JsonInput.StringFieldUnpairedLowSurrogate
Recommended.Proto3.JsonInput.TimestampHas3FractionalDigits.Validator
Recommended.Proto3.JsonInput.TimestampHas6FractionalDigits.Validator
Recommended.Proto3.ProtobufInput.ValidDataRepeated.BOOL.PackedInput.DefaultOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.BOOL.PackedInput.PackedOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.BOOL.UnpackedInput.DefaultOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.BOOL.UnpackedInput.PackedOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.DOUBLE.PackedInput.DefaultOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.DOUBLE.PackedInput.PackedOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.DOUBLE.UnpackedInput.DefaultOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.DOUBLE.UnpackedInput.PackedOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.ENUM.PackedInput.DefaultOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.ENUM.PackedInput.PackedOutput.ProtobufOutput
Recommended.Proto2.ProtobufInput.ValidDataRepeated.BOOL.PackedInput.PackedOutput.ProtobufOutput
Recommended.Proto2.ProtobufInput.ValidDataRepeated.BOOL.UnpackedInput.PackedOutput.ProtobufOutput
Recommended.Proto2.ProtobufInput.ValidDataRepeated.DOUBLE.PackedInput.PackedOutput.ProtobufOutput
Recommended.Proto2.ProtobufInput.ValidDataRepeated.DOUBLE.UnpackedInput.PackedOutput.ProtobufOutput
Recommended.Proto2.ProtobufInput.ValidDataRepeated.ENUM.PackedInput.PackedOutput.ProtobufOutput
Recommended.Proto2.ProtobufInput.ValidDataRepeated.ENUM.UnpackedInput.PackedOutput.ProtobufOutput
Recommended.Proto2.ProtobufInput.ValidDataRepeated.FIXED32.PackedInput.PackedOutput.ProtobufOutput
Recommended.Proto2.ProtobufInput.ValidDataRepeated.FIXED32.UnpackedInput.PackedOutput.ProtobufOutput
Recommended.Proto2.ProtobufInput.ValidDataRepeated.FIXED64.PackedInput.PackedOutput.ProtobufOutput
Recommended.Proto2.ProtobufInput.ValidDataRepeated.FIXED64.UnpackedInput.PackedOutput.ProtobufOutput
Recommended.Proto2.ProtobufInput.ValidDataRepeated.FLOAT.PackedInput.PackedOutput.ProtobufOutput
Recommended.Proto2.ProtobufInput.ValidDataRepeated.FLOAT.UnpackedInput.PackedOutput.ProtobufOutput
Recommended.Proto2.ProtobufInput.ValidDataRepeated.INT32.PackedInput.PackedOutput.ProtobufOutput
Recommended.Proto2.ProtobufInput.ValidDataRepeated.INT32.UnpackedInput.PackedOutput.ProtobufOutput
Recommended.Proto2.ProtobufInput.ValidDataRepeated.INT64.PackedInput.PackedOutput.ProtobufOutput
Recommended.Proto2.ProtobufInput.ValidDataRepeated.INT64.UnpackedInput.PackedOutput.ProtobufOutput
Recommended.Proto2.ProtobufInput.ValidDataRepeated.SFIXED32.PackedInput.PackedOutput.ProtobufOutput
Recommended.Proto2.ProtobufInput.ValidDataRepeated.SFIXED32.UnpackedInput.PackedOutput.ProtobufOutput
Recommended.Proto2.ProtobufInput.ValidDataRepeated.SFIXED64.PackedInput.PackedOutput.ProtobufOutput
Recommended.Proto2.ProtobufInput.ValidDataRepeated.SFIXED64.UnpackedInput.PackedOutput.ProtobufOutput
Recommended.Proto2.ProtobufInput.ValidDataRepeated.SINT32.PackedInput.PackedOutput.ProtobufOutput
Recommended.Proto2.ProtobufInput.ValidDataRepeated.SINT32.UnpackedInput.PackedOutput.ProtobufOutput
Recommended.Proto2.ProtobufInput.ValidDataRepeated.SINT64.PackedInput.PackedOutput.ProtobufOutput
Recommended.Proto2.ProtobufInput.ValidDataRepeated.SINT64.UnpackedInput.PackedOutput.ProtobufOutput
Recommended.Proto2.ProtobufInput.ValidDataRepeated.UINT32.PackedInput.PackedOutput.ProtobufOutput
Recommended.Proto2.ProtobufInput.ValidDataRepeated.UINT32.UnpackedInput.PackedOutput.ProtobufOutput
Recommended.Proto2.ProtobufInput.ValidDataRepeated.UINT64.PackedInput.PackedOutput.ProtobufOutput
Recommended.Proto2.ProtobufInput.ValidDataRepeated.UINT64.UnpackedInput.PackedOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataOneofBinary.MESSAGE.Merge.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.BOOL.PackedInput.UnpackedOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.BOOL.UnpackedInput.UnpackedOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.DOUBLE.PackedInput.UnpackedOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.DOUBLE.UnpackedInput.UnpackedOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.ENUM.PackedInput.UnpackedOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.ENUM.UnpackedInput.DefaultOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.ENUM.UnpackedInput.PackedOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.ENUM.UnpackedInput.UnpackedOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.FIXED32.PackedInput.DefaultOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.FIXED32.PackedInput.PackedOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.FIXED32.UnpackedInput.DefaultOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.FIXED32.UnpackedInput.PackedOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.FIXED64.PackedInput.DefaultOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.FIXED64.PackedInput.PackedOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.FIXED64.UnpackedInput.DefaultOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.FIXED64.UnpackedInput.PackedOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.FLOAT.PackedInput.DefaultOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.FLOAT.PackedInput.PackedOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.FLOAT.UnpackedInput.DefaultOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.FLOAT.UnpackedInput.PackedOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.INT32.PackedInput.DefaultOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.INT32.PackedInput.PackedOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.INT32.UnpackedInput.DefaultOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.INT32.UnpackedInput.PackedOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.INT64.PackedInput.DefaultOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.INT64.PackedInput.PackedOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.INT64.UnpackedInput.DefaultOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.INT64.UnpackedInput.PackedOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.SFIXED32.PackedInput.DefaultOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.SFIXED32.PackedInput.PackedOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.SFIXED32.UnpackedInput.DefaultOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.SFIXED32.UnpackedInput.PackedOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.SFIXED64.PackedInput.DefaultOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.SFIXED64.PackedInput.PackedOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.SFIXED64.UnpackedInput.DefaultOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.SFIXED64.UnpackedInput.PackedOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.SINT32.PackedInput.DefaultOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.SINT32.PackedInput.PackedOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.SINT32.UnpackedInput.DefaultOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.SINT32.UnpackedInput.PackedOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.SINT64.PackedInput.DefaultOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.SINT64.PackedInput.PackedOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.SINT64.UnpackedInput.DefaultOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.SINT64.UnpackedInput.PackedOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.UINT32.PackedInput.DefaultOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.UINT32.PackedInput.PackedOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.UINT32.UnpackedInput.DefaultOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.UINT32.UnpackedInput.PackedOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.UINT64.PackedInput.DefaultOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.UINT64.PackedInput.PackedOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.UINT64.UnpackedInput.DefaultOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.UINT64.UnpackedInput.PackedOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataScalarBinary.ENUM[3].ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataScalarBinary.ENUM[4].ProtobufOutput
Required.DurationProtoInputTooLarge.JsonOutput
Required.DurationProtoInputTooSmall.JsonOutput
Required.Proto2.JsonInput.StoresDefaultPrimitive.Validator
Required.Proto3.JsonInput.DoubleFieldMaxNegativeValue.JsonOutput
Required.Proto3.JsonInput.DoubleFieldMaxNegativeValue.ProtobufOutput
Required.Proto3.JsonInput.DoubleFieldMinPositiveValue.JsonOutput
Required.Proto3.JsonInput.DoubleFieldMinPositiveValue.ProtobufOutput
Required.Proto3.JsonInput.DoubleFieldNan.JsonOutput
Required.Proto3.JsonInput.DurationMinValue.JsonOutput
Required.Proto3.JsonInput.DurationRepeatedValue.JsonOutput
Required.Proto3.JsonInput.FloatFieldInfinity.JsonOutput
Required.Proto3.JsonInput.FloatFieldNan.JsonOutput
Required.Proto3.JsonInput.FloatFieldNegativeInfinity.JsonOutput
Required.Proto3.JsonInput.IgnoreUnknownJsonFalse.ProtobufOutput
Required.Proto3.JsonInput.IgnoreUnknownJsonNull.ProtobufOutput
Required.Proto3.JsonInput.IgnoreUnknownJsonNumber.ProtobufOutput
Required.Proto3.JsonInput.IgnoreUnknownJsonObject.ProtobufOutput
Required.Proto3.JsonInput.IgnoreUnknownJsonString.ProtobufOutput
Required.Proto3.JsonInput.IgnoreUnknownJsonTrue.ProtobufOutput
Required.Proto3.JsonInput.OneofFieldDuplicate
Required.Proto3.JsonInput.RejectTopLevelNull
Required.Proto3.JsonInput.StringFieldSurrogatePair.JsonOutput
Required.Proto3.JsonInput.StringFieldSurrogatePair.ProtobufOutput
Required.Proto3.ProtobufInput.DoubleFieldNormalizeQuietNan.JsonOutput
Required.Proto3.ProtobufInput.DoubleFieldNormalizeSignalingNan.JsonOutput
Required.Proto3.ProtobufInput.FloatFieldNormalizeQuietNan.JsonOutput
Required.Proto3.ProtobufInput.FloatFieldNormalizeSignalingNan.JsonOutput
Required.Proto3.ProtobufInput.ValidDataMap.STRING.MESSAGE.MissingDefault.JsonOutput
Required.Proto3.ProtobufInput.ValidDataRepeated.FLOAT.PackedInput.JsonOutput
Required.Proto3.ProtobufInput.ValidDataRepeated.FLOAT.UnpackedInput.JsonOutput
Required.Proto3.ProtobufInput.ValidDataScalar.FLOAT[2].JsonOutput
Required.TimestampProtoInputTooLarge.JsonOutput
Required.TimestampProtoInputTooSmall.JsonOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.FIXED32.PackedInput.UnpackedOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.FIXED32.UnpackedInput.UnpackedOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.FIXED64.PackedInput.UnpackedOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.FIXED64.UnpackedInput.UnpackedOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.FLOAT.PackedInput.UnpackedOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.FLOAT.UnpackedInput.UnpackedOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.INT32.PackedInput.UnpackedOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.INT32.UnpackedInput.UnpackedOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.INT64.PackedInput.UnpackedOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.INT64.UnpackedInput.UnpackedOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.SFIXED32.PackedInput.UnpackedOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.SFIXED32.UnpackedInput.UnpackedOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.SFIXED64.PackedInput.UnpackedOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.SFIXED64.UnpackedInput.UnpackedOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.SINT32.PackedInput.UnpackedOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.SINT32.UnpackedInput.UnpackedOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.SINT64.PackedInput.UnpackedOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.SINT64.UnpackedInput.UnpackedOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.UINT32.PackedInput.UnpackedOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.UINT32.UnpackedInput.UnpackedOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.UINT64.PackedInput.UnpackedOutput.ProtobufOutput
Recommended.Proto3.ProtobufInput.ValidDataRepeated.UINT64.UnpackedInput.UnpackedOutput.ProtobufOutput

3
kokoro/linux/dockerfile/test/ruby/Dockerfile
View File

@ -26,13 +26,14 @@ RUN apt-get update && apt-get install -y \
RUN gpg --keyserver hkp://keys.gnupg.net --recv-keys \
409B6B1796C275462A1703113804BB82D39DC0E3 \
7D2BAF1CF37B13E2069D6956105BD0E739499BDB
RUN \curl -sSL https://get.rvm.io | bash -s stable
RUN \curl -sSL https://get.rvm.io | bash -s master
RUN /bin/bash -l -c "rvm install 2.3.8"
RUN /bin/bash -l -c "rvm install 2.4.5"
RUN /bin/bash -l -c "rvm install 2.5.1"
RUN /bin/bash -l -c "rvm install 2.6.0"
RUN /bin/bash -l -c "rvm install 2.7.0"
RUN /bin/bash -l -c "rvm install 3.0.0"
RUN /bin/bash -l -c "echo 'gem: --no-ri --no-rdoc' > ~/.gemrc"
RUN /bin/bash -l -c "echo 'export PATH=/usr/local/rvm/bin:$PATH' >> ~/.bashrc"

18
kokoro/linux/ruby30/build.sh Executable file
View File

@ -0,0 +1,18 @@
#!/bin/bash
#
# This is the top-level script we give to Kokoro as the entry point for
# running the "pull request" project:
#
# This script selects a specific Dockerfile (for building a Docker image) and
# a script to run inside that image. Then we delegate to the general
# build_and_run_docker.sh script.
# Change to repo root
cd $(dirname $0)/../../..
export DOCKERHUB_ORGANIZATION=protobuftesting
export DOCKERFILE_DIR=kokoro/linux/dockerfile/test/ruby
export DOCKER_RUN_SCRIPT=kokoro/linux/pull_request_in_docker.sh
export OUTPUT_DIR=testoutput
export TEST_SET="ruby30"
./kokoro/linux/build_and_run_docker.sh

11
kokoro/linux/ruby30/continuous.cfg Normal file
View File

@ -0,0 +1,11 @@
# Config file for running tests in Kokoro
# Location of the build script in repository
build_file: "protobuf/kokoro/linux/ruby30/build.sh"
timeout_mins: 120
action {
define_artifacts {
regex: "**/sponge_log.xml"
}
}

11
kokoro/linux/ruby30/presubmit.cfg Normal file
View File

@ -0,0 +1,11 @@
# Config file for running tests in Kokoro
# Location of the build script in repository
build_file: "protobuf/kokoro/linux/ruby30/build.sh"
timeout_mins: 120
action {
define_artifacts {
regex: "**/sponge_log.xml"
}
}

2
kokoro/macos/prepare_build_macos_rc
View File

@ -80,5 +80,5 @@ if [[ "${KOKORO_INSTALL_RVM:-}" == "yes" ]] ; then
curl -sSL https://rvm.io/mpapis.asc | gpg --import -
curl -sSL https://rvm.io/pkuczynski.asc | gpg --import -
curl -sSL https://get.rvm.io | bash -s stable --ruby
curl -sSL https://get.rvm.io | bash -s master --ruby
fi

13
kokoro/macos/ruby30/build.sh Executable file
View File

@ -0,0 +1,13 @@
#!/bin/bash
#
# Build file to set up and run tests
# Change to repo root
cd $(dirname $0)/../../..
# Prepare worker environment to run tests
KOKORO_INSTALL_RUBY=yes
KOKORO_INSTALL_RVM=yes
source kokoro/macos/prepare_build_macos_rc
./tests.sh ruby30

5
kokoro/macos/ruby30/continuous.cfg Normal file
View File

@ -0,0 +1,5 @@
# Config file for running tests in Kokoro
# Location of the build script in repository
build_file: "protobuf/kokoro/macos/ruby30/build.sh"
timeout_mins: 1440

5
kokoro/macos/ruby30/presubmit.cfg Normal file
View File

@ -0,0 +1,5 @@
# Config file for running tests in Kokoro
# Location of the build script in repository
build_file: "protobuf/kokoro/macos/ruby30/build.sh"
timeout_mins: 1440

17
kokoro/release/ruby/macos/ruby/ruby_build_environment.sh
View File

@ -21,13 +21,13 @@ rm -rf ~/.rake-compiler
CROSS_RUBY=$(mktemp tmpfile.XXXXXXXX)
curl https://raw.githubusercontent.com/rake-compiler/rake-compiler/v1.1.0/tasks/bin/cross-ruby.rake > "$CROSS_RUBY"
curl https://raw.githubusercontent.com/rake-compiler/rake-compiler/72184e51779b6a3b9b8580b036a052fdc3181ced/tasks/bin/cross-ruby.rake > "$CROSS_RUBY"
# See https://github.com/grpc/grpc/issues/12161 for verconf.h patch details
patch "$CROSS_RUBY" << EOF
--- cross-ruby.rake 2018-04-10 11:32:16.000000000 -0700
+++ patched 2018-04-10 11:40:25.000000000 -0700
@@ -141,8 +141,10 @@
--- cross-ruby.rake 2020-12-11 11:17:53.000000000 +0900
+++ patched 2020-12-11 11:18:52.000000000 +0900
@@ -111,10 +111,12 @@
"--host=#{MINGW_HOST}",
"--target=#{MINGW_TARGET}",
"--build=#{RUBY_BUILD}",
@ -36,10 +36,13 @@ patch "$CROSS_RUBY" << EOF
+ '--disable-shared',
'--disable-install-doc',
+ '--without-gmp',
'--with-ext='
'--with-ext=',
- 'LDFLAGS=-pipe -s',
+ 'LDFLAGS=-pipe',
]
@@ -159,6 +161,7 @@
# Force Winsock2 for Ruby 1.8, 1.9 defaults to it
@@ -130,6 +132,7 @@
# make
file "#{build_dir}/ruby.exe" => ["#{build_dir}/Makefile"] do |t|
chdir File.dirname(t.prerequisites.first) do
@ -55,7 +58,7 @@ set +x # rvm commands are very verbose
rvm use 2.7.0
set -x
ruby --version | grep 'ruby 2.7.0'
for v in 2.7.0 ; do
for v in 3.0.0 2.7.0 ; do
ccache -c
rake -f "$CROSS_RUBY" cross-ruby VERSION="$v" HOST=x86_64-darwin11 MAKE="$MAKE"
done

10
ruby/Rakefile
View File

@ -51,6 +51,12 @@ if RUBY_PLATFORM == "java"
system("mvn --batch-mode package")
end
else
unless ENV['IN_DOCKER'] == 'true'
# We need wyhash in-tree.
FileUtils.mkdir_p("ext/google/protobuf_c/third_party/wyhash")
FileUtils.cp("../third_party/wyhash/wyhash.h", "ext/google/protobuf_c/third_party/wyhash/wyhash.h")
end
Rake::ExtensionTask.new("protobuf_c", spec) do |ext|
unless RUBY_PLATFORM =~ /darwin/
# TODO: also set "no_native to true" for mac if possible. As is,
@ -73,7 +79,7 @@ else
['x86-mingw32', 'x64-mingw32', 'x86_64-linux', 'x86-linux'].each do |plat|
RakeCompilerDock.sh <<-"EOT", platform: plat
bundle && \
IN_DOCKER=true rake native:#{plat} pkg/#{spec.full_name}-#{plat}.gem RUBY_CC_VERSION=2.7.0:2.6.0:2.5.0:2.4.0:2.3.0
IN_DOCKER=true rake native:#{plat} pkg/#{spec.full_name}-#{plat}.gem RUBY_CC_VERSION=3.0.0:2.7.0:2.6.0:2.5.0:2.4.0:2.3.0
EOT
end
end
@ -81,7 +87,7 @@ else
if RUBY_PLATFORM =~ /darwin/
task 'gem:native' do
system "rake genproto"
system "rake cross native gem RUBY_CC_VERSION=2.7.0:2.6.0:2.5.1:2.4.0:2.3.0"
system "rake cross native gem RUBY_CC_VERSION=3.0.0:2.7.0:2.6.0:2.5.1:2.4.0:2.3.0"
end
else
task 'gem:native' => [:genproto, 'gem:windows']

6
ruby/compatibility_tests/v3.0.0/tests/basic.rb
View File

@ -1264,10 +1264,10 @@ module BasicTest
m = MapMessage.new(:map_string_int32 => {"a" => 1})
expected = '{"mapStringInt32":{"a":1},"mapStringMsg":{}}'
expected_preserve = '{"map_string_int32":{"a":1},"map_string_msg":{}}'
assert MapMessage.encode_json(m) == expected
assert_equal expected, MapMessage.encode_json(m, :emit_defaults => true)
json = MapMessage.encode_json(m, :preserve_proto_fieldnames => true)
assert json == expected_preserve
json = MapMessage.encode_json(m, :preserve_proto_fieldnames => true, :emit_defaults => true)
assert_equal expected_preserve, json
m2 = MapMessage.decode_json(MapMessage.encode_json(m))
assert m == m2

349
ruby/ext/google/protobuf_c/convert.c Normal file
View File

@ -0,0 +1,349 @@
// Protocol Buffers - Google's data interchange format
// Copyright 2008 Google Inc. All rights reserved.
// https://developers.google.com/protocol-buffers/
//
// Redistribution and use in source and binary forms, with or without
// modification, are permitted provided that the following conditions are
// met:
//
// * Redistributions of source code must retain the above copyright
// notice, this list of conditions and the following disclaimer.
// * Redistributions in binary form must reproduce the above
// copyright notice, this list of conditions and the following disclaimer
// in the documentation and/or other materials provided with the
// distribution.
// * Neither the name of Google Inc. nor the names of its
// contributors may be used to endorse or promote products derived from
// this software without specific prior written permission.
//
// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
// -----------------------------------------------------------------------------
// Ruby <-> upb data conversion functions.
//
// This file Also contains a few other assorted algorithms on upb_msgval.
//
// None of the algorithms in this file require any access to the internal
// representation of Ruby or upb objects.
// -----------------------------------------------------------------------------
#include "convert.h"
#include "message.h"
#include "protobuf.h"
#include "third_party/wyhash/wyhash.h"
static upb_strview Convert_StringData(VALUE str, upb_arena *arena) {
upb_strview ret;
if (arena) {
char *ptr = upb_arena_malloc(arena, RSTRING_LEN(str));
memcpy(ptr, RSTRING_PTR(str), RSTRING_LEN(str));
ret.data = ptr;
} else {
// Data is only needed temporarily (within map lookup).
ret.data = RSTRING_PTR(str);
}
ret.size = RSTRING_LEN(str);
return ret;
}
static bool is_ruby_num(VALUE value) {
return (TYPE(value) == T_FLOAT ||
TYPE(value) == T_FIXNUM ||
TYPE(value) == T_BIGNUM);
}
static void Convert_CheckInt(const char* name, upb_fieldtype_t type,
VALUE val) {
if (!is_ruby_num(val)) {
rb_raise(cTypeError,
"Expected number type for integral field '%s' (given %s).", name,
rb_class2name(CLASS_OF(val)));
}
// NUM2{INT,UINT,LL,ULL} macros do the appropriate range checks on upper
// bound; we just need to do precision checks (i.e., disallow rounding) and
// check for < 0 on unsigned types.
if (TYPE(val) == T_FLOAT) {
double dbl_val = NUM2DBL(val);
if (floor(dbl_val) != dbl_val) {
rb_raise(rb_eRangeError,
"Non-integral floating point value assigned to integer field "
"'%s' (given %s).",
name, rb_class2name(CLASS_OF(val)));
}
}
if (type == UPB_TYPE_UINT32 || type == UPB_TYPE_UINT64) {
if (NUM2DBL(val) < 0) {
rb_raise(
rb_eRangeError,
"Assigning negative value to unsigned integer field '%s' (given %s).",
name, rb_class2name(CLASS_OF(val)));
}
}
}
static int32_t Convert_ToEnum(VALUE value, const char* name,
const upb_enumdef* e) {
int32_t val;
switch (TYPE(value)) {
case T_FLOAT:
case T_FIXNUM:
case T_BIGNUM:
Convert_CheckInt(name, UPB_TYPE_INT32, value);
val = NUM2INT(value);
break;
case T_STRING:
if (!upb_enumdef_ntoi(e, RSTRING_PTR(value), RSTRING_LEN(value), &val)) {
goto unknownval;
}
break;
case T_SYMBOL:
if (!upb_enumdef_ntoiz(e, rb_id2name(SYM2ID(value)), &val)) {
goto unknownval;
}
break;
default:
rb_raise(cTypeError,
"Expected number or symbol type for enum field '%s'.", name);
}
return val;
unknownval:
rb_raise(rb_eRangeError, "Unknown symbol value for enum field '%s'.", name);
}
upb_msgval Convert_RubyToUpb(VALUE value, const char* name, TypeInfo type_info,
upb_arena* arena) {
upb_msgval ret;
switch (type_info.type) {
case UPB_TYPE_FLOAT:
if (!is_ruby_num(value)) {
rb_raise(cTypeError, "Expected number type for float field '%s' (given %s).",
name, rb_class2name(CLASS_OF(value)));
}
ret.float_val = NUM2DBL(value);
break;
case UPB_TYPE_DOUBLE:
if (!is_ruby_num(value)) {
rb_raise(cTypeError, "Expected number type for double field '%s' (given %s).",
name, rb_class2name(CLASS_OF(value)));
}
ret.double_val = NUM2DBL(value);
break;
case UPB_TYPE_BOOL: {
if (value == Qtrue) {
ret.bool_val = 1;
} else if (value == Qfalse) {
ret.bool_val = 0;
} else {
rb_raise(cTypeError, "Invalid argument for boolean field '%s' (given %s).",
name, rb_class2name(CLASS_OF(value)));
}
break;
}
case UPB_TYPE_STRING: {
VALUE utf8 = rb_enc_from_encoding(rb_utf8_encoding());
if (CLASS_OF(value) == rb_cSymbol) {
value = rb_funcall(value, rb_intern("to_s"), 0);
} else if (CLASS_OF(value) != rb_cString) {
rb_raise(cTypeError, "Invalid argument for string field '%s' (given %s).",
name, rb_class2name(CLASS_OF(value)));
}
if (rb_obj_encoding(value) != utf8) {
// Note: this will not duplicate underlying string data unless necessary.
value = rb_str_encode(value, utf8, 0, Qnil);
if (rb_enc_str_coderange(value) == ENC_CODERANGE_BROKEN) {
rb_raise(rb_eEncodingError, "String is invalid UTF-8");
}
}
ret.str_val = Convert_StringData(value, arena);
break;
}
case UPB_TYPE_BYTES: {
VALUE bytes = rb_enc_from_encoding(rb_ascii8bit_encoding());
if (CLASS_OF(value) != rb_cString) {
rb_raise(cTypeError, "Invalid argument for bytes field '%s' (given %s).",
name, rb_class2name(CLASS_OF(value)));
}
if (rb_obj_encoding(value) != bytes) {
// Note: this will not duplicate underlying string data unless necessary.
// TODO(haberman): is this really necessary to get raw bytes?
value = rb_str_encode(value, bytes, 0, Qnil);
}
ret.str_val = Convert_StringData(value, arena);
break;
}
case UPB_TYPE_MESSAGE:
ret.msg_val =
Message_GetUpbMessage(value, type_info.def.msgdef, name, arena);
break;
case UPB_TYPE_ENUM:
ret.int32_val = Convert_ToEnum(value, name, type_info.def.enumdef);
break;
case UPB_TYPE_INT32:
case UPB_TYPE_INT64:
case UPB_TYPE_UINT32:
case UPB_TYPE_UINT64:
Convert_CheckInt(name, type_info.type, value);
switch (type_info.type) {
case UPB_TYPE_INT32:
ret.int32_val = NUM2INT(value);
break;
case UPB_TYPE_INT64:
ret.int64_val = NUM2LL(value);
break;
case UPB_TYPE_UINT32:
ret.uint32_val = NUM2UINT(value);
break;
case UPB_TYPE_UINT64:
ret.uint64_val = NUM2ULL(value);
break;
default:
break;
}
break;
default:
break;
}
return ret;
}
VALUE Convert_UpbToRuby(upb_msgval upb_val, TypeInfo type_info, VALUE arena) {
switch (type_info.type) {
case UPB_TYPE_FLOAT:
return DBL2NUM(upb_val.float_val);
case UPB_TYPE_DOUBLE:
return DBL2NUM(upb_val.double_val);
case UPB_TYPE_BOOL:
return upb_val.bool_val ? Qtrue : Qfalse;
case UPB_TYPE_INT32:
return INT2NUM(upb_val.int32_val);
case UPB_TYPE_INT64:
return LL2NUM(upb_val.int64_val);
case UPB_TYPE_UINT32:
return UINT2NUM(upb_val.uint32_val);
case UPB_TYPE_UINT64:
return ULL2NUM(upb_val.int64_val);
case UPB_TYPE_ENUM: {
const char* name =
upb_enumdef_iton(type_info.def.enumdef, upb_val.int32_val);
if (name) {
return ID2SYM(rb_intern(name));
} else {
return INT2NUM(upb_val.int32_val);
}
}
case UPB_TYPE_STRING: {
VALUE str_rb = rb_str_new(upb_val.str_val.data, upb_val.str_val.size);
rb_enc_associate(str_rb, rb_utf8_encoding());
rb_obj_freeze(str_rb);
return str_rb;
}
case UPB_TYPE_BYTES: {
VALUE str_rb = rb_str_new(upb_val.str_val.data, upb_val.str_val.size);
rb_enc_associate(str_rb, rb_ascii8bit_encoding());
rb_obj_freeze(str_rb);
return str_rb;
}
case UPB_TYPE_MESSAGE:
return Message_GetRubyWrapper((upb_msg*)upb_val.msg_val,
type_info.def.msgdef, arena);
default:
rb_raise(rb_eRuntimeError, "Convert_UpbToRuby(): Unexpected type %d",
(int)type_info.type);
}
}
upb_msgval Msgval_DeepCopy(upb_msgval msgval, TypeInfo type_info,
upb_arena* arena) {
upb_msgval new_msgval;
switch (type_info.type) {
default:
memcpy(&new_msgval, &msgval, sizeof(msgval));
break;
case UPB_TYPE_STRING:
case UPB_TYPE_BYTES: {
size_t n = msgval.str_val.size;
char *mem = upb_arena_malloc(arena, n);
new_msgval.str_val.data = mem;
new_msgval.str_val.size = n;
memcpy(mem, msgval.str_val.data, n);
break;
}
case UPB_TYPE_MESSAGE:
new_msgval.msg_val =
Message_deep_copy(msgval.msg_val, type_info.def.msgdef, arena);
break;
}
return new_msgval;
}
bool Msgval_IsEqual(upb_msgval val1, upb_msgval val2, TypeInfo type_info) {
switch (type_info.type) {
case UPB_TYPE_BOOL:
return memcmp(&val1, &val2, 1) == 0;
case UPB_TYPE_FLOAT:
case UPB_TYPE_INT32:
case UPB_TYPE_UINT32:
case UPB_TYPE_ENUM:
return memcmp(&val1, &val2, 4) == 0;
case UPB_TYPE_DOUBLE:
case UPB_TYPE_INT64:
case UPB_TYPE_UINT64:
return memcmp(&val1, &val2, 8) == 0;
case UPB_TYPE_STRING:
case UPB_TYPE_BYTES:
return val1.str_val.size != val2.str_val.size ||
memcmp(val1.str_val.data, val2.str_val.data,
val1.str_val.size) == 0;
case UPB_TYPE_MESSAGE:
return Message_Equal(val1.msg_val, val2.msg_val, type_info.def.msgdef);
default:
rb_raise(rb_eRuntimeError, "Internal error, unexpected type");
}
}
uint64_t Msgval_GetHash(upb_msgval val, TypeInfo type_info, uint64_t seed) {
switch (type_info.type) {
case UPB_TYPE_BOOL:
return wyhash(&val, 1, seed, _wyp);
case UPB_TYPE_FLOAT:
case UPB_TYPE_INT32:
case UPB_TYPE_UINT32:
case UPB_TYPE_ENUM:
return wyhash(&val, 4, seed, _wyp);
case UPB_TYPE_DOUBLE:
case UPB_TYPE_INT64:
case UPB_TYPE_UINT64:
return wyhash(&val, 8, seed, _wyp);
case UPB_TYPE_STRING:
case UPB_TYPE_BYTES:
return wyhash(val.str_val.data, val.str_val.size, seed, _wyp);
case UPB_TYPE_MESSAGE:
return Message_Hash(val.msg_val, type_info.def.msgdef, seed);
default:
rb_raise(rb_eRuntimeError, "Internal error, unexpected type");
}
}

72
ruby/ext/google/protobuf_c/convert.h Normal file
View File

@ -0,0 +1,72 @@
// Protocol Buffers - Google's data interchange format
// Copyright 2008 Google Inc. All rights reserved.
// https://developers.google.com/protocol-buffers/
//
// Redistribution and use in source and binary forms, with or without
// modification, are permitted provided that the following conditions are
// met:
//
// * Redistributions of source code must retain the above copyright
// notice, this list of conditions and the following disclaimer.
// * Redistributions in binary form must reproduce the above
// copyright notice, this list of conditions and the following disclaimer
// in the documentation and/or other materials provided with the
// distribution.
// * Neither the name of Google Inc. nor the names of its
// contributors may be used to endorse or promote products derived from
// this software without specific prior written permission.
//
// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#ifndef RUBY_PROTOBUF_CONVERT_H_
#define RUBY_PROTOBUF_CONVERT_H_
#include <ruby/ruby.h>
#include "protobuf.h"
#include "ruby-upb.h"
// Converts |ruby_val| to a upb_msgval according to |type_info|.
//
// The |arena| parameter indicates the lifetime of the container where this
// value will be assigned. It is used as follows:
// - If type is string or bytes, the string data will be copied into |arena|.
// - If type is message, and we need to auto-construct a message due to implicit
// conversions (eg. Time -> Google::Protobuf::Timestamp), the new message
// will be created in |arena|.
// - If type is message and the Ruby value is a message instance, we will fuse
// the message's arena into |arena|, to ensure that this message outlives the
// container.
upb_msgval Convert_RubyToUpb(VALUE ruby_val, const char *name,
TypeInfo type_info, upb_arena *arena);
// Converts |upb_val| to a Ruby VALUE according to |type_info|. This may involve
// creating a Ruby wrapper object.
//
// The |arena| parameter indicates the arena that owns the lifetime of
// |upb_val|. Any Ruby wrapper object that is created will reference |arena|
// and ensure it outlives the wrapper.
VALUE Convert_UpbToRuby(upb_msgval upb_val, TypeInfo type_info, VALUE arena);
// Creates a deep copy of |msgval| in |arena|.
upb_msgval Msgval_DeepCopy(upb_msgval msgval, TypeInfo type_info,
upb_arena *arena);
// Returns true if |val1| and |val2| are equal. Their type is given by
// |type_info|.
bool Msgval_IsEqual(upb_msgval val1, upb_msgval val2, TypeInfo type_info);
// Returns a hash value for the given upb_msgval.
uint64_t Msgval_GetHash(upb_msgval val, TypeInfo type_info, uint64_t seed);
#endif // RUBY_PROTOBUF_CONVERT_H_

2911
ruby/ext/google/protobuf_c/defs.c
View File

File diff suppressed because it is too large Load Diff

107
ruby/ext/google/protobuf_c/defs.h Normal file
View File

@ -0,0 +1,107 @@
// Protocol Buffers - Google's data interchange format
// Copyright 2008 Google Inc. All rights reserved.
// https://developers.google.com/protocol-buffers/
//
// Redistribution and use in source and binary forms, with or without
// modification, are permitted provided that the following conditions are
// met:
//
// * Redistributions of source code must retain the above copyright
// notice, this list of conditions and the following disclaimer.
// * Redistributions in binary form must reproduce the above
// copyright notice, this list of conditions and the following disclaimer
// in the documentation and/or other materials provided with the
// distribution.
// * Neither the name of Google Inc. nor the names of its
// contributors may be used to endorse or promote products derived from
// this software without specific prior written permission.
//
// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#ifndef RUBY_PROTOBUF_DEFS_H_
#define RUBY_PROTOBUF_DEFS_H_
#include <ruby/ruby.h>
#include "protobuf.h"
#include "ruby-upb.h"
// -----------------------------------------------------------------------------
// TypeInfo
// -----------------------------------------------------------------------------
// This bundles a upb_fieldtype_t and msgdef/enumdef when appropriate. This is
// convenient for functions that need type information but cannot necessarily
// assume a upb_fielddef will be available.
//
// For example, Google::Protobuf::Map and Google::Protobuf::RepeatedField can
// be constructed with type information alone:
//
// # RepeatedField will internally store the type information in a TypeInfo.
// Google::Protobuf::RepeatedField.new(:message, FooMessage)
typedef struct {
upb_fieldtype_t type;
union {
const upb_msgdef* msgdef; // When type == UPB_TYPE_MESSAGE
const upb_enumdef* enumdef; // When type == UPB_TYPE_ENUM
} def;
} TypeInfo;
static inline TypeInfo TypeInfo_get(const upb_fielddef *f) {
TypeInfo ret = {upb_fielddef_type(f), {NULL}};
switch (ret.type) {
case UPB_TYPE_MESSAGE:
ret.def.msgdef = upb_fielddef_msgsubdef(f);
break;
case UPB_TYPE_ENUM:
ret.def.enumdef = upb_fielddef_enumsubdef(f);
break;
default:
break;
}
return ret;
}
TypeInfo TypeInfo_FromClass(int argc, VALUE* argv, int skip_arg,
VALUE* type_class, VALUE* init_arg);
static inline TypeInfo TypeInfo_from_type(upb_fieldtype_t type) {
TypeInfo ret = {type};
assert(type != UPB_TYPE_MESSAGE && type != UPB_TYPE_ENUM);
return ret;
}
// -----------------------------------------------------------------------------
// Other utilities
// -----------------------------------------------------------------------------
VALUE Descriptor_DefToClass(const upb_msgdef *m);
// Returns the underlying msgdef, enumdef, or symtab (respectively) for the
// given Descriptor, EnumDescriptor, or DescriptorPool Ruby object.
const upb_enumdef *EnumDescriptor_GetEnumDef(VALUE enum_desc_rb);
const upb_symtab *DescriptorPool_GetSymtab(VALUE desc_pool_rb);
const upb_msgdef *Descriptor_GetMsgDef(VALUE desc_rb);
// Returns a upb field type for the given Ruby symbol
// (eg. :float => UPB_TYPE_FLOAT).
upb_fieldtype_t ruby_to_fieldtype(VALUE type);
// The singleton generated pool (a DescriptorPool object).
extern VALUE generated_pool;
// Call at startup to register all types in this module.
void Defs_register(VALUE module);
#endif // RUBY_PROTOBUF_DEFS_H_

1795
ruby/ext/google/protobuf_c/encode_decode.c
View File

File diff suppressed because it is too large Load Diff

9
ruby/ext/google/protobuf_c/extconf.rb
View File

@ -3,9 +3,9 @@
require 'mkmf'
if RUBY_PLATFORM =~ /darwin/ || RUBY_PLATFORM =~ /linux/
$CFLAGS += " -std=gnu90 -O3 -DNDEBUG -Wall -Wdeclaration-after-statement -Wsign-compare"
$CFLAGS += " -std=gnu99 -O3 -DNDEBUG -fvisibility=hidden -Wall -Wsign-compare -Wno-declaration-after-statement"
else
$CFLAGS += " -std=gnu90 -O3 -DNDEBUG"
$CFLAGS += " -std=gnu99 -O3 -DNDEBUG"
end
@ -14,8 +14,7 @@ if RUBY_PLATFORM =~ /linux/
$LDFLAGS += " -Wl,-wrap,memcpy"
end
$objs = ["protobuf.o", "defs.o", "storage.o", "message.o",
"repeated_field.o", "map.o", "encode_decode.o", "upb.o",
"wrap_memcpy.o"]
$objs = ["protobuf.o", "convert.o", "defs.o", "message.o",
"repeated_field.o", "map.o", "ruby-upb.o", "wrap_memcpy.o"]
create_makefile("google/protobuf_c")

760
ruby/ext/google/protobuf_c/map.c
View File

@ -28,170 +28,231 @@
// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#include "convert.h"
#include "defs.h"
#include "message.h"
#include "protobuf.h"
// -----------------------------------------------------------------------------
// Basic map operations on top of upb's strtable.
// Basic map operations on top of upb_map.
//
// Note that we roll our own `Map` container here because, as for
// `RepeatedField`, we want a strongly-typed container. This is so that any user
// errors due to incorrect map key or value types are raised as close as
// possible to the error site, rather than at some deferred point (e.g.,
// serialization).
//
// We build our `Map` on top of upb_strtable so that we're able to take
// advantage of the native_slot storage abstraction, as RepeatedField does.
// (This is not quite a perfect mapping -- see the key conversions below -- but
// gives us full support and error-checking for all value types for free.)
// -----------------------------------------------------------------------------
// Map values are stored using the native_slot abstraction (as with repeated
// field values), but keys are a bit special. Since we use a strtable, we need
// to store keys as sequences of bytes such that equality of those bytes maps
// one-to-one to equality of keys. We store strings directly (i.e., they map to
// their own bytes) and integers as native integers (using the native_slot
// abstraction).
// Note that there is another tradeoff here in keeping string keys as native
// strings rather than Ruby strings: traversing the Map requires conversion to
// Ruby string values on every traversal, potentially creating more garbage. We
// should consider ways to cache a Ruby version of the key if this becomes an
// issue later.
// Forms a key to use with the underlying strtable from a Ruby key value. |buf|
// must point to TABLE_KEY_BUF_LENGTH bytes of temporary space, used to
// construct a key byte sequence if needed. |out_key| and |out_length| provide
// the resulting key data/length.
#define TABLE_KEY_BUF_LENGTH 8 // sizeof(uint64_t)
static VALUE table_key(Map* self, VALUE key,
char* buf,
const char** out_key,
size_t* out_length) {
switch (self->key_type) {
case UPB_TYPE_BYTES:
case UPB_TYPE_STRING:
// Strings: use string content directly.
if (TYPE(key) == T_SYMBOL) {
key = rb_id2str(SYM2ID(key));
}
Check_Type(key, T_STRING);
key = native_slot_encode_and_freeze_string(self->key_type, key);
*out_key = RSTRING_PTR(key);
*out_length = RSTRING_LEN(key);
break;
case UPB_TYPE_BOOL:
case UPB_TYPE_INT32:
case UPB_TYPE_INT64:
case UPB_TYPE_UINT32:
case UPB_TYPE_UINT64:
native_slot_set("", self->key_type, Qnil, buf, key);
*out_key = buf;
*out_length = native_slot_size(self->key_type);
break;
default:
// Map constructor should not allow a Map with another key type to be
// constructed.
assert(false);
break;
}
return key;
}
static VALUE table_key_to_ruby(Map* self, upb_strview key) {
switch (self->key_type) {
case UPB_TYPE_BYTES:
case UPB_TYPE_STRING: {
VALUE ret = rb_str_new(key.data, key.size);
rb_enc_associate(ret,
(self->key_type == UPB_TYPE_BYTES) ?
kRubyString8bitEncoding : kRubyStringUtf8Encoding);
return ret;
}
case UPB_TYPE_BOOL:
case UPB_TYPE_INT32:
case UPB_TYPE_INT64:
case UPB_TYPE_UINT32:
case UPB_TYPE_UINT64:
return native_slot_get(self->key_type, Qnil, key.data);
default:
assert(false);
return Qnil;
}
}
static void* value_memory(upb_value* v) {
return (void*)(&v->val);
}
// -----------------------------------------------------------------------------
// Map container type.
// -----------------------------------------------------------------------------
typedef struct {
const upb_map *map; // Can convert to mutable when non-frozen.
upb_fieldtype_t key_type;
TypeInfo value_type_info;
VALUE value_type_class;
VALUE arena;
} Map;
static void Map_mark(void* _self) {
Map* self = _self;
rb_gc_mark(self->value_type_class);
rb_gc_mark(self->arena);
}
const rb_data_type_t Map_type = {
"Google::Protobuf::Map",
{ Map_mark, Map_free, NULL },
{ Map_mark, RUBY_DEFAULT_FREE, NULL },
.flags = RUBY_TYPED_FREE_IMMEDIATELY,
};
VALUE cMap;
Map* ruby_to_Map(VALUE _self) {
static Map* ruby_to_Map(VALUE _self) {
Map* self;
TypedData_Get_Struct(_self, Map, &Map_type, self);
return self;
}
void Map_mark(void* _self) {
Map* self = _self;
rb_gc_mark(self->value_type_class);
rb_gc_mark(self->parse_frame);
if (self->value_type == UPB_TYPE_STRING ||
self->value_type == UPB_TYPE_BYTES ||
self->value_type == UPB_TYPE_MESSAGE) {
upb_strtable_iter it;
for (upb_strtable_begin(&it, &self->table);
!upb_strtable_done(&it);
upb_strtable_next(&it)) {
upb_value v = upb_strtable_iter_value(&it);
void* mem = value_memory(&v);
native_slot_mark(self->value_type, mem);
}
}
}
void Map_free(void* _self) {
Map* self = _self;
upb_strtable_uninit(&self->table);
xfree(self);
}
VALUE Map_alloc(VALUE klass) {
static VALUE Map_alloc(VALUE klass) {
Map* self = ALLOC(Map);
memset(self, 0, sizeof(Map));
self->map = NULL;
self->value_type_class = Qnil;
self->value_type_info.def.msgdef = NULL;
self->arena = Qnil;
return TypedData_Wrap_Struct(klass, &Map_type, self);
}
VALUE Map_set_frame(VALUE map, VALUE val) {
Map* self = ruby_to_Map(map);
self->parse_frame = val;
VALUE Map_GetRubyWrapper(upb_map* map, upb_fieldtype_t key_type,
TypeInfo value_type, VALUE arena) {
PBRUBY_ASSERT(map);
VALUE val = ObjectCache_Get(map);
if (val == Qnil) {
val = Map_alloc(cMap);
Map* self;
ObjectCache_Add(map, val, Arena_get(arena));
TypedData_Get_Struct(val, Map, &Map_type, self);
self->map = map;
self->arena = arena;
self->key_type = key_type;
self->value_type_info = value_type;
if (self->value_type_info.type == UPB_TYPE_MESSAGE) {
const upb_msgdef *val_m = self->value_type_info.def.msgdef;
self->value_type_class = Descriptor_DefToClass(val_m);
}
}
return val;
}
static bool needs_typeclass(upb_fieldtype_t type) {
switch (type) {
case UPB_TYPE_MESSAGE:
case UPB_TYPE_ENUM:
return true;
default:
return false;
static VALUE Map_new_this_type(Map *from) {
VALUE arena_rb = Arena_new();
upb_map* map = upb_map_new(Arena_get(arena_rb), from->key_type,
from->value_type_info.type);
VALUE ret =
Map_GetRubyWrapper(map, from->key_type, from->value_type_info, arena_rb);
PBRUBY_ASSERT(ruby_to_Map(ret)->value_type_class == from->value_type_class);
return ret;
}
static TypeInfo Map_keyinfo(Map* self) {
TypeInfo ret;
ret.type = self->key_type;
ret.def.msgdef = NULL;
return ret;
}
static upb_map *Map_GetMutable(VALUE _self) {
rb_check_frozen(_self);
return (upb_map*)ruby_to_Map(_self)->map;
}
VALUE Map_CreateHash(const upb_map* map, upb_fieldtype_t key_type,
TypeInfo val_info) {
VALUE hash = rb_hash_new();
size_t iter = UPB_MAP_BEGIN;
TypeInfo key_info = TypeInfo_from_type(key_type);
if (!map) return hash;
while (upb_mapiter_next(map, &iter)) {
upb_msgval key = upb_mapiter_key(map, iter);
upb_msgval val = upb_mapiter_value(map, iter);
VALUE key_val = Convert_UpbToRuby(key, key_info, Qnil);
VALUE val_val = Scalar_CreateHash(val, val_info);
rb_hash_aset(hash, key_val, val_val);
}
return hash;
}
VALUE Map_deep_copy(VALUE obj) {
Map* self = ruby_to_Map(obj);
VALUE new_arena_rb = Arena_new();
upb_arena *arena = Arena_get(new_arena_rb);
upb_map* new_map =
upb_map_new(arena, self->key_type, self->value_type_info.type);
size_t iter = UPB_MAP_BEGIN;
while (upb_mapiter_next(self->map, &iter)) {
upb_msgval key = upb_mapiter_key(self->map, iter);
upb_msgval val = upb_mapiter_value(self->map, iter);
upb_msgval val_copy = Msgval_DeepCopy(val, self->value_type_info, arena);
upb_map_set(new_map, key, val_copy, arena);
}
return Map_GetRubyWrapper(new_map, self->key_type, self->value_type_info,
new_arena_rb);
}
const upb_map* Map_GetUpbMap(VALUE val, const upb_fielddef *field) {
const upb_fielddef* key_field = map_field_key(field);
const upb_fielddef* value_field = map_field_value(field);
TypeInfo value_type_info = TypeInfo_get(value_field);
Map* self;
if (!RB_TYPE_P(val, T_DATA) || !RTYPEDDATA_P(val) ||
RTYPEDDATA_TYPE(val) != &Map_type) {
rb_raise(cTypeError, "Expected Map instance");
}
self = ruby_to_Map(val);
if (self->key_type != upb_fielddef_type(key_field)) {
rb_raise(cTypeError, "Map key type does not match field's key type");
}
if (self->value_type_info.type != value_type_info.type) {
rb_raise(cTypeError, "Map value type does not match field's value type");
}
if (self->value_type_info.def.msgdef != value_type_info.def.msgdef) {
rb_raise(cTypeError, "Map value type has wrong message/enum class");
}
return self->map;
}
void Map_Inspect(StringBuilder* b, const upb_map* map, upb_fieldtype_t key_type,
TypeInfo val_type) {
bool first = true;
TypeInfo key_type_info = {key_type};
StringBuilder_Printf(b, "{");
if (map) {
size_t iter = UPB_MAP_BEGIN;
while (upb_mapiter_next(map, &iter)) {
upb_msgval key = upb_mapiter_key(map, iter);
upb_msgval val = upb_mapiter_value(map, iter);
if (first) {
first = false;
} else {
StringBuilder_Printf(b, ", ");
}
StringBuilder_PrintMsgval(b, key, key_type_info);
StringBuilder_Printf(b, "=>");
StringBuilder_PrintMsgval(b, val, val_type);
}
}
StringBuilder_Printf(b, "}");
}
static int merge_into_self_callback(VALUE key, VALUE val, VALUE _self) {
Map* self = ruby_to_Map(_self);
upb_arena *arena = Arena_get(self->arena);
upb_msgval key_val = Convert_RubyToUpb(key, "", Map_keyinfo(self), arena);
upb_msgval val_val = Convert_RubyToUpb(val, "", self->value_type_info, arena);
upb_map_set(Map_GetMutable(_self), key_val, val_val, arena);
return ST_CONTINUE;
}
// Used only internally -- shared by #merge and #initialize.
static VALUE Map_merge_into_self(VALUE _self, VALUE hashmap) {
if (TYPE(hashmap) == T_HASH) {
rb_hash_foreach(hashmap, merge_into_self_callback, _self);
} else if (RB_TYPE_P(hashmap, T_DATA) && RTYPEDDATA_P(hashmap) &&
RTYPEDDATA_TYPE(hashmap) == &Map_type) {
Map* self = ruby_to_Map(_self);
Map* other = ruby_to_Map(hashmap);
upb_arena *arena = Arena_get(self->arena);
upb_msg *self_msg = Map_GetMutable(_self);
size_t iter = UPB_MAP_BEGIN;
upb_arena_fuse(arena, Arena_get(other->arena));
if (self->key_type != other->key_type ||
self->value_type_info.type != other->value_type_info.type ||
self->value_type_class != other->value_type_class) {
rb_raise(rb_eArgError, "Attempt to merge Map with mismatching types");
}
while (upb_mapiter_next(other->map, &iter)) {
upb_msgval key = upb_mapiter_key(other->map, iter);
upb_msgval val = upb_mapiter_value(other->map, iter);
upb_map_set(self_msg, key, val, arena);
}
} else {
rb_raise(rb_eArgError, "Unknown type merging into Map");
}
return _self;
}
/*
@ -224,9 +285,9 @@ static bool needs_typeclass(upb_fieldtype_t type) {
* references to underlying objects will be shared if the value type is a
* message type.
*/
VALUE Map_init(int argc, VALUE* argv, VALUE _self) {
static VALUE Map_init(int argc, VALUE* argv, VALUE _self) {
Map* self = ruby_to_Map(_self);
int init_value_arg;
VALUE init_arg;
// We take either two args (:key_type, :value_type), three args (:key_type,
// :value_type, "ValueMessageType"), or four args (the above plus an initial
@ -236,8 +297,9 @@ VALUE Map_init(int argc, VALUE* argv, VALUE _self) {
}
self->key_type = ruby_to_fieldtype(argv[0]);
self->value_type = ruby_to_fieldtype(argv[1]);
self->parse_frame = Qnil;
self->value_type_info =
TypeInfo_FromClass(argc, argv, 1, &self->value_type_class, &init_arg);
self->arena = Arena_new();
// Check that the key type is an allowed type.
switch (self->key_type) {
@ -254,21 +316,12 @@ VALUE Map_init(int argc, VALUE* argv, VALUE _self) {
rb_raise(rb_eArgError, "Invalid key type for map.");
}
init_value_arg = 2;
if (needs_typeclass(self->value_type) && argc > 2) {
self->value_type_class = argv[2];
validate_type_class(self->value_type, self->value_type_class);
init_value_arg = 3;
}
self->map = upb_map_new(Arena_get(self->arena), self->key_type,
self->value_type_info.type);
ObjectCache_Add(self->map, _self, Arena_get(self->arena));
// Table value type is always UINT64: this ensures enough space to store the
// native_slot value.
if (!upb_strtable_init(&self->table, UPB_CTYPE_UINT64)) {
rb_raise(rb_eRuntimeError, "Could not allocate table.");
}
if (argc > init_value_arg) {
Map_merge_into_self(_self, argv[init_value_arg]);
if (init_arg != Qnil) {
Map_merge_into_self(_self, init_arg);
}
return Qnil;
@ -282,22 +335,16 @@ VALUE Map_init(int argc, VALUE* argv, VALUE _self) {
* Note that Map also includes Enumerable; map thus acts like a normal Ruby
* sequence.
*/
VALUE Map_each(VALUE _self) {
static VALUE Map_each(VALUE _self) {
Map* self = ruby_to_Map(_self);
size_t iter = UPB_MAP_BEGIN;
upb_strtable_iter it;
for (upb_strtable_begin(&it, &self->table);
!upb_strtable_done(&it);
upb_strtable_next(&it)) {
VALUE key = table_key_to_ruby(self, upb_strtable_iter_key(&it));
upb_value v = upb_strtable_iter_value(&it);
void* mem = value_memory(&v);
VALUE value = native_slot_get(self->value_type,
self->value_type_class,
mem);
rb_yield_values(2, key, value);
while (upb_mapiter_next(self->map, &iter)) {
upb_msgval key = upb_mapiter_key(self->map, iter);
upb_msgval val = upb_mapiter_value(self->map, iter);
VALUE key_val = Convert_UpbToRuby(key, Map_keyinfo(self), self->arena);
VALUE val_val = Convert_UpbToRuby(val, self->value_type_info, self->arena);
rb_yield_values(2, key_val, val_val);
}
return Qnil;
@ -309,17 +356,15 @@ VALUE Map_each(VALUE _self) {
*
* Returns the list of keys contained in the map, in unspecified order.
*/
VALUE Map_keys(VALUE _self) {
static VALUE Map_keys(VALUE _self) {
Map* self = ruby_to_Map(_self);
size_t iter = UPB_MAP_BEGIN;
VALUE ret = rb_ary_new();
upb_strtable_iter it;
for (upb_strtable_begin(&it, &self->table);
!upb_strtable_done(&it);
upb_strtable_next(&it)) {
VALUE key = table_key_to_ruby(self, upb_strtable_iter_key(&it));
rb_ary_push(ret, key);
while (upb_mapiter_next(self->map, &iter)) {
upb_msgval key = upb_mapiter_key(self->map, iter);
VALUE key_val = Convert_UpbToRuby(key, Map_keyinfo(self), self->arena);
rb_ary_push(ret, key_val);
}
return ret;
@ -331,22 +376,15 @@ VALUE Map_keys(VALUE _self) {
*
* Returns the list of values contained in the map, in unspecified order.
*/
VALUE Map_values(VALUE _self) {
static VALUE Map_values(VALUE _self) {
Map* self = ruby_to_Map(_self);
size_t iter = UPB_MAP_BEGIN;
VALUE ret = rb_ary_new();
upb_strtable_iter it;
for (upb_strtable_begin(&it, &self->table);
!upb_strtable_done(&it);
upb_strtable_next(&it)) {
upb_value v = upb_strtable_iter_value(&it);
void* mem = value_memory(&v);
VALUE value = native_slot_get(self->value_type,
self->value_type_class,
mem);
rb_ary_push(ret, value);
while (upb_mapiter_next(self->map, &iter)) {
upb_msgval val = upb_mapiter_value(self->map, iter);
VALUE val_val = Convert_UpbToRuby(val, self->value_type_info, self->arena);
rb_ary_push(ret, val_val);
}
return ret;
@ -359,18 +397,13 @@ VALUE Map_values(VALUE _self) {
* Accesses the element at the given key. Throws an exception if the key type is
* incorrect. Returns nil when the key is not present in the map.
*/
VALUE Map_index(VALUE _self, VALUE key) {
static VALUE Map_index(VALUE _self, VALUE key) {
Map* self = ruby_to_Map(_self);
upb_msgval key_upb = Convert_RubyToUpb(key, "", Map_keyinfo(self), NULL);
upb_msgval val;
char keybuf[TABLE_KEY_BUF_LENGTH];
const char* keyval = NULL;
size_t length = 0;
upb_value v;
key = table_key(self, key, keybuf, &keyval, &length);
if (upb_strtable_lookup2(&self->table, keyval, length, &v)) {
void* mem = value_memory(&v);
return native_slot_get(self->value_type, self->value_type_class, mem);
if (upb_map_get(self->map, key_upb, &val)) {
return Convert_UpbToRuby(val, self->value_type_info, self->arena);
} else {
return Qnil;
}
@ -384,33 +417,15 @@ VALUE Map_index(VALUE _self, VALUE key) {
* Throws an exception if the key type is incorrect. Returns the new value that
* was just inserted.
*/
VALUE Map_index_set(VALUE _self, VALUE key, VALUE value) {
static VALUE Map_index_set(VALUE _self, VALUE key, VALUE val) {
Map* self = ruby_to_Map(_self);
char keybuf[TABLE_KEY_BUF_LENGTH];
const char* keyval = NULL;
size_t length = 0;
upb_value v;
void* mem;
key = table_key(self, key, keybuf, &keyval, &length);
upb_arena *arena = Arena_get(self->arena);
upb_msgval key_upb = Convert_RubyToUpb(key, "", Map_keyinfo(self), NULL);
upb_msgval val_upb = Convert_RubyToUpb(val, "", self->value_type_info, arena);
rb_check_frozen(_self);
upb_map_set(Map_GetMutable(_self), key_upb, val_upb, arena);
if (TYPE(value) == T_HASH) {
VALUE args[1] = { value };
value = rb_class_new_instance(1, args, self->value_type_class);
}
mem = value_memory(&v);
native_slot_set("", self->value_type, self->value_type_class, mem, value);
// Replace any existing value by issuing a 'remove' operation first.
upb_strtable_remove2(&self->table, keyval, length, NULL);
if (!upb_strtable_insert2(&self->table, keyval, length, v)) {
rb_raise(rb_eRuntimeError, "Could not insert into table");
}
// Ruby hashmap's :[]= method also returns the inserted value.
return value;
return val;
}
/*
@ -420,15 +435,11 @@ VALUE Map_index_set(VALUE _self, VALUE key, VALUE value) {
* Returns true if the given key is present in the map. Throws an exception if
* the key has the wrong type.
*/
VALUE Map_has_key(VALUE _self, VALUE key) {
static VALUE Map_has_key(VALUE _self, VALUE key) {
Map* self = ruby_to_Map(_self);
upb_msgval key_upb = Convert_RubyToUpb(key, "", Map_keyinfo(self), NULL);
char keybuf[TABLE_KEY_BUF_LENGTH];
const char* keyval = NULL;
size_t length = 0;
key = table_key(self, key, keybuf, &keyval, &length);
if (upb_strtable_lookup2(&self->table, keyval, length, NULL)) {
if (upb_map_get(self->map, key_upb, NULL)) {
return Qtrue;
} else {
return Qfalse;
@ -442,22 +453,25 @@ VALUE Map_has_key(VALUE _self, VALUE key) {
* Deletes the value at the given key, if any, returning either the old value or
* nil if none was present. Throws an exception if the key is of the wrong type.
*/
VALUE Map_delete(VALUE _self, VALUE key) {
static VALUE Map_delete(VALUE _self, VALUE key) {
Map* self = ruby_to_Map(_self);
char keybuf[TABLE_KEY_BUF_LENGTH];
const char* keyval = NULL;
size_t length = 0;
upb_value v;
key = table_key(self, key, keybuf, &keyval, &length);
upb_msgval key_upb = Convert_RubyToUpb(key, "", Map_keyinfo(self), NULL);
upb_msgval val_upb;
VALUE ret;
rb_check_frozen(_self);
if (upb_strtable_remove2(&self->table, keyval, length, &v)) {
void* mem = value_memory(&v);
return native_slot_get(self->value_type, self->value_type_class, mem);
// TODO(haberman): make upb_map_delete() also capable of returning the deleted
// value.
if (upb_map_get(self->map, key_upb, &val_upb)) {
ret = Convert_UpbToRuby(val_upb, self->value_type_info, self->arena);
} else {
return Qnil;
ret = Qnil;
}
upb_map_delete(Map_GetMutable(_self), key_upb);
return ret;
}
/*
@ -466,17 +480,8 @@ VALUE Map_delete(VALUE _self, VALUE key) {
*
* Removes all entries from the map.
*/
VALUE Map_clear(VALUE _self) {
Map* self = ruby_to_Map(_self);
rb_check_frozen(_self);
// Uninit and reinit the table -- this is faster than iterating and doing a
// delete-lookup on each key.
upb_strtable_uninit(&self->table);
if (!upb_strtable_init(&self->table, UPB_CTYPE_INT64)) {
rb_raise(rb_eRuntimeError, "Unable to re-initialize table");
}
static VALUE Map_clear(VALUE _self) {
upb_map_clear(Map_GetMutable(_self));
return Qnil;
}
@ -486,24 +491,9 @@ VALUE Map_clear(VALUE _self) {
*
* Returns the number of entries (key-value pairs) in the map.
*/
VALUE Map_length(VALUE _self) {
static VALUE Map_length(VALUE _self) {
Map* self = ruby_to_Map(_self);
return ULL2NUM(upb_strtable_count(&self->table));
}
VALUE Map_new_this_type(VALUE _self) {
Map* self = ruby_to_Map(_self);
VALUE new_map = Qnil;
VALUE key_type = fieldtype_to_ruby(self->key_type);
VALUE value_type = fieldtype_to_ruby(self->value_type);
if (self->value_type_class != Qnil) {
new_map = rb_funcall(CLASS_OF(_self), rb_intern("new"), 3,
key_type, value_type, self->value_type_class);
} else {
new_map = rb_funcall(CLASS_OF(_self), rb_intern("new"), 2,
key_type, value_type);
}
return new_map;
return ULL2NUM(upb_map_size(self->map));
}
/*
@ -513,54 +503,23 @@ VALUE Map_new_this_type(VALUE _self) {
* Duplicates this map with a shallow copy. References to all non-primitive
* element objects (e.g., submessages) are shared.
*/
VALUE Map_dup(VALUE _self) {
static VALUE Map_dup(VALUE _self) {
Map* self = ruby_to_Map(_self);
VALUE new_map = Map_new_this_type(_self);
Map* new_self = ruby_to_Map(new_map);
VALUE new_map_rb = Map_new_this_type(self);
Map* new_self = ruby_to_Map(new_map_rb);
size_t iter = UPB_MAP_BEGIN;
upb_arena *arena = Arena_get(new_self->arena);
upb_map *new_map = Map_GetMutable(new_map_rb);
upb_strtable_iter it;
for (upb_strtable_begin(&it, &self->table);
!upb_strtable_done(&it);
upb_strtable_next(&it)) {
upb_strview k = upb_strtable_iter_key(&it);
upb_value v = upb_strtable_iter_value(&it);
void* mem = value_memory(&v);
upb_value dup;
void* dup_mem = value_memory(&dup);
native_slot_dup(self->value_type, dup_mem, mem);
upb_arena_fuse(arena, Arena_get(self->arena));
if (!upb_strtable_insert2(&new_self->table, k.data, k.size, dup)) {
rb_raise(rb_eRuntimeError, "Error inserting value into new table");
}
while (upb_mapiter_next(self->map, &iter)) {
upb_msgval key = upb_mapiter_key(self->map, iter);
upb_msgval val = upb_mapiter_value(self->map, iter);
upb_map_set(new_map, key, val, arena);
}
return new_map;
}
// Used by Google::Protobuf.deep_copy but not exposed directly.
VALUE Map_deep_copy(VALUE _self) {
Map* self = ruby_to_Map(_self);
VALUE new_map = Map_new_this_type(_self);
Map* new_self = ruby_to_Map(new_map);
upb_strtable_iter it;
for (upb_strtable_begin(&it, &self->table);
!upb_strtable_done(&it);
upb_strtable_next(&it)) {
upb_strview k = upb_strtable_iter_key(&it);
upb_value v = upb_strtable_iter_value(&it);
void* mem = value_memory(&v);
upb_value dup;
void* dup_mem = value_memory(&dup);
native_slot_deep_copy(self->value_type, self->value_type_class, dup_mem,
mem);
if (!upb_strtable_insert2(&new_self->table, k.data, k.size, dup)) {
rb_raise(rb_eRuntimeError, "Error inserting value into new table");
}
}
return new_map;
return new_map_rb;
}
/*
@ -579,12 +538,11 @@ VALUE Map_deep_copy(VALUE _self) {
VALUE Map_eq(VALUE _self, VALUE _other) {
Map* self = ruby_to_Map(_self);
Map* other;
upb_strtable_iter it;
// Allow comparisons to Ruby hashmaps by converting to a temporary Map
// instance. Slow, but workable.
if (TYPE(_other) == T_HASH) {
VALUE other_map = Map_new_this_type(_self);
VALUE other_map = Map_new_this_type(self);
Map_merge_into_self(other_map, _other);
_other = other_map;
}
@ -595,33 +553,27 @@ VALUE Map_eq(VALUE _self, VALUE _other) {
return Qtrue;
}
if (self->key_type != other->key_type ||
self->value_type != other->value_type ||
self->value_type_info.type != other->value_type_info.type ||
self->value_type_class != other->value_type_class) {
return Qfalse;
}
if (upb_strtable_count(&self->table) != upb_strtable_count(&other->table)) {
if (upb_map_size(self->map) != upb_map_size(other->map)) {
return Qfalse;
}
// For each member of self, check that an equal member exists at the same key
// in other.
for (upb_strtable_begin(&it, &self->table);
!upb_strtable_done(&it);
upb_strtable_next(&it)) {
upb_strview k = upb_strtable_iter_key(&it);
upb_value v = upb_strtable_iter_value(&it);
void* mem = value_memory(&v);
upb_value other_v;
void* other_mem = value_memory(&other_v);
if (!upb_strtable_lookup2(&other->table, k.data, k.size, &other_v)) {
size_t iter = UPB_MAP_BEGIN;
while (upb_mapiter_next(self->map, &iter)) {
upb_msgval key = upb_mapiter_key(self->map, iter);
upb_msgval val = upb_mapiter_value(self->map, iter);
upb_msgval other_val;
if (!upb_map_get(other->map, key, &other_val)) {
// Not present in other map.
return Qfalse;
}
if (!native_slot_eq(self->value_type, self->value_type_class, mem,
other_mem)) {
// Present, but value not equal.
if (!Msgval_IsEqual(val, other_val, self->value_type_info)) {
// Present but different value.
return Qfalse;
}
}
@ -629,6 +581,21 @@ VALUE Map_eq(VALUE _self, VALUE _other) {
return Qtrue;
}
/*
* call-seq:
* Message.freeze => self
*
* Freezes the message object. We have to intercept this so we can pin the
* Ruby object into memory so we don't forget it's frozen.
*/
static VALUE Map_freeze(VALUE _self) {
Map* self = ruby_to_Map(_self);
ObjectCache_Pin(self->map, _self, Arena_get(self->arena));
RB_OBJ_FREEZE(_self);
return _self;
}
/*
* call-seq:
* Map.hash => hash_value
@ -637,26 +604,18 @@ VALUE Map_eq(VALUE _self, VALUE _other) {
*/
VALUE Map_hash(VALUE _self) {
Map* self = ruby_to_Map(_self);
uint64_t hash = 0;
st_index_t h = rb_hash_start(0);
VALUE hash_sym = rb_intern("hash");
upb_strtable_iter it;
for (upb_strtable_begin(&it, &self->table); !upb_strtable_done(&it);
upb_strtable_next(&it)) {
VALUE key = table_key_to_ruby(self, upb_strtable_iter_key(&it));
upb_value v = upb_strtable_iter_value(&it);
void* mem = value_memory(&v);
VALUE value = native_slot_get(self->value_type,
self->value_type_class,
mem);
h = rb_hash_uint(h, NUM2LONG(rb_funcall(key, hash_sym, 0)));
h = rb_hash_uint(h, NUM2LONG(rb_funcall(value, hash_sym, 0)));
size_t iter = UPB_MAP_BEGIN;
TypeInfo key_info = {self->key_type};
while (upb_mapiter_next(self->map, &iter)) {
upb_msgval key = upb_mapiter_key(self->map, iter);
upb_msgval val = upb_mapiter_value(self->map, iter);
hash = Msgval_GetHash(key, key_info, hash);
hash = Msgval_GetHash(val, self->value_type_info, hash);
}
return INT2FIX(h);
return LL2NUM(hash);
}
/*
@ -667,24 +626,7 @@ VALUE Map_hash(VALUE _self) {
*/
VALUE Map_to_h(VALUE _self) {
Map* self = ruby_to_Map(_self);
VALUE hash = rb_hash_new();
upb_strtable_iter it;
for (upb_strtable_begin(&it, &self->table);
!upb_strtable_done(&it);
upb_strtable_next(&it)) {
VALUE key = table_key_to_ruby(self, upb_strtable_iter_key(&it));
upb_value v = upb_strtable_iter_value(&it);
void* mem = value_memory(&v);
VALUE value = native_slot_get(self->value_type,
self->value_type_class,
mem);
if (self->value_type == UPB_TYPE_MESSAGE) {
value = Message_to_h(value);
}
rb_hash_aset(hash, key, value);
}
return hash;
return Map_CreateHash(self->map, self->key_type, self->value_type_info);
}
/*
@ -698,34 +640,11 @@ VALUE Map_to_h(VALUE _self) {
VALUE Map_inspect(VALUE _self) {
Map* self = ruby_to_Map(_self);
VALUE str = rb_str_new2("{");
bool first = true;
VALUE inspect_sym = rb_intern("inspect");
upb_strtable_iter it;
for (upb_strtable_begin(&it, &self->table); !upb_strtable_done(&it);
upb_strtable_next(&it)) {
VALUE key = table_key_to_ruby(self, upb_strtable_iter_key(&it));
upb_value v = upb_strtable_iter_value(&it);
void* mem = value_memory(&v);
VALUE value = native_slot_get(self->value_type,
self->value_type_class,
mem);
if (!first) {
str = rb_str_cat2(str, ", ");
} else {
first = false;
}
str = rb_str_append(str, rb_funcall(key, inspect_sym, 0));
str = rb_str_cat2(str, "=>");
str = rb_str_append(str, rb_funcall(value, inspect_sym, 0));
}
str = rb_str_cat2(str, "}");
return str;
StringBuilder* builder = StringBuilder_New();
Map_Inspect(builder, self->map, self->key_type, self->value_type_info);
VALUE ret = StringBuilder_ToRubyString(builder);
StringBuilder_Free(builder);
return ret;
}
/*
@ -737,79 +656,11 @@ VALUE Map_inspect(VALUE _self) {
* in the new copy of this map. Returns the new copy of this map with merged
* contents.
*/
VALUE Map_merge(VALUE _self, VALUE hashmap) {
static VALUE Map_merge(VALUE _self, VALUE hashmap) {
VALUE dupped = Map_dup(_self);
return Map_merge_into_self(dupped, hashmap);
}
static int merge_into_self_callback(VALUE key, VALUE value, VALUE self) {
Map_index_set(self, key, value);
return ST_CONTINUE;
}
// Used only internally -- shared by #merge and #initialize.
VALUE Map_merge_into_self(VALUE _self, VALUE hashmap) {
if (TYPE(hashmap) == T_HASH) {
rb_hash_foreach(hashmap, merge_into_self_callback, _self);
} else if (RB_TYPE_P(hashmap, T_DATA) && RTYPEDDATA_P(hashmap) &&
RTYPEDDATA_TYPE(hashmap) == &Map_type) {
Map* self = ruby_to_Map(_self);
Map* other = ruby_to_Map(hashmap);
upb_strtable_iter it;
if (self->key_type != other->key_type ||
self->value_type != other->value_type ||
self->value_type_class != other->value_type_class) {
rb_raise(rb_eArgError, "Attempt to merge Map with mismatching types");
}
for (upb_strtable_begin(&it, &other->table);
!upb_strtable_done(&it);
upb_strtable_next(&it)) {
upb_strview k = upb_strtable_iter_key(&it);
// Replace any existing value by issuing a 'remove' operation first.
upb_value v;
upb_value oldv;
upb_strtable_remove2(&self->table, k.data, k.size, &oldv);
v = upb_strtable_iter_value(&it);
upb_strtable_insert2(&self->table, k.data, k.size, v);
}
} else {
rb_raise(rb_eArgError, "Unknown type merging into Map");
}
return _self;
}
// Internal method: map iterator initialization (used for serialization).
void Map_begin(VALUE _self, Map_iter* iter) {
Map* self = ruby_to_Map(_self);
iter->self = self;
upb_strtable_begin(&iter->it, &self->table);
}
void Map_next(Map_iter* iter) {
upb_strtable_next(&iter->it);
}
bool Map_done(Map_iter* iter) {
return upb_strtable_done(&iter->it);
}
VALUE Map_iter_key(Map_iter* iter) {
return table_key_to_ruby(iter->self, upb_strtable_iter_key(&iter->it));
}
VALUE Map_iter_value(Map_iter* iter) {
upb_value v = upb_strtable_iter_value(&iter->it);
void* mem = value_memory(&v);
return native_slot_get(iter->self->value_type,
iter->self->value_type_class,
mem);
}
void Map_register(VALUE module) {
VALUE klass = rb_define_class_under(module, "Map", rb_cObject);
rb_define_alloc_func(klass, Map_alloc);
@ -828,6 +679,7 @@ void Map_register(VALUE module) {
rb_define_method(klass, "length", Map_length, 0);
rb_define_method(klass, "dup", Map_dup, 0);
rb_define_method(klass, "==", Map_eq, 1);
rb_define_method(klass, "freeze", Map_freeze, 0);
rb_define_method(klass, "hash", Map_hash, 0);
rb_define_method(klass, "to_h", Map_to_h, 0);
rb_define_method(klass, "inspect", Map_inspect, 0);

66
ruby/ext/google/protobuf_c/map.h Normal file
View File

@ -0,0 +1,66 @@
// Protocol Buffers - Google's data interchange format
// Copyright 2008 Google Inc. All rights reserved.
// https://developers.google.com/protocol-buffers/
//
// Redistribution and use in source and binary forms, with or without
// modification, are permitted provided that the following conditions are
// met:
//
// * Redistributions of source code must retain the above copyright
// notice, this list of conditions and the following disclaimer.
// * Redistributions in binary form must reproduce the above
// copyright notice, this list of conditions and the following disclaimer
// in the documentation and/or other materials provided with the
// distribution.
// * Neither the name of Google Inc. nor the names of its
// contributors may be used to endorse or promote products derived from
// this software without specific prior written permission.
//
// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#ifndef RUBY_PROTOBUF_MAP_H_
#define RUBY_PROTOBUF_MAP_H_
#include <ruby/ruby.h>
#include "protobuf.h"
#include "ruby-upb.h"
// Returns a Ruby wrapper object for the given map, which will be created if
// one does not exist already.
VALUE Map_GetRubyWrapper(upb_map *map, upb_fieldtype_t key_type,
TypeInfo value_type, VALUE arena);
// Gets the underlying upb_map for this Ruby map object, which must have
// key/value type that match |field|. If this is not a map or the type doesn't
// match, raises an exception.
const upb_map *Map_GetUpbMap(VALUE val, const upb_fielddef *field);
// Implements #inspect for this map by appending its contents to |b|.
void Map_Inspect(StringBuilder *b, const upb_map *map, upb_fieldtype_t key_type,
TypeInfo val_type);
// Returns a new Hash object containing the contents of this Map.
VALUE Map_CreateHash(const upb_map* map, upb_fieldtype_t key_type,
TypeInfo val_info);
// Returns a deep copy of this Map object.
VALUE Map_deep_copy(VALUE obj);
// Ruby class of Google::Protobuf::Map.
extern VALUE cMap;
// Call at startup to register all types in this module.
void Map_register(VALUE module);
#endif // RUBY_PROTOBUF_MAP_H_

1376
ruby/ext/google/protobuf_c/message.c
View File

File diff suppressed because it is too large Load Diff

98
ruby/ext/google/protobuf_c/message.h Normal file
View File

@ -0,0 +1,98 @@
// Protocol Buffers - Google's data interchange format
// Copyright 2008 Google Inc. All rights reserved.
// https://developers.google.com/protocol-buffers/
//
// Redistribution and use in source and binary forms, with or without
// modification, are permitted provided that the following conditions are
// met:
//
// * Redistributions of source code must retain the above copyright
// notice, this list of conditions and the following disclaimer.
// * Redistributions in binary form must reproduce the above
// copyright notice, this list of conditions and the following disclaimer
// in the documentation and/or other materials provided with the
// distribution.
// * Neither the name of Google Inc. nor the names of its
// contributors may be used to endorse or promote products derived from
// this software without specific prior written permission.
//
// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#ifndef RUBY_PROTOBUF_MESSAGE_H_
#define RUBY_PROTOBUF_MESSAGE_H_
#include <ruby/ruby.h>
#include "protobuf.h"
#include "ruby-upb.h"
// Gets the underlying upb_msg* and upb_msgdef for the given Ruby message
// wrapper. Requires that |value| is indeed a message object.
const upb_msg *Message_Get(VALUE value, const upb_msgdef **m);
// Like Message_Get(), but checks that the object is not frozen and returns a
// mutable pointer.
upb_msg *Message_GetMutable(VALUE value, const upb_msgdef **m);
// Returns the Arena object for this message.
VALUE Message_GetArena(VALUE value);
// Converts |value| into a upb_msg value of the expected upb_msgdef type,
// raising an error if this is not possible. Used when assigning |value| to a
// field of another message, which means the message must be of a particular
// type.
//
// This will perform automatic conversions in some cases (for example, Time ->
// Google::Protobuf::Timestamp). If any new message is created, it will be
// created on |arena|, and any existing message will have its arena fused with
// |arena|.
const upb_msg* Message_GetUpbMessage(VALUE value, const upb_msgdef* m,
const char* name, upb_arena* arena);
// Gets or constructs a Ruby wrapper object for the given message. The wrapper
// object will reference |arena| and ensure that it outlives this object.
VALUE Message_GetRubyWrapper(upb_msg* msg, const upb_msgdef* m, VALUE arena);
// Implements #inspect for this message, printing the text to |b|.
void Message_PrintMessage(StringBuilder* b, const upb_msg* msg,
const upb_msgdef* m);
// Returns a hash value for the given message.
uint64_t Message_Hash(const upb_msg *msg, const upb_msgdef *m, uint64_t seed);
// Returns a deep copy of the given message.
upb_msg* Message_deep_copy(const upb_msg* msg, const upb_msgdef* m,
upb_arena *arena);
// Returns true if these two messages are equal.
bool Message_Equal(const upb_msg *m1, const upb_msg *m2, const upb_msgdef *m);
// Checks that this Ruby object is a message, and raises an exception if not.
void Message_CheckClass(VALUE klass);
// Returns a new Hash object containing the contents of this message.
VALUE Scalar_CreateHash(upb_msgval val, TypeInfo type_info);
// Creates a message class or enum module for this descriptor, respectively.
VALUE build_class_from_descriptor(VALUE descriptor);
VALUE build_module_from_enumdesc(VALUE _enumdesc);
// Returns the Descriptor/EnumDescriptor for the given message class or enum
// module, respectively. Returns nil if this is not a message class or enum
// module.
VALUE MessageOrEnum_GetDescriptor(VALUE klass);
// Call at startup to register all types in this module.
void Message_register(VALUE protobuf);
#endif // RUBY_PROTOBUF_MESSAGE_H_

392
ruby/ext/google/protobuf_c/protobuf.c
View File

@ -30,62 +30,342 @@
#include "protobuf.h"
#include <ruby/version.h>
#include "defs.h"
#include "map.h"
#include "message.h"
#include "repeated_field.h"
VALUE cError;
VALUE cParseError;
VALUE cTypeError;
VALUE c_only_cookie = Qnil;
static VALUE cached_empty_string = Qnil;
static VALUE cached_empty_bytes = Qnil;
static VALUE create_frozen_string(const char* str, size_t size, bool binary) {
VALUE str_rb = rb_str_new(str, size);
rb_enc_associate(str_rb,
binary ? kRubyString8bitEncoding : kRubyStringUtf8Encoding);
rb_obj_freeze(str_rb);
return str_rb;
const upb_fielddef* map_field_key(const upb_fielddef* field) {
const upb_msgdef *entry = upb_fielddef_msgsubdef(field);
return upb_msgdef_itof(entry, 1);
}
VALUE get_frozen_string(const char* str, size_t size, bool binary) {
if (size == 0) {
return binary ? cached_empty_bytes : cached_empty_string;
const upb_fielddef* map_field_value(const upb_fielddef* field) {
const upb_msgdef *entry = upb_fielddef_msgsubdef(field);
return upb_msgdef_itof(entry, 2);
}
// -----------------------------------------------------------------------------
// StringBuilder, for inspect
// -----------------------------------------------------------------------------
struct StringBuilder {
size_t size;
size_t cap;
char *data;
};
typedef struct StringBuilder StringBuilder;
static size_t StringBuilder_SizeOf(size_t cap) {
return sizeof(StringBuilder) + cap;
}
StringBuilder* StringBuilder_New() {
const size_t cap = 128;
StringBuilder* builder = malloc(sizeof(*builder));
builder->size = 0;
builder->cap = cap;
builder->data = malloc(builder->cap);
return builder;
}
void StringBuilder_Free(StringBuilder* b) {
free(b->data);
free(b);
}
void StringBuilder_Printf(StringBuilder* b, const char *fmt, ...) {
size_t have = b->cap - b->size;
size_t n;
va_list args;
va_start(args, fmt);
n = vsnprintf(&b->data[b->size], have, fmt, args);
va_end(args);
if (have <= n) {
while (have <= n) {
b->cap *= 2;
have = b->cap - b->size;
}
b->data = realloc(b->data, StringBuilder_SizeOf(b->cap));
va_start(args, fmt);
n = vsnprintf(&b->data[b->size], have, fmt, args);
va_end(args);
PBRUBY_ASSERT(n < have);
}
b->size += n;
}
VALUE StringBuilder_ToRubyString(StringBuilder* b) {
VALUE ret = rb_str_new(b->data, b->size);
rb_enc_associate(ret, rb_utf8_encoding());
return ret;
}
static void StringBuilder_PrintEnum(StringBuilder* b, int32_t val,
const upb_enumdef* e) {
const char *name = upb_enumdef_iton(e, val);
if (name) {
StringBuilder_Printf(b, ":%s", name);
} else {
// It is harder to memoize non-empty strings. The obvious approach would be
// to use a Ruby hash keyed by string as memo table, but looking up in such a table
// requires constructing a string (the very thing we're trying to avoid).
//
// Since few fields have defaults, we will just optimize the empty string
// case for now.
return create_frozen_string(str, size, binary);
StringBuilder_Printf(b, "%" PRId32, val);
}
}
void StringBuilder_PrintMsgval(StringBuilder* b, upb_msgval val,
TypeInfo info) {
switch (info.type) {
case UPB_TYPE_BOOL:
StringBuilder_Printf(b, "%s", val.bool_val ? "true" : "false");
break;
case UPB_TYPE_FLOAT: {
VALUE str = rb_inspect(DBL2NUM(val.float_val));
StringBuilder_Printf(b, "%s", RSTRING_PTR(str));
break;
}
case UPB_TYPE_DOUBLE: {
VALUE str = rb_inspect(DBL2NUM(val.double_val));
StringBuilder_Printf(b, "%s", RSTRING_PTR(str));
break;
}
case UPB_TYPE_INT32:
StringBuilder_Printf(b, "%" PRId32, val.int32_val);
break;
case UPB_TYPE_UINT32:
StringBuilder_Printf(b, "%" PRIu32, val.uint32_val);
break;
case UPB_TYPE_INT64:
StringBuilder_Printf(b, "%" PRId64, val.int64_val);
break;
case UPB_TYPE_UINT64:
StringBuilder_Printf(b, "%" PRIu64, val.uint64_val);
break;
case UPB_TYPE_STRING:
StringBuilder_Printf(b, "\"%.*s\"", (int)val.str_val.size, val.str_val.data);
break;
case UPB_TYPE_BYTES:
StringBuilder_Printf(b, "\"%.*s\"", (int)val.str_val.size, val.str_val.data);
break;
case UPB_TYPE_ENUM:
StringBuilder_PrintEnum(b, val.int32_val, info.def.enumdef);
break;
case UPB_TYPE_MESSAGE:
Message_PrintMessage(b, val.msg_val, info.def.msgdef);
break;
}
}
// -----------------------------------------------------------------------------
// Utilities.
// Arena
// -----------------------------------------------------------------------------
// Raises a Ruby error if |status| is not OK, using its error message.
void check_upb_status(const upb_status* status, const char* msg) {
if (!upb_ok(status)) {
rb_raise(rb_eRuntimeError, "%s: %s\n", msg, upb_status_errmsg(status));
}
void Arena_free(void* data) { upb_arena_free(data); }
static VALUE cArena;
const rb_data_type_t Arena_type = {
"Google::Protobuf::Internal::Arena",
{ NULL, Arena_free, NULL },
};
static VALUE Arena_alloc(VALUE klass) {
upb_arena *arena = upb_arena_new();
return TypedData_Wrap_Struct(klass, &Arena_type, arena);
}
// String encodings: we look these up once, at load time, and then cache them
// here.
rb_encoding* kRubyStringUtf8Encoding;
rb_encoding* kRubyStringASCIIEncoding;
rb_encoding* kRubyString8bitEncoding;
upb_arena *Arena_get(VALUE _arena) {
upb_arena *arena;
TypedData_Get_Struct(_arena, upb_arena, &Arena_type, arena);
return arena;
}
// Ruby-interned string: "descriptor". We use this identifier to store an
// instance variable on message classes we create in order to link them back to
// their descriptors.
VALUE Arena_new() {
return Arena_alloc(cArena);
}
void Arena_register(VALUE module) {
VALUE internal = rb_define_module_under(module, "Internal");
VALUE klass = rb_define_class_under(internal, "Arena", rb_cObject);
rb_define_alloc_func(klass, Arena_alloc);
rb_gc_register_address(&cArena);
cArena = klass;
}
// -----------------------------------------------------------------------------
// Object Cache
// -----------------------------------------------------------------------------
// A pointer -> Ruby Object cache that keeps references to Ruby wrapper
// objects. This allows us to look up any Ruby wrapper object by the address
// of the object it is wrapping. That way we can avoid ever creating two
// different wrapper objects for the same C object, which saves memory and
// preserves object identity.
//
// We intern this once at module load time then use the interned identifier at
// runtime in order to avoid the cost of repeatedly interning in hot paths.
const char* kDescriptorInstanceVar = "descriptor";
ID descriptor_instancevar_interned;
// We use Hash and/or WeakMap for the cache. WeakMap is faster overall
// (probably due to removal being integrated with GC) but doesn't work for Ruby
// <2.7 (see note below). We need Hash for Ruby <2.7 and for cases where we
// need to GC-root the object (notably when the object has been frozen).
#if RUBY_API_VERSION_CODE >= 20700
#define USE_WEAK_MAP 1
#else
#define USE_WEAK_MAP 0
#endif
static VALUE ObjectCache_GetKey(const void* key) {
char buf[sizeof(key)];
memcpy(&buf, &key, sizeof(key));
intptr_t key_int = (intptr_t)key;
PBRUBY_ASSERT((key_int & 3) == 0);
return LL2NUM(key_int >> 2);
}
// Strong object cache, uses regular Hash and GC-roots objects.
// - For Ruby <2.7, used for all objects.
// - For Ruby >=2.7, used only for frozen objects, so we preserve the "frozen"
// bit (since this information is not preserved at the upb level).
VALUE strong_obj_cache = Qnil;
static void StrongObjectCache_Init() {
rb_gc_register_address(&strong_obj_cache);
strong_obj_cache = rb_hash_new();
}
static void StrongObjectCache_Remove(void* key) {
VALUE key_rb = ObjectCache_GetKey(key);
PBRUBY_ASSERT(rb_hash_lookup(strong_obj_cache, key_rb) != Qnil);
rb_hash_delete(strong_obj_cache, key_rb);
}
static VALUE StrongObjectCache_Get(const void* key) {
VALUE key_rb = ObjectCache_GetKey(key);
return rb_hash_lookup(strong_obj_cache, key_rb);
}
static void StrongObjectCache_Add(const void* key, VALUE val,
upb_arena* arena) {
PBRUBY_ASSERT(StrongObjectCache_Get(key) == Qnil);
VALUE key_rb = ObjectCache_GetKey(key);
rb_hash_aset(strong_obj_cache, key_rb, val);
upb_arena_addcleanup(arena, (void*)key, StrongObjectCache_Remove);
}
// Weak object cache. This speeds up the test suite significantly, so we
// presume it speeds up real code also. However we can only use it in Ruby
// >=2.7 due to:
// https://bugs.ruby-lang.org/issues/16035
#if USE_WEAK_MAP
VALUE weak_obj_cache = Qnil;
static void WeakObjectCache_Init() {
rb_gc_register_address(&weak_obj_cache);
VALUE klass = rb_eval_string("ObjectSpace::WeakMap");
weak_obj_cache = rb_class_new_instance(0, NULL, klass);
}
static VALUE WeakObjectCache_Get(const void* key) {
VALUE key_rb = ObjectCache_GetKey(key);
VALUE ret = rb_funcall(weak_obj_cache, rb_intern("[]"), 1, key_rb);
return ret;
}
static void WeakObjectCache_Add(const void* key, VALUE val) {
PBRUBY_ASSERT(WeakObjectCache_Get(key) == Qnil);
VALUE key_rb = ObjectCache_GetKey(key);
rb_funcall(weak_obj_cache, rb_intern("[]="), 2, key_rb, val);
PBRUBY_ASSERT(WeakObjectCache_Get(key) == val);
}
#endif
// Public ObjectCache API.
static void ObjectCache_Init() {
StrongObjectCache_Init();
#if USE_WEAK_MAP
WeakObjectCache_Init();
#endif
}
void ObjectCache_Add(const void* key, VALUE val, upb_arena *arena) {
#if USE_WEAK_MAP
(void)arena;
WeakObjectCache_Add(key, val);
#else
StrongObjectCache_Add(key, val, arena);
#endif
}
// Returns the cached object for this key, if any. Otherwise returns Qnil.
VALUE ObjectCache_Get(const void* key) {
#if USE_WEAK_MAP
return WeakObjectCache_Get(key);
#else
return StrongObjectCache_Get(key);
#endif
}
void ObjectCache_Pin(const void* key, VALUE val, upb_arena *arena) {
#if USE_WEAK_MAP
PBRUBY_ASSERT(WeakObjectCache_Get(key) == val);
// This will GC-root the object, but we'll still use the weak map for
// actual lookup.
StrongObjectCache_Add(key, val, arena);
#else
// Value is already pinned, nothing to do.
#endif
}
/*
* call-seq:
* Google::Protobuf.discard_unknown(msg)
*
* Discard unknown fields in the given message object and recursively discard
* unknown fields in submessages.
*/
static VALUE Google_Protobuf_discard_unknown(VALUE self, VALUE msg_rb) {
const upb_msgdef *m;
upb_msg *msg = Message_GetMutable(msg_rb, &m);
if (!upb_msg_discardunknown(msg, m, 128)) {
rb_raise(rb_eRuntimeError, "Messages nested too deeply.");
}
return Qnil;
}
/*
* call-seq:
* Google::Protobuf.deep_copy(obj) => copy_of_obj
*
* Performs a deep copy of a RepeatedField instance, a Map instance, or a
* message object, recursively copying its members.
*/
VALUE Google_Protobuf_deep_copy(VALUE self, VALUE obj) {
VALUE klass = CLASS_OF(obj);
if (klass == cRepeatedField) {
return RepeatedField_deep_copy(obj);
} else if (klass == cMap) {
return Map_deep_copy(obj);
} else {
VALUE new_arena_rb = Arena_new();
upb_arena *new_arena = Arena_get(new_arena_rb);
const upb_msgdef *m;
const upb_msg *msg = Message_Get(obj, &m);
upb_msg* new_msg = Message_deep_copy(msg, m, new_arena);
return Message_GetRubyWrapper(new_msg, m, new_arena_rb);
}
}
// -----------------------------------------------------------------------------
// Initialization/entry point.
@ -93,44 +373,24 @@ ID descriptor_instancevar_interned;
// This must be named "Init_protobuf_c" because the Ruby module is named
// "protobuf_c" -- the VM looks for this symbol in our .so.
__attribute__ ((visibility ("default")))
void Init_protobuf_c() {
ObjectCache_Init();
VALUE google = rb_define_module("Google");
VALUE protobuf = rb_define_module_under(google, "Protobuf");
VALUE internal = rb_define_module_under(protobuf, "Internal");
descriptor_instancevar_interned = rb_intern(kDescriptorInstanceVar);
DescriptorPool_register(protobuf);
Descriptor_register(protobuf);
FileDescriptor_register(protobuf);
FieldDescriptor_register(protobuf);
OneofDescriptor_register(protobuf);
EnumDescriptor_register(protobuf);
MessageBuilderContext_register(internal);
OneofBuilderContext_register(internal);
EnumBuilderContext_register(internal);
FileBuilderContext_register(internal);
Builder_register(internal);
Arena_register(protobuf);
Defs_register(protobuf);
RepeatedField_register(protobuf);
Map_register(protobuf);
Message_register(protobuf);
cError = rb_const_get(protobuf, rb_intern("Error"));
cParseError = rb_const_get(protobuf, rb_intern("ParseError"));
cTypeError = rb_const_get(protobuf, rb_intern("TypeError"));
rb_define_singleton_method(protobuf, "discard_unknown",
Google_Protobuf_discard_unknown, 1);
rb_define_singleton_method(protobuf, "deep_copy",
Google_Protobuf_deep_copy, 1);
kRubyStringUtf8Encoding = rb_utf8_encoding();
kRubyStringASCIIEncoding = rb_usascii_encoding();
kRubyString8bitEncoding = rb_ascii8bit_encoding();
rb_gc_register_address(&c_only_cookie);
c_only_cookie = rb_class_new_instance(0, NULL, rb_cObject);
rb_gc_register_address(&cached_empty_string);
rb_gc_register_address(&cached_empty_bytes);
cached_empty_string = create_frozen_string("", 0, false);
cached_empty_bytes = create_frozen_string("", 0, true);
}

633
ruby/ext/google/protobuf_c/protobuf.h
View File

@ -35,631 +35,76 @@
#include <ruby/vm.h>
#include <ruby/encoding.h>
#include "upb.h"
// Forward decls.
struct DescriptorPool;
struct Descriptor;
struct FileDescriptor;
struct FieldDescriptor;
struct EnumDescriptor;
struct MessageLayout;
struct MessageField;
struct MessageHeader;
struct MessageBuilderContext;
struct EnumBuilderContext;
struct FileBuilderContext;
struct Builder;
typedef struct DescriptorPool DescriptorPool;
typedef struct Descriptor Descriptor;
typedef struct FileDescriptor FileDescriptor;
typedef struct FieldDescriptor FieldDescriptor;
typedef struct OneofDescriptor OneofDescriptor;
typedef struct EnumDescriptor EnumDescriptor;
typedef struct MessageLayout MessageLayout;
typedef struct MessageField MessageField;
typedef struct MessageOneof MessageOneof;
typedef struct MessageHeader MessageHeader;
typedef struct MessageBuilderContext MessageBuilderContext;
typedef struct OneofBuilderContext OneofBuilderContext;
typedef struct EnumBuilderContext EnumBuilderContext;
typedef struct FileBuilderContext FileBuilderContext;
typedef struct Builder Builder;
/*
It can be a bit confusing how the C structs defined below and the Ruby
objects interact and hold references to each other. First, a few principles:
- Ruby's "TypedData" abstraction lets a Ruby VALUE hold a pointer to a C
struct (or arbitrary memory chunk), own it, and free it when collected.
Thus, each struct below will have a corresponding Ruby object
wrapping/owning it.
- To get back from an underlying upb {msg,enum}def to the Ruby object, we
keep a global hashmap, accessed by get_def_obj/add_def_obj below.
The in-memory structure is then something like:
Ruby | upb
|
DescriptorPool ------------|-----------> upb_symtab____________________
| | (message types) \
| v \
Descriptor ---------------|-----------> upb_msgdef (enum types)|
|--> msgclass | | ^ |
| (dynamically built) | | | (submsg fields) |
|--> MessageLayout | | | /
|--------------------------|> decoder method| | /
\--------------------------|> serialize | | /
| handlers v | /
FieldDescriptor -----------|-----------> upb_fielddef /
| | /
| v (enum fields) /
EnumDescriptor ------------|-----------> upb_enumdef <----------'
|
|
^ | \___/
`---------------|-----------------' (get_def_obj map)
*/
// -----------------------------------------------------------------------------
// Ruby class structure definitions.
// -----------------------------------------------------------------------------
struct DescriptorPool {
VALUE def_to_descriptor; // Hash table of def* -> Ruby descriptor.
upb_symtab* symtab;
upb_handlercache* fill_handler_cache;
upb_handlercache* pb_serialize_handler_cache;
upb_handlercache* json_serialize_handler_cache;
upb_handlercache* json_serialize_handler_preserve_cache;
upb_pbcodecache* fill_method_cache;
upb_json_codecache* json_fill_method_cache;
};
struct Descriptor {
const upb_msgdef* msgdef;
MessageLayout* layout;
VALUE klass;
VALUE descriptor_pool;
};
struct FileDescriptor {
const upb_filedef* filedef;
VALUE descriptor_pool; // Owns the upb_filedef.
};
struct FieldDescriptor {
const upb_fielddef* fielddef;
VALUE descriptor_pool; // Owns the upb_fielddef.
};
struct OneofDescriptor {
const upb_oneofdef* oneofdef;
VALUE descriptor_pool; // Owns the upb_oneofdef.
};
struct EnumDescriptor {
const upb_enumdef* enumdef;
VALUE module; // begins as nil
VALUE descriptor_pool; // Owns the upb_enumdef.
};
struct MessageBuilderContext {
google_protobuf_DescriptorProto* msg_proto;
VALUE file_builder;
};
struct OneofBuilderContext {
int oneof_index;
VALUE message_builder;
};
struct EnumBuilderContext {
google_protobuf_EnumDescriptorProto* enum_proto;
VALUE file_builder;
};
struct FileBuilderContext {
upb_arena *arena;
google_protobuf_FileDescriptorProto* file_proto;
VALUE descriptor_pool;
};
struct Builder {
VALUE descriptor_pool;
VALUE default_file_builder;
};
extern VALUE cDescriptorPool;
extern VALUE cDescriptor;
extern VALUE cFileDescriptor;
extern VALUE cFieldDescriptor;
extern VALUE cEnumDescriptor;
extern VALUE cMessageBuilderContext;
extern VALUE cOneofBuilderContext;
extern VALUE cEnumBuilderContext;
extern VALUE cFileBuilderContext;
extern VALUE cBuilder;
extern VALUE cError;
extern VALUE cParseError;
extern VALUE cTypeError;
// We forward-declare all of the Ruby method implementations here because we
// sometimes call the methods directly across .c files, rather than going
// through Ruby's method dispatching (e.g. during message parse). It's cleaner
// to keep the list of object methods together than to split them between
// static-in-file definitions and header declarations.
void DescriptorPool_mark(void* _self);
void DescriptorPool_free(void* _self);
VALUE DescriptorPool_alloc(VALUE klass);
void DescriptorPool_register(VALUE module);
DescriptorPool* ruby_to_DescriptorPool(VALUE value);
VALUE DescriptorPool_build(int argc, VALUE* argv, VALUE _self);
VALUE DescriptorPool_lookup(VALUE _self, VALUE name);
VALUE DescriptorPool_generated_pool(VALUE _self);
extern VALUE generated_pool;
void Descriptor_mark(void* _self);
void Descriptor_free(void* _self);
VALUE Descriptor_alloc(VALUE klass);
void Descriptor_register(VALUE module);
Descriptor* ruby_to_Descriptor(VALUE value);
VALUE Descriptor_initialize(VALUE _self, VALUE cookie, VALUE descriptor_pool,
VALUE ptr);
VALUE Descriptor_name(VALUE _self);
VALUE Descriptor_each(VALUE _self);
VALUE Descriptor_lookup(VALUE _self, VALUE name);
VALUE Descriptor_each_oneof(VALUE _self);
VALUE Descriptor_lookup_oneof(VALUE _self, VALUE name);
VALUE Descriptor_msgclass(VALUE _self);
VALUE Descriptor_file_descriptor(VALUE _self);
extern const rb_data_type_t _Descriptor_type;
void FileDescriptor_mark(void* _self);
void FileDescriptor_free(void* _self);
VALUE FileDescriptor_alloc(VALUE klass);
void FileDescriptor_register(VALUE module);
FileDescriptor* ruby_to_FileDescriptor(VALUE value);
VALUE FileDescriptor_initialize(VALUE _self, VALUE cookie,
VALUE descriptor_pool, VALUE ptr);
VALUE FileDescriptor_name(VALUE _self);
VALUE FileDescriptor_syntax(VALUE _self);
void FieldDescriptor_mark(void* _self);
void FieldDescriptor_free(void* _self);
VALUE FieldDescriptor_alloc(VALUE klass);
void FieldDescriptor_register(VALUE module);
FieldDescriptor* ruby_to_FieldDescriptor(VALUE value);
VALUE FieldDescriptor_initialize(VALUE _self, VALUE cookie,
VALUE descriptor_pool, VALUE ptr);
VALUE FieldDescriptor_name(VALUE _self);
VALUE FieldDescriptor_type(VALUE _self);
VALUE FieldDescriptor_default(VALUE _self);
VALUE FieldDescriptor_label(VALUE _self);
VALUE FieldDescriptor_number(VALUE _self);
VALUE FieldDescriptor_submsg_name(VALUE _self);
VALUE FieldDescriptor_subtype(VALUE _self);
VALUE FieldDescriptor_has(VALUE _self, VALUE msg_rb);
VALUE FieldDescriptor_clear(VALUE _self, VALUE msg_rb);
VALUE FieldDescriptor_get(VALUE _self, VALUE msg_rb);
VALUE FieldDescriptor_set(VALUE _self, VALUE msg_rb, VALUE value);
upb_fieldtype_t ruby_to_fieldtype(VALUE type);
VALUE fieldtype_to_ruby(upb_fieldtype_t type);
void OneofDescriptor_mark(void* _self);
void OneofDescriptor_free(void* _self);
VALUE OneofDescriptor_alloc(VALUE klass);
void OneofDescriptor_register(VALUE module);
OneofDescriptor* ruby_to_OneofDescriptor(VALUE value);
VALUE OneofDescriptor_initialize(VALUE _self, VALUE cookie,
VALUE descriptor_pool, VALUE ptr);
VALUE OneofDescriptor_name(VALUE _self);
VALUE OneofDescriptor_each(VALUE _self);
void EnumDescriptor_mark(void* _self);
void EnumDescriptor_free(void* _self);
VALUE EnumDescriptor_alloc(VALUE klass);
VALUE EnumDescriptor_initialize(VALUE _self, VALUE cookie,
VALUE descriptor_pool, VALUE ptr);
void EnumDescriptor_register(VALUE module);
EnumDescriptor* ruby_to_EnumDescriptor(VALUE value);
VALUE EnumDescriptor_file_descriptor(VALUE _self);
VALUE EnumDescriptor_name(VALUE _self);
VALUE EnumDescriptor_lookup_name(VALUE _self, VALUE name);
VALUE EnumDescriptor_lookup_value(VALUE _self, VALUE number);
VALUE EnumDescriptor_each(VALUE _self);
VALUE EnumDescriptor_enummodule(VALUE _self);
extern const rb_data_type_t _EnumDescriptor_type;
void MessageBuilderContext_mark(void* _self);
void MessageBuilderContext_free(void* _self);
VALUE MessageBuilderContext_alloc(VALUE klass);
void MessageBuilderContext_register(VALUE module);
MessageBuilderContext* ruby_to_MessageBuilderContext(VALUE value);
VALUE MessageBuilderContext_initialize(VALUE _self,
VALUE _file_builder,
VALUE name);
VALUE MessageBuilderContext_optional(int argc, VALUE* argv, VALUE _self);
VALUE MessageBuilderContext_proto3_optional(int argc, VALUE* argv, VALUE _self);
VALUE MessageBuilderContext_required(int argc, VALUE* argv, VALUE _self);
VALUE MessageBuilderContext_repeated(int argc, VALUE* argv, VALUE _self);
VALUE MessageBuilderContext_map(int argc, VALUE* argv, VALUE _self);
VALUE MessageBuilderContext_oneof(VALUE _self, VALUE name);
void OneofBuilderContext_mark(void* _self);
void OneofBuilderContext_free(void* _self);
VALUE OneofBuilderContext_alloc(VALUE klass);
void OneofBuilderContext_register(VALUE module);
OneofBuilderContext* ruby_to_OneofBuilderContext(VALUE value);
VALUE OneofBuilderContext_initialize(VALUE _self,
VALUE descriptor,
VALUE builder);
VALUE OneofBuilderContext_optional(int argc, VALUE* argv, VALUE _self);
void EnumBuilderContext_mark(void* _self);
void EnumBuilderContext_free(void* _self);
VALUE EnumBuilderContext_alloc(VALUE klass);
void EnumBuilderContext_register(VALUE module);
EnumBuilderContext* ruby_to_EnumBuilderContext(VALUE value);
VALUE EnumBuilderContext_initialize(VALUE _self, VALUE _file_builder,
VALUE name);
VALUE EnumBuilderContext_value(VALUE _self, VALUE name, VALUE number);
void FileBuilderContext_mark(void* _self);
void FileBuilderContext_free(void* _self);
VALUE FileBuilderContext_alloc(VALUE klass);
void FileBuilderContext_register(VALUE module);
FileBuilderContext* ruby_to_FileBuilderContext(VALUE _self);
upb_strview FileBuilderContext_strdup(VALUE _self, VALUE rb_str);
upb_strview FileBuilderContext_strdup_name(VALUE _self, VALUE rb_str);
upb_strview FileBuilderContext_strdup_sym(VALUE _self, VALUE rb_sym);
VALUE FileBuilderContext_initialize(VALUE _self, VALUE descriptor_pool,
VALUE name, VALUE options);
VALUE FileBuilderContext_add_message(VALUE _self, VALUE name);
VALUE FileBuilderContext_add_enum(VALUE _self, VALUE name);
VALUE FileBuilderContext_pending_descriptors(VALUE _self);
void Builder_mark(void* _self);
void Builder_free(void* _self);
VALUE Builder_alloc(VALUE klass);
void Builder_register(VALUE module);
Builder* ruby_to_Builder(VALUE value);
VALUE Builder_build(VALUE _self);
VALUE Builder_initialize(VALUE _self, VALUE descriptor_pool);
VALUE Builder_add_file(int argc, VALUE *argv, VALUE _self);
VALUE Builder_add_message(VALUE _self, VALUE name);
VALUE Builder_add_enum(VALUE _self, VALUE name);
VALUE Builder_finalize_to_pool(VALUE _self, VALUE pool_rb);
// -----------------------------------------------------------------------------
// Native slot storage abstraction.
// -----------------------------------------------------------------------------
#define NATIVE_SLOT_MAX_SIZE sizeof(uint64_t)
size_t native_slot_size(upb_fieldtype_t type);
void native_slot_set(const char* name,
upb_fieldtype_t type,
VALUE type_class,
void* memory,
VALUE value);
// Atomically (with respect to Ruby VM calls) either update the value and set a
// oneof case, or do neither. If |case_memory| is null, then no case value is
// set.
void native_slot_set_value_and_case(const char* name,
upb_fieldtype_t type,
VALUE type_class,
void* memory,
VALUE value,
uint32_t* case_memory,
uint32_t case_number);
VALUE native_slot_get(upb_fieldtype_t type,
VALUE type_class,
const void* memory);
void native_slot_init(upb_fieldtype_t type, void* memory);
void native_slot_mark(upb_fieldtype_t type, void* memory);
void native_slot_dup(upb_fieldtype_t type, void* to, void* from);
void native_slot_deep_copy(upb_fieldtype_t type, VALUE type_class, void* to,
void* from);
bool native_slot_eq(upb_fieldtype_t type, VALUE type_class, void* mem1,
void* mem2);
VALUE native_slot_encode_and_freeze_string(upb_fieldtype_t type, VALUE value);
void native_slot_check_int_range_precision(const char* name, upb_fieldtype_t type, VALUE value);
uint32_t slot_read_oneof_case(MessageLayout* layout, const void* storage,
const upb_oneofdef* oneof);
bool is_value_field(const upb_fielddef* f);
extern rb_encoding* kRubyStringUtf8Encoding;
extern rb_encoding* kRubyStringASCIIEncoding;
extern rb_encoding* kRubyString8bitEncoding;
VALUE field_type_class(const MessageLayout* layout, const upb_fielddef* field);
#define MAP_KEY_FIELD 1
#define MAP_VALUE_FIELD 2
// Oneof case slot value to indicate that no oneof case is set. The value `0` is
// safe because field numbers are used as case identifiers, and no field can
// have a number of 0.
#define ONEOF_CASE_NONE 0
#include "ruby-upb.h"
#include "defs.h"
// These operate on a map field (i.e., a repeated field of submessages whose
// submessage type is a map-entry msgdef).
bool is_map_field(const upb_fielddef* field);
const upb_fielddef* map_field_key(const upb_fielddef* field);
const upb_fielddef* map_field_value(const upb_fielddef* field);
// These operate on a map-entry msgdef.
const upb_fielddef* map_entry_key(const upb_msgdef* msgdef);
const upb_fielddef* map_entry_value(const upb_msgdef* msgdef);
// -----------------------------------------------------------------------------
// Repeated field container type.
// Arena
// -----------------------------------------------------------------------------
typedef struct {
upb_fieldtype_t field_type;
VALUE field_type_class;
void* elements;
int size;
int capacity;
} RepeatedField;
// A Ruby object that wraps an underlying upb_arena. Any objects that are
// allocated from this arena should reference the Arena in rb_gc_mark(), to
// ensure that the object's underlying memory outlives any Ruby object that can
// reach it.
void RepeatedField_mark(void* self);
void RepeatedField_free(void* self);
VALUE RepeatedField_alloc(VALUE klass);
VALUE RepeatedField_init(int argc, VALUE* argv, VALUE self);
void RepeatedField_register(VALUE module);
extern const rb_data_type_t RepeatedField_type;
extern VALUE cRepeatedField;
RepeatedField* ruby_to_RepeatedField(VALUE value);
VALUE RepeatedField_new_this_type(VALUE _self);
VALUE RepeatedField_each(VALUE _self);
VALUE RepeatedField_index(int argc, VALUE* argv, VALUE _self);
void* RepeatedField_index_native(VALUE _self, int index);
int RepeatedField_size(VALUE _self);
VALUE RepeatedField_index_set(VALUE _self, VALUE _index, VALUE val);
void RepeatedField_reserve(RepeatedField* self, int new_size);
VALUE RepeatedField_push(VALUE _self, VALUE val);
void RepeatedField_push_native(VALUE _self, void* data);
VALUE RepeatedField_pop_one(VALUE _self);
VALUE RepeatedField_insert(int argc, VALUE* argv, VALUE _self);
VALUE RepeatedField_replace(VALUE _self, VALUE list);
VALUE RepeatedField_clear(VALUE _self);
VALUE RepeatedField_length(VALUE _self);
VALUE RepeatedField_dup(VALUE _self);
VALUE RepeatedField_deep_copy(VALUE _self);
VALUE RepeatedField_to_ary(VALUE _self);
VALUE RepeatedField_eq(VALUE _self, VALUE _other);
VALUE RepeatedField_hash(VALUE _self);
VALUE RepeatedField_inspect(VALUE _self);
VALUE RepeatedField_plus(VALUE _self, VALUE list);
// Defined in repeated_field.c; also used by Map.
void validate_type_class(upb_fieldtype_t type, VALUE klass);
VALUE Arena_new();
upb_arena *Arena_get(VALUE arena);
// -----------------------------------------------------------------------------
// Map container type.
// ObjectCache
// -----------------------------------------------------------------------------
typedef struct {
upb_fieldtype_t key_type;
upb_fieldtype_t value_type;
VALUE value_type_class;
VALUE parse_frame;
upb_strtable table;
} Map;
// Global object cache from upb array/map/message/symtab to wrapper object.
//
// This is a conceptually "weak" cache, in that it does not prevent "val" from
// being collected (though in Ruby <2.7 is it effectively strong, due to
// implementation limitations).
void Map_mark(void* self);
void Map_free(void* self);
VALUE Map_alloc(VALUE klass);
VALUE Map_init(int argc, VALUE* argv, VALUE self);
void Map_register(VALUE module);
VALUE Map_set_frame(VALUE self, VALUE val);
// Adds an entry to the cache. The "arena" parameter must give the arena that
// "key" was allocated from. In Ruby <2.7.0, it will be used to remove the key
// from the cache when the arena is destroyed.
void ObjectCache_Add(const void* key, VALUE val, upb_arena *arena);
extern const rb_data_type_t Map_type;
extern VALUE cMap;
// Returns the cached object for this key, if any. Otherwise returns Qnil.
VALUE ObjectCache_Get(const void* key);
Map* ruby_to_Map(VALUE value);
VALUE Map_new_this_type(VALUE _self);
VALUE Map_each(VALUE _self);
VALUE Map_keys(VALUE _self);
VALUE Map_values(VALUE _self);
VALUE Map_index(VALUE _self, VALUE key);
VALUE Map_index_set(VALUE _self, VALUE key, VALUE value);
VALUE Map_has_key(VALUE _self, VALUE key);
VALUE Map_delete(VALUE _self, VALUE key);
VALUE Map_clear(VALUE _self);
VALUE Map_length(VALUE _self);
VALUE Map_dup(VALUE _self);
VALUE Map_deep_copy(VALUE _self);
VALUE Map_eq(VALUE _self, VALUE _other);
VALUE Map_hash(VALUE _self);
VALUE Map_to_h(VALUE _self);
VALUE Map_inspect(VALUE _self);
VALUE Map_merge(VALUE _self, VALUE hashmap);
VALUE Map_merge_into_self(VALUE _self, VALUE hashmap);
typedef struct {
Map* self;
upb_strtable_iter it;
} Map_iter;
void Map_begin(VALUE _self, Map_iter* iter);
void Map_next(Map_iter* iter);
bool Map_done(Map_iter* iter);
VALUE Map_iter_key(Map_iter* iter);
VALUE Map_iter_value(Map_iter* iter);
// Pins the previously added object so it is GC-rooted. This turns the
// reference to "val" from weak to strong. We use this to guarantee that the
// "frozen" bit on the object will be remembered, even if the user drops their
// reference to this precise object.
//
// The "arena" parameter must give the arena that "key" was allocated from.
void ObjectCache_Pin(const void* key, VALUE val, upb_arena *arena);
// -----------------------------------------------------------------------------
// Message layout / storage.
// StringBuilder, for inspect
// -----------------------------------------------------------------------------
#define MESSAGE_FIELD_NO_HASBIT ((uint32_t)-1)
struct StringBuilder;
typedef struct StringBuilder StringBuilder;
struct MessageField {
uint32_t offset;
uint32_t hasbit;
};
StringBuilder* StringBuilder_New();
void StringBuilder_Free(StringBuilder* b);
void StringBuilder_Printf(StringBuilder* b, const char *fmt, ...);
VALUE StringBuilder_ToRubyString(StringBuilder* b);
struct MessageOneof {
uint32_t offset;
uint32_t case_offset;
};
// MessageLayout is owned by the enclosing Descriptor, which must outlive us.
struct MessageLayout {
const Descriptor* desc;
const upb_msgdef* msgdef;
void* empty_template; // Can memcpy() onto a layout to clear it.
MessageField* fields;
MessageOneof* oneofs;
uint32_t size;
uint32_t value_offset;
int value_count;
int repeated_count;
int map_count;
};
#define ONEOF_CASE_MASK 0x80000000
void create_layout(Descriptor* desc);
void free_layout(MessageLayout* layout);
bool field_contains_hasbit(MessageLayout* layout,
const upb_fielddef* field);
VALUE layout_get_default(const upb_fielddef* field);
VALUE layout_get(MessageLayout* layout,
const void* storage,
const upb_fielddef* field);
void layout_set(MessageLayout* layout,
void* storage,
const upb_fielddef* field,
VALUE val);
VALUE layout_has(MessageLayout* layout,
const void* storage,
const upb_fielddef* field);
void layout_clear(MessageLayout* layout,
const void* storage,
const upb_fielddef* field);
void layout_init(MessageLayout* layout, void* storage);
void layout_mark(MessageLayout* layout, void* storage);
void layout_dup(MessageLayout* layout, void* to, void* from);
void layout_deep_copy(MessageLayout* layout, void* to, void* from);
VALUE layout_eq(MessageLayout* layout, void* msg1, void* msg2);
VALUE layout_hash(MessageLayout* layout, void* storage);
VALUE layout_inspect(MessageLayout* layout, void* storage);
bool is_wrapper_type_field(const upb_fielddef* field);
VALUE ruby_wrapper_type(VALUE type_class, VALUE value);
// -----------------------------------------------------------------------------
// Message class creation.
// -----------------------------------------------------------------------------
// This should probably be factored into a common upb component.
typedef struct {
upb_byteshandler handler;
upb_bytessink sink;
char *ptr;
size_t len, size;
} stringsink;
void stringsink_uninit(stringsink *sink);
struct MessageHeader {
Descriptor* descriptor; // kept alive by self.class.descriptor reference.
stringsink* unknown_fields; // store unknown fields in decoding.
// Data comes after this.
};
extern rb_data_type_t Message_type;
VALUE build_class_from_descriptor(VALUE descriptor);
void* Message_data(void* msg);
void Message_mark(void* self);
void Message_free(void* self);
VALUE Message_alloc(VALUE klass);
VALUE Message_method_missing(int argc, VALUE* argv, VALUE _self);
VALUE Message_initialize(int argc, VALUE* argv, VALUE _self);
VALUE Message_dup(VALUE _self);
VALUE Message_deep_copy(VALUE _self);
VALUE Message_eq(VALUE _self, VALUE _other);
VALUE Message_hash(VALUE _self);
VALUE Message_inspect(VALUE _self);
VALUE Message_to_h(VALUE _self);
VALUE Message_index(VALUE _self, VALUE field_name);
VALUE Message_index_set(VALUE _self, VALUE field_name, VALUE value);
VALUE Message_descriptor(VALUE klass);
VALUE Message_decode(VALUE klass, VALUE data);
VALUE Message_encode(VALUE klass, VALUE msg_rb);
VALUE Message_decode_json(int argc, VALUE* argv, VALUE klass);
VALUE Message_encode_json(int argc, VALUE* argv, VALUE klass);
VALUE Google_Protobuf_discard_unknown(VALUE self, VALUE msg_rb);
VALUE Google_Protobuf_deep_copy(VALUE self, VALUE obj);
VALUE build_module_from_enumdesc(VALUE _enumdesc);
VALUE enum_lookup(VALUE self, VALUE number);
VALUE enum_resolve(VALUE self, VALUE sym);
VALUE enum_descriptor(VALUE self);
const upb_pbdecodermethod *new_fillmsg_decodermethod(
Descriptor* descriptor, const void *owner);
void add_handlers_for_message(const void *closure, upb_handlers *h);
// Maximum depth allowed during encoding, to avoid stack overflows due to
// cycles.
#define ENCODE_MAX_NESTING 63
// -----------------------------------------------------------------------------
// A cache of frozen string objects to use as field defaults.
// -----------------------------------------------------------------------------
VALUE get_frozen_string(const char* data, size_t size, bool binary);
// -----------------------------------------------------------------------------
// Global map from upb {msg,enum}defs to wrapper Descriptor/EnumDescriptor
// instances.
// -----------------------------------------------------------------------------
VALUE get_msgdef_obj(VALUE descriptor_pool, const upb_msgdef* def);
VALUE get_enumdef_obj(VALUE descriptor_pool, const upb_enumdef* def);
VALUE get_fielddef_obj(VALUE descriptor_pool, const upb_fielddef* def);
VALUE get_filedef_obj(VALUE descriptor_pool, const upb_filedef* def);
VALUE get_oneofdef_obj(VALUE descriptor_pool, const upb_oneofdef* def);
void StringBuilder_PrintMsgval(StringBuilder* b, upb_msgval val, TypeInfo info);
// -----------------------------------------------------------------------------
// Utilities.
// -----------------------------------------------------------------------------
void check_upb_status(const upb_status* status, const char* msg);
#define CHECK_UPB(code, msg) do { \
upb_status status = UPB_STATUS_INIT; \
code; \
check_upb_status(&status, msg); \
} while (0)
extern ID descriptor_instancevar_interned;
// A distinct object that is not accessible from Ruby. We use this as a
// constructor argument to enforce that certain objects cannot be created from
// Ruby.
extern VALUE c_only_cookie;
extern VALUE cTypeError;
#ifdef NDEBUG
#define UPB_ASSERT(expr) do {} while (false && (expr))
#define PBRUBY_ASSERT(expr) do {} while (false && (expr))
#else
#define UPB_ASSERT(expr) assert(expr)
#define PBRUBY_ASSERT(expr) assert(expr)
#endif
#define UPB_UNUSED(var) (void)var

618
ruby/ext/google/protobuf_c/repeated_field.c
View File

@ -28,49 +28,162 @@
// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#include "repeated_field.h"
#include "convert.h"
#include "defs.h"
#include "message.h"
#include "protobuf.h"
#include "third_party/wyhash/wyhash.h"
// -----------------------------------------------------------------------------
// Repeated field container type.
// -----------------------------------------------------------------------------
const rb_data_type_t RepeatedField_type = {
"Google::Protobuf::RepeatedField",
{ RepeatedField_mark, RepeatedField_free, NULL },
};
typedef struct {
const upb_array *array; // Can get as mutable when non-frozen.
TypeInfo type_info;
VALUE type_class; // To GC-root the msgdef/enumdef in type_info.
VALUE arena; // To GC-root the upb_array.
} RepeatedField;
VALUE cRepeatedField;
RepeatedField* ruby_to_RepeatedField(VALUE _self) {
static void RepeatedField_mark(void* _self) {
RepeatedField* self = (RepeatedField*)_self;
rb_gc_mark(self->type_class);
rb_gc_mark(self->arena);
}
const rb_data_type_t RepeatedField_type = {
"Google::Protobuf::RepeatedField",
{ RepeatedField_mark, RUBY_DEFAULT_FREE, NULL },
.flags = RUBY_TYPED_FREE_IMMEDIATELY,
};
static RepeatedField* ruby_to_RepeatedField(VALUE _self) {
RepeatedField* self;
TypedData_Get_Struct(_self, RepeatedField, &RepeatedField_type, self);
return self;
}
void* RepeatedField_memoryat(RepeatedField* self, int index, int element_size) {
return ((uint8_t *)self->elements) + index * element_size;
static upb_array *RepeatedField_GetMutable(VALUE _self) {
rb_check_frozen(_self);
return (upb_array*)ruby_to_RepeatedField(_self)->array;
}
VALUE RepeatedField_alloc(VALUE klass) {
RepeatedField* self = ALLOC(RepeatedField);
self->arena = Qnil;
self->type_class = Qnil;
self->array = NULL;
return TypedData_Wrap_Struct(klass, &RepeatedField_type, self);
}
VALUE RepeatedField_GetRubyWrapper(upb_array* array, TypeInfo type_info,
VALUE arena) {
PBRUBY_ASSERT(array);
VALUE val = ObjectCache_Get(array);
if (val == Qnil) {
val = RepeatedField_alloc(cRepeatedField);
RepeatedField* self;
ObjectCache_Add(array, val, Arena_get(arena));
TypedData_Get_Struct(val, RepeatedField, &RepeatedField_type, self);
self->array = array;
self->arena = arena;
self->type_info = type_info;
if (self->type_info.type == UPB_TYPE_MESSAGE) {
self->type_class = Descriptor_DefToClass(type_info.def.msgdef);
}
}
PBRUBY_ASSERT(ruby_to_RepeatedField(val)->type_info.type == type_info.type);
PBRUBY_ASSERT(ruby_to_RepeatedField(val)->type_info.def.msgdef ==
type_info.def.msgdef);
return val;
}
static VALUE RepeatedField_new_this_type(RepeatedField* from) {
VALUE arena_rb = Arena_new();
upb_array *array = upb_array_new(Arena_get(arena_rb), from->type_info.type);
VALUE ret = RepeatedField_GetRubyWrapper(array, from->type_info, arena_rb);
PBRUBY_ASSERT(ruby_to_RepeatedField(ret)->type_class == from->type_class);
return ret;
}
void RepeatedField_Inspect(StringBuilder* b, const upb_array* array,
TypeInfo info) {
bool first = true;
StringBuilder_Printf(b, "[");
size_t n = array ? upb_array_size(array) : 0;
for (size_t i = 0; i < n; i++) {
if (first) {
first = false;
} else {
StringBuilder_Printf(b, ", ");
}
StringBuilder_PrintMsgval(b, upb_array_get(array, i), info);
}
StringBuilder_Printf(b, "]");
}
VALUE RepeatedField_deep_copy(VALUE _self) {
RepeatedField* self = ruby_to_RepeatedField(_self);
VALUE new_rptfield = RepeatedField_new_this_type(self);
RepeatedField* new_self = ruby_to_RepeatedField(new_rptfield);
VALUE arena_rb = new_self->arena;
upb_array *new_array = RepeatedField_GetMutable(new_rptfield);
upb_arena *arena = Arena_get(arena_rb);
size_t elements = upb_array_size(self->array);
upb_array_resize(new_array, elements, arena);
size_t size = upb_array_size(self->array);
for (size_t i = 0; i < size; i++) {
upb_msgval msgval = upb_array_get(self->array, i);
upb_msgval copy = Msgval_DeepCopy(msgval, self->type_info, arena);
upb_array_set(new_array, i, copy);
}
return new_rptfield;
}
const upb_array* RepeatedField_GetUpbArray(VALUE val, const upb_fielddef *field) {
RepeatedField* self;
TypeInfo type_info = TypeInfo_get(field);
if (!RB_TYPE_P(val, T_DATA) || !RTYPEDDATA_P(val) ||
RTYPEDDATA_TYPE(val) != &RepeatedField_type) {
rb_raise(cTypeError, "Expected repeated field array");
}
self = ruby_to_RepeatedField(val);
if (self->type_info.type != type_info.type) {
rb_raise(cTypeError, "Repeated field array has wrong element type");
}
if (self->type_info.def.msgdef != type_info.def.msgdef) {
rb_raise(cTypeError, "Repeated field array has wrong message/enum class");
}
return self->array;
}
static int index_position(VALUE _index, RepeatedField* repeated_field) {
int index = NUM2INT(_index);
if (index < 0 && repeated_field->size > 0) {
index = repeated_field->size + index;
}
if (index < 0) index += upb_array_size(repeated_field->array);
return index;
}
VALUE RepeatedField_subarray(VALUE _self, long beg, long len) {
RepeatedField* self = ruby_to_RepeatedField(_self);
int element_size = native_slot_size(self->field_type);
upb_fieldtype_t field_type = self->field_type;
VALUE field_type_class = self->field_type_class;
size_t off = beg * element_size;
VALUE ary = rb_ary_new2(len);
int i;
static VALUE RepeatedField_subarray(RepeatedField* self, long beg, long len) {
size_t size = upb_array_size(self->array);
VALUE ary = rb_ary_new2(size);
long i;
for (i = beg; i < beg + len; i++, off += element_size) {
void* mem = ((uint8_t *)self->elements) + off;
VALUE elem = native_slot_get(field_type, field_type_class, mem);
for (i = beg; i < beg + len; i++) {
upb_msgval msgval = upb_array_get(self->array, i);
VALUE elem = Convert_UpbToRuby(msgval, self->type_info, self->arena);
rb_ary_push(ary, elem);
}
return ary;
@ -84,17 +197,14 @@ VALUE RepeatedField_subarray(VALUE _self, long beg, long len) {
* also includes Enumerable; combined with this method, the repeated field thus
* acts like an ordinary Ruby sequence.
*/
VALUE RepeatedField_each(VALUE _self) {
static VALUE RepeatedField_each(VALUE _self) {
RepeatedField* self = ruby_to_RepeatedField(_self);
upb_fieldtype_t field_type = self->field_type;
VALUE field_type_class = self->field_type_class;
int element_size = native_slot_size(field_type);
size_t off = 0;
int size = upb_array_size(self->array);
int i;
for (i = 0; i < self->size; i++, off += element_size) {
void* memory = (void *) (((uint8_t *)self->elements) + off);
VALUE val = native_slot_get(field_type, field_type_class, memory);
for (i = 0; i < size; i++) {
upb_msgval msgval = upb_array_get(self->array, i);
VALUE val = Convert_UpbToRuby(msgval, self->type_info, self->arena);
rb_yield(val);
}
return _self;
@ -107,11 +217,9 @@ VALUE RepeatedField_each(VALUE _self) {
*
* Accesses the element at the given index. Returns nil on out-of-bounds
*/
VALUE RepeatedField_index(int argc, VALUE* argv, VALUE _self) {
static VALUE RepeatedField_index(int argc, VALUE* argv, VALUE _self) {
RepeatedField* self = ruby_to_RepeatedField(_self);
int element_size = native_slot_size(self->field_type);
upb_fieldtype_t field_type = self->field_type;
VALUE field_type_class = self->field_type_class;
long size = upb_array_size(self->array);
VALUE arg = argv[0];
long beg, len;
@ -119,35 +227,36 @@ VALUE RepeatedField_index(int argc, VALUE* argv, VALUE _self) {
if (argc == 1){
if (FIXNUM_P(arg)) {
/* standard case */
void* memory;
upb_msgval msgval;
int index = index_position(argv[0], self);
if (index < 0 || index >= self->size) {
if (index < 0 || (size_t)index >= upb_array_size(self->array)) {
return Qnil;
}
memory = RepeatedField_memoryat(self, index, element_size);
return native_slot_get(field_type, field_type_class, memory);
}else{
msgval = upb_array_get(self->array, index);
return Convert_UpbToRuby(msgval, self->type_info, self->arena);
} else {
/* check if idx is Range */
switch (rb_range_beg_len(arg, &beg, &len, self->size, 0)) {
switch (rb_range_beg_len(arg, &beg, &len, size, 0)) {
case Qfalse:
break;
case Qnil:
return Qnil;
default:
return RepeatedField_subarray(_self, beg, len);
return RepeatedField_subarray(self, beg, len);
}
}
}
/* assume 2 arguments */
beg = NUM2LONG(argv[0]);
len = NUM2LONG(argv[1]);
if (beg < 0) {
beg += self->size;
beg += size;
}
if (beg >= self->size) {
if (beg >= size) {
return Qnil;
}
return RepeatedField_subarray(_self, beg, len);
return RepeatedField_subarray(self, beg, len);
}
/*
@ -157,128 +266,88 @@ VALUE RepeatedField_index(int argc, VALUE* argv, VALUE _self) {
* Sets the element at the given index. On out-of-bounds assignments, extends
* the array and fills the hole (if any) with default values.
*/
VALUE RepeatedField_index_set(VALUE _self, VALUE _index, VALUE val) {
static VALUE RepeatedField_index_set(VALUE _self, VALUE _index, VALUE val) {
RepeatedField* self = ruby_to_RepeatedField(_self);
upb_fieldtype_t field_type = self->field_type;
VALUE field_type_class = self->field_type_class;
int element_size = native_slot_size(field_type);
void* memory;
int size = upb_array_size(self->array);
upb_array *array = RepeatedField_GetMutable(_self);
upb_arena *arena = Arena_get(self->arena);
upb_msgval msgval = Convert_RubyToUpb(val, "", self->type_info, arena);
int index = index_position(_index, self);
if (index < 0 || index >= (INT_MAX - 1)) {
return Qnil;
}
if (index >= self->size) {
upb_fieldtype_t field_type = self->field_type;
int element_size = native_slot_size(field_type);
int i;
RepeatedField_reserve(self, index + 1);
for (i = self->size; i <= index; i++) {
void* elem = RepeatedField_memoryat(self, i, element_size);
native_slot_init(field_type, elem);
if (index >= size) {
upb_array_resize(array, index + 1, arena);
upb_msgval fill;
memset(&fill, 0, sizeof(fill));
for (int i = size; i < index; i++) {
// Fill default values.
// TODO(haberman): should this happen at the upb level?
upb_array_set(array, i, fill);
}
self->size = index + 1;
}
memory = RepeatedField_memoryat(self, index, element_size);
native_slot_set("", field_type, field_type_class, memory, val);
upb_array_set(array, index, msgval);
return Qnil;
}
static int kInitialSize = 8;
void RepeatedField_reserve(RepeatedField* self, int new_size) {
void* old_elems = self->elements;
int elem_size = native_slot_size(self->field_type);
if (new_size <= self->capacity) {
return;
}
if (self->capacity == 0) {
self->capacity = kInitialSize;
}
while (self->capacity < new_size) {
self->capacity *= 2;
}
self->elements = ALLOC_N(uint8_t, elem_size * self->capacity);
if (old_elems != NULL) {
memcpy(self->elements, old_elems, self->size * elem_size);
xfree(old_elems);
}
}
/*
* call-seq:
* RepeatedField.push(value)
* RepeatedField.push(value, ...)
*
* Adds a new element to the repeated field.
*/
VALUE RepeatedField_push(VALUE _self, VALUE val) {
static VALUE RepeatedField_push_vararg(int argc, VALUE* argv, VALUE _self) {
RepeatedField* self = ruby_to_RepeatedField(_self);
upb_fieldtype_t field_type = self->field_type;
int element_size = native_slot_size(field_type);
void* memory;
RepeatedField_reserve(self, self->size + 1);
memory = (void *) (((uint8_t *)self->elements) + self->size * element_size);
native_slot_set("", field_type, self->field_type_class, memory, val);
// native_slot_set may raise an error; bump size only after set.
self->size++;
return _self;
}
VALUE RepeatedField_push_vararg(VALUE _self, VALUE args) {
upb_arena *arena = Arena_get(self->arena);
upb_array *array = RepeatedField_GetMutable(_self);
int i;
for (i = 0; i < RARRAY_LEN(args); i++) {
RepeatedField_push(_self, rb_ary_entry(args, i));
for (i = 0; i < argc; i++) {
upb_msgval msgval = Convert_RubyToUpb(argv[i], "", self->type_info, arena);
upb_array_append(array, msgval, arena);
}
return _self;
}
// Used by parsing handlers.
void RepeatedField_push_native(VALUE _self, void* data) {
/*
* call-seq:
* RepeatedField.<<(value)
*
* Adds a new element to the repeated field.
*/
static VALUE RepeatedField_push(VALUE _self, VALUE val) {
RepeatedField* self = ruby_to_RepeatedField(_self);
upb_fieldtype_t field_type = self->field_type;
int element_size = native_slot_size(field_type);
void* memory;
upb_arena *arena = Arena_get(self->arena);
upb_array *array = RepeatedField_GetMutable(_self);
RepeatedField_reserve(self, self->size + 1);
memory = (void *) (((uint8_t *)self->elements) + self->size * element_size);
memcpy(memory, data, element_size);
self->size++;
}
upb_msgval msgval = Convert_RubyToUpb(val, "", self->type_info, arena);
upb_array_append(array, msgval, arena);
void* RepeatedField_index_native(VALUE _self, int index) {
RepeatedField* self = ruby_to_RepeatedField(_self);
upb_fieldtype_t field_type = self->field_type;
int element_size = native_slot_size(field_type);
return RepeatedField_memoryat(self, index, element_size);
}
int RepeatedField_size(VALUE _self) {
RepeatedField* self = ruby_to_RepeatedField(_self);
return self->size;
return _self;
}
/*
* Private ruby method, used by RepeatedField.pop
*/
VALUE RepeatedField_pop_one(VALUE _self) {
static VALUE RepeatedField_pop_one(VALUE _self) {
RepeatedField* self = ruby_to_RepeatedField(_self);
upb_fieldtype_t field_type = self->field_type;
VALUE field_type_class = self->field_type_class;
int element_size = native_slot_size(field_type);
int index;
void* memory;
size_t size = upb_array_size(self->array);
upb_array *array = RepeatedField_GetMutable(_self);
upb_msgval last;
VALUE ret;
if (self->size == 0) {
if (size == 0) {
return Qnil;
}
index = self->size - 1;
memory = RepeatedField_memoryat(self, index, element_size);
ret = native_slot_get(field_type, field_type_class, memory);
self->size--;
last = upb_array_get(self->array, size - 1);
ret = Convert_UpbToRuby(last, self->type_info, self->arena);
upb_array_resize(array, size - 1, Arena_get(self->arena));
return ret;
}
@ -288,15 +357,18 @@ VALUE RepeatedField_pop_one(VALUE _self) {
*
* Replaces the contents of the repeated field with the given list of elements.
*/
VALUE RepeatedField_replace(VALUE _self, VALUE list) {
static VALUE RepeatedField_replace(VALUE _self, VALUE list) {
RepeatedField* self = ruby_to_RepeatedField(_self);
upb_array *array = RepeatedField_GetMutable(_self);
int i;
Check_Type(list, T_ARRAY);
self->size = 0;
upb_array_resize(array, 0, Arena_get(self->arena));
for (i = 0; i < RARRAY_LEN(list); i++) {
RepeatedField_push(_self, rb_ary_entry(list, i));
}
return list;
}
@ -306,9 +378,10 @@ VALUE RepeatedField_replace(VALUE _self, VALUE list) {
*
* Clears (removes all elements from) this repeated field.
*/
VALUE RepeatedField_clear(VALUE _self) {
static VALUE RepeatedField_clear(VALUE _self) {
RepeatedField* self = ruby_to_RepeatedField(_self);
self->size = 0;
upb_array *array = RepeatedField_GetMutable(_self);
upb_array_resize(array, 0, Arena_get(self->arena));
return _self;
}
@ -318,23 +391,9 @@ VALUE RepeatedField_clear(VALUE _self) {
*
* Returns the length of this repeated field.
*/
VALUE RepeatedField_length(VALUE _self) {
static VALUE RepeatedField_length(VALUE _self) {
RepeatedField* self = ruby_to_RepeatedField(_self);
return INT2NUM(self->size);
}
VALUE RepeatedField_new_this_type(VALUE _self) {
RepeatedField* self = ruby_to_RepeatedField(_self);
VALUE new_rptfield = Qnil;
VALUE element_type = fieldtype_to_ruby(self->field_type);
if (self->field_type_class != Qnil) {
new_rptfield = rb_funcall(CLASS_OF(_self), rb_intern("new"), 2,
element_type, self->field_type_class);
} else {
new_rptfield = rb_funcall(CLASS_OF(_self), rb_intern("new"), 1,
element_type);
}
return new_rptfield;
return INT2NUM(upb_array_size(self->array));
}
/*
@ -344,42 +403,20 @@ VALUE RepeatedField_new_this_type(VALUE _self) {
* Duplicates this repeated field with a shallow copy. References to all
* non-primitive element objects (e.g., submessages) are shared.
*/
VALUE RepeatedField_dup(VALUE _self) {
static VALUE RepeatedField_dup(VALUE _self) {
RepeatedField* self = ruby_to_RepeatedField(_self);
VALUE new_rptfield = RepeatedField_new_this_type(_self);
VALUE new_rptfield = RepeatedField_new_this_type(self);
RepeatedField* new_rptfield_self = ruby_to_RepeatedField(new_rptfield);
upb_fieldtype_t field_type = self->field_type;
size_t elem_size = native_slot_size(field_type);
size_t off = 0;
upb_array *new_array = RepeatedField_GetMutable(new_rptfield);
upb_arena* arena = Arena_get(new_rptfield_self->arena);
int size = upb_array_size(self->array);
int i;
RepeatedField_reserve(new_rptfield_self, self->size);
for (i = 0; i < self->size; i++, off += elem_size) {
void* to_mem = (uint8_t *)new_rptfield_self->elements + off;
void* from_mem = (uint8_t *)self->elements + off;
native_slot_dup(field_type, to_mem, from_mem);
new_rptfield_self->size++;
}
upb_arena_fuse(arena, Arena_get(self->arena));
return new_rptfield;
}
// Internal only: used by Google::Protobuf.deep_copy.
VALUE RepeatedField_deep_copy(VALUE _self) {
RepeatedField* self = ruby_to_RepeatedField(_self);
VALUE new_rptfield = RepeatedField_new_this_type(_self);
RepeatedField* new_rptfield_self = ruby_to_RepeatedField(new_rptfield);
upb_fieldtype_t field_type = self->field_type;
size_t elem_size = native_slot_size(field_type);
size_t off = 0;
int i;
RepeatedField_reserve(new_rptfield_self, self->size);
for (i = 0; i < self->size; i++, off += elem_size) {
void* to_mem = (uint8_t *)new_rptfield_self->elements + off;
void* from_mem = (uint8_t *)self->elements + off;
native_slot_deep_copy(field_type, self->field_type_class, to_mem, from_mem);
new_rptfield_self->size++;
for (i = 0; i < size; i++) {
upb_msgval msgval = upb_array_get(self->array, i);
upb_array_append(new_array, msgval, arena);
}
return new_rptfield;
@ -394,17 +431,16 @@ VALUE RepeatedField_deep_copy(VALUE _self) {
*/
VALUE RepeatedField_to_ary(VALUE _self) {
RepeatedField* self = ruby_to_RepeatedField(_self);
upb_fieldtype_t field_type = self->field_type;
size_t elem_size = native_slot_size(field_type);
size_t off = 0;
VALUE ary = rb_ary_new2(self->size);
int size = upb_array_size(self->array);
VALUE ary = rb_ary_new2(size);
int i;
for (i = 0; i < self->size; i++, off += elem_size) {
void* mem = ((uint8_t *)self->elements) + off;
VALUE elem = native_slot_get(field_type, self->field_type_class, mem);
rb_ary_push(ary, elem);
for (i = 0; i < size; i++) {
upb_msgval msgval = upb_array_get(self->array, i);
VALUE val = Convert_UpbToRuby(msgval, self->type_info, self->arena);
rb_ary_push(ary, val);
}
return ary;
}
@ -436,28 +472,38 @@ VALUE RepeatedField_eq(VALUE _self, VALUE _other) {
self = ruby_to_RepeatedField(_self);
other = ruby_to_RepeatedField(_other);
if (self->field_type != other->field_type ||
self->field_type_class != other->field_type_class ||
self->size != other->size) {
size_t n = upb_array_size(self->array);
if (self->type_info.type != other->type_info.type ||
self->type_class != other->type_class ||
upb_array_size(other->array) != n) {
return Qfalse;
}
{
upb_fieldtype_t field_type = self->field_type;
size_t elem_size = native_slot_size(field_type);
size_t off = 0;
int i;
for (i = 0; i < self->size; i++, off += elem_size) {
void* self_mem = ((uint8_t *)self->elements) + off;
void* other_mem = ((uint8_t *)other->elements) + off;
if (!native_slot_eq(field_type, self->field_type_class, self_mem,
other_mem)) {
return Qfalse;
}
for (size_t i = 0; i < n; i++) {
upb_msgval val1 = upb_array_get(self->array, i);
upb_msgval val2 = upb_array_get(other->array, i);
if (!Msgval_IsEqual(val1, val2, self->type_info)) {
return Qfalse;
}
return Qtrue;
}
return Qtrue;
}
/*
* call-seq:
* RepeatedField.freeze => self
*
* Freezes the repeated field. We have to intercept this so we can pin the Ruby
* object into memory so we don't forget it's frozen.
*/
static VALUE RepeatedField_freeze(VALUE _self) {
RepeatedField* self = ruby_to_RepeatedField(_self);
ObjectCache_Pin(self->array, _self, Arena_get(self->arena));
RB_OBJ_FREEZE(_self);
return _self;
}
/*
@ -468,22 +514,15 @@ VALUE RepeatedField_eq(VALUE _self, VALUE _other) {
*/
VALUE RepeatedField_hash(VALUE _self) {
RepeatedField* self = ruby_to_RepeatedField(_self);
st_index_t h = rb_hash_start(0);
VALUE hash_sym = rb_intern("hash");
upb_fieldtype_t field_type = self->field_type;
VALUE field_type_class = self->field_type_class;
size_t elem_size = native_slot_size(field_type);
size_t off = 0;
int i;
uint64_t hash = 0;
size_t n = upb_array_size(self->array);
for (i = 0; i < self->size; i++, off += elem_size) {
void* mem = ((uint8_t *)self->elements) + off;
VALUE elem = native_slot_get(field_type, field_type_class, mem);
h = rb_hash_uint(h, NUM2LONG(rb_funcall(elem, hash_sym, 0)));
for (size_t i = 0; i < n; i++) {
upb_msgval val = upb_array_get(self->array, i);
hash = Msgval_GetHash(val, self->type_info, hash);
}
h = rb_hash_end(h);
return INT2FIX(h);
return LL2NUM(hash);
}
/*
@ -495,34 +534,39 @@ VALUE RepeatedField_hash(VALUE _self) {
* be either another repeated field or a Ruby array.
*/
VALUE RepeatedField_plus(VALUE _self, VALUE list) {
VALUE dupped = RepeatedField_dup(_self);
VALUE dupped_ = RepeatedField_dup(_self);
if (TYPE(list) == T_ARRAY) {
int i;
for (i = 0; i < RARRAY_LEN(list); i++) {
VALUE elem = rb_ary_entry(list, i);
RepeatedField_push(dupped, elem);
RepeatedField_push(dupped_, elem);
}
} else if (RB_TYPE_P(list, T_DATA) && RTYPEDDATA_P(list) &&
RTYPEDDATA_TYPE(list) == &RepeatedField_type) {
RepeatedField* self = ruby_to_RepeatedField(_self);
RepeatedField* list_rptfield = ruby_to_RepeatedField(list);
RepeatedField* dupped = ruby_to_RepeatedField(dupped_);
upb_array *dupped_array = RepeatedField_GetMutable(dupped_);
upb_arena* arena = Arena_get(dupped->arena);
int size = upb_array_size(list_rptfield->array);
int i;
if (self->field_type != list_rptfield->field_type ||
self->field_type_class != list_rptfield->field_type_class) {
if (self->type_info.type != list_rptfield->type_info.type ||
self->type_class != list_rptfield->type_class) {
rb_raise(rb_eArgError,
"Attempt to append RepeatedField with different element type.");
}
for (i = 0; i < list_rptfield->size; i++) {
void* mem = RepeatedField_index_native(list, i);
RepeatedField_push_native(dupped, mem);
for (i = 0; i < size; i++) {
upb_msgval msgval = upb_array_get(list_rptfield->array, i);
upb_array_append(dupped_array, msgval, arena);
}
} else {
rb_raise(rb_eArgError, "Unknown type appending to RepeatedField");
}
return dupped;
return dupped_;
}
/*
@ -541,93 +585,6 @@ VALUE RepeatedField_concat(VALUE _self, VALUE list) {
return _self;
}
void validate_type_class(upb_fieldtype_t type, VALUE klass) {
if (rb_ivar_get(klass, descriptor_instancevar_interned) == Qnil) {
rb_raise(rb_eArgError,
"Type class has no descriptor. Please pass a "
"class or enum as returned by the DescriptorPool.");
}
if (type == UPB_TYPE_MESSAGE) {
VALUE desc = rb_ivar_get(klass, descriptor_instancevar_interned);
if (!RB_TYPE_P(desc, T_DATA) || !RTYPEDDATA_P(desc) ||
RTYPEDDATA_TYPE(desc) != &_Descriptor_type) {
rb_raise(rb_eArgError, "Descriptor has an incorrect type.");
}
if (rb_get_alloc_func(klass) != &Message_alloc) {
rb_raise(rb_eArgError,
"Message class was not returned by the DescriptorPool.");
}
} else if (type == UPB_TYPE_ENUM) {
VALUE enumdesc = rb_ivar_get(klass, descriptor_instancevar_interned);
if (!RB_TYPE_P(enumdesc, T_DATA) || !RTYPEDDATA_P(enumdesc) ||
RTYPEDDATA_TYPE(enumdesc) != &_EnumDescriptor_type) {
rb_raise(rb_eArgError, "Descriptor has an incorrect type.");
}
}
}
void RepeatedField_init_args(int argc, VALUE* argv,
VALUE _self) {
RepeatedField* self = ruby_to_RepeatedField(_self);
VALUE ary = Qnil;
if (argc < 1) {
rb_raise(rb_eArgError, "Expected at least 1 argument.");
}
self->field_type = ruby_to_fieldtype(argv[0]);
if (self->field_type == UPB_TYPE_MESSAGE ||
self->field_type == UPB_TYPE_ENUM) {
if (argc < 2) {
rb_raise(rb_eArgError, "Expected at least 2 arguments for message/enum.");
}
self->field_type_class = argv[1];
if (argc > 2) {
ary = argv[2];
}
validate_type_class(self->field_type, self->field_type_class);
} else {
if (argc > 2) {
rb_raise(rb_eArgError, "Too many arguments: expected 1 or 2.");
}
if (argc > 1) {
ary = argv[1];
}
}
if (ary != Qnil) {
int i;
if (!RB_TYPE_P(ary, T_ARRAY)) {
rb_raise(rb_eArgError, "Expected array as initialize argument");
}
for (i = 0; i < RARRAY_LEN(ary); i++) {
RepeatedField_push(_self, rb_ary_entry(ary, i));
}
}
}
// Mark, free, alloc, init and class setup functions.
void RepeatedField_mark(void* _self) {
RepeatedField* self = (RepeatedField*)_self;
upb_fieldtype_t field_type = self->field_type;
int element_size = native_slot_size(field_type);
int i;
rb_gc_mark(self->field_type_class);
for (i = 0; i < self->size; i++) {
void* memory = (((uint8_t *)self->elements) + i * element_size);
native_slot_mark(self->field_type, memory);
}
}
void RepeatedField_free(void* _self) {
RepeatedField* self = (RepeatedField*)_self;
xfree(self->elements);
xfree(self);
}
/*
* call-seq:
* RepeatedField.new(type, type_class = nil, initial_elems = [])
@ -639,18 +596,30 @@ void RepeatedField_free(void* _self) {
* EnumDescriptor#enummodule, respectively. An initial list of elements may also
* be provided.
*/
VALUE RepeatedField_alloc(VALUE klass) {
RepeatedField* self = ALLOC(RepeatedField);
self->elements = NULL;
self->size = 0;
self->capacity = 0;
self->field_type = -1;
self->field_type_class = Qnil;
return TypedData_Wrap_Struct(klass, &RepeatedField_type, self);
}
VALUE RepeatedField_init(int argc, VALUE* argv, VALUE _self) {
RepeatedField* self = ruby_to_RepeatedField(_self);
upb_arena *arena;
VALUE ary = Qnil;
VALUE RepeatedField_init(int argc, VALUE* argv, VALUE self) {
RepeatedField_init_args(argc, argv, self);
self->arena = Arena_new();
arena = Arena_get(self->arena);
if (argc < 1) {
rb_raise(rb_eArgError, "Expected at least 1 argument.");
}
self->type_info = TypeInfo_FromClass(argc, argv, 0, &self->type_class, &ary);
self->array = upb_array_new(arena, self->type_info.type);
ObjectCache_Add(self->array, _self, arena);
if (ary != Qnil) {
if (!RB_TYPE_P(ary, T_ARRAY)) {
rb_raise(rb_eArgError, "Expected array as initialize argument");
}
for (int i = 0; i < RARRAY_LEN(ary); i++) {
RepeatedField_push(_self, rb_ary_entry(ary, i));
}
}
return Qnil;
}
@ -667,7 +636,7 @@ void RepeatedField_register(VALUE module) {
rb_define_method(klass, "[]", RepeatedField_index, -1);
rb_define_method(klass, "at", RepeatedField_index, -1);
rb_define_method(klass, "[]=", RepeatedField_index_set, 2);
rb_define_method(klass, "push", RepeatedField_push_vararg, -2);
rb_define_method(klass, "push", RepeatedField_push_vararg, -1);
rb_define_method(klass, "<<", RepeatedField_push, 1);
rb_define_private_method(klass, "pop_one", RepeatedField_pop_one, 0);
rb_define_method(klass, "replace", RepeatedField_replace, 1);
@ -679,6 +648,7 @@ void RepeatedField_register(VALUE module) {
rb_define_method(klass, "clone", RepeatedField_dup, 0);
rb_define_method(klass, "==", RepeatedField_eq, 1);
rb_define_method(klass, "to_ary", RepeatedField_to_ary, 0);
rb_define_method(klass, "freeze", RepeatedField_freeze, 0);
rb_define_method(klass, "hash", RepeatedField_hash, 0);
rb_define_method(klass, "+", RepeatedField_plus, 1);
rb_define_method(klass, "concat", RepeatedField_concat, 1);

62
ruby/ext/google/protobuf_c/repeated_field.h Normal file
View File

@ -0,0 +1,62 @@
// Protocol Buffers - Google's data interchange format
// Copyright 2008 Google Inc. All rights reserved.
// https://developers.google.com/protocol-buffers/
//
// Redistribution and use in source and binary forms, with or without
// modification, are permitted provided that the following conditions are
// met:
//
// * Redistributions of source code must retain the above copyright
// notice, this list of conditions and the following disclaimer.
// * Redistributions in binary form must reproduce the above
// copyright notice, this list of conditions and the following disclaimer
// in the documentation and/or other materials provided with the
// distribution.
// * Neither the name of Google Inc. nor the names of its
// contributors may be used to endorse or promote products derived from
// this software without specific prior written permission.
//
// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#ifndef RUBY_PROTOBUF_REPEATED_FIELD_H_
#define RUBY_PROTOBUF_REPEATED_FIELD_H_
#include <ruby/ruby.h>
#include "protobuf.h"
#include "ruby-upb.h"
// Returns a Ruby wrapper object for the given upb_array, which will be created
// if one does not exist already.
VALUE RepeatedField_GetRubyWrapper(upb_array* msg, TypeInfo type_info,
VALUE arena);
// Gets the underlying upb_array for this Ruby RepeatedField object, which must
// have a type that matches |f|. If this is not a repeated field or the type
// doesn't match, raises an exception.
const upb_array* RepeatedField_GetUpbArray(VALUE value, const upb_fielddef* f);
// Implements #inspect for this repeated field by appending its contents to |b|.
void RepeatedField_Inspect(StringBuilder* b, const upb_array* array,
TypeInfo info);
// Returns a deep copy of this RepeatedField object.
VALUE RepeatedField_deep_copy(VALUE obj);
// Ruby class of Google::Protobuf::RepeatedField.
extern VALUE cRepeatedField;
// Call at startup to register all types in this module.
void RepeatedField_register(VALUE module);
#endif // RUBY_PROTOBUF_REPEATED_FIELD_H_

8913
ruby/ext/google/protobuf_c/ruby-upb.c Executable file
View File

File diff suppressed because it is too large Load Diff

5043
ruby/ext/google/protobuf_c/upb.h → ruby/ext/google/protobuf_c/ruby-upb.h Normal file → Executable file
View File

File diff suppressed because it is too large Load Diff

1198
ruby/ext/google/protobuf_c/storage.c
View File

File diff suppressed because it is too large Load Diff

13817
ruby/ext/google/protobuf_c/upb.c
View File

File diff suppressed because it is too large Load Diff

2
ruby/google-protobuf.gemspec
View File

@ -17,7 +17,7 @@ Gem::Specification.new do |s|
else
s.files += Dir.glob('ext/**/*')
s.extensions= ["ext/google/protobuf_c/extconf.rb"]
s.add_development_dependency "rake-compiler-dock", ">= 1.0.1", "< 2.0"
s.add_development_dependency "rake-compiler-dock", ">= 1.1.0", "< 2.0"
end
s.test_files = ["tests/basic.rb",
"tests/stress.rb",

33
ruby/tests/basic.rb
View File

@ -216,7 +216,7 @@ module BasicTest
m.map_string_int32 = {}
end
assert_raise TypeError do
assert_raise Google::Protobuf::TypeError do
m = MapMessage.new(:map_string_int32 => { 1 => "I am not a number" })
end
end
@ -447,6 +447,7 @@ module BasicTest
:optional_int32=>0,
:optional_int64=>0,
:optional_msg=>nil,
:optional_msg2=>nil,
:optional_string=>"foo",
:optional_uint32=>0,
:optional_uint64=>0,
@ -484,9 +485,9 @@ module BasicTest
m = MapMessage.new(:map_string_int32 => {"a" => 1})
expected = {mapStringInt32: {a: 1}, mapStringMsg: {}, mapStringEnum: {}}
expected_preserve = {map_string_int32: {a: 1}, map_string_msg: {}, map_string_enum: {}}
assert_equal JSON.parse(MapMessage.encode_json(m), :symbolize_names => true), expected
assert_equal JSON.parse(MapMessage.encode_json(m, :emit_defaults=>true), :symbolize_names => true), expected
json = MapMessage.encode_json(m, :preserve_proto_fieldnames => true)
json = MapMessage.encode_json(m, :preserve_proto_fieldnames => true, :emit_defaults=>true)
assert_equal JSON.parse(json, :symbolize_names => true), expected_preserve
m2 = MapMessage.decode_json(MapMessage.encode_json(m))
@ -496,7 +497,7 @@ module BasicTest
def test_json_maps_emit_defaults_submsg
# TODO: Fix JSON in JRuby version.
return if RUBY_PLATFORM == "java"
m = MapMessage.new(:map_string_msg => {"a" => TestMessage2.new})
m = MapMessage.new(:map_string_msg => {"a" => TestMessage2.new(foo: 0)})
expected = {mapStringInt32: {}, mapStringMsg: {a: {foo: 0}}, mapStringEnum: {}}
actual = MapMessage.encode_json(m, :emit_defaults => true)
@ -504,6 +505,30 @@ module BasicTest
assert_equal JSON.parse(actual, :symbolize_names => true), expected
end
def test_json_emit_defaults_submsg
# TODO: Fix JSON in JRuby version.
return if RUBY_PLATFORM == "java"
m = TestSingularFields.new(singular_msg: proto_module::TestMessage2.new)
expected = {
singularInt32: 0,
singularInt64: "0",
singularUint32: 0,
singularUint64: "0",
singularBool: false,
singularFloat: 0,
singularDouble: 0,
singularString: "",
singularBytes: "",
singularMsg: {},
singularEnum: "Default",
}
actual = proto_module::TestMessage.encode_json(m, :emit_defaults => true)
assert_equal expected, JSON.parse(actual, :symbolize_names => true)
end
def test_respond_to
# This test fails with JRuby 1.7.23, likely because of an old JRuby bug.
return if RUBY_PLATFORM == "java"

17
ruby/tests/basic_proto2.rb
View File

@ -237,23 +237,6 @@ module BasicTestProto2
assert_equal expected_result, m.to_h
end
def test_map_keyword_disabled
pool = Google::Protobuf::DescriptorPool.new
e = assert_raise ArgumentError do
pool.build do
add_file 'test_file.proto', syntax: :proto2 do
add_message "MapMessage" do
map :map_string_int32, :string, :int32, 1
map :map_string_msg, :string, :message, 2, "TestMessage2"
end
end
end
end
assert_match(/Cannot add a native map/, e.message)
end
def test_respond_to
# This test fails with JRuby 1.7.23, likely because of an old JRuby bug.
return if RUBY_PLATFORM == "java"

4
ruby/tests/basic_test.proto
View File

@ -44,6 +44,8 @@ message TestMessage {
repeated bytes repeated_bytes = 20;
repeated TestMessage2 repeated_msg = 21;
repeated TestEnum repeated_enum = 22;
optional TestSingularFields optional_msg2 = 23;
}
message TestSingularFields {
@ -61,7 +63,7 @@ message TestSingularFields {
}
message TestMessage2 {
int32 foo = 1;
optional int32 foo = 1;
}
enum TestEnum {

73
ruby/tests/common_tests.rb
View File

@ -115,14 +115,14 @@ module CommonTests
m = proto_module::TestMessage.new(
:optional_int32 => -42,
:optional_enum => :A,
:optional_msg => proto_module::TestMessage2.new,
:optional_msg => proto_module::TestMessage2.new(foo: 0),
:repeated_string => ["hello", "there", "world"])
expected = "<#{proto_module}::TestMessage: optional_int32: -42, optional_int64: 0, optional_uint32: 0, optional_uint64: 0, optional_bool: false, optional_float: 0.0, optional_double: 0.0, optional_string: \"\", optional_bytes: \"\", optional_msg: <#{proto_module}::TestMessage2: foo: 0>, optional_enum: :A, repeated_int32: [], repeated_int64: [], repeated_uint32: [], repeated_uint64: [], repeated_bool: [], repeated_float: [], repeated_double: [], repeated_string: [\"hello\", \"there\", \"world\"], repeated_bytes: [], repeated_msg: [], repeated_enum: []>"
expected = "<#{proto_module}::TestMessage: optional_int32: -42, optional_msg: <#{proto_module}::TestMessage2: foo: 0>, optional_enum: :A, repeated_int32: [], repeated_int64: [], repeated_uint32: [], repeated_uint64: [], repeated_bool: [], repeated_float: [], repeated_double: [], repeated_string: [\"hello\", \"there\", \"world\"], repeated_bytes: [], repeated_msg: [], repeated_enum: []>"
assert_equal expected, m.inspect
assert_equal expected, m.to_s
m = proto_module::OneofMessage.new(:b => -42)
expected = "<#{proto_module}::OneofMessage: a: \"\", b: -42, c: nil, d: :Default>"
expected = "<#{proto_module}::OneofMessage: b: -42>"
assert_equal expected, m.inspect
assert_equal expected, m.to_s
end
@ -428,7 +428,7 @@ module CommonTests
assert m.length == 0
assert m == {}
assert_raise TypeError do
assert_raise Google::Protobuf::TypeError do
m[1] = 1
end
assert_raise RangeError do
@ -492,7 +492,7 @@ module CommonTests
m = Google::Protobuf::Map.new(:string, :int32)
m["asdf"] = 1
assert_raise TypeError do
assert_raise Google::Protobuf::TypeError do
m[1] = 1
end
assert_raise Encoding::UndefinedConversionError do
@ -505,7 +505,7 @@ module CommonTests
m[bytestring] = 1
# Allowed -- we will automatically convert to ASCII-8BIT.
m["asdf"] = 1
assert_raise TypeError do
assert_raise Google::Protobuf::TypeError do
m[1] = 1
end
end
@ -547,6 +547,7 @@ module CommonTests
"b" => proto_module::TestMessage.new(:optional_int32 => 84) })
m2 = m.dup
assert m.to_h == m2.to_h
assert m == m2
assert m.object_id != m2.object_id
assert m["a"].object_id == m2["a"].object_id
@ -1103,16 +1104,6 @@ module CommonTests
m = proto_module::TestMessage.new
expected = {
optionalInt32: 0,
optionalInt64: "0",
optionalUint32: 0,
optionalUint64: "0",
optionalBool: false,
optionalFloat: 0,
optionalDouble: 0,
optionalString: "",
optionalBytes: "",
optionalEnum: "Default",
repeatedInt32: [],
repeatedInt64: [],
repeatedUint32: [],
@ -1137,17 +1128,7 @@ module CommonTests
m = proto_module::TestMessage.new(optional_msg: proto_module::TestMessage2.new)
expected = {
optionalInt32: 0,
optionalInt64: "0",
optionalUint32: 0,
optionalUint64: "0",
optionalBool: false,
optionalFloat: 0,
optionalDouble: 0,
optionalString: "",
optionalBytes: "",
optionalMsg: {foo: 0},
optionalEnum: "Default",
optionalMsg: {},
repeatedInt32: [],
repeatedInt64: [],
repeatedUint32: [],
@ -1172,16 +1153,6 @@ module CommonTests
m = proto_module::TestMessage.new(repeated_msg: [proto_module::TestMessage2.new])
expected = {
optionalInt32: 0,
optionalInt64: "0",
optionalUint32: 0,
optionalUint64: "0",
optionalBool: false,
optionalFloat: 0,
optionalDouble: 0,
optionalString: "",
optionalBytes: "",
optionalEnum: "Default",
repeatedInt32: [],
repeatedInt64: [],
repeatedUint32: [],
@ -1191,7 +1162,7 @@ module CommonTests
repeatedDouble: [],
repeatedString: [],
repeatedBytes: [],
repeatedMsg: [{foo: 0}],
repeatedMsg: [{}],
repeatedEnum: []
}
@ -1256,8 +1227,9 @@ module CommonTests
assert_equal json, struct.to_json
assert_raise(RuntimeError, "Maximum recursion depth exceeded during encoding") do
proto_module::MyRepeatedStruct.encode(
proto_module::MyRepeatedStruct.new(structs: [proto_module::MyStruct.new(struct: struct)]))
struct = Google::Protobuf::Struct.new
struct.fields["foobar"] = Google::Protobuf::Value.new(struct_value: struct)
Google::Protobuf::Struct.encode(struct)
end
end
@ -1786,4 +1758,25 @@ module CommonTests
assert m1.hash != m2.hash
assert_nil h[m2]
end
def test_object_gc
m = proto_module::TestMessage.new(optional_msg: proto_module::TestMessage2.new)
m.optional_msg
GC.start(full_mark: true, immediate_sweep: true)
m.optional_msg.inspect
end
def test_object_gc_freeze
m = proto_module::TestMessage.new
m.repeated_float.freeze
GC.start(full_mark: true)
# Make sure we remember that the object is frozen.
# The wrapper object contains this information, so we need to ensure that
# the previous GC did not collect it.
assert m.repeated_float.frozen?
GC.start(full_mark: true, immediate_sweep: true)
assert m.repeated_float.frozen?
end
end

2
ruby/travis-test.sh
View File

@ -18,7 +18,7 @@ test_version() {
rake gc_test &&
cd ../conformance && make test_jruby &&
cd ../ruby/compatibility_tests/v3.0.0 && ./test.sh"
elif [ "$version" == "ruby-2.6.0" -o "$version" == "ruby-2.7.0" ] ; then
elif [ "$version" == "ruby-2.6.0" -o "$version" == "ruby-2.7.0" -o "$version" == "ruby-3.0.0" ] ; then
bash --login -c \
"rvm install $version && rvm use $version && \
which ruby && \

6
src/google/protobuf/compiler/ruby/ruby_generator.cc
View File

@ -242,8 +242,7 @@ void GenerateOneof(const OneofDescriptor* oneof, io::Printer* printer) {
bool GenerateMessage(const Descriptor* message, io::Printer* printer,
std::string* error) {
if (message->extension_range_count() > 0 || message->extension_count() > 0) {
*error = "Extensions are not yet supported for proto2 .proto files.";
return false;
GOOGLE_LOG(WARNING) << "Extensions are not yet supported for proto2 .proto files.";
}
// Don't generate MapEntry messages -- we use the Ruby extension's native
@ -543,8 +542,7 @@ bool GenerateFile(const FileDescriptor* file, io::Printer* printer,
// TODO: Remove this when ruby supports extensions for proto2 syntax.
if (file->syntax() == FileDescriptor::SYNTAX_PROTO2 &&
file->extension_count() > 0) {
*error = "Extensions are not yet supported for proto2 .proto files.";
return false;
GOOGLE_LOG(WARNING) << "Extensions are not yet supported for proto2 .proto files.";
}
printer->Print("Google::Protobuf::DescriptorPool.generated_pool.build do\n");

5
tests.sh
View File

@ -441,6 +441,10 @@ build_ruby27() {
internal_build_cpp # For conformance tests.
cd ruby && bash travis-test.sh ruby-2.7.0 && cd ..
}
build_ruby30() {
internal_build_cpp # For conformance tests.
cd ruby && bash travis-test.sh ruby-3.0.0 && cd ..
}
build_jruby() {
internal_build_cpp # For conformance tests.
@ -723,6 +727,7 @@ Usage: $0 { cpp |
ruby25 |
ruby26 |
ruby27 |
ruby30 |
jruby |
ruby_all |
php7.0 |

2
third_party/wyhash/wyhash.h vendored
View File

@ -116,7 +116,7 @@ static inline uint64_t wyhash(const void *key, uint64_t len, uint64_t seed, cons
return _wyfinish(p,len,seed,secret,i);
}
//utility functions
const uint64_t _wyp[5] = {0xa0761d6478bd642full, 0xe7037ed1a0b428dbull, 0x8ebc6af09c88c6e3ull, 0x589965cc75374cc3ull, 0x1d8e4e27c47d124full};
static const uint64_t _wyp[5] = {0xa0761d6478bd642full, 0xe7037ed1a0b428dbull, 0x8ebc6af09c88c6e3ull, 0x589965cc75374cc3ull, 0x1d8e4e27c47d124full};
static inline uint64_t wyhash64(uint64_t A, uint64_t B){ A^=_wyp[0]; B^=_wyp[1]; _wymum(&A,&B); return _wymix(A^_wyp[0],B^_wyp[1]);}
static inline uint64_t wyrand(uint64_t *seed){ *seed+=_wyp[0]; return _wymix(*seed,*seed^_wyp[1]);}
static inline double wy2u01(uint64_t r){ const double _wynorm=1.0/(1ull<<52); return (r>>12)*_wynorm;}