AuroraMiddleware/protobuf
1
0
Fork 0
Code Issues 1 Pull Requests Projects Releases Wiki Activity
c798ba18a9
protobuf/docs/options.md

227 lines
5.8 KiB
Markdown
Raw Normal View History

Add global extension registry to the repo.
2018-07-03 00:00:39 +00:00
# Protobuf Global Extension Registry
This file contains a global registry of known extensions for descriptor.proto,
so that any developer who wishes to use multiple 3rd party projects, each with
their own extensions, can be confident that there won't be collisions in
extension numbers.
If you need an extension number for your custom option (see [custom options](
https://developers.google.com/protocol-buffers/docs/proto#customoptions)),
Replace repo links.
2018-08-22 18:55:30 +00:00
please [send us a pull request](https://github.com/protocolbuffers/protobuf/pulls) to
add an entry to this doc, or [create an issue](https://github.com/protocolbuffers/protobuf/issues)
Add global extension registry to the repo.
2018-07-03 00:00:39 +00:00
with info about your project (name and website) so we can add an entry for you.
## Existing Registered Extensions
1. C# port of protocol buffers
Update options.md | fix website url for C# port Update options.md | fix website url for C# port of protocol buffers.. From `http://github.com/jskeet/dotnet-protobufs` to `https://github.com/jskeet/protobuf-csharp-port`
2018-08-24 14:19:22 +00:00
* Website: https://github.com/jskeet/protobuf-csharp-port
Add global extension registry to the repo.
2018-07-03 00:00:39 +00:00
* Extensions: 1000
1. Perl/XS port of protocol buffers
* Website: http://code.google.com/p/protobuf-perlxs
* Extensions: 1001
1. Objective-C port of protocol buffers
* Website: http://code.google.com/p/protobuf-objc
* Extensions: 1002
1. Google Wave Federation Protocol open-source release (FedOne)
* Website: http://code.google.com/p/wave-protocol
* Extensions: 1003
1. PHP code generator plugin
* Website: ???
* Extensions: 1004
1. GWT code generator plugin (third-party!)
* Website: http://code.google.com/p/protobuf-gwt/
* Extensions: 1005
1. Unix Domain RPC code generator plugin
* Website: http://go/udrpc
* Extensions: 1006
1. Object-C generator plugin (Plausible Labs)
* Website: http://www.plausible.coop
* Extensions: 1007
1. TBD (code42.com)
* Website: ???
* Extensions: 1008
1. Goby Underwater Autonomy Project
Update options.md Update my projects' (DCCL / Goby) URLs. Propose new extension for CGSN Mooring project.
2018-08-24 21:10:24 +00:00
* Website: https://github.com/GobySoft/goby
Add global extension registry to the repo.
2018-07-03 00:00:39 +00:00
* Extensions: 1009
1. Nanopb
* Website: http://kapsi.fi/~jpa/nanopb
* Extensions: 1010
1. Bluefin AUV Communication Extensions
* Website: http://www.bluefinrobotics.com
* Extensions: 1011
1. Dynamic Compact Control Language
Update options.md Update my projects' (DCCL / Goby) URLs. Propose new extension for CGSN Mooring project.
2018-08-24 21:10:24 +00:00
* Website: http://github.com/GobySoft/dccl
Add global extension registry to the repo.
2018-07-03 00:00:39 +00:00
* Extensions: 1012
1. ScaleOut StateServer® Native C++ API
* Website: http://www.scaleoutsoftware.com
* Extensions: 1013
1. FoundationDB SQL Layer
* Website: https://github.com/FoundationDB/sql-layer
* Extensions: 1014
1. Fender
* Website: https://github.com/hassox/fender
* Extensions: 1015
1. Vortex
* Website: http://www.prismtech.com/vortex
* Extensions: 1016
1. tresorit
* Website: https://tresorit.com/
* Extensions: 1017
1. CRIU (Checkpoint Restore In Userspace)
* Website: http://criu.org/Main_Page
* Extensions: 1018
1. protobuf-c
* Website: https://github.com/protobuf-c/protobuf-c
* Extensions: 1019
1. ScalePB
* Website: http://trueaccord.github.io/ScalaPB/
* Extensions: 1020
1. protoc-gen-bq-schema
* Website: https://github.com/GoogleCloudPlatform/protoc-gen-bq-schema
* Extensions: 1021
1. grpc-gateway
* Website: https://github.com/gengo/grpc-gateway
* Extensions: 1022
1. Certificate Transparency
* Website: https://github.com/google/certificate-transparency
* Extensions: 1023
1. JUNOS Router Telemetry
* Website: http://www.juniper.net
* Extensions: 1024
1. Spine Event Engine
* Website: https://github.com/SpineEventEngine/core-java
* Extensions: 1025
1. Aruba cloud platform
* Website: ???
* Extensions: 1026 -> 1030
1. Voltha
* Website: ???
* Extensions: 1031 -> 1033
1. gator
* Website: ???
* Extensions: 1034
1. protoc-gen-flowtypes
* Website: https://github.com/tmc/grpcutil/tree/master/protoc-gen-flowtypes
* Extensions: 1035
1. ProfaneDB
* Website: https://gitlab.com/ProfaneDB/ProfaneDB
* Extensions: 1036
1. protobuf-net
* Website: https://github.com/mgravell/protobuf-net
* Extensions: 1037
1. FICO / StreamEngine
* Website: http://www.fico.com/
* Extensions: 1038
1. GopherJS
* Website: https://github.com/johanbrandhorst/protobuf
* Extensions: 1039
1. ygot
* Website: https://github.com/openconfig/ygot
* Extensions: 1040
1. go-grpcmw
* Website: https://github.com/MarquisIO/go-grpcmw
* Extensions: 1041
1. grpc-gateway protoc-gen-swagger
* Website: https://github.com/grpc-ecosystem/grpc-gateway
* Extensions: 1042
1. AN Message
* Website: TBD
* Extensions: 1043
1. protofire
* Website: https://github.com/ribrdb/protofire
* Extensions: 1044
1. Gravity
* Website: https://github.com/aphysci/gravity
* Extensions: 1045
1. SEMI Standards – I&C Technical Committee
* Website: http://downloads.semi.org/web/wstdsbal.nsf/9c2b317e76523cca88257641005a47f5/88a5863a580e323088256e7b00707489!OpenDocument
* Extensions: 1046
1. Elixir plugin
* Website: https://github.com/tony612/grpc-elixir
* Extensions: 1047
1. API client generators
* Website: ???
* Extensions: 1048-1056
Update options.md (#4952) Added Netifi Proteus to Registered Extensions
2018-07-21 15:34:17 +00:00
1. Netifi Proteus
* Website: https://github.com/netifi-proteus
* Extensions: 1057
Update options.md Update my projects' (DCCL / Goby) URLs. Propose new extension for CGSN Mooring project.
2018-08-24 21:10:24 +00:00
1. CGSN Mooring Project
* Website: https://bitbucket.org/ooicgsn/cgsn-mooring
* Extensions: 1058
delete all duplicate empty blanks (#5758) Signed-off-by: Xiang Dai <764524258@qq.com>
2019-02-21 03:28:50 +00:00
options: reserved extension range for CSI
2018-11-06 16:33:18 +00:00
1. Container Storage Interface
* Website: https://github.com/container-storage-interface/spec
* Extensions: 1059-1069
Update options.md Update my projects' (DCCL / Goby) URLs. Propose new extension for CGSN Mooring project.
2018-08-24 21:10:24 +00:00
Add twirpql ID to global registry
2019-06-11 13:26:25 +00:00
1. TwirpQL Plugin
* Website: https://twirpql.dev
* Extensions: 1070
Add extension number for protoc-gen-validate This project is used by the [Envoy proxy](https://www.envoyproxy.io/) and several others. The existing code is using the extension value [919191](https://github.com/envoyproxy/protoc-gen-validate/blob/54b08a686ff6548d8a248218664254c422fc4599/validate/validate.proto#L29) but since maintaining binary compatibility is not important, it shouldn't be a problem to change once this allocation is approved.
2019-05-24 11:44:51 +00:00
1. Protoc-gen-validate
* Website: https://github.com/envoyproxy/protoc-gen-validate
Merge branch 'master' into patch-1
2019-06-20 18:10:06 +00:00
* Extensions: 1071
Requesting extension id 1072 This library supports an idiomatic proto3 protobuf generator for kotlin. The library will be open sourced by Toast Inc under the Apache2 license, and is currently used in production at Toast. The following is the readme.md that will be released with the code by the end of Q4 2019. Supports only the Protocol Buffers language version 3. #### Features - Clean data class generation - Oneof types handled as sealed classes - JavaDoc comments on generated code - Deprecation option pass-through to Kotlin's `@Deprecated` annotation - Protokt-specific options: non-null types, wrapper types, interface implementation, and more - Tight integration with Protobuf's Java library: compatibility with its well-known types and usage of CodedInputStream and CodedOutputStream for best performance #### Not yet implemented - Kotlin native support - Kotlin JS support - Support for gRPC service generation - Protobuf JSON support See examples in [protokt-testing](https://github.com/toasttab/protokt/tree/master/protokt-testing). ### Generated Code Generated code is placed in `<buildDir>/generated-sources/main/protokt`. A simple example: ```proto syntax = "proto3"; package com.protokt.sample; message Sample { string sample_field = 1; } ``` will produce: ```kotlin /* * Generated by protokt. Do not modify. */ package com.protokt.sample import com.toasttab.protokt.rt.* data class Sample( val sampleField: String, val unknown: Map<Int, Unknown> = emptyMap() ) : KtMessage { @Suppress("UNUSED") constructor( sampleField: String = "" ) : this( sampleField, emptyMap() ) override val messageSize by lazy { sizeof() } override fun serialize(serializer: KtMessageSerializer) { if (sampleField.isNotEmpty()) { serializer.write(Tag(10)).write(sampleField) } if (unknown.isNotEmpty()) { serializer.writeUnknown(unknown) } } private fun sizeof(): Int { var res = 0 if (sampleField.isNotEmpty()) { res += sizeof(Tag(1)) + sizeof(sampleField) } res += unknown.entries.sumBy { it.value.sizeof() } return res } companion object Deserializer : KtDeserializer<Sample> { override fun deserialize(deserializer: KtMessageDeserializer): Sample { var sampleField = "" val unknown = mutableMapOf<Int, Unknown>() while (true) { when (deserializer.readTag()) { 0 -> return Sample( sampleField, unknown ) 10 -> sampleField = deserializer.readString() else -> { val unk = deserializer.readUnknown() unknown[unk.fieldNum] = unknown[unk.fieldNum].let { when (it) { null -> unk else -> when (val v = it.value) { is ListVal -> Unknown(unk.fieldNum, ListVal(v.value + unk.value)) else -> Unknown(unk.fieldNum, ListVal(listOf(v, unk.value))) } } } } } } } } } ``` #### Runtime Notes ##### Package The Kotlin package of a generated file can be overridden from protobuf package with the `(protokt).package` option: ```proto syntax = "proto3"; import "protokt.proto"; package com.example; option (protokt).package = "com.package"; ``` ##### Message Each protokt message implements the `KtMessage` interface. `KtMessage` defines the `serialize()` method and its overloads which can serialize to a byte array, a `KtMessageSerializer`, or on the JVM, an `OutputStream`. Each protokt message has a companion object `Deserializer` that implements the `KtDeserializer` interface, which provides the `deserialize()` method and its overloads to construct an instance of the message from a byte array, a Java InputStream, or others. In order to enjoy the full benefits of Kotlin data classes, byte arrays are wrapped in the protokt `Bytes` class, which provides appropriate `equals()` and `hashCode()` implementations. ##### Enums Enum fields are generated as data classes with a single integer field. Kotlin enum classes are closed and cannot retain unknown values, and protobuf requires that unknown enum values are preserved for reserialization. This compromise exposes a constructor taking an integer, but the `from(value: Int)` on an enum's `Deserializer` should be preferred as it avoids instantiation when possible. Other notes: - `optimize_for` is ignored. - `repeated` fields are deserialized to Lists. - `map` fields are deserialized to Maps. - `oneof` fields are represented as data class subtypes of a sealed base class with a single property. ### Extensions See examples of each option in the [protokt-options](https://github.com/toasttab/protokt/tree/master/protokt-testing/protokt-options/src/main/proto) module. All protokt-specific options require importing `protokt.proto` in the protocol file. #### Wrapper Types Sometimes a field on a protobuf message corresponds to a concrete nonprimitive type. In standard protobuf the user would be responsible for this extra transformation, but the protokt wrapper type option allows specification of a converter that will automatically encode and decode custom types to protobuf primitives and well-known types. Some standard types are implemented in [protokt-extensions](https://github.com/toasttab/protokt/tree/master/protokt-extensions/src/main/kotlin/com/toasttab/protokt/ext). Wrap a field by invoking the `(protokt_property).wrap` option: ```proto message DateWrapperMessage { int64 date = 1 [ (protokt_property).wrap = "java.util.Date" ]; } ``` Converters implement the `Converter` interface: ```kotlin interface Converter<S: Any, T: Any> { val wrapper: KClass<S> fun wrap(unwrapped: T): S fun unwrap(wrapped: S): T } ``` and protokt will reference the converter's methods to wrap and unwrap from protobuf primitives: ```kotlin object DateConverter : Converter<Date, Long> { override val wrapper = Date::class override fun wrap(unwrapped: Long) = Date(unwrapped) override fun unwrap(wrapped: Date) = wrapped.time } ``` ```kotlin data class WrapperModel( val date: java.util.Date, ... ) : KtMessage { ... override fun serialize(serializer: KtMessageSerializer) { serializer.write(Tag(10)).write(Int64(DateConverter.unwrap(date))) ... } override fun deserialize(deserializer: KtMessageDeserializer): WrapperModel { var date = 0L while (true) { when (deserializer.readTag()) { 0 -> return WrapperModel( DateConverter.wrap(date), ... ) ... } } } } ``` Converters can also implement the `OptimizedSizeofConverter` interface adding `sizeof()`, which allows them to optimize the calculation of the wrapper's size rather than unwrap the object twice. For example, a UUID is always 16 bytes: ```kotlin object UuidConverter : OptimizedSizeofConverter<UUID, ByteArray> { override val wrapper = UUID::class private val sizeofProxy = ByteArray(16) override fun sizeof(wrapped: UUID) = sizeof(sizeofProxy) override fun wrap(unwrapped: ByteArray): UUID { require(unwrapped.size == 16) { "input must have size 16; had ${unwrapped.size}" } return ByteBuffer.wrap(unwrapped) .run { UUID(long, long) } } override fun unwrap(wrapped: UUID) = ByteBuffer.allocate(16) .putLong(wrapped.mostSignificantBits) .putLong(wrapped.leastSignificantBits) .array() } ``` Rather than convert a UUID to a byte array both for size calculation and for serialization (which is what a naïve implementation would do), UuidConverter always returns the size of a constant 16-byte array. If the wrapper type is in the same package as the generated protobuf message, then it does not need a fully-qualified name. Custom wrapper type converters can be in the same module as protobuf types that reference them. In order to use any wrapper type defined in `protokt-extensions`, the module must be included as a dependency: ```groovy dependencies { implementation 'com.toasttab.protokt:protokt-extensions:0.0.3' } ``` #### Interface implementation To avoid the need to create domain-specific objects from protobuf messages you can declare that a protobuf message implements a custom interface with properties and default methods. ```kotlin package com.protokt.sample interface Model { val id: String } ``` ```proto package com.protokt.sample; message ImplementsSampleMessage { option (protokt_class).implements = "Model"; string id = 1; } ``` If the wrapper interface is in the same package as the generated protobuf message, then it does not need a fully-qualified name. Wrapper interfaces cannot be used by protobuf messages in the same module that defines them; the dependency must be declared with`protoktExtensions` in `build.gradle`: ```groovy dependencies { protoktExtensions project(':api-module') } ``` #### Nonnull fields If there is a message that has no meaning whatsoever when a particular field is missing, you can emulate proto2's `required` key word by using the `(protokt_oneof).non_null` option: ```proto message Sample { } message NonNullSampleMessage { Sample non_null_sample = 1 [ (protokt_property).non_null = true ]; } ``` Generated code will not have a nullable type so the field can be referenced without using Kotlin's `!!`. Oneof fields can also be declared non-null: ```proto message NonNullSampleMessage { oneof non_null_oneof { option (protokt_oneof).non_null = true; string message = 2; } } ``` Note that deserialization of a message with a non-nullable field will fail if the message being decoded does not contain an instance of the required field. #### BytesSlice When reading messages that contain other serialized messages as `bytes` fields, protokt can keep a reference to the originating byte array to prevent a large copy operation on deserialization. This can be desirable when the wrapping message is a thin metadata shim and doesn't include much memory overhead: ```proto message SliceModel { int64 version = 1; bytes encoded_message = 2 [ (protokt_property).bytes_slice = true ]; } ``` ### Usage #### Gradle ```groovy buildscript { dependencies { classpath "com.toasttab.protokt:protokt-gradle-plugin:0.0.3" } } apply plugin: 'com.toasttab.protokt' ``` This will automatically download and install protokt, apply the Google protobuf plugin, and configure all the necessary boilerplate. By default it will also add `protokt-runtime` to the api scope of the project, and `protobuf-java` to the implementation scope. If your project is pure Kotlin you may run into the following error: ``` Execution failed for task ':compileJava'. > error: no source files ``` To work around it, disable all `JavaCompile` tasks in the project: ```groovy tasks.withType(JavaCompile) { enabled = false } ``` or: ```groovy compileJava.enabled = false ``` #### Command line code generation ```bash protokt-codegen$ ./gradlew assemble [OR ./gradlew installDist] protokt-codegen$ ./run-protokt.sh -h protokt-codegen$ ./run-protokt.sh \ -out=../kotlin \ -pkg=com.toasttab.protokt.conformance \ -file=conformance.proto \ -cp=../build/libs/protokt-codegen-0.0.3-SNAPSHOT-all.jar \ -plugin=../bin/protokt.sh ``` ### Contribution To enable rapid development of the code generator, the protobuf conformance tests have been compiled and included in the protokt-testing project. They run on Mac OS 10.14+ and Ubuntu 16.04 x86-64. Publish the plugin to the integration repository: ```bash protokt$ ./gradlew publishToIntegrationRepository ``` Then run the tests from `protokt-testing`: ```bash protokt-testing$ ./gradlew protokt-conformance-tests:test ``` All integration tests can be run with: ``` protokt-testing$ ./gradlew test ```
2019-07-10 13:58:38 +00:00
1. protokt
* Website: https://github.com/toasttab/protokt (Currently Private but will be open soon.)
* Extensions: 1072
Add option for dart
2019-09-27 07:21:27 +00:00
1. Dart port of protocol buffers
* Website https://github.com/dart-lang/protobuf
* Extensions: 1073
Added a descriptor option number for ocaml-protoc-plugin
2019-10-29 21:05:40 +00:00
1. Ocaml-protoc-plugin
* Website: https://github.com/issuu/ocaml-protoc-plugin
* Extensions: 1074
Allocated a custom option number for Analyze Re Graphene
2019-12-05 23:00:56 +00:00
1. Analyze Re Graphene
* Website: https://analyzere.com
* Extensions: 1075
Reserve 1076, 1077 for Wire since and until release notes: no
2019-12-17 02:15:44 +00:00
1. Wire since and until
* Website: https://square.github.io/wire/
* Extensions: 1076, 1077
Reference in New Issue Copy Permalink