Commit Graph

43 Commits

Author SHA1 Message Date
Vasiliy Tolstov
7ffe9402db
Add micro Registered Extension Number ()
unistack-org/micro (microservices framework) uses protoc-gen-go-micro generator that has extension support for additional framework related options.

It is currently not using any registered extension number, so i want to stabilise this and i think that this is a good thing.

See here for an example of the extenstion in action.

Co-authored-by: Adam Cozzette <acozzette@google.com>
2022-04-08 09:11:27 -07:00
Jeff Sawatzky
e3b00bb58a
Add Google Gnostic Registered Extension Number ()
google/gnostic's protoc-gen-openapi generator has extension support for additional openapi options.
It is [currently using `1042`](https://github.com/google/gnostic/blob/master/openapiv3/annotations.proto) which is registered with grpc-gateway's protoc-gen-openapiv2 extension which probably isn't a good thing.
See [here](https://github.com/google/gnostic/blob/master/cmd/protoc-gen-openapi/examples/tests/openapiv3annotations/message.proto) for an example of the extenstion in action.
2022-04-08 09:08:38 -07:00
yeqown
a69354f31b
apply a registration for protoc-gen-fieldmask extension ()
* Update options.md

* add one blank line
2022-03-19 08:46:51 -07:00
BartH
2495d220fa
Added Embedded Proto to options.md to reserve the extenion number for the custom options. () 2022-02-11 08:08:25 -08:00
Dane Harrigan
29b3d01572
Add protoc-gen-go-svc to options.md () 2021-12-07 16:47:05 -08:00
Upils
1e1c1cd884
Reserve extension id range for protoc-gen-checker () 2021-10-21 08:24:38 -07:00
Oleg Golberg
cb7be38519
Add Protokt to the list of 3rd party implementations () 2021-09-24 09:41:06 -07:00
Prawn
8c39a1c4fd
An additional range so we can have FieldOptions, MessageOptions, EnumOptions, FileOptions, and OneOfOptions () 2021-09-23 14:09:32 -07:00
Prawn
e9b977fdce
Adding an extension number for protoc-gen-jsonschema () 2021-09-20 15:17:07 -07:00
Adam Cozzette
0b0e404be9
Allocate 10 descriptor.proto extension numbers for COSI ()
Fixes .
2021-08-27 15:01:57 -07:00
Andrew Parmet
18b1b21aba
update protokt options reference () 2021-07-26 11:11:02 -07:00
Marcel Lanz
01ac995c94
I've requested a block of 5 extension numbers. We already have two concrete extensions that we need ids for. Our project is under active development and we can imagine a number of other extensions that we will want in future, so we thought we'd start by registering a block of 5, instead of raising a new PR here each time we need a new one () 2021-06-28 11:52:54 -07:00
Petr Jediný
3737676d83
Reserve 1109 for Glitchdot () 2021-05-26 12:19:33 -07:00
chenzhihui
43aca57613 Allocate extension 1108 to Kratos API Errors 2021-03-10 08:20:18 -08:00
Galen Warren
6f4573afbd docs: register Coach Client Connect extension number (1107) 2021-03-09 09:14:23 -08:00
Upils
01dcb94846
Expand reserved IDs for protoc-gen-psql and protoc-gen-sanititize ()
Fix for 59b3d97.
2021-02-11 16:00:49 -08:00
Paul MARS
59b3d97f6d Reserve extension id for protoc-gen-psql & protoc-gen-sanitize 2021-02-05 10:34:04 -08:00
brucearctor
6c970b5a05 Update options.md - Added 'Astounding' Project 2020-12-28 13:44:56 -08:00
Nadav Samet
9637a3b57c Allocate extension 1089 to ScalaPB validate
ScalaPB validate is a sub-package of ScalaPB that allows user to generate data validation code for ScalaPB generated classes.
2020-12-01 15:53:51 -08:00
Nadav Samet
b5d2cc08f1 Update scalapb URL and fix typo in name 2020-11-02 12:45:58 -08:00
Robert Yokota
300dfcc5bf Reserve 1088 for Confluent Schema Registry 2020-09-25 11:12:03 -07:00
Benoît Quenaudon
8d74659c7a Reserve 1087 for Wire wire_package
release notes: no
2020-08-10 10:19:56 -07:00
Adam Cozzette
cf601047eb Allocated option number 1086 for ADLINK EdgeSDK
Fixes .
2020-04-02 09:07:16 -07:00
Robert Morris
a015ca8cf8
Request SummaFT extension ()
SummaFT develops a protoc plugin for both internal and external use that allows specific extensions to be made to extend the platform for GraphQL and OpenAPI 3.
2020-03-19 14:20:01 -07:00
James Roper
ab968155e5 Add Cloudstate extensions 2020-03-19 08:40:39 -07:00
Adam Cozzette
a80775c391 Allocated extension number 1079 for grpc-graphql-gateway
Fixes issue .
2020-03-17 08:27:19 -07:00
Mark Schaller
88579a44fb Reserve extension for Bazel failure detail metadata
The Bazel project (https://github.com/bazelbuild/bazel) intends to
enrich its server app's (protobuf-defined) command service's status
message with a structured representation of what, if anything, went
wrong while processing a command. We wish to use custom options to add
metadata to the protobuf constructs involved.

release notes: no
2020-02-25 14:40:36 -08:00
Jesse Wilson
6263268b8c Reserve 1076, 1077 for Wire since and until
release notes: no
2019-12-17 10:27:25 -08:00
Adam Cozzette
16b0c054a1 Allocated a custom option number for Analyze Re Graphene 2019-12-06 10:33:25 -08:00
Adam Cozzette
403df1d047 Added a descriptor option number for ocaml-protoc-plugin 2019-10-29 14:34:50 -07:00
Sigurd Meldgaard
f4a3e06101 Add option for dart 2019-09-27 14:41:42 -07:00
Ben Gordon
f5362e11fd
Requesting extension id 1072
This library supports an idiomatic proto3 protobuf generator for kotlin.
The library will be open sourced by Toast Inc under the Apache2 license, and is currently used in production at Toast.
The following is the readme.md that will be released with the code by the end of Q4 2019.

Supports only the Protocol Buffers language version 3.

#### Features
- Clean data class generation
- Oneof types handled as sealed classes
- JavaDoc comments on generated code
- Deprecation option pass-through to Kotlin's `@Deprecated` annotation
- Protokt-specific options: non-null types, wrapper types, interface implementation,
and more
- Tight integration with Protobuf's Java library: compatibility with its well-known
types and usage of CodedInputStream and CodedOutputStream for best performance

#### Not yet implemented
- Kotlin native support
- Kotlin JS support
- Support for gRPC service generation
- Protobuf JSON support

See examples in [protokt-testing](https://github.com/toasttab/protokt/tree/master/protokt-testing).

### Generated Code
Generated code is placed in `<buildDir>/generated-sources/main/protokt`.

A simple example:
```proto
syntax = "proto3";

package com.protokt.sample;

message Sample {
  string sample_field = 1;
}
```

will produce:
```kotlin
/*
 * Generated by protokt. Do not modify.
 */
package com.protokt.sample

import com.toasttab.protokt.rt.*

data class Sample(
    val sampleField: String,
    val unknown: Map<Int, Unknown> = emptyMap()
) : KtMessage {
    @Suppress("UNUSED")
    constructor(
        sampleField: String = ""
    ) : this(
        sampleField,
        emptyMap()
    )

    override val messageSize by lazy { sizeof() }

    override fun serialize(serializer: KtMessageSerializer) {
        if (sampleField.isNotEmpty()) {
            serializer.write(Tag(10)).write(sampleField)
        }
        if (unknown.isNotEmpty()) {
            serializer.writeUnknown(unknown)
        }
    }

    private fun sizeof(): Int {
        var res = 0
        if (sampleField.isNotEmpty()) {
            res += sizeof(Tag(1)) + sizeof(sampleField)
        }
        res += unknown.entries.sumBy { it.value.sizeof() }
        return res
    }

    companion object Deserializer : KtDeserializer<Sample> {
        override fun deserialize(deserializer: KtMessageDeserializer): Sample {
            var sampleField = ""
            val unknown = mutableMapOf<Int, Unknown>()
            while (true) {
                when (deserializer.readTag()) {
                    0 ->
                        return Sample(
                            sampleField,
                            unknown
                        )
                    10 -> sampleField = deserializer.readString()
                    else -> {
                        val unk = deserializer.readUnknown()
                        unknown[unk.fieldNum] = unknown[unk.fieldNum].let {
                            when (it) {
                                null -> unk
                                else ->
                                    when (val v = it.value) {
                                        is ListVal ->
                                            Unknown(unk.fieldNum, ListVal(v.value + unk.value))
                                        else ->
                                            Unknown(unk.fieldNum, ListVal(listOf(v, unk.value)))
                                    }
                            }
                        }
                    }
                }
            }
        }
    }
}
```

#### Runtime Notes
##### Package
The Kotlin package of a generated file can be overridden from protobuf package with the `(protokt).package` option:
```proto
syntax = "proto3";

import "protokt.proto";

package com.example;

option (protokt).package = "com.package";
```

##### Message
Each protokt message implements the `KtMessage` interface. `KtMessage` defines the `serialize()`
method and its overloads which can serialize to a byte array, a `KtMessageSerializer`, or on the JVM,
an `OutputStream`.

Each protokt message has a companion object `Deserializer` that implements the `KtDeserializer`
interface, which provides the `deserialize()` method and its overloads to construct an
instance of the message from a byte array, a Java InputStream, or others.

In order to enjoy the full benefits of Kotlin data classes, byte arrays are wrapped in the
protokt `Bytes` class, which provides appropriate `equals()` and `hashCode()` implementations.

##### Enums
Enum fields are generated as data classes with a single integer field. Kotlin enum classes are
closed and cannot retain unknown values, and protobuf requires that unknown enum values are
preserved for reserialization. This compromise exposes a constructor taking an integer, but the
`from(value: Int)` on an enum's `Deserializer` should be preferred as it avoids instantiation
when possible.


Other notes:
- `optimize_for` is ignored.
- `repeated` fields are deserialized to Lists.
- `map` fields are deserialized to Maps.
- `oneof` fields are represented as data class subtypes of a sealed base class with a single property.

### Extensions
See examples of each option in the [protokt-options](https://github.com/toasttab/protokt/tree/master/protokt-testing/protokt-options/src/main/proto)
module. All protokt-specific options require importing `protokt.proto` in the protocol file.
#### Wrapper Types
Sometimes a field on a protobuf message corresponds to a concrete nonprimitive type. In
standard protobuf the user would be responsible for this extra transformation, but the
protokt wrapper type option allows specification of a converter that will automatically
encode and decode custom types to protobuf primitives and well-known types. Some standard
types are implemented in
[protokt-extensions](https://github.com/toasttab/protokt/tree/master/protokt-extensions/src/main/kotlin/com/toasttab/protokt/ext).

Wrap a field by invoking the `(protokt_property).wrap` option:
```proto
message DateWrapperMessage {
  int64 date = 1 [
    (protokt_property).wrap = "java.util.Date"
  ];
}
```

Converters implement the `Converter` interface:
```kotlin
interface Converter<S: Any, T: Any> {
    val wrapper: KClass<S>

    fun wrap(unwrapped: T): S

    fun unwrap(wrapped: S): T
}
```

and protokt will reference the converter's methods to wrap and unwrap from protobuf primitives:
```kotlin
object DateConverter : Converter<Date, Long> {
    override val wrapper = Date::class

    override fun wrap(unwrapped: Long) =
        Date(unwrapped)

    override fun unwrap(wrapped: Date) =
        wrapped.time
}
```

```kotlin
data class WrapperModel(
    val date: java.util.Date,
    ...
) : KtMessage {
    ...
    override fun serialize(serializer: KtMessageSerializer) {
        serializer.write(Tag(10)).write(Int64(DateConverter.unwrap(date)))
        ...
    }

    override fun deserialize(deserializer: KtMessageDeserializer): WrapperModel {
        var date = 0L

        while (true) {
            when (deserializer.readTag()) {
                0 ->
                    return WrapperModel(
                        DateConverter.wrap(date),
                        ...
                    )
                ...
            }
        }
    }
}
```

Converters can also implement the `OptimizedSizeofConverter` interface adding `sizeof()`,
which allows them to optimize the calculation of the wrapper's size rather than unwrap
the object twice. For example, a UUID is always 16 bytes:

```kotlin
object UuidConverter : OptimizedSizeofConverter<UUID, ByteArray> {
    override val wrapper = UUID::class

    private val sizeofProxy = ByteArray(16)

    override fun sizeof(wrapped: UUID) =
        sizeof(sizeofProxy)

    override fun wrap(unwrapped: ByteArray): UUID {
        require(unwrapped.size == 16) {
            "input must have size 16; had ${unwrapped.size}"
        }

        return ByteBuffer.wrap(unwrapped)
            .run { UUID(long, long) }
    }

    override fun unwrap(wrapped: UUID) =
        ByteBuffer.allocate(16)
            .putLong(wrapped.mostSignificantBits)
            .putLong(wrapped.leastSignificantBits)
            .array()
}
```

Rather than convert a UUID to a byte array both for size calculation and for serialization
(which is what a naïve implementation would do), UuidConverter always returns the size of a
constant 16-byte array.

If the wrapper type is in the same package as the generated protobuf message, then it
does not need a fully-qualified name. Custom wrapper type converters can be in the same module as
protobuf types that reference them. In order to use any wrapper type defined in
`protokt-extensions`, the module must be included as a dependency:

```groovy
dependencies {
    implementation 'com.toasttab.protokt:protokt-extensions:0.0.3'
}
```

#### Interface implementation
To avoid the need to create domain-specific objects from protobuf messages you can declare
that a protobuf message implements a custom interface with properties and default methods.

```kotlin
package com.protokt.sample

interface Model {
    val id: String
}
```

```proto
package com.protokt.sample;

message ImplementsSampleMessage {
  option (protokt_class).implements = "Model";

  string id = 1;
}
```

If the wrapper interface is in the same package as the generated protobuf message, then it
does not need a fully-qualified name. Wrapper interfaces cannot be used by protobuf messages
in the same module that defines them; the dependency must be declared with`protoktExtensions`
in `build.gradle`:

```groovy
dependencies {
    protoktExtensions project(':api-module')
}
```

#### Nonnull fields
If there is a message that has no meaning whatsoever when a particular field is missing, you
can emulate proto2's `required` key word by using the `(protokt_oneof).non_null` option:

```proto
message Sample {
}

message NonNullSampleMessage {
  Sample non_null_sample = 1 [
    (protokt_property).non_null = true
  ];
}
```

Generated code will not have a nullable type so the field can be referenced without using
Kotlin's `!!`.

Oneof fields can also be declared non-null:

```proto
message NonNullSampleMessage {
  oneof non_null_oneof {
    option (protokt_oneof).non_null = true;

    string message = 2;
  }
}
```

Note that deserialization of a message with a non-nullable field will fail if the
message being decoded does not contain an instance of the required field.

#### BytesSlice
When reading messages that contain other serialized messages as `bytes` fields, protokt can
keep a reference to the originating byte array to prevent a large copy operation on
deserialization. This can be desirable when the wrapping message is a thin metadata shim and
doesn't include much memory overhead:

```proto
message SliceModel {
  int64 version = 1;

  bytes encoded_message = 2 [
    (protokt_property).bytes_slice = true
  ];
}
```

### Usage

#### Gradle

```groovy
buildscript {
    dependencies {
        classpath "com.toasttab.protokt:protokt-gradle-plugin:0.0.3"
    }
}

apply plugin: 'com.toasttab.protokt'
```

This will automatically download and install protokt, apply the Google protobuf plugin,
and configure all the necessary boilerplate. By default it will also add `protokt-runtime`
to the api scope of the project, and `protobuf-java` to the implementation scope.

If your project is pure Kotlin you may run into the following error:

```
Execution failed for task ':compileJava'.
> error: no source files
```

To work around it, disable all `JavaCompile` tasks in the project:

```groovy
tasks.withType(JavaCompile) {
    enabled = false
}
```

or:
```groovy
compileJava.enabled = false
```

#### Command line code generation

```bash
protokt-codegen$ ./gradlew assemble [OR ./gradlew installDist]

protokt-codegen$ ./run-protokt.sh -h

protokt-codegen$ ./run-protokt.sh \
  -out=../kotlin \
  -pkg=com.toasttab.protokt.conformance \
  -file=conformance.proto \
  -cp=../build/libs/protokt-codegen-0.0.3-SNAPSHOT-all.jar \
  -plugin=../bin/protokt.sh
```

### Contribution
To enable rapid development of the code generator, the protobuf conformance tests have been 
compiled and included in the protokt-testing project. They run on Mac OS 10.14+ and Ubuntu
16.04 x86-64.

Publish the plugin to the integration repository:
```bash
protokt$ ./gradlew publishToIntegrationRepository
```

Then run the tests from `protokt-testing`:
```bash
protokt-testing$ ./gradlew protokt-conformance-tests:test
```

All integration tests can be run with:
```
protokt-testing$ ./gradlew test
```
2019-07-10 09:58:38 -04:00
Alex Konradi
619ffac4f7
Merge branch 'master' into patch-1 2019-06-20 14:10:06 -04:00
marwan-at-work
36cd11a2bf Add twirpql ID to global registry 2019-06-11 07:34:41 -07:00
Alex Konradi
b44a1364ec
Add extension number for protoc-gen-validate
This project is used by the [Envoy proxy](https://www.envoyproxy.io/) and several others.

The existing code is using the extension value [919191](54b08a686f/validate/validate.proto (L29)) but since maintaining binary compatibility is not important, it shouldn't be a problem to change once this allocation is approved.
2019-05-24 07:44:51 -04:00
Xiang Dai
e479410564 delete all duplicate empty blanks ()
Signed-off-by: Xiang Dai <764524258@qq.com>
2019-02-20 19:28:50 -08:00
James DeFelice
4358133b85 options: reserved extension range for CSI 2018-11-06 09:01:59 +00:00
Feng Xiao
0bfe1ca277
Merge pull request from GobySoft/request-new-extension
Request new option extension (and update URLs for existing projects)
2018-08-29 14:15:33 -07:00
Toby Schneider
4fd7bdee48
Update options.md
Update my projects' (DCCL / Goby) URLs.
Propose new extension for CGSN Mooring project.
2018-08-24 17:10:24 -04:00
Gohar Aziz
f3a74befcd
Update options.md | fix website url for C# port
Update options.md | fix website url for C# port of protocol buffers.. From `http://github.com/jskeet/dotnet-protobufs` to `https://github.com/jskeet/protobuf-csharp-port`
2018-08-24 10:19:22 -04:00
Feng Xiao
afe98de32a Replace repo links. 2018-08-22 11:55:30 -07:00
Ryland Degnan
5b7d0dc8aa Update options.md ()
Added Netifi Proteus to Registered Extensions
2018-07-21 08:34:17 -07:00
Feng Xiao
ef7052b473 Add global extension registry to the repo. 2018-07-02 17:07:55 -07:00