Lots of CI tests have been failing recently with error messages like
this from "docker pull":
Error response from daemon: unexpected EOF
This seems to be an issue affecting other projects as well:
https://forums.docker.com/t/docker-pull-causes-error-response-from-daemon-unexpected-eof/88608
This commit tries to mitigate the problem by periodically retrying
failed Docker pulls.
* python: generate documentation with Sphinx and Read the Docs
Background:
Formerly, the Python protobuf reference documentation was built with
[Epydoc](http://epydoc.sourceforge.net/). This package has not been
updated since 2008, and it has inconsistent formatting (see internal
issue 131415575) with most Python documentation. Sphinx is used for the
official docs.python.org docs as well as most other Python packages,
including the Google client libraries and related packages, such as
https://googleapis.dev/python/google-api-core/latest/
To build the docs with Sphinx:
1. Install the needed packages (`sphinx`, `sphinxcontrib-napoleon` for
Google-style docstring support). I've created a conda environment file
to make this easier:
```
conda env create -f python/docs/environment.yml
```
2. (Optional) Generate reference docs files and regenerate index:
```
cd python
python generate_docs.py
cd ..
```
3. Run Sphinx.
```
cd python/docs
make html
```
About this change:
The script at `python/generate_docs.py` creates a ReStructured Text file
for each public module in the protobuf Python package. The script also
updates the table of contents in `python/docs/index.rst` to point to
these module references.
Future work:
Testing the docs build on PRs requires contributors to actually do some
setup work to configure builds on their fork. It'd be better if CI had a
docs build session to verify that the Sphinx docs generation at least
runs.
There are many warnings due to not-quite-correct docstrings in the
actual Python code itself. I'm choosing to ignore these errors to keep
the PR small, but I recommend you fix these and then enable "fail on
warnings" in the docs build on CI.
* add docs to EXTRA_DIST
* add instructions to build documentation to generate_docs.py
* exclude python/odcs from cpp_distcheck
* Add failing tests for issues with wrapped values where the value is the default
* Add test for wrapped values without a value set
* Bugfix for wrapper types with default values.
The previous optimizations for wrapper types had a bug that prevented
wrappers from registering as "present" if the "value" field was not
present on the wire.
In practice the "value" field will not be serialized when it is zero,
according to proto3 semantics, but due to the optimization this
prevented it from creating a new object to represent the presence of the
field.
The fix is to ensure that if the wrapper message is present on the wire,
we always initialize its value to zero.
Co-authored-by: Dan Quan <dan@quan.io>
Instead of calling initOnce of dependencies, initialize metadata of dependencies in the same file.
Needs to pass aggregate_metadata option to protoc to trigger, e.g.:
--php_out=aggregate_metadata=foo#bar:generated_dir
For each input file, transitive dependencies (including itself), whose package name has the prefix of foo or bar, will be aggregated, in which their metadata string will be aggregated in the same internalAddGeneratedFile call. For other dependencies, initOnce is called as before.
This feature is EXPERIMENTAL. DO NOT USE!!!
There are have been a few issues around people using case sensitive file systems
what Xcode/clang does when looking at the paths. In attempts to solve one set of
warnings, new warnings/errors happened in different setup. So, to hopefully put
these problem away for got, move the WKTs to be at the same level as the other
headers.
- Revert "Override CocoaPods module to lowercase (#6464)"
This reverts commit 479ba8226b.
- Move WKTs to the objectivec directory and make the old headers shim back to
the new locations.
- Update objectivec/generate_well_known_types.sh to check them one at a time
and to deal with the new locations for them.
Fixes#6803
Remove dead links and update the ones that were moved.
Use https when possible.
Isolate inactive projects (not updated in the last few years) in a subsection.
release notes: no
This should reduce binary size slightly, small performance improvement, and improve linkage by forcing references to all used classes.
Note that this maintains backwards compatibility for sources generated by older protoc for the time being. If you want the benefits
you will need to recompile your protos with the newer protoc.
* Blacklist .proto source files is Bazel allows us to
This is a partial revert of 7b28278c7d to unblock, e.g., https://github.com/grpc/grpc/pull/21590 or https://github.com/lyft/envoy-mobile/issues/617 until Bazel is fixed.
Note: this is a forward-compatible change that automatically switches to the behavior intended by 7b28278c7d when a compatible Bazel is released without requiring users to upgrade Protobuf. We will revert this change when Bazel is fixed.
* Remove trailing ,
* Update BUILD
* Add scripts to test multirequest
* chmod ug+x multirequest.sh
* Add continuous test
* Compile c extension
* Class entry is obsolete in the second request
1) Needes to use class name in persistent map
2) Invalidate class entry stored in descriptor
* Add new files to dist
* Fix compile_extension
* Cleanup outputs for phpize
Clang currently displays an error if source files generated with protoc are compiled with -Wcomma. This change fixes this as suggested by the compiler itself.
VLAs complicate static analysis and bloat stack size. Replace VLA allocation
with calls to malloc and free. This will alos the code to build with -Wvla.
This corresponds to the newest reading of RFC 3629, and results
in the largest possible number of character entities by any
valid parser. This may result in a buffer which is oversized,
but never undersized.
This is after further discussion with acozzette in this PR;
https://github.com/protocolbuffers/protobuf/pull/6844
Signed-off-by: William A Rowe Jr wrowe@pivotal.io
Signed-off-by: Yechiel Kalmenson ykalmenson@pivotal.io
In consuming this useful string utility, it was discovered
that the interpretation of leading byte codes 0xf8-0xff
did not conform to either the RFC 3629 nor ISO/IEC 10646
definitions of utf-8.
The IETF RFC describes only 1-4 byte encodings (a limited
number of 4 byte encodings at that), and plainly states in
section 1. Introduction;
o The octet values C0, C1, F5 to FF never appear.
Alternately, the ISO definition "R.2 Specification of UTF-8"
preseented in the original IETF RFC 2279 clearly define the
meaning of leading byte values F5 through FD, and RFC 3629
Section 10. Security paragraph 3 calls out this alternate
reading (alterative to "never appears".) F5-F7 begin an
invalid (in the domain of unicode code points) 4-byte UTF-8
sequence (similar to F0-F4), while F8-FC begin a 5-byte
sequence, FC and FD begin a 6 byte sequence.
The curent code is wrong in that it doesn't treat the codes
F8-FF as invalid 1-byte characters, nor does it treat the
codes F8-FD as the correct number of bytes. No valid parser
will land these lead characters 4 bytes forward. Most will
treat these as the 5 or 6 byte utf-32 character and may then
treat the resulting character as invalid, while some parsers
may reject all leading F5-FF characters as a single byte of
erronious input, followed by each invalid continuation byte.
We propose the conventional reading of F8-FD as 5 and 6 byte
sequences as originally defined, while FE-FF must be read
as single byte invalid code points.
Signed-off-by: William A Rowe Jr <wrowe@pivotal.io>
Signed-off-by: Yechiel Kalmenson <ykalmenson@pivotal.io>
Plugins (and some built-in generators) have `--<lang>_opt` flag that
allows passing parameters one-by-one instead of passing them as
`--<lang>_out=<params>:<out_base>`. This PR changes protoc to
allow using `--<lang>_opt` for all (built-in) generators.
