* Fix a typo
* Fix lots of spelling errors
* Fix a few more spelling mistakes
* s/parsable/parseable/
* Don't touch the third party files
* Cloneable is the preferred C# term
* Copyable is the preferred C++ term
* Revert "s/parsable/parseable/"
This reverts commit 534ecf7675.
* Revert unparseable->unparsable corrections
Newer versions of virtualenv lack the --no-site-packages option, so I
had to remove it to keep the release.sh script working. I read that this
option has already been the default for a long time, so removing it
shouldn't chany any behavior.
For the setup.py script, I was getting some errors about the bdist_wheel
argument to setup.py, but I was able to fix that by adding 'wheel' to
setup_requires.
Since our (public) generated APIs document their use of repeated fields, we should probably document its API, too.
This includes some changes to generate_docs.py to explicitly allow named modules, since the repeated field implementation lives in the "internal" tree.
* Set execute bit on files if and only if they begin with (#!).
Git only tracks the 'x' (executable) bit on each file. Prior to this
CL, our files were a random mix of executable and non-executable.
This change imposes some order by making files executable if and only
if they have shebang (#!) lines at the beginning.
We don't have any executable binaries checked into the repo, so
we shouldn't need to worry about that case.
* Added fix_permissions.sh script to set +x iff a file begins with (#!).
This change updates docstrings and comments so that they will produce nicer
formatting and cross-references from Sphinx. There are a few broad categories of
changes:
- Paramter and attribute docs are updated so that types will be recognized by
Napoleon (https://sphinxcontrib-napoleon.readthedocs.io/en/latest/) This
usually just means moving a colon in the docstring, so
`name: (type) description` becomes `name (type): description`.
- References to other symbols can be cross-references if they have the right
format. For example, "attr_name" might become ":attr:`attr_name`".
https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#cross-referencing-python-objects
- For fenced code blocks, adding a double-colon `::` signifies a literal block.
https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html#literal-blocks
- Some bits of docstrings move from docstring to comments. For TODOs, this
means we won't be putting stale (or otherwise unrelated) noise into the docs.
For `Message.DESCRIPTOR`, the change means it gets appropriate documentation.
- There are some wording tweaks for consistency, and some new docstrings
(especially for methods in Message).
For types, I used the convention of `list[Foo]` and `dict(foo, bar)`, which seem
to be common among other Python rst docstrings. Sphinx should generally
recognize both, and cross-links them correctly (both internally and to Python
library documentation). Upgrading to Python3-style type annotations would allow
us to use `sphinx-autodoc-typehints`; the changes in this commit are very
similar to typing-based hints.
Background:
This is a follow-up to the PR that adds sphinx docs.
Read the Docs is a hosting platform for documentation, primarily Python
docs. It supports builds at commit time as well as at specific git
labels to support versioned docs. I have claimed the
protobuf.readthedocs.io project and can add any Googlers who need access
to be able to configure and trigger builds.
https://readthedocs.org/projects/protobuf/builds/ It's also relatively
easy to create a new project to test the documentation builds from a
fork, such as https://readthedocs.org/projects/tswast-protobuf/builds/
About this change:
Once web hooks are configured, Read the Docs will automatically build
the docs for the latest changes on the master branch.
I needed to update `python/setup.py` to support installation from the
root of the repository because Read the Docs does not `cd python` before
installing the protobuf package with `setup.py install`. To support
this, I updated the file paths to use the absolute path to files. The
`__file__` special variable comes in handy for this, as it provides the
path to the `setup.py` file.
A banner is added to the docs when published to readthedocs. This links
to the official documentation and the future home of the stable API
reference on googleapis.dev.
**Background:**
This is a follow-up to the PR that adds sphinx docs.
Googleapis.dev hosts client library documentation for several languages as
well as documentation for core packages such as google-api-core. It is owned
and maintained by the Cloud DevRel Core team.
**About this change:**
The `kokoro/docs/publish-python.sh` script builds the docs with sphinx, then
uses the [googleapis/docuploader](https://github.com/googleapis/docuploader)
tool to publish to googleapis.dev. The publish script is triggered manually
with a Kokoro RELEASE type job. Googlers, see go link
protobuf-docs-python-design for additional internal references.
* python: generate documentation with Sphinx and Read the Docs
Background:
Formerly, the Python protobuf reference documentation was built with
[Epydoc](http://epydoc.sourceforge.net/). This package has not been
updated since 2008, and it has inconsistent formatting (see internal
issue 131415575) with most Python documentation. Sphinx is used for the
official docs.python.org docs as well as most other Python packages,
including the Google client libraries and related packages, such as
https://googleapis.dev/python/google-api-core/latest/
To build the docs with Sphinx:
1. Install the needed packages (`sphinx`, `sphinxcontrib-napoleon` for
Google-style docstring support). I've created a conda environment file
to make this easier:
```
conda env create -f python/docs/environment.yml
```
2. (Optional) Generate reference docs files and regenerate index:
```
cd python
python generate_docs.py
cd ..
```
3. Run Sphinx.
```
cd python/docs
make html
```
About this change:
The script at `python/generate_docs.py` creates a ReStructured Text file
for each public module in the protobuf Python package. The script also
updates the table of contents in `python/docs/index.rst` to point to
these module references.
Future work:
Testing the docs build on PRs requires contributors to actually do some
setup work to configure builds on their fork. It'd be better if CI had a
docs build session to verify that the Sphinx docs generation at least
runs.
There are many warnings due to not-quite-correct docstrings in the
actual Python code itself. I'm choosing to ignore these errors to keep
the PR small, but I recommend you fix these and then enable "fail on
warnings" in the docs build on CI.
* add docs to EXTRA_DIST
* add instructions to build documentation to generate_docs.py
* exclude python/odcs from cpp_distcheck
* Add changelog for 3.9.x
* Revert "Make php message class final to avoid mocking (#6277)" (#6324)
This reverts commit 7f84a94366.
This is just temporary. Eventually, we still want to roll forward this
change. Some users are complaining they need more time to clean up their
code.
* Update extract_includes.bat.in
File io_win32.h is not in directory google\protobuf\stubs under directory google\protobuf\io
* Set oneof case in array constructor (#6351)
Forgot to set it previously.
* Update protobuf version (#6366)
* Drop building wheel for python 3.4 (#6406)
https://github.com/matthew-brett/multibuild/pull/240
* Fix binary compatibility in FieldCodec factory methods (#6380) (#6424)
* Fix binary compatibility in FieldCodec factory messages
* Make default value parameter for current factories required
* Route old methods through default value overloads
* Remove ExtensionRegistry.Add(params) overload
* Rename ExtensionRegistry.Add(IEnumerable<Extension>) overload to AddRange
* Edit naming of parameters in Extension classes
* * Fix add API warnings to docs for extension APIs
* Rename internal ExtensionSet.GetValue to TryGetValue
* Disable javadoc error (#6371)
* Disable javadoc error
Actual fixes of the javadoc will be followed up
* Remove duplicated configuration
* Update javadoc plugin version
* Updated Bazel test script to use most recent Bazel version (#6413) (#6433)
I'm not exactly sure why, but this fixes the failing Bazel presubmit
test. Using the most recent version seems like a good idea anyway so
that we can make sure we're compatible with any new Bazel changes.
* [bazel] Add fixes for --incompatible_load_{cc,java,proto}_rules_from_bzl
* No need to update version in generated code (#6471)
generate_descriptor will handle that
* Update protobuf version (#6472)
Currently, if you access a ListValue from a Struct and attempted to
assign it to another Struct, you would get an exception:
> s1 = spb.Struct()
> s1['a'] = [1]
> s2 = spb.Struct()
> s2['a'] = s1['a']
ValueError: Unexpected type
This fixes that case.
* Fix construction of messages using the C++ Python implementation when a map field is passed from one message to another.
* Add a test on message map field construction
* python 3 support
* review comments
* add test
* Collapse code into one
* Add several fixes for python toolchain
* Fix versin regex
* Make script exit on error
* Fix version regex
* Fix version regex
* Fix version regex
* Fix version regex
* Make test run on the current commit
* Fix test
* Fix test
* Use git to retrieve current commit
* Fix tests
* Fix tests
* Also make linux and mac work on the current commit
* Fix test
* Down-integrate internal changes to github.
* fix python conformance test
* fix csharp conformance test
* add back java map_lite_test.proto's optimize for option
* fix php conformance test
* Rebuild python docker image
* Update docker image
* Change call to build_and_run_docker.sh
* Set up python tests for different versions
* Fix comments
* Add kokoro build for python source package
* Use libc++ for xcode 10 (#5303)
The xcode 10 removes the deprecated libstdc++ library. We could set
"MACOSX_DEPLOYMENT_TARGET" to "10.9" to use libc++ instead.
* Add python 3.7 build
* Add build for python 3.7 on linux and windows
* Remove unused source build
* Add comment
* Fix $MACOSX_DEPLOYMENT_TARGET mismatch
* Fix MACOSX_DEPLOYMENT_TARGET mismatch
* Add missing import for sysconfig
* Add missing imports
* Fix parsing empty Struct Values from Json
This fixes a bug. When parsing a struct from JSON like
struct = json_format.Parse('{"k": {}}', Struct())
then the struct's "k" value would end up not initialized, and accessing
the value would raise an error.
In[1]: struct['k']
ValueError: Value not set
That seems to be because the Struct field of the Value was not set.
In[2]: struct
Out[2]:
fields {
key: "k"
value {
}
}
This commit makes sure that the Value's Struct field is initialized even
if the Struct has no values itself.
This commit also extends a test to cover this case.
* Additionally test for empty list
* ParseFromString returns bytes parsed
ParseFromString is documented as returning the number of bytes parsed,
and the C++ implementation does this, so the Python implementation
should too.
See #5165.
* Fix docstring
* Add a test for ParseFromString length behavior
Many classes within collections were moved to collections.abc in Python
3.3 -- their usage as imported directly from collections is now
deprecated as of Python 3.7 and will be removed soon.
The fallback import added in this PR can be removed entirely when
support for versions of Python prior to 3.3 is dropped.
1. Fix C++ tests.
* Add missing files to Makefile.am and fix distcheck in tests.sh
* Remove BUILT_SOURCES from conformance/Makefile.am.
* Add some missing override keyword.
* Add a type cast to int64 because our StrCat() in stubs can't handle size_t.
2. Fix Java tests.
* Add missing test dependency on guava in pom.xml.
* Include newly referenced test data in test resources.
* Manually fix map_lite_test.proto which is overwritten because it's mapped
from map_test.proto in google3.
* Add back "optimize_for = LITE_RUNTIME" which is still needed to keep the
opensource test passing as it's still running lite tests.
* Add a type cast in newBuilder() because without it the code doesn't compile
with openjdk javac 1.8 (the compiler can't figure if it's the right type
due to complex generic typing).
3. Fix Python tests.
* Remove/replace references to <hash_map>.
* Suppress more warnings in setup.py.
* Replace incorrect header inclusion for google/protobuf/proto_api.h.
* Add strings::EndsWith to google/protobuf/stubs/strutil.h because it's
referenced in the updated python C extension code.
* Replace proto2 with google::protobuf. The proto2 name is leaked to
opensource because we removed the subsitition rule for proto2 namespace
but only fixed C++ source code and forgot to update python C extension code.
Compilation of Python wrappers fails with Python 3.7 because
the Python folks changed their C API such that
PyUnicode_AsUTF8AndSize() now returns a const char* rather
than a char*. Add a patch to work around. Relates #4086.
The compiler and util subpackages are created by the build_py class in
setup.py. This has caused an issue in the protobuf package in
conda-forge (https://github.com/conda-forge/protobuf-feedstock/issues/40),
which is fixed by this commit.
Otherwise, pip complains "cannot find a version satisfies the
requirement." I suspect it was due to caching issues. I usued to
manually add a `sleep 30` between the upload and the installation test
which is quite unstable.
Add _file_desc_by_toplevel_extension back for compatibility issue with 3.4.1.
_file_desc_by_toplevel_extension is a temporary solution for FieldDescriptor.file.
FieldDescriptor.file was added in code gen. However, we need to keep it for some time for compatibility reason (with 3.4.1 only). May try to remove it after 2 years.