[Python] Release 4.21.0 broke multiple Google Cloud client libraries ("TypeError: Descriptors cannot not be created directly.")

[lgruen]

Example error for the google-cloud-logging==3.1.1 library using protobuf==4.21.0:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.10/site-packages/google/cloud/logging/__init__.py", line 18, in <module>
    from google.cloud.logging_v2 import __version__
  File "/usr/local/lib/python3.10/site-packages/google/cloud/logging_v2/__init__.py", line 25, in <module>
    from google.cloud.logging_v2.client import Client
  File "/usr/local/lib/python3.10/site-packages/google/cloud/logging_v2/client.py", line 25, in <module>
    from google.cloud.logging_v2._helpers import _add_defaults_to_filter
  File "/usr/local/lib/python3.10/site-packages/google/cloud/logging_v2/_helpers.py", line 25, in <module>
    from google.cloud.logging_v2.entries import LogEntry
  File "/usr/local/lib/python3.10/site-packages/google/cloud/logging_v2/entries.py", line 31, in <module>
    import google.cloud.audit.audit_log_pb2  # noqa: F401
  File "/usr/local/lib/python3.10/site-packages/google/cloud/audit/audit_log_pb2.py", line 62, in <module>
    _descriptor.FieldDescriptor(
  File "/usr/local/lib/python3.10/site-packages/google/protobuf/descriptor.py", line 560, in __new__
    _message.Message._CheckCalledFromGeneratedFile()
TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates

Note that this works fine with the previous protobuf==3.20.1 release.

To reproduce:

docker run -it python:3.10 /bin/bash
# Within the Docker container:
pip3 install google-cloud-logging==3.1.1 protobuf==4.21.0
python3 -c "import google.cloud.logging"

[chelseas]

Same issue when using onnx to convert network formats.

[JanuszL]

Same for the TensorFlow 1.15.5.

[gpmahanta]

Same issue for the google-cloud-datastore as well.

[hagen00]

Upgrading the following packages to latest fixed it for me. Not sure which packages actually needed upgrading.

google-api-core==2.8.0
google-auth==2.6.6
google-cloud-core==2.3.0
google-cloud-storage==2.3.0
google-resumable-media==2.3.3
googleapis-common-protos==1.56.1

[stephanie-gillespie]

I've set protobuf==3.20.1 and it resolved the issue

[peterdudfield]

I've set protobuf==3.20.1 and it resolved the issue

Thanks, this also worked for me

[gpmahanta]

I've set protobuf==3.20.1 and it resolved the issue

yes, this thing works but this is not a permanent fix.

Many libraries like google-cloud-datastore are having the dependency on protobuf >= 3.19.0 and most of the platforms like Databricks are already pre-installed with protobuf == 3.17.2 by default which is causing an issue in various platforms and frameworks.

[EduMorenoJ]

setting protobuf==3.20.x or setting PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python works