python CopyFrom segmentation fault on version 4.24.0

[sfc-gh-mussakli]

What version of protobuf and what language are you using?
Version: v4.24.0
Language: Python upb

What operating system (Linux, Windows, ...) and version?
Repros on Linux and MacOS

What runtime / compiler are you using (e.g., python version or gcc version)
python 3.8
Installed Apple clang version 14.0.3 (clang-1403.0.22.14.1)

What did you do?
We have a proto message that fundamentally looks like this:

syntax = "proto3";

package message;

message TestMessage {
    optional int64 field_one = 1;
    optional int64 field_two = 2;
    map<int32, double> map_field = 3;
}

And we have a piece of python that tries to copy an empty version of one of these messages into the other:

message = message_pb2.TestMessage()
message2 = message_pb2.TestMessage()
message2.CopyFrom(message)

This is resulting on a segmentation fault on exactly the CopyFrom line.

This failure started with us picking up version 4.24.0 with yesterday's release. Reverting back to any version prior fixes the issue.

Unfortunately building the above example proto as an MVP did not repro the segmentation fault for me. If there are any diagnostics I can provide that would make root causing this easier, happy to help.

What did you expect to see
No seg fault, behavior as before.

What did you see instead?
Segmentation fault failure on CopyFrom

Make sure you include information that can help us debug (full error message, exception listing, stack trace, logs).

Anything else we should know about your project / environment

[arcra]

I've seen this issue with python 3.9 and 3.11 as well. It's blocking our project's CI runs.

[anandolee]

Not sure why but I can NOT reproduce in google3 internal CL with

message TestMessage {
    optional int64 field_one = 1;
    optional int64 field_two = 2;
    map<int32, double> map_field = 3;
}

message = message_pb2.TestMessage()
message2 = message_pb2.TestMessage()
message2.CopyFrom(message)

Nor opensource on my linux:

$ pip show protobuf
Name: protobuf
Version: 4.24.0
$ python
Python 3.11.4 (main, Jun  7 2023, 10:13:09) [GCC 12.2.0] on linux
>>> from google.protobuf.internal import api_implementation
>>> print(api_implementation.Type())
upb
>>> from google.protobuf import timestamp_pb2
>>> msg = timestamp_pb2.Timestamp()
>>> msg2 = timestamp_pb2.Timestamp()
>>> msg2.CopyFrom(msg)
>>> from google.protobuf import struct_pb2
>>> msg = struct_pb2.Struct()
>>> msg2 =  struct_pb2.Struct()
>>> msg2.CopyFrom(msg)

[arcra]

Just a thought, could it be that the protos that we're intending to use are "compiled" with a previous protoc version, and the issue is that this new version throws in that case? In our project, it was failing when trying to use some protos from a TensorFlow dependency (which was released before this new protobuf release from yesterday).

[anandolee]

Can you help to verify:
1, Does the segment fault still raise if not depend on TensorFlow.
2, Does the segment fault still raise for the message that do not have map:

message TestMessage {
    optional int64 field_one = 1;
    optional int64 field_two = 2;
}

message = message_pb2.TestMessage()
message2 = message_pb2.TestMessage()
message2.CopyFrom(message)

3, Does the segment fault still raise for struct_pb2.Struct()

from google.protobuf import struct_pb2
msg = struct_pb2.Struct()
msg2 =  struct_pb2.Struct()
msg2.CopyFrom(msg)

[sfc-gh-mussakli]

Just a thought, could it be that the protos that we're intending to use are "compiled" with a previous protoc version, and the issue is that this new version throws in that case? In our project, it was failing when trying to use some protos from a TensorFlow dependency (which was released before this new protobuf release from yesterday).

This might actually be possibly how it's happening, as I could not produce a simple repro outside of our internal product.

Can you help to verify: 1, Does the segment fault still raise if not depend on TensorFlow. 2, Does the segment fault still raise for the message that do not have map:

message TestMessage {
    optional int64 field_one = 1;
    optional int64 field_two = 2;
}

message = message_pb2.TestMessage()
message2 = message_pb2.TestMessage()
message2.CopyFrom(message)

3, Does the segment fault still raise for struct_pb2.Struct()

from google.protobuf import struct_pb2
msg = struct_pb2.Struct()
msg2 =  struct_pb2.Struct()
msg2.CopyFrom(msg)

I am working to answer your other questions, but I can confirm (1) that we do not depend on TensorFlow.

[sfc-gh-mussakli]

Can you help to verify: 1, Does the segment fault still raise if not depend on TensorFlow. 2, Does the segment fault still raise for the message that do not have map:

message TestMessage {
    optional int64 field_one = 1;
    optional int64 field_two = 2;
}

message = message_pb2.TestMessage()
message2 = message_pb2.TestMessage()
message2.CopyFrom(message)

3, Does the segment fault still raise for struct_pb2.Struct()

from google.protobuf import struct_pb2
msg = struct_pb2.Struct()
msg2 =  struct_pb2.Struct()
msg2.CopyFrom(msg)

Confirming for (3) that this does not fail with struct_pb2.Struct() and removing for (2), the segmentation fault still occurs with no map.