Fixed deadlock in producer.send_async #87
Conversation
| producer.sendAsync(msg, callback); | ||
| Py_END_ALLOW_THREADS | ||
|
|
||
| if (PyErr_CheckSignals() == -1) { |
There was a problem hiding this comment.
What if an exception was already set in the interpreter global before the callback fired?
There was a problem hiding this comment.
This is in the main thread, irrespective of the callback.
send_async() is a potentially blocking operation (when block_if_queue_full=True)
There was a problem hiding this comment.
Sure. Something I've seen before is issues around an error already being "pre-raise" in a Python thread when some external code (the pulsar client C++ in this case) decides to invoke an unrelated Python callback. Does pybind handle situations like this one?
There was a problem hiding this comment.
All set per conversation on #84
| @@ -34,6 +34,16 @@ MessageId Producer_send(Producer& producer, const Message& message) { | |||
| return messageId; | |||
| } | |||
|
|
|||
| void Producer_sendAsync(Producer& producer, const Message& msg, SendCallback callback) { | |||
| Py_BEGIN_ALLOW_THREADS | |||
| producer.sendAsync(msg, callback); | |||
There was a problem hiding this comment.
Is something inside this line handling getting the GIL around callback? Since callback is python code, running it outside of the GIL can break all sorts of stuff.
There was a problem hiding this comment.
@zbentley PyBind is already acquiring the GIL before it enters the Python callback code.
There was a problem hiding this comment.
All set per conversation on #84
Fix #84
Release the GIL while calling
producer.sendAsync()to avoid a deadlock when PyBind is triggering the Python callback.Main Thread
producer.send_async()ClientConnetionlockPulsar client internal thread
ClientConnectionThe problem is the different behavior in PyBind from Boost::Python.
We always need to make sure we release the GIL before making any call to C++ that potentially acquires any mutexes