Feature/local exception translator

docs/advanced/exceptions.rst

@@ -75,9 +75,10 @@ Registering custom translators
 
 If the default exception conversion policy described above is insufficient,
 pybind11 also provides support for registering custom exception translators.
-To register a simple exception conversion that translates a C++ exception into
-a new Python exception using the C++ exception's ``what()`` method, a helper
-function is available:
+Similar to pybind11 classes, exception translators can be local to the module
+they are defined in or global to the entire python session.  To register a simple
+exception conversion that translates a C++ exception into a new Python exception
+using the C++ exception's ``what()`` method, a helper function is available:
 
 .. code-block:: cpp
 
@@ -87,29 +88,39 @@ This call creates a Python exception class with the name ``PyExp`` in the given
 module and automatically converts any encountered exceptions of type ``CppExp``
 into Python exceptions of type ``PyExp``.
 
+A matching function is available for registering a local exception translator:
+
+.. code-block:: cpp
+
+    py::register_local_exception<CppExp>(module, "PyExp");
+
+
 It is possible to specify base class for the exception using the third
 parameter, a `handle`:
 
 .. code-block:: cpp
 
     py::register_exception<CppExp>(module, "PyExp", PyExc_RuntimeError);
+    py::register_local_exception<CppExp>(module, "PyExp", PyExc_RuntimeError);
 
 Then `PyExp` can be caught both as `PyExp` and `RuntimeError`.
 
 The class objects of the built-in Python exceptions are listed in the Python
 documentation on `Standard Exceptions <https://docs.python.org/3/c-api/exceptions.html#standard-exceptions>`_.
 The default base class is `PyExc_Exception`.
 
-When more advanced exception translation is needed, the function
-``py::register_exception_translator(translator)`` can be used to register
+When more advanced exception translation is needed, the functions
+``py::register_exception_translator(translator)`` and
+``py::register_local_exception_translator(translator)`` can be used to register
 functions that can translate arbitrary exception types (and which may include
-additional logic to do so).  The function takes a stateless callable (e.g.  a
+additional logic to do so).  The functions takes a stateless callable (e.g. a
 function pointer or a lambda function without captured variables) with the call
 signature ``void(std::exception_ptr)``.
 
 When a C++ exception is thrown, the registered exception translators are tried
 in reverse order of registration (i.e. the last registered translator gets the
-first shot at handling the exception).
+first shot at handling the exception). All local translators will be tried
+before a global translator is tried.
 
 Inside the translator, ``std::rethrow_exception`` should be used within
 a try block to re-throw the exception.  One or more catch clauses to catch
@@ -168,6 +179,56 @@ section.
     with ``-fvisibility=hidden``. Therefore exceptions that are used across ABI boundaries need to be explicitly exported, as exercised in ``tests/test_exceptions.h``.
     See also: "Problems with C++ exceptions" under `GCC Wiki <https://gcc.gnu.org/wiki/Visibility>`_.
 
+
+Local vs Global Exception Translators
+=====================================
+
+Similar to how the ``py::module_local`` flag allows uesrs to limit a bound class
+to a single compiled module to prevent name collisions. When a global exception
+translator is registered, it will be applied across all modules in the reverse
+order of registration. This can create behavior where the order of module import
+influences how exceptions are translated.
+
+If module1 has the following translator:
+
+.. code-block:: cpp
+
+      py::register_exception_translator([](std::exception_ptr p) {
+        try {
+            if (p) std::rethrow_exception(p);
+        } catch (const std::invalid_argument &e) {
+            PyErr_SetString("module1 handled this")
+        }
+      }
+
+and module2 has the following similar translator:
+
+.. code-block:: cpp
+
+      py::register_exception_translator([](std::exception_ptr p) {
+        try {
+            if (p) std::rethrow_exception(p);
+        } catch (const std::invalid_argument &e) {
+            PyErr_SetString("module2 handled this")
+        }
+      }
+
+then which translator handles the invalid_argument will be determined by the
+order that module1 and module2 are imported. Since exception translators are
+applied in the reverse order of registration, which ever module was imported
+last will "win" and that translator will be applied.
+
+If there are multiple pybind11 modules that share exception types (either
+standard built-in or custom) loaded into a single python instance and
+consistent error handling behavior is needed, then local translators should be
+used.
+
+Changing the previous example to use register_local_exception_translator would
+mean that when invalid_argument is thrown in the module2 code, the module2
+translator will always handle it, while in module1, the module1 translator will
+do the same.
+>>>>>>> 854aa95a (Update documentation for new local exception feature)
+
 .. _handling_python_exceptions_cpp:
 
 Handling exceptions from Python in C++

include/pybind11/detail/class.h

@@ -209,7 +209,7 @@ extern "C" inline void pybind11_meta_dealloc(PyObject *obj) {
         internals.direct_conversions.erase(tindex);
 
         if (tinfo->module_local)
-            registered_local_types_cpp().erase(tindex);
+            get_local_internals().registered_local_types_cpp.erase(tindex);
         else
             internals.registered_types_cpp.erase(tindex);
         internals.registered_types_py.erase(tinfo->type);

include/pybind11/detail/internals.h

@@ -313,12 +313,19 @@ PYBIND11_NOINLINE inline internals &get_internals() {
     return **internals_pp;
 }
 
-/// Works like `internals.registered_types_cpp`, but for module-local registered types:
-inline type_map<type_info *> &registered_local_types_cpp() {
-    static type_map<type_info *> locals{};
-    return locals;
+
+struct local_internals {
+  type_map<type_info *> registered_local_types_cpp;
+  std::forward_list<void (*) (std::exception_ptr)> registered_local_exception_translators;
+};
+
+/// Works like `get_internals`, but for things which are locally registered.
+inline local_internals &get_local_internals() {
+  static local_internals locals;
+  return locals;
 }
 
+
 /// Constructs a std::string with the given arguments, stores it in `internals`, and returns its
 /// `c_str()`.  Such strings objects have a long storage duration -- the internal strings are only
 /// cleared when the program exits or after interpreter shutdown (when embedding), and so are

include/pybind11/detail/type_caster_base.h

@@ -160,7 +160,7 @@ PYBIND11_NOINLINE inline detail::type_info* get_type_info(PyTypeObject *type) {
 }
 
 inline detail::type_info *get_local_type_info(const std::type_index &tp) {
-    auto &locals = registered_local_types_cpp();
+    auto &locals = get_local_internals().registered_local_types_cpp;
     auto it = locals.find(tp);
     if (it != locals.end())
         return it->second;

include/pybind11/pybind11.h

@@ -840,8 +840,12 @@ class cpp_function : public function {
 #endif
         } catch (...) {
             /* When an exception is caught, give each registered exception
-               translator a chance to translate it to a Python exception
-               in reverse order of registration.
+               translator a chance to translate it to a Python exception. First
+               all module-local translators will be tried in reverse order of
+               registration. If none of the module-locale translators handle
+               the exception (or there are no module-locale translators) then
+               the global translators will be tried, also in reverse order of
+               registration.
 
                A translator may choose to do one of the following:
 
@@ -851,6 +855,18 @@ class cpp_function : public function {
                 - delegate translation to the next translator by throwing a new type of exception. */
 
             auto last_exception = std::current_exception();
+
+            auto & registered_local_exception_translators = get_local_internals().registered_local_exception_translators;
+            for (auto& translator : registered_local_exception_translators) {
+                try {
+                    translator(last_exception);
+                } catch (...) {
+                    last_exception = std::current_exception();
+                    continue;
+                }
+                return nullptr;
+            }
+
             auto &registered_exception_translators = get_internals().registered_exception_translators;
             for (auto& translator : registered_exception_translators) {
                 try {
@@ -1134,7 +1150,7 @@ class generic_type : public object {
         auto tindex = std::type_index(*rec.type);
         tinfo->direct_conversions = &internals.direct_conversions[tindex];
         if (rec.module_local)
-            registered_local_types_cpp()[tindex] = tinfo;
+            get_local_internals().registered_local_types_cpp[tindex] = tinfo;
         else
             internals.registered_types_cpp[tindex] = tinfo;
         internals.registered_types_py[(PyTypeObject *) m_ptr] = { tinfo };
@@ -1320,7 +1336,7 @@ class class_ : public detail::generic_type {
         generic_type::initialize(record);
 
         if (has_alias) {
-            auto &instances = record.module_local ? registered_local_types_cpp() : get_internals().registered_types_cpp;
+            auto &instances = record.module_local ? get_local_internals().registered_local_types_cpp : get_internals().registered_types_cpp;
             instances[std::type_index(typeid(type_alias))] = instances[std::type_index(typeid(type))];
         }
     }
@@ -2026,6 +2042,19 @@ void register_exception_translator(ExceptionTranslator&& translator) {
         std::forward<ExceptionTranslator>(translator));
 }
 
+
+/**
+  * Add a new module-local exception translator. Locally registered functions
+  * will be tried before any globally registered exception translators, which
+  * will only be invoked if the moduke-local handlers do not deal with
+  * the exception.
+  */
+template <typename ExceptionTranslator>
+void register_local_exception_translator(ExceptionTranslator&& translator) {
+    detail::get_local_internals().registered_local_exception_translators.push_front(
+        std::forward<ExceptionTranslator>(translator));
+}
+
 /**
  * Wrapper to generate a new Python exception type.
  *
@@ -2085,6 +2114,32 @@ exception<CppException> &register_exception(handle scope,
     return ex;
 }
 
+/**
+ * Registers a Python exception in `m` of the given `name` and installs an exception translator to
+ * translate the C++ exception to the created Python exception using the exceptions what() method.
+ * This translator will only be used for exceptions that are thrown in this module and will be
+ * tried before global exception translators, including those registered with register_exception.
+ * This is intended for simple exception translations; for more complex translation, register the
+ * exception object and translator directly.
+ */
+template <typename CppException>
+exception<CppException> &register_local_exception(handle scope,
+                                                  const char *name,
+                                                  handle base = PyExc_Exception) {
+    auto &ex = detail::get_exception_object<CppException>();
+    if (!ex) ex = exception<CppException>(scope, name, base);
+
+    register_local_exception_translator([](std::exception_ptr p) {
+        if (!p) return;
+        try {
+            std::rethrow_exception(p);
+        } catch (const CppException &e) {
+            detail::get_exception_object<CppException>()(e.what());
+        }
+    });
+    return ex;
+}
+
 PYBIND11_NAMESPACE_BEGIN(detail)
 PYBIND11_NOINLINE inline void print(const tuple &args, const dict &kwargs) {
     auto strings = tuple(args.size());

tests/local_bindings.h

@@ -35,6 +35,25 @@ using NonLocalVec2 = std::vector<NonLocal2>;
 using NonLocalMap = std::unordered_map<std::string, NonLocalType>;
 using NonLocalMap2 = std::unordered_map<std::string, uint8_t>;
 
+
+// Exception that will be caught via the module local translator.
+class LocalException : public std::exception {
+public:
+    explicit LocalException(const char * m) : message{m} {}
+    const char * what() const noexcept override {return message.c_str();}
+private:
+    std::string message = "";
+};
+
+// Exception that will be registered with register_local_exception_translator
+class LocalSimpleException : public std::exception {
+public:
+    explicit LocalSimpleException(const char * m) : message{m} {}
+    const char * what() const noexcept override {return message.c_str();}
+private:
+    std::string message = "";
+};
+
 PYBIND11_MAKE_OPAQUE(LocalVec);
 PYBIND11_MAKE_OPAQUE(LocalVec2);
 PYBIND11_MAKE_OPAQUE(LocalMap);

tests/pybind11_cross_module_tests.cpp

@@ -29,11 +29,14 @@ PYBIND11_MODULE(pybind11_cross_module_tests, m) {
     bind_local<ExternalType2>(m, "ExternalType2", py::module_local());
 
     // test_exceptions.py
+    py::register_local_exception<LocalSimpleException>(m, "LocalSimpleException");
     m.def("raise_runtime_error", []() { PyErr_SetString(PyExc_RuntimeError, "My runtime error"); throw py::error_already_set(); });
     m.def("raise_value_error", []() { PyErr_SetString(PyExc_ValueError, "My value error"); throw py::error_already_set(); });
     m.def("throw_pybind_value_error", []() { throw py::value_error("pybind11 value error"); });
     m.def("throw_pybind_type_error", []() { throw py::type_error("pybind11 type error"); });
     m.def("throw_stop_iteration", []() { throw py::stop_iteration(); });
+    m.def("throw_local_error", []() { throw LocalException("just local"); });
+    m.def("throw_local_simple_error", []() { throw LocalSimpleException("external mod"); });
     py::register_exception_translator([](std::exception_ptr p) {
       try {
           if (p) std::rethrow_exception(p);
@@ -42,6 +45,17 @@ PYBIND11_MODULE(pybind11_cross_module_tests, m) {
       }
     });
 
+    // translate the local exception into a key error but only in this module
+    py::register_local_exception_translator([](std::exception_ptr p) {
+      try {
+          if (p) {
+            std::rethrow_exception(p);
+          }
+      } catch (const LocalException &e) {
+        PyErr_SetString(PyExc_KeyError, e.what());
+      }
+    });
+
     // test_local_bindings.py
     // Local to both:
     bind_local<LocalType, 1>(m, "LocalType", py::module_local())
@@ -94,7 +108,7 @@ PYBIND11_MODULE(pybind11_cross_module_tests, m) {
     m.def("get_mixed_lg", [](int i) { return MixedLocalGlobal(i); });
 
     // test_internal_locals_differ
-    m.def("local_cpp_types_addr", []() { return (uintptr_t) &py::detail::registered_local_types_cpp(); });
+    m.def("local_cpp_types_addr", []() { return (uintptr_t) &py::detail::get_local_internals().registered_local_types_cpp; });
 
     // test_stl_caster_vs_stl_bind
     py::bind_vector<std::vector<int>>(m, "VectorInt");

tests/test_exceptions.cpp

@@ -6,9 +6,10 @@
     All rights reserved. Use of this source code is governed by a
     BSD-style license that can be found in the LICENSE file.
 */
-
 #include "test_exceptions.h"
 
+#include "local_bindings.h"
+
 #include "pybind11_tests.h"
 #include <utility>
 
@@ -68,6 +69,17 @@ class MyException5_1 : public MyException5 {
     using MyException5::MyException5;
 };
 
+
+// Exception that will be caught via the module local translator.
+class MyException6 : public std::exception {
+public:
+    explicit MyException6(const char * m) : message{m} {}
+    const char * what() const noexcept override {return message.c_str();}
+private:
+    std::string message = "";
+};
+
+
 struct PythonCallInDestructor {
     PythonCallInDestructor(const py::dict &d) : d(d) {}
     ~PythonCallInDestructor() { d["good"] = true; }
@@ -138,14 +150,29 @@ TEST_SUBMODULE(exceptions, m) {
     // A slightly more complicated one that declares MyException5_1 as a subclass of MyException5
     py::register_exception<MyException5_1>(m, "MyException5_1", ex5.ptr());
 
+    //py::register_local_exception<LocalSimpleException>(m, "LocalSimpleException")
+
+    py::register_local_exception_translator([](std::exception_ptr p) {
+      try {
+          if (p) {
+            std::rethrow_exception(p);
+          }
+      } catch (const MyException6 &e) {
+        PyErr_SetString(PyExc_RuntimeError, e.what());
+      }
+    });
+
     m.def("throws1", []() { throw MyException("this error should go to a custom type"); });
     m.def("throws2", []() { throw MyException2("this error should go to a standard Python exception"); });
     m.def("throws3", []() { throw MyException3("this error cannot be translated"); });
     m.def("throws4", []() { throw MyException4("this error is rethrown"); });
     m.def("throws5", []() { throw MyException5("this is a helper-defined translated exception"); });
     m.def("throws5_1", []() { throw MyException5_1("MyException5 subclass"); });
+    m.def("throws6", []() { throw MyException6("MyException6 only handled in this module"); });
     m.def("throws_logic_error", []() { throw std::logic_error("this error should fall through to the standard handler"); });
-    m.def("throws_overflow_error", []() {throw std::overflow_error(""); });
+    m.def("throws_overflow_error", []() { throw std::overflow_error(""); });
+    m.def("throws_local_error", []() { throw LocalException("never caught"); });
+    m.def("throws_local_simple_error", []() { throw LocalSimpleException("this mod"); });
     m.def("exception_matches", []() {
         py::dict foo;
         try {