Is Cereal Thread Safe?

[ssbanerje]

I had a question about the thread safety of cereal. I have the following code, which uses cereal to serialize an int and a std::vector into a cereal::BinaryOutputArchive. It does 10000 of such conversions in parallel.

#include <iostream>
#include <cassert>
#include <vector>
#include <tuple>

#include <tbb/tbb.h>

#include <cereal/cereal.hpp>
#include <cereal/archives/binary.hpp>
#include <cereal/types/vector.hpp>
#include <cereal/types/tuple.hpp>

std::string hexdump(const uint8_t* const begin, const size_t nbytes) {
  std::string str{""};
  const uint8_t* end = begin + nbytes;
  str.reserve((end - begin) * 2);
  const char *hex_chars = "0123456789ABCDEF";
  uint64_t ctr = 0;
  for (const uint8_t* it=begin; it < end; ++it) {
    uint8_t tmp = *it;
    ctr = (ctr + 1) % 16;
    char hex_string[4];
    hex_string[3] = '\0'; hex_string[2] = ' ';
    hex_string[1] = hex_chars[tmp % 16];
    tmp /= 16;
    hex_string[0] = hex_chars[tmp % 16];
    str += hex_string;
    if (ctr == 0) {
      str += "\n";
    }
  }
  return str;
}

class message {
 protected:
  char* buffer_;
  size_t buffer_length_;

 public:
  message() : buffer_(nullptr), buffer_length_(0) {}

  message(size_t length) {
    buffer_length_ = length;
    buffer_ = new char[length];
  }

  ~message() {
    if (buffer_) {
      delete [] buffer_;
      buffer_ = nullptr;
    }
  }

  inline char* buffer() const { return buffer_; }

  inline void set_buffer(char* buf, const int len) { buffer_ = buf; buffer_length_ = len; }

  inline unsigned buffer_length() const { return buffer_length_; }

  inline std::string dump_message() const {
    std::stringstream sstream;
    sstream << hexdump(reinterpret_cast<uint8_t*>(buffer_), buffer_length_);
    return sstream.str();
  }
};


class message_factory {
 public:
  template <typename... Args>
  static message create_message(Args&&... args) {
    std::stringstream stream;
    cereal::BinaryOutputArchive archive(stream);
    archive(std::forward<Args>(args)...)  ;
    auto buffer = stream.str().c_str();
    const size_t buffer_length = stream.str().size();
    message m(buffer_length);
    ::memcpy(m.buffer(), buffer, buffer_length*sizeof(char));
    return m;
  }

  template <typename... Args>
  inline static std::tuple<Args...> read_message_fields(message& mesg) {
    assert(mesg.buffer() != nullptr);
    std::stringstream stream(std::string(mesg.buffer(), mesg.buffer_length()));
    cereal::BinaryInputArchive archive(stream);
    std::tuple<Args...> data;
    archive(data);
    return data;
  }
};




int main() {
  std::vector<int> x{1,2,3,4,5,6,7,8,9,0};
  tbb::task_group g;
  for (int i = 0; i < 10000; i++) {
    g.run([&x] () {
      auto m = message_factory::create_message(1, x);
      auto v = message_factory::read_message_fields<int, std::vector<int> >(m);
      if (std::get<0>(v) != 1 || x != std::get<1>(v)) {
        std::cerr << m.dump_message() << std::endl;
      }
      assert(std::get<0>(v) == 1);
      assert(x == std::get<1>(v));
    });
  }
  g.wait();
  return EXIT_SUCCESS;
}

The output of this code is -

00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0A 00 00 00 03 00 00 00 04 00 00 00 05 00 00 00
06 00 00 00 07 00 00 00 08 00 00 00 09 00 00 00
00 00 00 00
Assertion failed: (std::get<0>(v) == 1), function operator(), file .../test.cc, line 107.
[1]    26153 abort      ./TestApplication

I am guessing this is some sort of race condition as it does not always happen. Also, if I get rid of the threading code, this bug never shows up.

Any idea what is happening here? Is this a cereal bug?

I am using the latest version of clang on OS X

Apple LLVM version 6.1.0 (clang-602.0.53) (based on LLVM 3.6.0svn)
Target: x86_64-apple-darwin14.4.0
Thread model: posix

[AzothAmmo]

cereal does not enforce atomic operations but can easily be made thread safe by using a mutex to control access to the serialization archive.

cereal is completely thread safe when operating on multiple distinct archives with one exception:

anything that touches cereal::StaticObject (mainly encountered when using polymorphism) can potentially be unsafe as these are global objects. See this discussion list response for more information.

[ssbanerje]

Thanks @AzothAmmo I dont think I am hitting either of those situations. In any case I will look into this.

[ChrisBFX]

Sorry for opening this again, but we run into this Issue in our project.
Versions::find(hash, version) is not thread-safe. (std::unordered_map::emplace is the problem).
This is actually a big problem for us. Are there any official plans to make the library thread safe?

[AzothAmmo]

It should be fairly easy to make cereal thread safe as there are only a few places where global state is accessed (we can't easily get rid of this global state). I think a good solution would be to have a preprocessor flag conditionally compile in a mutex/lock around these areas, so people not needing the thread safety won't be hit by a performance penalty.