Document the best way to serialize/deserialize user defined types to json

[gnzlbg]

It would be nice to add an example to the documentation about the best ways to make an user defined type serializable/deserializable to json. For example, given:

struct point {
  float x[3];
};

json v = point{{0., 1., 2.}};
assert(v.is_array());
assert(v[0] == 0.0 and v[1] == 1.0 and v[2] == 2.0);

point p = v; 
assert(p.x[0] == 0.0 and p.x[1] == 1.0 and p.x[2] == 2.0);

• Is it possible to make json serialization/deserialization without modifying point itself?
• Are there any customization points to allow this?

For example, to serialize point I think it should probably be enough to:

• define non-member non-friend begin and end functions that are found by ADL.
• have this basic_json array constructor use decltype(begin(std::declval<point>())), decltype(end(std::declval<point>())) to find the iterator types, and std::iterator_traits<T> to find the rest of the required types (instead of typename T::value_type) .

Such that adding:

float*       begin(point& p)       { return p.x; }
float const* begin(point const& p) { return p.x; }
float*       end(point& p)       { return p.x + 3; }
float const* end(point const& p) { return p.x + 3; }

to point's namespace should be enough to get it to work. Right now it fails because (for some reason), point is required to have iterator, const_iterator, and value_type type members. But since those can be obtained from point iterators by using std::iterator_traits<decltype(begin(std::declval<point>()))> they are not really necessary.

For other types that are not arrays maybe a couple of customization points could be added (or documented). That way it will be enough to add to_json/from_json overloads to the types namespace to allow them being found by argument dependent lookup.

[nlohmann]

I currently have not looked into this. Though I haven't used it yet, it seems cereal is the library to check out if you want to serialized to JSON with C++11. As you pointed out, there may be a recipe for doing this with this library.

[gnzlbg]

I am currently using this already for array-like/container-like user-defined types (by modifying my types slightly to make them look like containers).

If you are interested in having this functionality in the library let me know and I will come up with a proposal that we can refine first here before I start implementing it.

The rough idea would be to:

• add two customization points (as recommended in N4381), e.g. to_json / from_json (straw man names, we can bikeshed this later) to the library which call some to_json/from_json free functions
• provide some free functions for common types (arrays, containers, ...). These are already implemented in the library, the work here would be to move them to free functions such that they get picked up by the customization point mechanism.

And hopefully that will be enough (worst case we will need to add a type trait that users can specialize, but I hope this won't be necessary).

In a nutshell, when a user tries to assign an object to/from a json object, the customization points will be called by the basic_json constructors/conversion operator.

We have then two different cases:

• types the library knows about (e.g. std::vector). In this case we need to provide free functions for them in our namespace nlohmann::json (or some detail namespace). In practice for this example we would just have one single function for all containers with begin/end as we do now.
• types the library doesn't know about (e.g. user-defined types, optional, ...). In this case the user needs to add the two free functions to the type's namespace (without modifying the type), which allows them to be found by argument dependent look up.

[nlohmann]

Nice idea! Are you aware of approaches like https://github.com/Loki-Astari/ThorsSerializer/blob/master/doc/full.md which seem to have found also a simple way for serialization?

[gnzlbg]

Yes, Boost.Hana and Boost.Fusion allow to do it this way by adapting types to become tuple sequences. (e.g. see BOOST_FUSION_ADAPT_STRUCT_... macros). These work for simple structs fine, but if you need more complex initialization, at the end of the day you are going to have to write the from_/to_ logic somewhere. For example if you have:

struct A {
  float a;
  float a_squared;
}; 

and want to serialize only a and compute a_squared on deserialization.

It can be done, but it makes those approaches less nice (they are very nice for the simplest case though).

The second downside is that the macros hide a "medium" amount of metaprogramming, typically involving mapping the structs to a tuple, and then using tuple algorithms to loop over the tuple. It's not hard, but is not simple either (EDIT: what I mean here is that we either use a library that provides the tuple algorithms like Hana or range-v3 or some parts of them, or we need to provide at least a reimplementation of them somewhere which will be about 500-1000 LOC + tests and is worth keeping in mind).

Finally, the customization point approach allows the users to do serialization however they want. If they want to use any of those libraries, they can! But it doesn't tie them to any particular library. For example, something like:

namespace foo {

struct A {  
  int x;
  float y;
}; 

BOOST_FUSION_ADAPT_STRUCT( // I forgot the exact syntax
    A, 
    A::x, "x", 
    A::y, "y"
);  

void to_json(A const& a, basic_json& j) {
    boost::fusion::for_each(a, [&j](auto&& i) {  // loops over every element of A
         // i is a pair (reference to value, key const*)
         j[i.second] = i.first; 
    }); 
}

void from_json(A& a, basic_json& j) {
    boost::fusion::for_each(a, [&j](auto&& i) { i.first = j[i.second]; });
}

}  // namespace foo

EDIT: So with the customization points what I want is to allow users to easily make their types work like JSON objects. For that they are going to need to serialize/deserialize their types, but I think it would be out of scope for this library to provide any sort of serialization mechanism. There are many ways to do that, many libraries that do this already, and it is not a trivial problem. Some users might be already using some library for this in their projects, so it would be nice if they can reuse that easily.

[d-frey]

We had a similar problem in our JSON library taocpp/json, our solution was using a traits class as a template parameter. This allows easy customization for further types, but also, by replacing the traits class, you can even overwrite the default behavior of the standard types.

For example, check out the customization for std::optional:

  template< typename T >
  struct traits< optional< T > >
  {
     template< template< typename ... > class Traits >
     static void assign( basic_value< Traits > & v, const optional< T > & o ) noexcept
     {
        if( o ) {
           v = * o;
        }
        else {
           v = null;
        }
     }

     template< template< typename ... > class Traits >
     static void extract( const basic_value< Traits > & v, optional< T > & o )
     {
        if( v.is_null() ) {
           o = nullopt;
        }
        else {
           o = v.template as< T >();
        }
     }
  };

We have code bases with about 100 registered UDTs and it works pretty well for us.

Feel free to contact us if you have any questions.

[gnzlbg]

One problem with the trait approach as commented in issue #328 is that partial template specialization makes it a bit harder to add implementations for "sets of types" constrained by some predicate. For example, by using free functions one can use std::enable_if to easily add an implementation for OptionalLike objects that works on std::optional, boost::optional, and anything implementing the optional interface. However, doing so with partial template specialization, while possible, does requires some "harder" tricks, like using void_t, which would require the trait class to take a second dummy default template parameter.

I've found that free functions and function overloading are easier to explain/implement/use than trait classes and partial template specialization, and also make it easier to do more complicated things like overloading based on a predicate.

[d-frey]

@gnzlbg Please see my other comment in issue #328 on how to solve the enable_if-problem for a traits class.

[rianquinn]

To support user defined types, we did template partial specialization as well here: QMJson

Our library is light years behind this one (which is why we are using this one for Bareflank), but it did work well. The down side with our approach is that it did require a to_/from_ implementation that also required registration similar to how Qt does it, but with that approach, we could basically handle any custom types we wanted.

[nlohmann]

FYI: A nice serialization/deserialization for arbitrary types is now implemented: see #328.

[tobocop2]

If anyone is trying to use this library to serialize/de-serialize types in an arbitrary namespace, I threw this together with heavy inspiration from @gnzlbg

The boost stuff is poorly documented so I figured I'd share with anyone else who comes across this issue.

If you are able to use the latest release and can build with c++14, this should do the job:

// fusion_jsonify.h

#include <boost/fusion/include/is_sequence.hpp>
#include <boost/fusion/include/algorithm.hpp>
#include <boost/fusion/include/adapt_struct.hpp>
#include <boost/mpl/range_c.hpp>

#include <nlohmann/json.hpp>

#define FUSION_JSONIFY()                                                                                           \
  template<typename T,                                                                                             \
           typename = typename std::enable_if<                                                                     \
               boost::fusion::traits::is_sequence<T>::value                                                        \
           >::type>                                                                                                \
  void to_json(nlohmann::json& j, T const& t)                                                                      \
  {                                                                                                                \
      boost::fusion::for_each(                                                                                     \
          boost::mpl::range_c<unsigned, 0, boost::fusion::result_of::size<T>::value>(),                            \
          [&](auto index)                                                                                          \
          {                                                                                                        \
              j[boost::fusion::extension::struct_member_name<T,index>::call()] =                                   \
                  boost::fusion::at_c<index>(t);                                                                   \
          }                                                                                                        \
      );                                                                                                           \
                                                                                                                   \
  }                                                                                                                \
                                                                                                                   \
  template<typename T,                                                                                             \
           typename = typename std::enable_if<                                                                     \
               boost::fusion::traits::is_sequence<T>::value                                                        \
           >::type>                                                                                                \
  void from_json(const nlohmann::json& j, T &t)                                                                    \
  {                                                                                                                \
      boost::fusion::for_each(                                                                                     \
          boost::mpl::range_c<unsigned, 0, boost::fusion::result_of::size<T>::value>(),                            \
          [&](auto index)                                                                                          \
          {                                                                                                        \
              using member_type = typename boost::fusion::result_of::value_at<T, boost::mpl::int_<index> >::type;  \
              boost::fusion::at_c<index>(t) =                                                                      \
                  j.at(boost::fusion::extension::struct_member_name<T,index>::call()).template get<member_type>(); \
          }                                                                                                        \
      );                                                                                                           \
  }

// foo_jsonify.h
#include "fusion_jsonify.h"

// adds to_json and from_json for all structures defined in namespace Foo
namespace Foo
{
    FUSION_JSONIFY()
}

Any boost adapted struct would then get the json functions for free wherever the above header is included.