Serialization and Marshaling (again)

[Affie]

Just capturing ideas for discussion.

History

Another thing, we actively decided after a 2 year process not to define the "schema" for serialization. We originally had a protobuf packing approach which was hard schema type dependent and which supported forward-backward multilanguage compatibility. After much pain, we decided that top level packing MUST be JSON, and that the JSON string must at top level include the format in which the data is serialized. This is to mimic how the Internet has standardized it's data exchange format. So StructTypes.jl might be a real problem given the history on serialization.
So to stress, its probably not a good idea going down a path to unilaterally "change everything" on serialization.
In this case, it will be better to understand what the problems are with JSON.jl JSON2.jl JSON3.jl, because clearly they haven't figured it all out either. But we likely won't go down writing JSON4.jl either.
Also remember that the whole point of JSON is that serialization should be marginally or completely available outside Julia. The ZMQ interface for example is about multilanguage support and it uses the same serialization. The DB storage is definitely not a Julia implementation, later we will add a FuseDFG driver, etc, etc. A little while back we came up with the requirement that ZMQ messages should be exactly the same as FileDFG content. The schema discussion becomes a whole issue there. There are 100 different schema's available... So serialization only has 2nd best answers for everything. Whenever you pick the "best" answer for one requirement, then all others get real upset. So we have to accept a 2nd best solution for everybody and JSON is the best bet at this time.

So JSON

Options

JSON.jl + unmarshal.jl

...

JSON2.jl

Deprecated in favor of JSON3

JSON3.jl

See this for the 2 trait options: Struct() or Mutable()
In short Struct() is fast, but the order is important (So has to be preserved if created/modfied from outside).

Example:

julia> using JSON3
julia> using DistributedFactorGraphs
julia> using TimeZones
julia> using UUIDs
julia> bse = BlobStoreEntry(:a, uuid4(), :abc, "", "", "", "", ZonedDateTime("2020-08-11T14:32:27.000+02:00"))
BlobStoreEntry(:a, UUID("a533d151-764a-425d-9e57-a8f194577866"), :abc, "", "", "", "", ZonedDateTime(2020, 8, 11, 14, 32, 27, tz"UTC+02:00"))
julia> JSON3.StructTypes.StructType(::Type{BlobStoreEntry}) = JSON3.StructTypes.Struct()
julia> json3_str = JSON3.write(bse)
"{\"label\":\"a\",\"id\":\"a533d151-764a-425d-9e57-a8f194577866\",\"blobstore\":\"abc\",\"hash\":\"\",\"origin\":\"\",\"description\":\"\",\"mimeType\":\"\",\"createdTimestamp\":\"2020-08-11T14:32:27.000+02:00\"}"
julia> JSON3.read(json3_str, BlobStoreEntry)
BlobStoreEntry(:a, UUID("a533d151-764a-425d-9e57-a8f194577866"), :abc, "", "", "", "", ZonedDateTime(2020, 8, 11, 14, 32, 27, tz"UTC+02:00"))

[dehann]

To answer your earlier question, here is the first issue:

julia> bse = BlobStoreEntry(:a, uuid4(), :abc, "", "", "", "", ZonedDateTime("2020-08-11T14:32:27.12+02:00"))
ERROR: ArgumentError: Unable to parse string "2020-08-11T14:32:27.12+02:00" using format dateformat"yyyy-mm-ddTHH:MM:SS.ssszzz". Unable to parse date time. Expected directive DatePart(sss) at char 21
Stacktrace:
 [1] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Dates/src/parse.jl:104 [inlined]
 [2] tryparsenext_core(::String, ::Int64, ::Int64, ::Dates.DateFormat{Symbol("yyyy-mm-ddTHH:MM:SS.ssszzz"),Tuple{Dates.DatePart{'y'},Dates.Delim{Char,1},Dates.DatePart{'m'},Dates.Delim{Char,1},Dates.DatePart{'d'},Dates.Delim{Char,1},Dates.DatePart{'H'},Dates.Delim{Char,1},Dates.DatePart{'M'},Dates.Delim{Char,1},Dates.DatePart{'S'},Dates.Delim{Char,1},Dates.DatePart{'s'},Dates.DatePart{'z'}}}, ::Bool) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Dates/src/parse.jl:38
 [3] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Dates/src/parse.jl:150 [inlined]
 [4] tryparsenext_internal at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Dates/src/parse.jl:125 [inlined]
 [5] parse at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Dates/src/parse.jl:282 [inlined]
 [6] ZonedDateTime(::String, ::Dates.DateFormat{Symbol("yyyy-mm-ddTHH:MM:SS.ssszzz"),Tuple{Dates.DatePart{'y'},Dates.Delim{Char,1},Dates.DatePart{'m'},Dates.Delim{Char,1},Dates.DatePart{'d'},Dates.Delim{Char,1},Dates.DatePart{'H'},Dates.Delim{Char,1},Dates.DatePart{'M'},Dates.Delim{Char,1},Dates.DatePart{'S'},Dates.Delim{Char,1},Dates.DatePart{'s'},Dates.DatePart{'z'}}}) at /home/dehann/.julia/packages/TimeZones/v0mfN/src/parse.jl:88
 [7] ZonedDateTime(::String) at /home/dehann/.julia/packages/TimeZones/v0mfN/src/parse.jl:87
 [8] top-level scope at REPL[8]:1

EDIT: we can hodge podge this together using something similar to function in #589

[dehann]

From SLACK

• Just for clarification - marshaling is converting Dicts to structs and back. Serialization is technically converting the dict to a string, at least the way we talk about it
• we need to preserve the Dict <-> String behavior
• the Dict must store as a field inside itself stating what the type of that "object" is. That was the conclusion previously

[Affie]

here is the first issue

That is a TimeZones.jl bug: JuliaTime/TimeZones.jl#83

This is how I temporary patched it in variable timestamp:
Dates.format(v.timestamp, "yyyy-mm-ddTHH:MM:SS.ssszzz")

But we need a more permanent solution...
EDIT: BTW, JSON3.jl uses "yyyy-mm-ddTHH:MM:SS.ssszzz" as format from what I can tell. (thats why it gets my vote)

[dehann]

• Top level must be JSON (like the Internet)
  • Must be usable outside Julia (multilang)
• Order is important in dict (problem with JSON2 JSON3),
• nothings and nulls must be allowed
• save the type into the structure
  • specific field (TBD)
• separate marshaling and serialization
  • struct -> dict -> string/binary
• users should be able to pass packets along
  • not all users need to be able to unpack every packet
• ROS and (Python + ZMQ) similarly GraffSDK are strong usage examples
• Keep future FuseDFG driver in mind too (libfuse)

[dehann]

• JSON2 and JSON3 has out of order methods, but slower
• there are issues on checking order

[GearsAD]

Technologies to consider:

• https://juliadata.github.io/StructTypes.jl/stable/

These are two things we need to support in the future:

• We want to serialize UTF8/Unicode, then we can serialize Distributions.jl directly (which we want to do)
• We want to be able to override serialization types, so that we can natively serialize things like matrices. This will make factor serialization much simpler.
• We need to support null (this is an omitted field)

[dehann]

JSON standard bearers

• NewtonSoft C# opensource (Microsoft)
• JavaScript is hard
• Rust likely good

[dehann]

• Build JSON object properly
  • Don't overdo nesting of JSON in JSON in JSON...
    • In FileDFG PPE packing as JSON2 string (approx)...
  • Missing a test to ensure Dict of Dict is clean (escapes properly).

[GearsAD]

Javascript is the OG JSON - JSON stands for Java Script Object Notation. So they're arguably the gold standard.

[GearsAD]

The three use-cases we need to address when we refactor serialization packing/unpacking are:

• Persist to file and back
• Decorate and persist to Neo4j and back (decorating is important)
• Stream serialization and deserialization (we need to be able to handle a use-case like building a ROS driver)

[GearsAD]

So next steps:

• Build simple test cases for these situations (independent of DFG)

[GearsAD]

On @Affie's point - we should updated FileDFG so that it writes independent elements to subfolders. It will then resemble the Neo4J structure. This minimizes the stringification of subelements, but it also allows us to easily make a Fuse driver.