Standardize [Packed]State

[Affie]

TABLE IS STILL A WIP (Starting discussion on val)

	State	Old PackedState	New State Fields
✓	initialized::Bool	initialized::Bool	initialized::Bool
✓	separator::Vector{Symbol}	separator::Vector{Symbol}	separator::Vector{Symbol}
✓	× id::Union{UUID, Nothing}	× id::Union{UUID, Nothing}	REMOVE
✓	× dimIDs::Vector{Int}	× dimIDs::Vector{Int}	REMOVE
✓	× solveInProgress::Int	× solveInProgress::Int	REMOVE (#1181#issuecomment-3466836820)
✓	× BayesNetOutVertIDs::Vector{Symbol}	× BayesNetOutVertIDs::Vector{Symbol}	REMOVE
✓	× dims::Int	× dims::Int	REMOVE
✓	× BayesNetVertID::Symbol	× BayesNetVertID::Symbol	REMOVE
✓	× eliminated::Bool	× eliminated::Bool	REMOVE
✓	× dontmargin::Bool	× dontmargin::Bool	REMOVE
✓	✎ solveKey::Symbol	✎ solveKey::Symbol	label::Symbol
✓	✎ infoPerCoord::Vector{Float64}	✎ infoPerCoord::Vector{Float64}	observability
✓	✎ ismargin::Bool	✎ ismargin::Bool	marginalized
✓	✎ solvedCount::Int	✎ solvedCount::Int	solves
☐		✎ variableType::String	statetype::Symbol
☐		✎ _version::VersionNumber	type::TypeMetadata
☐	val::Vector{P}		see discussion
☐		vecval::Vector{Float64}
☐		dimval::Int
☐	bw::Matrix{Float64}		see discussion
☐		vecbw::Vector{Float64}
☐		dimbw::Int
☐	covar::Vector{SMatrix{N, N, Float64}}	covar::Vector{Float64}	TODO
☐	events::Dict{Symbol, Threads.Condition}

Keep in mind (#1177)

• that adding new fields in the future counts as feature expansion and will not break existing serialization.
• this is the chance for improving Clarity of variable names can be improved Caesar.jl#65.
  • Aim for a single noun and avoid camel case.
• We should not have a PackedState

[Affie]

val

• Name not very descriptive, we can maybe update to reflect that this field is points on the manifolds, ie. points::Vector{P}.

Packing val::Vector{P} for serialization

Currently:

• packState converts p ∈ M to coordinates and only works for Lie groups (with a TODO referring to Serialization and Marshaling (again) #590)
• stores in fields (vecvel,dimval)

Options for a point like [[1, 2], [3 4; 5 6]]:

• Force P to be compatible with a flat column major structure SArray/MArray/ArrayPartition: [1,2,3,5,4,6]
• Nesting: [[1,2],[[3,5],[4,6]]]
  • storage? string only?
  • Possibly ambiguous?
• Complex struct coordinates plus expansion point
  • have to still store the expansion point, basis, etc.
• pack/unpack functions for points based on the variable type (such as Pose3) as
  • Vector{Float64}
  • Vector{Vector{Float64}}
  • String - base64 encoded
  • String - json value
  • String - json string

Regardless, it's maybe a good idea to add a point_encoding field to store the encoding, eg. "coords-v1", "points-v1", etc

[Affie]

With #1207, manifold point serialization/deserialization now uses:

• A versioned AbstractStateType eg. Position{1}

"statekind": {
  "pkg": "IncrementalInferenceTypes",
  "name": "Position{1}",
  "version": "0.1.0"
},

• DFGJSONStyle overloads for default SArray, ArrayPartition, Complex, etc.
• StrucUtils.jl is currently the way to customize serialization, but we can wire in pack, unpack if needed.

[dehann]

Hmm, let me think on the storage of manifold points.

In the mean time, lets try clear some of the other fields that should be easier:

• dimIDs, yeah can remove. That was an idea from before JuliaManifolds existed. More likely to be replaced by infoPerCoord usage.
• eliminated is definitely used, don't remove from PackedState, and also goes along with Simultaneous solvers using same CSM / Tree? IncrementalInference.jl#673
• BayesNetOutVertIDs, visual inspection by search seems like unused field. My style of doing would be to remove the field and then run all tests through Caesar and then see if anything related pops up. If not, then good to remove. If really unused in all repos through Caesar, then I'm fine to remove.
• BayesNetVertID, seems used via quick search, why did you want to remove it maybe I'm missing the reasoning? (see image below). My guess is the field name is probably very old and outdated. Maybe we need to fix the name?
• _version should maybe just be version? This would be so for all types...

---

[dehann]

Dehann headscratching on serde for manifold points

Affie's whole comment: #1181 (comment)

I think anything other than String or Nested can quickly become super specific and requires a lot internal knowledge before you can unpack the data in any way. The ground rule for me remains that if you happen to come across a PackedDFG node somewhere, you can always start by loading into a dict (say in Python) that object before you know anything about it's contents json.parse(_I_found_this_file_that_looks_like.json). Now you start exploring that dict and find either a type or mime field, which you use to start Googling, hey what is Nva, CJL, or DFG, etc.

This is like the prime directive when it comes to DFG serialization. What I do know so far is that anything from the comment #1181 (comment) that violates blind json packing/unpacking is already a no go. I also quite strongly believe that any time a person somewhere in the wild starts unpacking these DFG json objects, that they want to look at the numerical estimates. So rather than hide the numbers in some obscure encoding, we should make them visible; or clearly tell them what to Google via the mime= ; encoding=... examples at the bottom of this comment.

---

This all ties in with the outstanding work of registering .nvy (.nva is taken) extension for FileDFG tars?

Regardless, it's maybe a good idea to add a point_encoding field to store the encoding, eg. "coords-v1", "points-v1", etc

is there not something like mime for manifold points, should we not register (or at least document) a bunch of mime extension options for this purpose?

---

Here is ChatGPT output on query :

`mime extension examples using ';'`

**Usage of Parameters**

Parameters can provide additional information about the content, such as character set or encoding. For example:

• text/plain; charset=UTF-8 specifies that the plain text file uses UTF-8 encoding.
• application/json; charset=UTF-8 indicates that the JSON data is also encoded in UTF-8.

Understanding these examples helps ensure proper handling of files by web servers and browsers.

---

Here is more ChatGPT output on the query ,

`existing encodings for riemannian manifold data`

Riemannian manifold data can be encoded using various methods, including persistent homology, which provides topology-inspired representations. These encodings help in analyzing the geometric and topological features of the data.[ OpenReview](https://openreview.net/forum?id=7yswRA8zzw) Overview of Riemannian Manifold Data Encodings

Riemannian manifolds are mathematical structures that allow for the study of curved spaces. Various encoding methods exist to represent data on these manifolds effectively.
Common Encoding Methods

1. Persistent Homology (PH)

   Description: PH is a topological data analysis method that captures the shape of data across multiple scales.
   Application: It generates topology-inspired representations, useful for understanding the intrinsic properties of data.
   Sensitivity: PH is sensitive to perturbations, making it effective for detecting features in data.

2. Pull-back Geometry

   Description: This novel methodology focuses on understanding the intrinsic properties of persistent homology without relying on external models.
   Advantage: It provides deeper insights into the structure of data represented on Riemannian manifolds.

Other Notable Methods
3. Geodesic Distance Encoding

Description: This method uses the shortest paths between points on the manifold to encode relationships.
Use Case: It is particularly useful in applications where the geometry of the data is crucial.

4. Heat Kernel Signature

   Description: This approach encodes the manifold's geometry by analyzing the heat diffusion process.
   Benefit: It captures both local and global geometric features, making it robust for various applications.

Conclusion

These encoding methods provide diverse ways to represent and analyze data on Riemannian manifolds, each with unique strengths and applications. Persistent homology and pull-back geometry are particularly noteworthy for their focus on intrinsic data properties.

---

Using prescribed tangent basis vectors (aka encoding coordinates with known bases)

A tangent vector can be expressed in terms of local coordinates as v=vi∂xi∂​ where vi are the components in the coordinate basis.

---

Mental steps to formalize

My sense is also that we can have more than one encoding, e.g. application/json; ABC which would be text, but say we are storing HomotopyBeliefs with 1000's of data points for high resolution belief estimation, then maybe we want to go to something like application/bson; ABC for binary representation on the bulk of data content.

So my first reaction is to maybe go for something like: mime: "application/json; [SOMETHING STANDARD]", but I don't quite know what yet... Just listing ideas:

• mime: iapplication/json; nva.tangent=se(3)
• mime: application/bson; group=nva.SGal(3)
• ...?

The other idea that comes up is to maybe, we should replace .type with .mime and rather do something like:

{
  "label": "continuation_01",
  "eliminated": true,
  # ...
  "mime": "application/json; type=DistributedFactorGraphs.PackedState, encoding=Caesar.Pose3.SGal(3)"
}

Here, I'm trying to keep the idea of different encodings for the same thing alive. So storing the raw application/json; encoding=Caesar.Pose3.SGal(3) might be really heavy but still persists. But, we can then add more encodings like application/json; encoding=RoMETypes.Pose3.sgal(3).b64 which stores only coordinates for the standard CJL bases in base64 binary format -- assuming NavAbility/Caesar.jl advertises a basis vector for sgal(3) tangents.

Or, the lightest of all would be mime: application/bson+gzip; type=DistributedFactorGraphs.PackedState, encoding=RoMETypes.RotVelPos3.sgal(3).b64, which means the whole PackedState blob exists as tightly wrapped binary rather than raw text json (exploiting the rule here that bson also requires json compliance, but stores in binary rather than text). To be clear, any json.write(jobj) can always be converted to bson using bson.write(json.parse(jstr)) and visa versa.

The problem with this is that if we start using other mimes, that needs to be known outside of the data blob. So in the real world, these payloads would only work if say the DB is known to only store application/bson+gzip, because there would be no way to know that apriori. So any data out in the wild sort of has to be application/json, small maybe application/bson but that already hurts adoption. Wild data would never be application/bson+gzip because no one would ever guess how to unpack it -- so these are for internal benefit when large data costs money, maybe things like archiving older data etc.

Additional note on why b64, because we need to stick to the rules that any DFG+Packable object is always jsonable. I.e. VariableCompute is not Packable, but VariableDFG is.

---

Recap

We can change .type to .mime and add to application/json the option ; type=DistributedFactorGraphs.PackedState as well as , encoding=CJL.Pose3.SGal(3) and the in the Caesar package advertises what that means, i.e. we use Nesting: [[1,2],[[3,5],[4,6]]] to store the whole SGal(3) affine matrix (aka group element).

We can then also advertise more efficient storage such as encoding=RoMETypes.Pose3.sgal(3) which have a documented basis with which to reconstruct the manifold point -- this asks of an uninitiated newcomer who is messing with raw data to do a bit more Googling.

[Affie]

#1181 (comment)

Sorry, I’m not fully following what you’re suggesting here.
We already have a pattern for Observation that works well, so I’d prefer not to introduce additional complexity with MIME-type variations and make things more involved than the Observation case. I’d suggest landing somewhere between the current flattened coordinate approach and a pack approach (similar to Observation).

An idea for large binary points — we already have Blobs, and if we decide to follow the existing pack/unpack pattern, we can simply store a reference to a BlobEntry (already includes the mimetype and metadata for the Blob). That way you can get your application/bson.

[Affie]

DF: sure blob approach works for large case, that also fits in with the "parch" verb usage.

I'm not sure how we'd use dedicated pack and unpack without describing the encoding somewhere? My first choice would be to avoid pack and unpack functions for manifold points, but that seems unlikely. This feels similar to the DistributionsJSON discussion, that said I like to avoid having a hard schema definition like Distributions if possible. My concern with coordinate approach is when loading to saving, that the performance of parsing and transforming all the points becomes heavy.

[Affie]

My concern with coordinate approach is when loading to saving, that the performance of parsing and transforming all the points becomes heavy.

Agree, we keep points ∈ M then.

I'm not sure how we'd use dedicated pack and unpack without describing the encoding somewhere?

We already have the <:AbstractStateType{N} (Pose2, Point2, etc.) to dispatch on. I do not think we should support more than one representation per AbstractStateType{N}.

My first choice would be to avoid pack and unpack functions for manifold points, but that seems unlikely. This feels similar to the DistributionsJSON discussion, that said I like to avoid having a hard schema definition like Distributions if possible.

I can't see a way to not have "a hard schema definition", or do you mean an extra type? I also don't see a way of not having pack and unpack functions; however, we can provide defaults for most types (eg. Matrix, ArrayPartition, Vector, etc.).

[Affie]

I'll confirm if these are indeed dead

[Affie]

eliminated, BayesNetOutVertIDs, and BayesNetVertID only occurs in dead code or is set but never used.

[Affie]

✅ infoPerCoord -> observability

[Affie]

✅ solvedCount -> solves

[Affie]

ismargin

• maybe marginalized

dontmargin

TODO likely unused.

---

I'm fine with marginalized.

[Affie]

✅ ismargin -> marginalized

[Affie]

dontmargin is set in constructors but never used, the name is not the best, therefor we can remove it and add it back with a better name when implemented.

[Affie]

Removed dontmargin

[Affie]

covar

This was prepared for parametric solve, but it is not populated yet. Maybe move to v1.1?

---

DF: just checking, if the parametric covariance value moves, that would be a breaking change - so we may have to sort this out for v1.0.

[Affie]

Good catch! We have to sort covar out as well and it fits in with the bw/val discussion.

[Affie]

This will move to .belief

[Affie]

bw

• Name not very descriptive, maybe more explicit as bandwidth?
  Reasons:
  • Clarity: bandwidth is immediately understandable; bw requires domain knowledge
  • Consistency: Other fields are spelled out
  • Self-documenting: The full word explains what it represents, external consumers shouldn't need to guess abbreviations.

---

DF: we should probably first resolve HomotopyBelief, which is sort out where convince/bandwidth info is stored.

[Affie]

This will move to .belief

[dehann]

That leaves the harder ones, val/bw/covar

we should solve these in light of future HomotopyBelief -- root of that belief tree has a mean and covariance, the leaves of that belief tree have kernels with bandwidths (or smallest singular values, depends on how we implement). I think this may result in a wholesale replacement of val/bw/covar.

[Affie]

• Do we store only the leaves and rebuild the tree or store the whole tree?
• Do we support different kernel types for the leaves?

[Affie]

My best crystal ball guess is that we need to store and serialize the "bumps" (kernel + parameters) -- eg ConcentratedGaussian(mu, cov)

[dehann]

Here is the underlying question to answer first:

• [Design] possible refactor manellic tree and leaf kernels to mixture or nonparametrics ApproxManifoldProducts.jl#327

[Affie]

The combination of val/bw/covar and more will be combined in a new type in a new field: belef
We are exploring looser interface requirements for belief.

[Affie]

belief WIP

New field to capture old bw, val, and covar.

We are exploring:

• having this type B <: Any with B required to follow an interface specification.
• expanding the current generic bw, val, covar approach.

WIP requirements and roles:

• Replacement of PPEs
  • Without any addition calculations (eg. can be used for plotting, indexing, etc. directly from the database)
• Support for the parametric solver
  • previously used val[1] and bw
• Support for the non-parametric solver
  • previously used val and bw
• Support for homotopy solver
• Support for x-stroke solving where number of modes are increased on each stroke.
• Serialization and deserialization
  • Without the need to serialize manifolds.
  • Based on <: StateType
  • Platform/language agnostic
• Must at least be a concrete type (isconcretetype) all the way up to VariableDFG. We have states::OrderedDict{Symbol, State{T, P}} where T <: StateType. At a minimum this can be states::OrderedDict{Symbol, State{T}} where T <: StateType

The AbstractBelief interface used in factors (#1146 (comment)) should support:

• rand(::AbstractRNG, belief)
• length(belief)

[Affie]

We can consider the same caching idea as with factors to cache the full HomotopyDensisty if needed.

[dehann]

Thanks for starting to capture DFG's requirements on new .belief::HomotopyDensity. What would help me at the moment is make the "required interfaces" list as crisp as possible for DFG/IIF/RoME/CJL/ needs please.

I have this discussion as the "Define Requirements" level in V-Model engineering terms.

Not sure what you mean with the comment on struct BeliefRepresentation above ... looks like a structure design -- if so the struct design of HomotopyDensity belongs upstream to DistributedFactorGraphs and we should probably keep the struct design discussion in that repo, currently centered around this issue (please add any other relevant links there):

• [Design] Should HomotopyDensity support heterogeneous kernel types (aka be a mixture)? ApproxManifoldProducts.jl#326

There are quite a few structs already in AMP and the the path I'm on now is the next iteration of ManellicTree and ManifoldKernelDensity towards a refactored HomotopyDensity. The existing structures already do many of the "Homotopy" things we previously spoke about. My sense is we don't need a revolutionary change in AMP, it's just plain engineering tasks to get the implementation to a good enough level. Aside, and for the notes here, there is a good chance that HomotopyDensity will spin out as it's own package upstream/subproject to AMP once the interface requirements become a bit more clear.

• There is still going to be some evolution on the internal fields of HomotopyDensity after consolidation of existing structs is complete. See progress at: Manellic code shuffle towards DFG v1 compliance ApproxManifoldProducts.jl#325
  • I'm currently working on a solution for partials as part of the upgrade from ManellicTree to HomotopyDensity. Looks like the path for that is to stay with <:AbstactVector{<:Integer}, and expect underlying support for ArrayPartition of tuples of integers. Perhaps good to know the following is true: ArrayPartition{Tuple{Int...},...} <: AbstractVector{<:Integer} -- i.e. a 2dof partial for something like SE2 now becomes partial=ArrayPartition((2,), (1,)) instead of legacy partial=[2,3].

[Affie]

Not sure what you mean with the comment on struct BeliefRepresentation above ... looks like a structure design -- if so the struct design of HomotopyDensity belongs upstream to DistributedFactorGraphs and we should probably keep the struct design discussion in that repo

BeliefRepresentation is a DFG only design that moves the old state fields val, bw, and covar to the new state.belief field and adds possible new fields. It follows the currently working design of a more generic container while improving serialization support and remaining fully type stable. I think we need that work session to discuss and finalize this.

[Affie]

See #1207 for a working prototype.

[Affie]

permute should also be stored in #1207