Add a proposal for a data update that avoids list indices

[gspencergoog]

Description

This PR has a proposal for a different data binding method in the 0.9 protocol.

I'm not entirely convinced that it is the best thing since sliced bread, but it has its good and bad points.

The main driving idea is that LLMs are bad at list indices, and that list mutations are non-local (they affect the indexes of later items in the list), so this data model spec attempts to avoid them. Avoiding indices means avoiding JSON pointers, though, which is too bad, since they are well known.

The proposed solution also tends to make the data "flatter" and definitely requires more unique IDs to be generated and maintained, but LLMs seem to be better at that than indices.

See the specification/0.9/docs/data_proposal.md file for more details.

[jacobsimionato]

I personally feel strongly that it would be better to keep the protocol simple and solve this problem on the agent side. But I don't want to get in the way of you advancing the v0.9 proposal, so approving this for if you do decide to incorporate it!

[jacobsimionato]

In a case where the agent and client is mutating the same data, perhaps a proper 2-way sync mechanism is needed anyway, and that would be another way of solving the incorrect index problem? I wonder if we should be focusing more on the idea of agent-client data model sync, and then solving these problems with agent-side utilities.

E.g. for this use case, if we had some automated two-way sync of the data model, then perhaps on the agent side we could have a tool call like:

  replaceListItem(listPath: "/user/roles/", currentValue: "Admin", newValue, "Owner")

In this case, it would still be possible for agents to define data models which always have ids within list items, so that they can identify them via id for mutation.

That way, the complexity around LLM manipulation of data can be kept inside the agent, and potentially tailored to the specific agent use case. The protocol remains simpler.

[jacobsimionato]

In fact, I think most of this proposal could be implemented as an inference utility, at least for cases where the agent is the only actor modifying its own data.

1. Agent stores current state of data model in Hybrid Adjacency Map format
2. LLM is prompted to create data model updates using this format as you demonstrate
3. Agent framework applies the changes to generate new state
4. Agent framework converts new and old states to traditional nested format, and computes JSON patch diff
5. Agent sends JSON patch to client

[jacobsimionato]

I must admit, I do love that the acronym is HAM!

[jacobsimionato]

I worry that this optimizes the data update use case at the expense of other use cases. E.g. some use cases are:

1. LLM defines new data model from scratch. This proposal worsens latency and potentially reliability because the format is more verbose.
2. LLM updates existing data model. This proposal improves reliability.
3. Data model created or updated directly via tool result or database lookup, without LLM's involvement. This proposal adds technical complexity because the regular JSON needs to be converted to HAM.

I personally suspect that the majority of usage will be 1 and 3, at least initially. So maybe it's good to keep the protocol simple and performant out of the box for those immediate use cases, then let people add the technical complexity of HAM for cases where incremental LLM-driven manipulation of data is important?

[gspencergoog]

I personally feel strongly that it would be better to keep the protocol simple and solve this problem on the agent side. But I don't want to get in the way of you advancing the v0.9 proposal, so approving this for if you do decide to incorporate it!

Yeah, I think I'm not well enough convinced that this is the way to go to incorporate it. I think I'd like to try some other solutions, similar to what you suggest above: keep the protocol simple and introduce complexity between the agent and the LLM to get correct output and convert it to the standard format. It would be nicer for clients to deal with JSON pointers and raw JSON objects, even if we have to use something like this behind the scenes to get the quality we need from the LLM.

And your point about mutations not actually being that common is also relevant. Perhaps if it fails too often, we just tell the LLM that it can't add/remove individual items in lists and has to output an entirely new list if it needs to do that.

And the iterative correction loop in 0.9 might just mitigate some of the issues that crop up to, at least for referencing invalid indices.

Anyhow, I'm glad I tried this, and we can revisit the concepts here if they turn out to be useful. Thanks for reviewing it.