feat(operations): remove attributes over the operations API (full_record, __unset__)

[dawsontoth]

What

Adds full_record: true to update and upsert, selecting a full replace instead of the
default merge.

{ "operation": "update", "database": "dev", "table": "dog", "full_record": true,
  "records": [{ "id": 1, "dog_name": "Penny" }] }

The stored record becomes exactly that — any attribute not in the payload is removed.

Why

There is currently no way to remove an attribute from a record through the operations API.
update and upsert both land on Table.patch for a record that already exists
(ResourceBridge.upsertRecords), which merges: an omitted attribute keeps its stored value, and
null stores a null. A full replace exists only over REST (PUT /Table/id → resource.put),
which a client speaking the operations API can't reach — and can't reach at all through a proxied
operations endpoint.

Reported downstream as HarperFast/studio#1643, where Studio's row editor silently dropped a
property deletion. The workaround available to clients today is to delete the record and
insert it again, which:

• leaves the record absent between two writes — concurrent reads, relationship resolutions
  from the far side, and replicas all observe the gap;
• resets __createdtime__ (a real put retains it — Table._writeUpdate keeps the stored
  created time on a full update and only stamps a new one for a new entry);
• shows subscribers a delete followed by an insert instead of one write.

A single full-replace write has none of those properties.

Design

The flag is orthogonal to each operation's create rule, so one flag covers both useful shapes
without a new verb:

request	behaviour
update	merge, requires an existing record (unchanged)
upsert	merge, creates a missing record (unchanged)
update + full_record	full replace, still skips a record that isn't there
upsert + full_record	full replace, still creates one — i.e. exactly REST PUT /Table/id

No effect on insert, which never writes over an existing record.

The merge default is deliberately untouched. 5da23c3ef originally used Table.put and
hand-copied missing properties from the existing record to emulate a merge; 5bd363946
("Update/upsert records should use patch") replaced that with a real Table.patch, and
b302a16e7 added the put fallback for a missing record. So merge semantics is load-bearing v4
compatibility, and changing upsert in place would be breaking. Hence a flag rather than new
semantics — even though the newer resource/type layer already reads upsert as PUT
(defineTable.ts: UPSERT (PUT): full replace, and defineResource.ts treats upsert and
patch as distinct projections).

Two details worth a look

.strict() on the validator key, not the whole object. validateBySchema passes unknown keys
through and discards Joi's converted value, so without .strict() a string "false" would
validate and then reach the bridge still a string — letting truthiness decide a destructive
question. Strict on the one key rejects it instead; making the whole object strict would tighten
the long-standing contract of database/schema/table/records. Same rationale as
analyticsValidator's .strict().

A role with attribute_permissions on the table is refused. Attribute permissions are checked
against the attributes a request supplies (checkAttributePerms), which is sufficient for a
merge but not for a replace, where an omitted attribute is removed — so an attribute-scoped role
could otherwise erase an attribute it has no update permission for simply by leaving it out.
REST closes the same gap in Table.allowUpdate ("if this is a full put operation that removes
missing properties, we don't want to remove properties that the user doesn't have permission to
remove") by restoring those attributes from the stored record, which needs the stored record;
authorization runs before anything is read, so it denies rather than silently narrowing the write.
Roles that scope no attribute — every role without deliberate attribute_permissions on the table
— are unaffected. Happy to swap this for a restore-based approach if you'd rather it mirror REST.

__unset__ — removing an attribute without resending the record

full_record can drop an attribute, but only by replacing the whole record, so a caller who wants
to remove one field has to resend every other field — and anything it fails to resend is dropped
too. __unset__ is the narrower tool:

{ "operation": "update", "database": "dev", "table": "dog",
  "records": [{ "id": 1, "__unset__": ["age"] }] }

age is removed; every attribute the request didn't mention keeps its value. An array of names
rather than a map with ignored values — the values would carry no meaning, and a shape that accepts
anything invites callers to read significance into it.

Resolved server-side in upsertRecords, against the record the write transaction has already
loaded, and written as a full replace. Doing it here rather than in the client is the point: the
merge and the write are one transaction, where a client emulating it with read-then-replace races
every other writer.

Reserved like __createdtime__/__updatedtime__ (terms.UNSET_ATTRIBUTES) and never stored. Both
copies of the attribute-collection loop skip it — insertUpdateValidate.js and its documented async
twin in insert.ts — because upsertRecords feeds that list to Table.addAttributes, so on an
open (non-schemaDefined) table an unrecognised __unset__ would be registered as a real table
attribute. Refused: the primary key (in the bridge, the only layer that knows which attribute that
is — _writeUpdate re-asserts it on a full update, so unsetting it would otherwise be silently
ignored), the system timestamps, and any malformed value.

Attribute permissions are checked precisely here, which is the part worth reviewing:
getRecordAttributes contributes the names __unset__ removes instead of the key itself, so
removing an attribute needs the same update permission that writing it would. A role may unset
what it can write and nothing else. That's strictly better than the blanket full_record denial
above, because the removals are named — if you'd prefer, full_record could later be narrowed the
same way by resolving its removals against the stored record.

One behaviour found while testing, pinned rather than changed

On a legacy open (non-schemaDefined) table, the operations-API attribute projection reports
every registered attribute, filling in null for one the record doesn't have. So after removing
an attribute from an open table, a get_attributes read still shows key: null — identically to a
record that never had it. SELECT * reflects the stored record and omits it.

It's a read-path artifact, not storage: verified by inserting a record that never had the attribute
and getting the same null. It applies equally to full_record, and it predates both. Called out
because it makes a successful removal look like a failed one, which is exactly the symptom
studio#1643 was originally reported as — a client removing attributes from open tables needs to know
the read lies. There's a test pinning it so it can't drift silently.

Testing

integrationTests/database/full-record-write.test.ts — 21 probes, all passing on rocksdb.

For full_record:

• the removal itself, and the merge default still merging (the control)
• an explicit null staying a stored null (omission is the only thing that removes)
• __createdtime__ surviving and __updatedtime__ re-stamping
• both create rules (update skips a missing record, upsert creates one)
• parity with REST PUT — same stored attribute set through both doors
• a relationship foreign key still resolving from the far side after a replace
• the attribute-permission denial, and that the same role can still merge
• a non-boolean full_record rejected rather than coerced

For __unset__: the removal with everything else merged, several at once, __createdtime__
preserved, a relationship table surviving the server-side merge with its foreign key intact, an
unset of an attribute the record doesn't have being a no-op, the attribute-permission check
(denied for an attribute the role can't update, allowed for one it can), the primary-key and
system-timestamp refusals, malformed values rejected, __unset__ not becoming an attribute of an
open table, and the open-table read-path projection above.

npm run test:integration -- "integrationTests/database/full-record-write.test.ts"
ℹ tests 21   ℹ pass 21   ℹ fail 0

Every guard was mutation-checked — reverting it has to turn a test red. Making the bridge ignore
full_record fails 5; dropping the server-side merge fails 4; not expanding __unset__ for the
permission check fails 1; letting __unset__ register as an attribute fails 1. That last one is
why the open-table probe creates its own table: the first version asserted against a
schema-defined table and passed with the guard removed, i.e. it tested nothing.

Also ran, green, for regressions around authorization and write semantics:
integrationTests/security/{user-role-management,sql-unqualified-table-authz,query-row-allowread-checkpermission}
and integrationTests/database/{auto-fields,bulk-conditional-mutation,patch-disjoint-field-merge}
— 59 tests, re-run after __unset__ landed.

Not covered / not done:

• Replication is untested here. A full_record write replicates as a put, the same as REST
  PUT already does, but I have not proven it — no multi-node probe in this suite.
• LMDB run not done (HARPER_STORAGE_ENGINE=lmdb). The suite is written to run under it.
• Unit suites not run — a local Harper instance held the RocksDB lock and stopping it wasn't
  mine to do. Nothing in this change has unit coverage; it's all integration.
• Docs not written. reference/operations-api/operations.md in the documentation repo needs
  both full_record and __unset__ on the update and upsert entries, plus a note about the
  open-table read projection. Happy to open that PR once the shape here is agreed.
• __unset__ on insert is accepted and ignored (there is no existing record to remove from).
  Rejecting it instead would be easy if you'd rather it be an error.
• No cross-model review pass yet.

Draft until the naming (full_record, __unset__) and the two attribute-permission decisions get
a maintainer opinion — all public API surface, all cheap to change now and awkward later.

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request introduces a full_record option for update and upsert operations in the operations API, enabling a full replace instead of a merge. This allows omitted attributes to be removed while preserving the original creation timestamp and maintaining relationship integrity. The changes also include strict boolean validation for the new flag, authorization checks to prevent attribute-scoped roles from bypassing permissions via full replaces, and comprehensive integration tests. We have no feedback to provide on these changes.

[claude]

Found one blocker (inline, still unaddressed): the full_record authorization denial in operation_authorization.ts:844 isn't scoped to update/upsert, so it spuriously denies insert/search/other ops for attribute-scoped roles.

[dawsontoth]

Superseded by #2347, which carries these two commits plus the attribute-permission scoping fix and the target-database authorization fix. #2347 is based on main, not on this branch, so it is reviewable as one PR — closing this to keep the review in one place.

Branch feat/ops-api-full-record left in place for now since #2347's head is stacked on top of it; nothing depends on this PR staying open.