Skip to content

feat: add BaseLanceNamespace that provides an implementation of all APIs through the Lance Java SDK#323

Open
wombatu-kun wants to merge 1 commit intolance-format:mainfrom
wombatu-kun:java-impl-BaseLanceNamespace
Open

feat: add BaseLanceNamespace that provides an implementation of all APIs through the Lance Java SDK#323
wombatu-kun wants to merge 1 commit intolance-format:mainfrom
wombatu-kun:java-impl-BaseLanceNamespace

Conversation

@wombatu-kun
Copy link
Copy Markdown

@wombatu-kun wombatu-kun commented Mar 30, 2026

Closes #169

Summary

Add BaseLanceNamespace — a filesystem-directory-based implementation of the LanceNamespace interface that delegates table operations to the Lance Java SDK (lance-core).

New module: lance-namespace-base

Key implementation details

  • Arrow IPC for data serialization across the interface boundary
  • Symlinks for registerTable -- references external datasets without copying
  • Cursor-based pagination on all list operations (limit + pageToken using sorted cursor tokens for string lists, simple limit truncation for object lists)
  • Vector search parameters: k, nprobes, ef, refineFactor, distanceType, bypassVectorIndex (mapped to SDK's setUseIndex(false))
  • Full-text search: single-column match and multi-column multiMatch
  • Merge insert: supports whenMatchedUpdateAll (with optional filter), whenNotMatchedInsertAll, whenNotMatchedBySourceDelete (with optional filter)
  • Scalar index types: BTREE, BITMAP, INVERTED/FTS; default IVF_PQ for vector indexes
  • Tag error detection: heuristic string-matching on SDK exceptions (documented as fragile)
  • Storage options: storage.* config prefix passed through to all Dataset operations, enabling cloud backends

Known limitations

  • fastSearch, lowerBound, upperBound query parameters are not forwarded (no SDK support in lance-core 5.0.0-beta.1)
  • pageToken on listTableVersions, listTableTags, listTableIndices is not supported (only limit truncation)
  • Tag-not-found detection relies on SDK error message string matching

@github-actions github-actions bot added enhancement New feature or request java Java features labels Mar 30, 2026
@github-actions
Copy link
Copy Markdown
Contributor

ACTION NEEDED
Lance follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

For details on the error please inspect the "PR Title Check" action.

@wombatu-kun wombatu-kun changed the title feat(java): add BaseLanceNamespace that provides an implementation of all APIs through the Lance core SDK feat: add BaseLanceNamespace that provides an implementation of all APIs through the Lance Java SDK Mar 30, 2026
@wombatu-kun wombatu-kun force-pushed the java-impl-BaseLanceNamespace branch from 0c475b4 to dca901d Compare March 31, 2026 07:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request java Java features

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BaseLanceNamespace Java

1 participant