Repository navigation
Conversation
… a Python list find_nearest turned every query into a Python list with tolist(). The codec then walked the list (one _prep call per element) and json.dumps printed a float repr per element, on every search, and the binary float32 side buffer (tag B, read by the library straight from the C buffer) that exists for ndarray parameters was never used. A numeric 1-d query (an ndarray of any numeric dtype, or a list/tuple of numbers) now goes to the codec as a float32 array; anything else (text or None entries, a generator, numpy missing) stays a list and reaches the engine as before (numpy would turn a None into nan, so the dtype is checked on the untyped conversion first). The engine reads the vector as float32 either way, so the rounding is the same. Tests first: test_native_find_nearest_params.py counts codec._prep calls for a dim-768 query (768 per-element calls before, 0 after, for float32 and float64 arrays and for a list), checks the query travels in the side buffer, and pins that every form (float32/float64 array, tuple, list of numpy floats, generator, non-contiguous view, integer array, find_nearest_by_key) returns the same neighbours at the same distances, that filter and ef_search still apply, and that a wrong-dimension or non-numeric query is still refused. Counts only; no speed claim. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This was referenced Oct 25, 2026
Author
|
Closing here: the bindings moved to ArcadeData/arcadedb-python. This change has no counterpart there yet and no queued A/B. If it is wanted, it goes there as its own PR: it is now a line in the tracking issue ArcadeData/arcadedb-python#19. The branch stays. |
tae898
added a commit
to ArcadeData/arcadedb-python
that referenced
this pull request
Nov 4, 2026
…a Python list (#203) * Native find_nearest: send the query vector as one float32 array, not a Python list NativeVectorIndex._run_search turned every query into a list with tolist(). The codec then checked the list for float32-exactness (_exact_float_list: a pass of numpy over the list, tag BL) or, for a float64 query whose values float32 rounds, printed JSON number text (about 611 us at 1536 dimensions, #107). A numeric 1-d query (an ndarray of any numeric dtype, or a list/tuple of numbers, which numpy walks in C) now goes to the codec as a float32 array and travels through the float32 side buffer (tag B) directly. The engine reads the vector as float32 whatever it is given, so the rounding is unchanged. Anything else (text or None entries, a generator, ragged input, numpy missing) stays a list and reaches the engine as before. Port of humemai/arcadedb-embedded-python#523 (backlog row N-07) onto main after #107: the per-element _prep calls it removed are gone since #107, what remains is tolist() plus the exactness check, and the float64 text path. Untimed; a timed A/B decides it. Tests: test_find_nearest_query_forms.py (both backends: every form answers the same neighbours at the same distances, refusals unchanged; passes before and after) and test_native_find_nearest_params.py (native: where the floats travel and that the list exactness check is not run; 5 of its 6 tests fail on the code before). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * tests: isort order for the find_nearest query-form tests Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * tests: find_nearest query forms compare JVM distances to a few float32 ulps, native exactly The JVM leg of the PR run failed test_every_form_of_the_query_returns_the_same_neighbours_at_the_same_distances on one distance that differed in the last float32 bit between two calls with the same query values (0.9530812501907349 against 0.9530811309814453, same neighbour order); the same test passed on this head in the dispatched JVM run. Exact distances are not a contract on the JVM build. On the native build, the path this PR changes, the comparison stays exact, because a tolerance of float32 size could not tell a query that float32 rounds from its float32 form, which is what the float64 test guards. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backlog row N-07 (bindings study 2026-10-25).
NativeVectorIndex._run_search(_native/schema_objects.py, behindfind_nearestandfind_nearest_by_key) turned the query into a Python list withquery_vector.tolist(). Incodec.encode_paramsa list of floats is not a plain value, so every element went through_prep(one Python call each) andjson.dumpsprinted a float repr per element, on every search. The binary float32 side buffer (tagB, read byJson.javastraight from the C buffer) exists for ndarray parameters and was never reached fromfind_nearest, even for callers that already held an array.What changed (Python only, one helper in
_native/schema_objects.py; no Java change, no image rebuild):_query_vector_paramconverts a numeric 1-d query (an ndarray of any numeric dtype, or a list/tuple of numbers, which numpy walks in C) to a float32 array, andencode_paramssends that through the side buffer. The engine reads the vector as float32 whatever it is given, so the rounding is unchanged. Anything else stays a list and reaches the engine as before: text or None entries (numpy would silently turn a None into nan, so the dtype is checked on the untyped conversion first), a generator, ragged input, or numpy missing. The JPype wrapper is untouched.What was measured, counts only (
codec._prepcalls for one dim-768find_nearest, new test filetest_native_find_nearest_params.py): per-element_prepcalls 768 -> 0 for a float32 array, a float64 array and a list (the test showed 1536 for the float64 array plus the list before), and the total is now at most the statement's own few parameters, independent of the dimension. A second test checks that the search's parameters carry the query as"B"withdimfloats beside the text. The count tests fail onnative-preview(3 failures) and pass here.No speed claim is made. At dim 768 the removed work is 768 Python calls and 768 float reprs per search against a search of about 0.9 ms (50k x 128-d, untimed profile); whether it shows up is a timed A/B through
tools/ab.py, not run here.Semantics: pinned by tests that pass before and after: every form of the query (float32/float64 array, tuple, list of numpy floats, generator, non-contiguous view, integer array,
find_nearest_by_key) returns the same neighbours at the same distances (exact equality);allowed_ridsandef_searchstill apply; a wrong-dimension query and a query of None or text entries are still refused withArcadeDBError. Also rantest_vector.py,test_vector_sql.py,test_vector_params_verification.py,test_vector_second_pass.py,test_numpy_support.pyandtest_parameter_binding.pyagainst the 26.10.2.dev1 library with this checkout's Python (109 passed, 38 skipped as NOTYET/NA-JVM parity cases, 0 failed).Other vector entry points in the file:
find_nearest_approximatehas no statement form on native yet (not_yet), so it does not reach this code. The index warm-up in_IndexShim([0.0] * dimsonce per index, to make the engine build its graph) has the same list shape, but it runs once per index and not per query, and the idea would need a numpy import there for no per-query gain; left as is. The result side (each neighbour's whole record, vector included, shipped back: row N-08) is a separate item and not touched.🤖 Generated with Claude Code