Add randomized tests for adding/removing keys#16
Merged
Conversation
bc81210 to
0dd98e2
Compare
fe2a314 to
154ed5d
Compare
5878c88 to
7d3c854
Compare
7d3c854 to
e0343e3
Compare
tzdybal
reviewed
Nov 25, 2022
tzdybal
left a comment
There was a problem hiding this comment.
I left some comments - mostly about simplifying the code. The tracing solution seems very clean 👍
| return b | ||
| } | ||
|
|
||
| func FuzzBatchAddReverse(f *testing.F) { |
There was a problem hiding this comment.
I know it's just a test, but this function is huge and complex. Can we for example:
- extract
keyas a function, instead of using lambda? - have separate helper functions for
SetandRemove? - probably introducing a struct with methods will make most sense (so we don't need to pass to many params)
Member
Author
There was a problem hiding this comment.
Could you please clarify "probably introducing a struct with methods will make most sense (so we don't need to pass to many params)" in the updated code?
There was a problem hiding this comment.
Something like:
type testContext struct {
r io.Reader
tree *MutableTree
dst *DeepSubTree
keys set.Set[string]
}
func (tc *testContext) getKey(getRandom bool) ([]byte, error) {
// ...
}
func (tc *testContext) removeInDst(key []byte) error {
// ...
}
func (tc *testContext) setInDst([key, value []byte, isNewKey, isFirstKey bool) error {
// ....
}
tzdybal
approved these changes
Nov 29, 2022
Manav-Aggarwal
added a commit
that referenced
this pull request
Dec 20, 2022
* Add go fuzz tests * Add membership proof for existing keys * Build tree after adding membership proof * Make batch add fuzz tests work * Do not commit version twice for dst * Save version out of dst.Set condition * Set rootHash to workingHash * Fix edge cases * Refactor DST Non-Existence Proof * Change cacheSize * Add test data and sibling nodes for each node in path * Fix GetSiblingNodes * Add more test data * testing: fuzz: failing test case * Use set for keys * Add more test data * Refactor existence proofs code * Add required info for remove operation * Add children of siblings as well * Remove debug code * Add testdata that breaks current code * Fix bug * Add failing testcase * Add breaking testcase * IAVL with tracing * Fuzz tests pass * Refactor tracing code * Remove redundant code * Remove working hash in calls to AddExistenceProof * Clean up flag * Make build tree a private method * Add back whitespace in node.go * Add new ci for fuzz * Refactor more * Refactor out getKey method * Change name to reapInclusionProofs * Refactor set and remove in DST * Factor out add existence proofs from Remove DST for consistency with Set * Refactor into testContext * Clean up setInDST Co-authored-by: Tomasz Zdybał <tomek@zdybal.lap.pl>
Manav-Aggarwal
added a commit
that referenced
this pull request
Dec 20, 2022
* Add go fuzz tests * Add membership proof for existing keys * Build tree after adding membership proof * Make batch add fuzz tests work * Do not commit version twice for dst * Save version out of dst.Set condition * Set rootHash to workingHash * Fix edge cases * Refactor DST Non-Existence Proof * Change cacheSize * Add test data and sibling nodes for each node in path * Fix GetSiblingNodes * Add more test data * testing: fuzz: failing test case * Use set for keys * Add more test data * Refactor existence proofs code * Add required info for remove operation * Add children of siblings as well * Remove debug code * Add testdata that breaks current code * Fix bug * Add failing testcase * Add breaking testcase * IAVL with tracing * Fuzz tests pass * Refactor tracing code * Remove redundant code * Remove working hash in calls to AddExistenceProof * Clean up flag * Make build tree a private method * Add back whitespace in node.go * Add new ci for fuzz * Refactor more * Refactor out getKey method * Change name to reapInclusionProofs * Refactor set and remove in DST * Factor out add existence proofs from Remove DST for consistency with Set * Refactor into testContext * Clean up setInDST Co-authored-by: Tomasz Zdybał <tomek@zdybal.lap.pl>
Manav-Aggarwal
added a commit
that referenced
this pull request
Jan 18, 2023
* feat: Deep tree structure with Updates (#13) * ICS23-style approach for DeepSubTree creation * Fix deepsubtree, all checks pass * Update documentation for deep subtrees * Fix deepsubtree to accomodate left/right paths both * Refactor test code * Refactor TestDeepSubtreeStepByStep * Address comments * Refactor code * Disable gocritic and unused * Add description * Refactor code to check err in Set * Modify traversal conditions to be clearer * feat: build deep subtree from ICS23 inclusion proofs (#9) * feat: build deep subtree from ICS23 inclusion proofs * feat: handle non-existence proofs * linter: goimports deepsubtree.go * refactor: addExistenceProofProof -> addExistenceProof * fix: un-hardcode size of byte array * Add more comments * Refactor ndb.Commit call outside for loop * verify that in the case that dst.root != nil that the root node hash matches the provided hash and check dst.root != nil first * Add strings and use strings.Repeat * Delete addPath and AddPath * Remove print statements * Refactor recomputeHash usage Co-authored-by: Matthew Sevey <mjsevey@gmail.com> * return err directly Co-authored-by: Matthew Sevey <mjsevey@gmail.com> * Address linting checks * Use tm-db instead of cosmos-db * Fix linter * Update comment Co-authored-by: Matthew Sevey <mjsevey@gmail.com> * Add error checking for node hash * turn lengthByte into a const * Refactor linkNode * Update comment Co-authored-by: Tomasz Zdybał <tomek@zdybal.lap.pl> Co-authored-by: Matthew Sevey <mjsevey@gmail.com> * feat: Support Adds in a Deep Subtree (#8) * ICS23-style approach for DeepSubTree creation * Fix deepsubtree, all checks pass * Update documentation for deep subtrees * Fix deepsubtree to accomodate left/right paths both * Refactor test code * Refactor TestDeepSubtreeStepByStep * Address comments * Refactor code * Disable gocritic and unused * Add description * Refactor code to check err in Set * Modify traversal conditions to be clearer * feat: build deep subtree from ICS23 inclusion proofs (#9) * feat: build deep subtree from ICS23 inclusion proofs * feat: handle non-existence proofs * linter: goimports deepsubtree.go * refactor: addExistenceProofProof -> addExistenceProof * fix: un-hardcode size of byte array * Add more comments * Refactor ndb.Commit call outside for loop * verify that in the case that dst.root != nil that the root node hash matches the provided hash and check dst.root != nil first * Add strings and use strings.Repeat * Delete addPath and AddPath * Remove print statements * Refactor recomputeHash usage Co-authored-by: Matthew Sevey <mjsevey@gmail.com> * return err directly Co-authored-by: Matthew Sevey <mjsevey@gmail.com> * Address linting checks * Use tm-db instead of cosmos-db * Fix linter * Add test to check add operation in deepsubtree: WIP * Add insert key functionality WIP * Try replicating non-inclusion proofs * Add sibling nodes to nonExistenceProofs * Refactor test code * Finish adding add functionality to dst * Add Remove functionality to dst * Fix linting errors * Fix GetSiblingNode * Remove more print statements * Add comment for each case in recursive set * Change which methods are exported and document exported methods * feat: Support Empty Hashes and Add constructor (#11) * Export siblings * Add deepsubtree constructor * Support empty root hashes * Use working hash instead of root.hash * Use tm-db instead of cosmos-db * Return nil in BuildTree * Address comments * Address more comments Co-authored-by: Tomasz Zdybał <tomek@zdybal.lap.pl> Co-authored-by: Matthew Sevey <mjsevey@gmail.com> * Add randomized tests for adding/removing keys (#16) * Add go fuzz tests * Add membership proof for existing keys * Build tree after adding membership proof * Make batch add fuzz tests work * Do not commit version twice for dst * Save version out of dst.Set condition * Set rootHash to workingHash * Fix edge cases * Refactor DST Non-Existence Proof * Change cacheSize * Add test data and sibling nodes for each node in path * Fix GetSiblingNodes * Add more test data * testing: fuzz: failing test case * Use set for keys * Add more test data * Refactor existence proofs code * Add required info for remove operation * Add children of siblings as well * Remove debug code * Add testdata that breaks current code * Fix bug * Add failing testcase * Add breaking testcase * IAVL with tracing * Fuzz tests pass * Refactor tracing code * Remove redundant code * Remove working hash in calls to AddExistenceProof * Clean up flag * Make build tree a private method * Add back whitespace in node.go * Add new ci for fuzz * Refactor more * Refactor out getKey method * Change name to reapInclusionProofs * Refactor set and remove in DST * Factor out add existence proofs from Remove DST for consistency with Set * Refactor into testContext * Clean up setInDST Co-authored-by: Tomasz Zdybał <tomek@zdybal.lap.pl> * Add tracing to Trees and Deep Subtrees (#18) * Add go fuzz tests * Add membership proof for existing keys * Build tree after adding membership proof * Make batch add fuzz tests work * Do not commit version twice for dst * Save version out of dst.Set condition * Set rootHash to workingHash * Fix edge cases * Refactor DST Non-Existence Proof * Change cacheSize * Add test data and sibling nodes for each node in path * Fix GetSiblingNodes * Add more test data * testing: fuzz: failing test case * Use set for keys * Add more test data * Refactor existence proofs code * Add required info for remove operation * Add children of siblings as well * Remove debug code * Add testdata that breaks current code * Fix bug * Add failing testcase * Add breaking testcase * IAVL with tracing * Fuzz tests pass * Refactor tracing code * Remove redundant code * Remove working hash in calls to AddExistenceProof * Clean up flag * Make build tree a private method * Add back whitespace in node.go * Add new ci for fuzz * Refactor more * Refactor out getKey method * Change name to reapInclusionProofs * Refactor set and remove in DST * Factor out add existence proofs from Remove DST for consistency with Set * Refactor into testContext * Clean up setInDST * Add method for get * Export methods * Add witness data to deep subtree * Verify operation in witness data * Refactor and verify operation for get and remove * Add set witness data * Add tracing to tree * add getter for witness data * Verify existence proofs in dst * Cleanup * Reset witness data on tracing enabled * Add node to keysAccessed even not in cache * Add initial root hash * Refactor GetInitialRootHash * Modify GetInitialRootHash * Add test data * Add get to dst tests: fails right now * Refactor and add tracing for root key as well * Add docs * Add more docs * Update comments * Update log message Co-authored-by: Tomasz Zdybał <tomek@zdybal.lap.pl> * allocate length Co-authored-by: Tomasz Zdybał <tomek@zdybal.lap.pl> * fix: remove `RangeProofs` (cosmos#586) Co-authored-by: colin axnér <25233464+colin-axner@users.noreply.github.com> Co-authored-by: Marko <marbar3778@yahoo.com> * Update go.mod file * Add new line at end of .golangci.yml Co-authored-by: Tomasz Zdybał <tomek@zdybal.lap.pl> Co-authored-by: Matthew Sevey <mjsevey@gmail.com> Co-authored-by: cool-developer <51834436+cool-develope@users.noreply.github.com> Co-authored-by: colin axnér <25233464+colin-axner@users.noreply.github.com> Co-authored-by: Marko <marbar3778@yahoo.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
We'd like to add randomized tests for
SetandRemovethat add new keys and delete existing keys respectively. Mainly, we'd like to make sure that Deep Subtrees are constructed with all the data they might need to rebalance correctly when new keys are added or existing keys are removed.Adds following tests:
[x] - Batch-add random keys.
[x] - Randomize both the operations,
SetandRemove, and the keys they are operating on so calls to both are interleaved together in no particular order. Note that the tree should need various rebalances when doing the above operations.Make sure the tree rebalances correctly at each stage (check against existing IAVL tree:
mutable tree).Overall, it adds strong substantial tests for an IAVL Deep Subtree.
Since this tests the
Removeimplementation, we can close out #14 as well.Resolves: #15, resolves: #14
Checklist