feat(ethexe): improve database migrations

[liferooter]

• Moves ethexe_db_init into ethexe_db. This will allow to access crate's private types in migrations.
• Introduces well-organized migration framework.
• Fixes some minor mistakes in existing migration.

[semanticdiff-com]

Review changes with  

Changed Files

File	Status
ethexe/db/src/database.rs	54% smaller
ethexe/service/src/tests/utils/env.rs	27% smaller
Cargo.lock	Unsupported file format
Cargo.toml	Unsupported file format
ethexe/common/Cargo.toml	Unsupported file format
ethexe/db/Cargo.toml	Unsupported file format
ethexe/db/init/Cargo.toml	Unsupported file format
ethexe/db/init/src/version1.rs	0% smaller
ethexe/db/src/lib.rs	0% smaller
ethexe/db/src/migrations/init.rs	0% smaller
ethexe/db/src/migrations/migration.rs	0% smaller
ethexe/db/src/migrations/mod.rs	Unsupported file format
ethexe/db/src/migrations/v0.rs	0% smaller
ethexe/db/src/migrations/v1.rs	0% smaller
ethexe/observer/Cargo.toml	Unsupported file format
ethexe/observer/src/tests.rs	Unsupported file format
ethexe/service/Cargo.toml	Unsupported file format
ethexe/service/src/lib.rs	Unsupported file format

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a robust database migration system for ethexe by creating a new ethexe-db-init crate. It refactors how global database configuration and state are managed, moving from a single LatestData structure to more granular DBConfig and DBGlobals with thread-safe access. Additionally, it enhances runtime type information for various data structures and updates the ProtocolTimelines to include slot duration, ensuring better consistency and extensibility for future database schema changes.

Highlights

• Database Migration System: A new ethexe-db-init crate has been introduced to manage database migrations, defining DBConfig and DBGlobals structures to store database configuration and global state, replacing the LatestData structure.
• Refactored Database Access: The Database struct in ethexe-db has been refactored to use Arc<RwLock<DBConfig>> and Arc<RwLock<DBGlobals>> for thread-safe access to global configuration and state, and now returns Result<Self> on creation.
• scale-info::TypeInfo Derivations: Many common data structures across ethexe-common, ethexe-db-init, and gsigner now derive scale-info::TypeInfo, which is crucial for runtime type information and potentially for SCALE codec compatibility checks.
• ProtocolTimelines Update: The ProtocolTimelines structure now includes a slot field, and its usage has been updated across various components to reflect this change.
• Removed LatestData Trait/Struct: The LatestData struct and its associated LatestDataStorageRO/RW traits have been removed, with their responsibilities absorbed by the new DBConfig and DBGlobals.

Changelog

• .gitignore
  • Added .vscode/settings.json to the ignore list.
• Cargo.lock
  • Updated dependencies, added indoc and scale-info, and included ethexe-db-init package.
  • Removed gear-core-processor and sha2 0.10.9 from some dependency lists.
• Cargo.toml
  • Added ethexe/db/init to workspace members and dependencies.
  • Added ethexe-sdk to ignored packages.
• core/Cargo.toml
  • Removed the [package.metadata.cargo-shear] section.
• ethexe/cli/src/commands/check.rs
  • Replaced LatestData with DBGlobals and GlobalsStorageRO in imports and Checker struct.
  • Updated db.latest_data() calls to db.globals().
• ethexe/common/Cargo.toml
  • Added scale-info dependency with derive feature.
  • Added indoc and scale-info with docs feature to dev-dependencies.
• ethexe/common/src/db.rs
  • Added Address import.
  • Derived TypeInfo for BlockMeta and AnnounceMeta.
  • Removed protocol_timelines methods from OnChainStorageRO/RW.
  • Removed LatestData struct and LatestDataStorageRO/RW traits.
  • Introduced DBConfig and DBGlobals structs with TypeInfo derivation, and new GlobalsStorageRO/RW and ConfigStorageRO traits.
  • Added a test to ensure types are unchanged.
• ethexe/common/src/events/mirror.rs
  • Derived TypeInfo for all event structs.
• ethexe/common/src/events/mod.rs
  • Added TypeInfo import and derived TypeInfo for BlockEvent.
• ethexe/common/src/events/router.rs
  • Derived TypeInfo for all event structs.
• ethexe/common/src/gear.rs
  • Added TypeInfo import.
  • Derived TypeInfo for Message, StateTransition, and ValueClaim.
  • Added GenesisBlockInfo struct.
• ethexe/common/src/hash.rs
  • Added TypeInfo import.
  • Derived TypeInfo for HashOf.
  • Added AsRef<[u8]> implementation for HashOf.
  • Changed #[cfg(feature = "std")] to #[cfg(feature = "mock")] for HashOf::random().
• ethexe/common/src/injected.rs
  • Added Vec import and TypeInfo, build::Fields imports.
  • Manually implemented TypeInfo for InjectedTransaction.
• ethexe/common/src/mock.rs
  • Removed tap::Tap re-export from top, moved it after other imports.
  • Added Address import.
  • Replaced protocol_timelines and latest_data fields in BlockChain with config and globals.
  • Updated BlockChain::setup and BlockChain::mock to use DBConfig and DBGlobals.
  • Added Mock implementations for DBConfig and DBGlobals.
• ethexe/common/src/primitives.rs
  • Added TypeInfo import.
  • Derived TypeInfo for BlockHeader, SimpleBlockData, Announce, StateHashWithQueueSize, CodeBlobInfo, and ProtocolTimelines.
  • Added slot field to ProtocolTimelines.
  • Updated tests for ProtocolTimelines.
• ethexe/common/src/utils.rs
  • Removed ProtocolTimelines, SimpleBlockData, ValidatorsVec, LatestData, LatestDataStorageRW imports.
  • Removed setup_start_block_in_db and setup_genesis_in_db functions.
• ethexe/common/src/validators.rs
  • Added TypeInfo and build::Fields imports.
  • Implemented TypeInfo for ValidatorsVec.
  • Added FromIterator<Address> implementation for ValidatorsVec under mock feature.
• ethexe/compute/Cargo.toml
  • Added ethexe-db with mock feature to dev-dependencies.
• ethexe/compute/src/compute.rs
  • Replaced LatestDataStorageRO/RW with ConfigStorageRO and GlobalsStorageRW.
  • Updated usage of db.latest_data() to db.globals() and db.config().
• ethexe/compute/src/lib.rs
  • Renamed BlockValidatorsCommittedForEraNotFound error to CommittedEraNotFound.
  • Removed LatestDataNotFound error.
• ethexe/compute/src/prepare.rs
  • Replaced LatestDataStorageRW with GlobalsStorageRW.
  • Updated usage of db.block_validators_committed_for_era and db.mutate_latest_data to db.globals_mutate.
• ethexe/consensus/Cargo.toml
  • Added ethexe-db with mock feature to dev-dependencies.
• ethexe/consensus/src/announces.rs
  • Replaced LatestDataStorageRO with GlobalsStorageRO.
  • Updated usage of db.latest_data() to db.globals().
• ethexe/consensus/src/connect/mod.rs
  • Replaced db::OnChainStorageRO with db::{ConfigStorageRO, OnChainStorageRO}.
  • Added timelines field to ConnectService and initialized it from db.config().
  • Updated usage of db.protocol_timelines() to self.timelines.
• ethexe/consensus/src/validator/core.rs
  • Updated BlockChain::mock and ctx.core.timelines to use chain.config.timelines.
  • Changed ValidatorsVec creation to use into_iter().collect().
• ethexe/consensus/src/validator/initial.rs
  • Updated ctx.core.timelines to use chain.config.timelines.
• ethexe/consensus/src/validator/mock.rs
  • Added db::* import.
  • Updated mock_validator_context to set DBConfig instead of protocol_timelines directly.
• ethexe/consensus/src/validator/mod.rs
  • Replaced db::OnChainStorageRO with db::ConfigStorageRO.
  • Updated db.protocol_timelines() to db.config().timelines.
• ethexe/db/Cargo.toml
  • Added ethexe-common with mock feature to dev-dependencies.
  • Added mock feature.
• ethexe/db/init/Cargo.toml
  • Added a new crate for database initialization and migrations.
• ethexe/db/init/src/init.rs
  • Added functions for initializing and migrating the database.
• ethexe/db/init/src/lib.rs
  • Defined database versions, migrations, and initialization configuration.
• ethexe/db/init/src/migration.rs
  • Introduced a Migration trait and testing utilities for migrations.
• ethexe/db/init/src/v0.rs
  • Defined database schema for version 0, including LatestData and ProtocolTimelines.
• ethexe/db/init/src/v1.rs
  • Defined database schema for version 1 and implemented the migration logic from v0.
• ethexe/db/src/database.rs
  • Introduced DatabaseRef struct.
  • Refactored Database to use Arc<RwLock> for global state and introduced DatabaseRef for trait implementations.
  • Implemented AnnounceStorageRO/RW, OnChainStorageRO/RW, BlockMetaStorageRO/RW for DatabaseRef.
  • Implemented GlobalsStorageRO/RW and ConfigStorageRO for Database.
  • Updated Key enum for Globals and Config.
  • Modified Key::to_bytes to append zero bytes for Globals/Config keys.
  • Added version(), config(), globals(), set_config(), set_globals() methods to dyn KVDatabase.
  • Updated Database::new and Database::memory to handle DBConfig and DBGlobals.
• ethexe/db/src/iterator.rs
  • Removed LatestDataStorageRO from the DatabaseIteratorStorage trait.
• ethexe/db/src/lib.rs
  • Exported DatabaseRef and VERSION.
  • Added is_empty() to KVDatabase trait.
  • Made take() unsafe in KVDatabase.
• ethexe/db/src/mem.rs
  • Implemented is_empty() and made take() unsafe for MemDb.
• ethexe/db/src/overlay.rs
  • Made take() unsafe and added is_empty() (unimplemented) for KVOverlay.
• ethexe/db/src/rocks.rs
  • Made take() unsafe and implemented is_empty() for RocksDatabase.
• ethexe/ethereum/src/abi/gear.rs
  • Implemented From conversion for GenesisBlockInfo.
• ethexe/ethereum/src/abi/mod.rs
  • Reordered use alloy::sol; statement.
• ethexe/ethereum/src/lib.rs
  • Updated RouterQuery::from_provider to accept a direct address.
• ethexe/ethereum/src/middleware/mod.rs
  • Updated MiddlewareQuery::from_provider to accept a generic address type.
• ethexe/ethereum/src/router/mod.rs
  • Updated RouterQuery constructors to accept generic address types.
• ethexe/network/Cargo.toml
  • Added ethexe-db mock feature to dev-dependencies.
• ethexe/network/src/db_sync/mod.rs
  • Updated DbSyncDatabase trait to use ConfigStorageRO and GlobalsStorageRO.
• ethexe/network/src/db_sync/responses.rs
  • Updated database access to use ConfigStorageRO and GlobalsStorageRO, and removed a specific error.
• ethexe/network/src/lib.rs
  • Updated network service to use Database directly and pass timelines from config.
• ethexe/network/src/validator/list.rs
  • Modified ValidatorList::new to accept ProtocolTimelines directly.
• ethexe/observer/Cargo.toml
  • Added ethexe-db and ethexe-db-init mock features to dev-dependencies.
• ethexe/observer/src/lib.rs
  • Introduced ObserverConfig struct.
  • Updated RuntimeConfig with timelines field.
  • Removed slot_duration_secs from RuntimeConfig.
  • Refactored ObserverService::new to accept Database and ObserverConfig, and to use db.config().timelines.
  • Removed pre_process_genesis_for_db function.
• ethexe/observer/src/sync.rs
  • Updated ChainSync to use Database directly and access global state via db.globals().
• ethexe/observer/src/tests.rs
  • Updated observer tests to use new database initialization and observer configuration.
• ethexe/observer/src/utils.rs
  • Added a TODO comment regarding batch size.
• ethexe/processor/Cargo.toml
  • Added ethexe-db mock feature to dev-dependencies.
• ethexe/processor/src/handling/run.rs
  • Updated a test to use Database::memory().
• ethexe/processor/src/lib.rs
  • Added a TODO comment about task processing order.
• ethexe/prometheus/src/lib.rs
  • Updated database access to use GlobalsStorageRO.
• ethexe/rpc/src/utils.rs
  • Updated RPC utility functions to use GlobalsStorageRO.
• ethexe/runtime/Cargo.toml
  • Removed core-processor dependency.
• ethexe/service/Cargo.toml
  • Added ethexe-db-init dependency and mock features for ethexe-db and ethexe-db-init.
• ethexe/service/src/fast_sync.rs
  • Updated fast sync logic to use new database structures and access methods.
• ethexe/service/src/lib.rs
  • Integrated ethexe-db-init for database initialization and migrations.
  • Updated observer service creation.
• ethexe/service/src/tests/mod.rs
  • Updated tests to use new database initialization and global state access.
• ethexe/service/src/tests/utils/env.rs
  • Updated test environment setup to use new database initialization and observer configuration.
• gsigner/Cargo.toml
  • Added scale-info dependency and feature.
• gsigner/src/schemes/secp256k1/address.rs
  • Derived TypeInfo for Address.
• gsigner/src/schemes/secp256k1/digest.rs
  • Derived TypeInfo for Digest.
• gsigner/src/schemes/secp256k1/signature.rs
  • Derived TypeInfo for Signature, SignedData, and SignedMessage.
• gstd/src/common/errors.rs
  • Updated documentation for error messages.
• pallets/gear-builtin/Cargo.toml
  • Removed sha2 dependency.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a significant and valuable improvement by adding a database migration framework and refactoring the database schema. The separation of configuration (DBConfig) from dynamic global state (DBGlobals) is a solid architectural change that will improve maintainability and likely performance. The changes are extensive but well-contained within the new ethexe-db-init crate and the database access layer. I've identified one critical issue regarding a database version mismatch that would prevent the node from starting, and two medium-severity suggestions to improve the migration script by cleaning up old data. Overall, this is a strong contribution to the project's long-term health.

[gemini-code-assist]

Code Review

This pull request significantly improves the database migration process by refactoring the logic from ethexe-db-init into a more structured migrations module within ethexe-db. This change enhances maintainability and clarity. The introduction of a Migration trait and versioned migration files is a solid design. The addition of a test to verify the stability of data types used in migrations is an excellent safety measure. However, I've found a couple of critical issues where .unwrap() is used on fallible RPC connections, which could cause the node to panic during startup. These should be addressed to make the initialization and migration process more robust.