Skip to content

Comments

[590] Add Iceberg Glue Catalog Sync implementation#632

Merged
the-other-tim-brown merged 1 commit intoapache:mainfrom
kroushan-nit:590-IcebergCatalogSync-v2
Feb 4, 2025
Merged

[590] Add Iceberg Glue Catalog Sync implementation#632
the-other-tim-brown merged 1 commit intoapache:mainfrom
kroushan-nit:590-IcebergCatalogSync-v2

Conversation

@kroushan-nit
Copy link
Contributor

@kroushan-nit kroushan-nit commented Jan 30, 2025

#590

Important Read

  • Please ensure the GitHub issue is mentioned at the beginning of the PR

What is the purpose of the pull request

Added support for syncing a source table to Glue catalog in Iceberg table format

Brief change log

  • Created a new module xtable-aws
  • Added GlueCatalogSyncClient implementation for syncing InternalTable to Glue catalog in Iceberg table format
  • Added GlueCatalogConversionSource implementation for creating SourceTable from Glue table identifier
  • Added GlueSchemaExtractor for generating glue compatible schema from InternalSchema

Verify this pull request

  • Added unit tests
  • Manually verified the change by running a job locally

Copy link
Contributor

@the-other-tim-brown the-other-tim-brown left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall things are looking good to me, @vinishjail97 let me know if there are any other concerns from your side that should be addressed before merging.

@kroushan-nit kroushan-nit force-pushed the 590-IcebergCatalogSync-v2 branch from e44406b to aee8a5f Compare February 4, 2025 17:06
@the-other-tim-brown the-other-tim-brown merged commit 8b0b5ea into apache:main Feb 4, 2025
2 checks passed
@vinishjail97 vinishjail97 mentioned this pull request Apr 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants