Skip to content

Fix MaxID logic for GCSToBigQueryOperator - #26768

Merged
potiuk merged 3 commits into
apache:mainfrom
patricker:FixBQMaxIDSelect
Oct 1, 2022
Merged

potiuk merged 3 commits into
apache:mainfrom
patricker:FixBQMaxIDSelect

Conversation

@patricker

Copy link
Copy Markdown
Contributor

related: #26283
closes: #26767

The max_id_key parameter, when used, causes an XCom serialization failure when trying to retrieve the value back out of XCom. This is because instead of storing a single column value in XCom, we were accidentally storing the entire Row.

The Unit Test was updated to reflect the return type of get_job().result(). This operation actually returns a Row iterator, but returning an array of Row works well for the test.

Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>

@uranusjr uranusjr left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the caveat that I have absolutely no idea whether the old or new logic is correct, lacking knowledge on the API.

@patricker

patricker commented Sep 29, 2022 •

Copy link
Copy Markdown
Contributor Author

@uranusjr If you have access to GCP/BigQuery, this was the quick/simple script I used to test out all the logic first. https://cloud.google.com/bigquery/docs/samples/bigquery-query-results-dataframe (just removed the dataframe part, and played around with the iterator/row access)

@potiuk potiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool. You can also apply the suggestion from @bhirsz .
I am going to merge that one after we release the other providers (and prepare rc2 of the google provider).

@potiuk
potiuk merged commit b7203cd into apache:main Oct 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:google Google (including GCP) related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MaxID logic for GCSToBigQueryOperator Causes XCom Serialization Error

4 participants