Prevent race condition in BQ sink jobId generation#877
Conversation
pyalex
requested review from
davidheryanto,
khorshuheng,
woop and
zhilingc
as code owners
|
/test test-end-to-end-batch-dataflow |
5 similar comments
|
/test test-end-to-end-batch-dataflow |
|
/test test-end-to-end-batch-dataflow |
|
/test test-end-to-end-batch-dataflow |
|
/test test-end-to-end-batch-dataflow |
|
/test test-end-to-end-batch-dataflow |
|
/test test-end-to-end-batch-dataflow |
5 similar comments
|
/test test-end-to-end-batch-dataflow |
|
/test test-end-to-end-batch-dataflow |
|
/test test-end-to-end-batch-dataflow |
|
/test test-end-to-end-batch-dataflow |
|
/test test-end-to-end-batch-dataflow |
pyalex
force-pushed
the
jobid-race-condition
branch
from
4e10d19 to
9d49852
Compare
pyalex
force-pushed
the
jobid-race-condition
branch
from
9d49852 to
88a2a0e
Compare
|
/retest |
|
/test publish-docker-images |
|
/test test-end-to-end-batch-dataflow |
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: khorshuheng, pyalex The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
* fix race condition * [bq] temp file prefix in global window
* fix race condition * [bq] temp file prefix in global window
3 participants
What this PR does / why we need it:
Getting rid of any DoFn state completely, so there won't be any race conditions.
TempFilePrefixis generated globally, since each data flush will be accompanied with generated UUID.JobIdPrefixis generated in the end of the window (soCombine.globallywithout triggers can be used) since it's not needed earlier.Which issue(s) this PR fixes:
Fixes #
Does this PR introduce a user-facing change?: