GCStoGCS operator not working when replace=False and source_object != destination_object

The GCStoGCS operator using the replace=False flag and specifying source_object != destination_object, will not work properly, because it is comparing apples with pears.

Cause: the prefix in the pears case needs to be calculated from the destination_object, however, it is coming from the source_object.

Thanks for opening your first issue here! Be sure to follow the issue template!

Possible solution:

existing_objects = hook.list(self.destination_bucket, prefix=prefix_, delimiter=delimiter)

# add an additional case
if self.destination_object is not None:
    existing_objects = existing_objects.replace(prefix_, self.destination_object, 1)

objects = set(objects) - set(existing_objects)

@aslantar

Would you like to provide a PR with a fix @aslantar ? happy to review it.

Yes. I never contributed, but I'll find out and let you know.

@potiuk

@potiuk From my side it is ready for review #16991. I'd especially be interested how I can improve wrt. your coding guidelines.

Navigation Menu

Repository navigation

GCStoGCS operator not working when replace=False and source_object != destination_object #16967

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

GCStoGCS operator not working when replace=False and source_object != destination_object #16967

Description

Activity

boring-cyborg commented on Jul 13, 2021

aslantar commented on Jul 13, 2021

potiuk commented on Jul 13, 2021

aslantar commented on Jul 13, 2021

aslantar commented on Jul 15, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions