Remote backups get partially deleted during the long upload

[tadus21]

Hello,
I am trying to upload backup of relatively big CH instance (usually around 7TB) into GCS but for some reason during the upload, some of the data that was already uploaded gets deleted from the GCS. Backup server does not produce any errors and indicates that everything was completed successfully.
Backup is created by using /backup/create_remote call with delete-source and resume args. It runs for around 20hours. Once its finished, when I list backups with backup/list I see this metadata:

{
  "name": "<...>-full-20260112132914",
  "created": "2026-01-12 13:44:04",
  "size": 7558433732294, // 7.55TB
  "data_size": 7553075730803, // 7.55TB
  "metadata_size": 886470,
  "rbac_size": 6144,
  "compressed_size": 7558432839680,
  "location": "remote",
  "required": "",
  "desc": "tar, regular"
}

and I also see this backup/actions:

{"command":"create_remote --delete-source --resume <...>-full-20260112132914","status":"success","start":"2026-01-12 13:29:14","finish":"2026-01-13 09:43:50"}

however when I check the storage bucket I see that some of the tables and parts are missing, and gsutil indicates that the size of the backup is much smaller:

gsutil du -sh gs://<...>/<...>-full-20260112132914
20.58 GiB    gs://<...>/<...>-full-20260112132914

I observed that initially during upload there were much more files in the bucket but at some point they were deleted. What might cause this partial deletion of files? Could it be that combination of delete-source and resume flags might cause this? For example the deleted local parts are then deleted in the GCS if the job is resumed? But I cannot find any evidence that the job was stopped and resumed at any point

We are using 2.6.33 version and run backup container in a server mode

[Slach]

how exactly you execute clickhouse-backup?

upgrade clickhouse-backup to 2.6.41, it contains feature with *.pid which prevent execution two parallel operation with the same backup even when you execute clickhouse-backup directly without using REST api

[tadus21]

I used allow_parallel: true, because before creating new backups I am listing existing ones, and there is a condition where if you perform any operation right away after listing backups, it would fail because list operation is still treated as in progress even though response is already returned.

Also in our case we never trigger two backups in parallel. In backup/actions response it can be seen that we only start new backup once the previous one is completed so they never overlap:

{"command":"create_remote --delete-source --resume <...>-full-20260107133324","status":"success","start":"2026-01-07 13:33:24","finish":"2026-01-08 11:11:34"}
{"command":"list","status":"success","start":"2026-01-07 13:39:52","finish":"2026-01-07 13:39:52"}
<... multiple list ...>
{"command":"list","status":"success","start":"2026-01-08 13:29:08","finish":"2026-01-08 13:29:09"}
{"command":"create_remote --delete-source --resume <...>-full-20260108132909","status":"success","start":"2026-01-08 13:29:09","finish":"2026-01-09 09:15:23"}
{"command":"list","status":"success","start":"2026-01-08 13:35:29","finish":"2026-01-08 13:35:29"}
<... multiple list ...>
{"command":"list","status":"success","start":"2026-01-09 13:29:01","finish":"2026-01-09 13:29:02"}
{"command":"create_remote --delete-source --resume <...>-full-20260109132902","status":"success","start":"2026-01-09 13:29:02","finish":"2026-01-10 10:18:12"}
{"command":"list","status":"success","start":"2026-01-09 13:35:17","finish":"2026-01-09 13:35:17"}
<... multiple list ...>
{"command":"list","status":"success","start":"2026-01-10 13:29:05","finish":"2026-01-10 13:29:06"}
{"command":"create_remote --delete-source --resume <...>-full-20260110132906","status":"success","start":"2026-01-10 13:29:06","finish":"2026-01-11 09:37:04"}

If this was an issue with backups_to_keep_remote I would assume that the whole backup folder would be deleted, but it is not, we still have metadata.json file, metadata, shadow folders. Just some tables are missing from metadata and shadow folders (still referenced in metadata.json) and tables that are still there, have parts are missing.
Might it be some other issue?

[Slach]

could you upgrade to clickhouse-backup 2.6.41 and check again?

actually need backup logs with --env=GCS_DEBUG=1 to figure out when old backup will partially delete

[tadus21]

I see. These are good suggestions. I will try to deploy new version of backup with GCS_DEBUG enabled and will get back there with the results later. Thanks!

[Slach]

be carefull GCS_DEBUG can generate LOT of logs for TB size backup

[Slach]

check GCS_DEBUG in non prudction first

[Slach]

it would fail because list operation is still treated as in progress even though response is already returned.

this is weird, list operation is synchronous and shall be finished after return response

[tadus21]

In list http handler current command is stopped as defer statement on line 860:

defer status.Current.Stop(commandId, err)

which makes the response to be returned before the defer is executed on line 940:

api.sendJSONEachRow(w, http.StatusOK, backupsJSON)

So if you make another request right away you can trigger this condition where list command is not "officially" ended.

[Slach]

ok. will fix it in 2.6.42