`No such file or directory` error on try `dvc pull`

[maltempi]

Hello everyone,

Problem

I'm currently teaching an MLOps crash course, and some students are facing an issue with dvc-gdrive. They're unable to sync with gdrive at a certain point. I've been troubleshooting the problem and discovered that the directories that are being logged as 'no such file or directory' have a duplicate directory in gdrive. This is because gdrive allows multiple directories with the same name.

 dvc pull
Collecting                                                                                                                 |3.06k [00:05,  557entry/s]
ERROR: failed to transfer '00c597b3f6e01f32dbeb5dc41ba24c96' - [Errno 2] No such file or directory: '1APGY4tjfFhNr9lK4UdgGc81FgcNvCqLd/files/md5/00/c597b3f6e01f32dbeb5dc41ba24c96'
ERROR: failed to transfer '34fa1599305a5796ea5158cd2a29045b' - [Errno 2] No such file or directory: '1APGY4tjfFhNr9lK4UdgGc81FgcNvCqLd/files/md5/34/fa1599305a5796ea5158cd2a29045b'

Troubleshooting

Here is the Google Drive folder (public dir) : https://drive.google.com/drive/folders/1BQ8SfIRmsdZi4KAXtcyFZOKs9_6BL4BU .
And this is the ds.dvc file:

outs:
- md5: 99224f4a9ed7e9ac086c48831f0ef676.dir
  size: 68986649
  nfiles: 3060
  hash: md5
  path: ds

The files only exist in one of the folders. For example, 00c597b3f6e01f32dbeb5dc41ba24c96 (as logged in the log attached to this gh issue) exists in the first one only. In the Google Drive folder, you'll find the duplicates:

DVC Doctor

 dvc doctor
DVC version: 3.30.1 (pip)
-------------------------
Platform: Python 3.10.13 on Windows-10-10.0.22621-SP0
Subprojects:
        dvc_data = 2.22.0
        dvc_objects = 1.2.0
        dvc_render = 0.6.0
        dvc_task = 0.3.0
        scmrepo = 1.5.0
Supports:
        gdrive (pydrive2 = 1.16.0),
        http (aiohttp = 3.8.4, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.8.4, aiohttp-retry = 2.8.3),
        s3 (s3fs = 2023.10.0, boto3 = 1.28.64)
Config:
        Global: C:\Users\maltempi\AppData\Local\iterative\dvc
        System: C:\ProgramData\iterative\dvc
Cache types: hardlink
Cache directory: NTFS on D:\
Caches: local
Remotes: gdrive
Workspace directory: NTFS on D:\
Repo: dvc, git
Repo.site_cache_dir: C:\ProgramData\iterative\dvc\Cache\repo\db21a55dd073a7d61d29785b6476808

Thank you very much in advance. Please, let me know if you need any additional info.

[shcheklein]

@maltempi It is possible that multiple people simultaneously initialized DVC and pushed some initial data in different machines ?

@skshetry btw, can it be related to the race condition that was not solved (I don't remember by now the full context)

[maltempi]

Hi @shcheklein , thanks for your really fast answer!

Do you mean at the exact same time? I don't think so...

One additional info, I don't know if this matters for reproducibility, but the majority of the students are using Windows environment with conda.

[maltempi]

Hi @shcheklein . I asked the students to try again from scratch, but with dvc push -j 1 , so this way we can know if concurrence is causing this.

[shcheklein]

@maltempi I'm curious even if multiple people did this command simultaneously. But, yes, please try initialize it again. AFAIR the expectation is that when you run it for the first time it creates all these 00, 01, etc subdirectories once, specifically to "lock" those names. So, please try to push a single file and see first the structure, then do a few files from different machines or in parallel and also see the result. Thanks would be helpful, thanks!

[gribas]

Hi,

The upload (dvc push) was done by a single user. Sometimes folders are duplicated, sometimes there's three or four folders with the same name. The doppelgangers share a common trait, they have only one file in them, while the "official" folder has all the others.

From the looks of it, dvc push creates the structure (00, 01,...,ff) on demand, and not as a first step before spawning the file upload processes. Pushing two files generated only three hash folders (6a,91,96 in my experiment).

With many files and parallel process it's easy to have a collision... so I'm guessing a few process checked for the same folder name and all got a bFileNotFound. Then they tried to create it, all at the same time.... And they all succeeded, for GoogleDrive allows more than one folder with the same name. Later processes just picked the first or last created folder (dunno) and use it to upload the files.

[shcheklein]

Yes, we had at some point a mechanism to initialize all "root" directories at once as far as I remember. I'll check what happened to it.

[shcheklein]

Okay, we actually dropped that logic and instead we were using locks + I think the implementation was compatible with multiple directories (in case multiple people run push simultaneously). Most likely got broken in fsspec migration and/or in this the precursor to this PR https://github.com/iterative/PyDrive2/pull/289/files .

When you see those duplicates, do you run it in a single threaded mode btw?

[skshetry]

In dvc, we don't create directories in parallel. We only create the ones that are needed for the current transfer, and in series.

https://github.com/iterative/dvc-objects/blob/b3a4b48955181173db30f640404c57243d6997b3/src/dvc_objects/db.py#L161-L162

[shcheklein]

Yes, it was not about DVC. It was the very first impl of the driver (when it was part of DVC). It doesn't matter since even that driver was changed to work with duplicate directories + use locks to prevent them from happening. Both seems to be broken in the current fsspec implementation. We need to confirm and research this.