process_dependency_metrics file not found - Facade race condition

[MoralCode]

Repo: https://github.com/flannel-io/flannel

Task: augur.tasks.git.dependency_tasks.tasks.process_dependency_metrics

Stack Trace

Traceback (most recent call last):
  File "/augur/.venv/lib/python3.11/site-packages/celery/app/trace.py", line 453, in trace_task
    R = retval = fun(*args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^
  File "/augur/.venv/lib/python3.11/site-packages/celery/app/trace.py", line 736, in __protected_call__
    return self.run(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/augur/augur/tasks/git/dependency_tasks/tasks.py", line 14, in process_dependency_metrics
    generate_deps_data(logger, repo_git)
  File "/augur/augur/tasks/git/dependency_tasks/core.py", line 32, in generate_deps_data
    deps = dep_calc.get_deps(path,logger)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/augur/augur/tasks/git/dependency_tasks/dependency_util/dependency_calculator.py", line 43, in get_deps
    deps.extend(get_language_deps(path, lib_module, name,logger))
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/augur/augur/tasks/git/dependency_tasks/dependency_util/dependency_calculator.py", line 48, in get_language_deps
    files = language.get_files(path)
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/augur/augur/tasks/git/dependency_tasks/dependency_util/c_deps.py", line 8, in get_files
    files = list(p.glob('**/*.c'))
            ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/pathlib.py", line 953, in glob
    for p in selector.select_from(self):
  File "/usr/local/lib/python3.11/pathlib.py", line 407, in _select_from
    for starting_point in self._iterate_directories(parent_path, is_dir, scandir):
  File "/usr/local/lib/python3.11/pathlib.py", line 397, in _iterate_directories
    for p in self._iterate_directories(path, is_dir, scandir):
  File "/usr/local/lib/python3.11/pathlib.py", line 386, in _iterate_directories
    with scandir(parent_path) as scandir_it:
         ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/pathlib.py", line 938, in _scandir
    return os.scandir(self)
           ^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/augur/facade/32856-git.832008.xyz-flannel-io-flannel/flannel/docs'

this facade task has the facade core collection chain (poorly named) and the dependency metrics in the same group, allowing them to run in parallel. This appears to result in a race condition where this task runs before facade has cloned or updated the repo.

https://github.com/chaoss/augur/blob/413383af55d08411a5ce40cf3afaecf602f66de6/augur/tasks/git/facade_tasks.py#L502-L507

[MoralCode]

This also seems to be affecting augur.tasks.github.facade_github.tasks.insert_facade_contributors, although the error may be masked by #3458 not being completed

This happened to the same repo

Stack Trace

Traceback (most recent call last):
  File "/augur/.venv/lib/python3.11/site-packages/celery/backends/redis.py", line 533, in on_chord_part_return
    resl = [unpack(tup, decode) for tup in resl]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/augur/.venv/lib/python3.11/site-packages/celery/backends/redis.py", line 533, in <listcomp>
    resl = [unpack(tup, decode) for tup in resl]
            ^^^^^^^^^^^^^^^^^^^
  File "/augur/.venv/lib/python3.11/site-packages/celery/backends/redis.py", line 439, in _unpack_chord_result
    raise ChordError(f'Dependency {tid} raised {retval!r}')
celery.exceptions.ChordError: Dependency 71121513-6ae8-4d7b-b933-fb856cc53565 raised FileNotFoundError(2, 'No such file or directory')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/augur/.venv/lib/python3.11/site-packages/celery/app/trace.py", line 520, in trace_task
    task.backend.mark_as_done(
  File "/augur/.venv/lib/python3.11/site-packages/celery/backends/base.py", line 159, in mark_as_done
    self.on_chord_part_return(request, state, result)
  File "/augur/.venv/lib/python3.11/site-packages/celery/backends/redis.py", line 552, in on_chord_part_return
    return self.chord_error_from_stack(callback, exc)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/augur/.venv/lib/python3.11/site-packages/celery/backends/base.py", line 304, in chord_error_from_stack
    return backend.fail_from_current_stack(callback.id, exc=exc)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/augur/.venv/lib/python3.11/site-packages/celery/backends/base.py", line 311, in fail_from_current_stack
    self.mark_as_failure(task_id, exc, exception_info.traceback)
  File "/augur/.venv/lib/python3.11/site-packages/celery/backends/base.py", line 167, in mark_as_failure
    self.store_result(task_id, exc, state,
  File "/augur/.venv/lib/python3.11/site-packages/celery/backends/base.py", line 526, in store_result
    self._store_result(task_id, result, state, traceback,
  File "/augur/.venv/lib/python3.11/site-packages/celery/backends/base.py", line 975, in _store_result
    current_meta = self._get_task_meta_for(task_id)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/augur/.venv/lib/python3.11/site-packages/celery/backends/base.py", line 997, in _get_task_meta_for
    meta = self.get(self.get_key_for_task(task_id))
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/augur/.venv/lib/python3.11/site-packages/celery/backends/base.py", line 871, in get_key_for_task
    raise ValueError(f'task_id must not be empty. Got {task_id} instead.')
ValueError: task_id must not be empty. Got None instead.

[MoralCode]

https://github.com/cilium/cilium-cli
https://github.com/openshift/cluster-kube-controller-manager-operator

and quite a lot of others

are also affected repos. Given this may be a race condition though, i dont think the repos themselves matter that much

[MoralCode]

related: #3459