Remove unnecessary fixed sleep by adding predicate-based path check

[Red-Eyed]

What does this PR do?

Replaces the fixed time.sleep(0.5) with a predicate-based check using wait_for_predicate.
This removes unnecessary waiting when the dataset path already exists, while still handling cases where it appears slightly later.
This significantly speeds up cache loading when there are thousands of optimized dirs

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84%. Comparing base (8ab1975) to head (ef539da).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@         Coverage Diff         @@
##           main   #700   +/-   ##
===================================
  Coverage    84%    84%           
===================================
  Files        52     52           
  Lines      7095   7103    +8     
===================================
+ Hits       5970   5986   +16     
+ Misses     1125   1117    -8     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull Request Overview

This PR improves performance when loading datasets by replacing a fixed 0.5-second sleep with a predicate-based check that only waits when necessary. This optimization significantly speeds up cache loading scenarios with thousands of optimized directories.

• Introduces a wait_for_predicate utility function for timeout-based conditional waiting
• Replaces the fixed sleep with a conditional check that returns immediately if the path exists
• Maintains the same 0.5-second timeout for cases where the path doesn't exist initially

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[tchaton]

LGTM !