This repository was archived by the owner on Jul 18, 2024. It is now read-only.
This repository was archived by the owner on Jul 18, 2024. It is now read-only.
Refactor doc_loader.py to load documents concurrently using Ray actors or Spark tasks, instead of loading them all at once and then putting them into a dataset #510
Description
Refactor doc_loader.py to load documents concurrently using Ray actors or Spark tasks, instead of loading them all at once and then putting them into a dataset