Skip to content
This repository was archived by the owner on Apr 8, 2025. It is now read-only.

DPR - enable loading of custom BERT models#746

Merged
Timoeller merged 2 commits intomasterfrom
dpr_config
Mar 31, 2021
Merged

DPR - enable loading of custom BERT models#746
Timoeller merged 2 commits intomasterfrom
dpr_config

Conversation

@Timoeller
Copy link
Contributor

@Timoeller Timoeller commented Mar 26, 2021

Add proper loading and saving of DPR models. It also supports other Bert models (e.g. with different vocab)

  • We want to use processor.save and model.save for saving and load those models in the same way.
  • A test is added for saving and loading the processor and model
  • Adds shuffling of DPR data right after loading the file

Code tested in haystack as well, needs adjustments to loading, but simplifies code a lot.

@Timoeller Timoeller changed the title WIP: DPR - enable loading of custom BERT models DPR - enable loading of custom BERT models Mar 30, 2021
@Timoeller Timoeller requested a review from julian-risch March 30, 2021 15:21
Copy link
Member

@julian-risch julian-risch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested on my side with gbert. Works like a charm.

@Timoeller Timoeller merged commit 19fef0e into master Mar 31, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants