Need way to mirror definitions

[jeffmcaffer]

There is interest in people replicating definition data locally to support robustness/performance, infrastructure control, and privacy.

Principles:

• Readonly -- clearlydefined.io is still the source of truth with all curations, harvesting, ... done in the production service.
• Definitions only -- Harvested data and curations are not included in the mirroring process.
• Only point queries -- The production service supports arbitrary queries over definition documents. The local copy only needs point queries based on component coordinates.

Options:

• rsync-style -- The definitions are just blobs so in theory we could mirror those as files and allow people to read from disk. That exposes the user to internal details of ClearlyDefined.
• slave service -- Implement a path through the service code that is read-only and has the mirroring activity built in. This would shut down any write paths, not have a crawler, ... and implement whatever mirroring protocol we decide is best.

Random thoughts/topics

• Must all definitions be aggressively computed? Currently we (re)compute definitions on demand in the event of schema changes. We could have the local service fall back to the remote service if the schemas don't match.
• First replication is different. That could be a bulk download of a dump where as keeping up to date continuously replicates recent actions.
• periodic or continuous. Need to determine if the use cases require up to the minute replication or if periodic (hourly, daily) replication is enough.
• should be related to the need for an "event stream" that enables people to track new definitions.
• Local scenarios may use different data store technology from the main service. A simple version would just put the data in the local file system. So this is not a straight record for record mirror. Rather it should use an API to read and write the data using the correct structures.
• Local servers, being read-only, need not ever compute a definition.

cc: @jeffmendoza

[geneh]

@jeffmcaffer Today the /definitions/:type/:provider/:namespace/:name/:revision endpoint checks whether a definition is cached. If it isn't cached, the definition is retrieved from a pre-computed definition blob and then cached. If the definition doesn't exist, it is computed based on the harvested data. If there is no harvested data for the definition, the crawler is requested to harvest the corresponding data.

If the harvested data and the curations are out of scope and the definition cannot be found in the definitions replicated store, the crawler won't be notified about the missing definition and it won't be queued up.
Is this understanding correct?
Would in-memory caching work here or should this type of caching be disabled due to a potential risk of running out of memory?

Regarding the definitions syncing, the following options are possible starting from the least amount of effort. I am also open to any other options and ideas.

• The internally-hosted service would access the production definition blob storage in Azure by using a read-only+disabled blob listing SAS token with an expiration date.
• The internally-hosted service would access a replicated Azure blob storage using an anonymous SAS token with the same permissions as described above. The blobs would be replicated daily to a different container by an Azure Data Factory pipeline.
• Azure Data Factory can also push blobs incrementally to a remote file store using credentials. It would be hard to scale this solution to additional users and become too expensive, so I would advise against it.
• The internally-hosted service would access a local file store. If the definition isn't available or the schema version has changed, it will retrieve the data from the hosted service and persist it locally. A hybrid solution may also be copying all the blobs initially and maybe periodically. Pros: the data is mostly up-to-date and the crawler is notified about any missing components. Cons: the service may still be dependent on the hosted service. The definitions may become out of sync/stale for recomputed definitions.
• A script can be developed to copy all or some of the blobs based on the blobs' prefix. The script could potentially pull only the blobs that have changed since the last successful run. Azure blob storage does not have an API to retrieve blobs starting from a specific time, so all the blobs would need to be listed using pagination. Pros: the service will be completely self-contained. Cons: the script will have to be scheduled to run daily and can potentially fail due to the high volume of data.

Additionally, Azure Data Factory provides blobs compressing capability. It looks like this capability is only for every blob individually, so it won't be possible to compress all or incremental blobs in one archive. I haven't tested this option but it seems to be the case based on the official documentation.

[geneh]

@bduranc @iamwillbar @pombredanne Could you please review and comment here?

[pombredanne]

My preferred option would be to use rsync as this would eschew a whole lot of complexity. I do not think that exposing the data files/blob is an issue.