[Draft] Add automated tests to provide metrics on retrieval time for already existent data

[significance]

The Milestone Smoke Test provides an automated test, performed against mainnet, that establishes a regularly captured metric for random data uploaded to the swarm, then downloaded from a different part of the network.

While this is an invaluable metric in terms of evaluating network health and node performance, it does not enable visibility of the time taken to retrieve data that has been in existence for some time on the network, and therefore should be available without any need for synchronisation.

In addition, it would be useful to establish a long running test which checks for data availability and identifies any gaps in service provision.

It is suggested that the OSM dataset could provide such a dataset, as it's stamp is maintained and each of its many chunks presumed to be available at any time.

[gacevicljubisa]

Metrics are merged to bee, and we have deployed 4 mainnet nodes in dev-bee-perf namespace. Those nodes are nuked every 24 hours using beekeeper. And after every nuke, they are given different random neighborhood.

Histogram Metrics added:

• bee_init_warmup_duration_seconds
• bee_init_full_sync_duration_minutes

@nugaon can you add any other relevant info?

[nugaon]

@gacevicljubisa I think it is related to provide metric data on fetching long-living data on Swarm.
This could identify problems whether there is any chunk loss for stored data during the time.

I guess the download speed should be provided for such data because of getting result without caching the nodes. In that case this check should run rarely... and for data availability check it is a great idea.