Add a metric recording the overall time spent building each crate

[Nemo157]

In some ways this is simply an inverse of the existing "builds/hour" derived metric, but I think having a histogram will help visualize what proportion of fast vs slow crates we have.

[GuillaumeGomez]

Looks good to me! (Please wait for someone else than me to approve before merging as I'm not much knowledgeable about this area in the code)

[syphar]

LGMT in general.

I'm not 100% certain how useful this metric will be around build failures, this probably depends on the kind of failure / when it fails.

But in my mind #1011 would add more visibility, presumably even recording per-target data / logs / metrics.

[Nemo157]

Yeah, per-target logs were the main reason I was working on #1011. I'm not sure if there's very useful metrics we can extract to prometheus after it, but I do think we can record per-target times and surface those on the builds page, and I have some thoughts on trying to show warnings to crate owners where we could potentially inform them if they're approaching the timeout.

[Nemo157]

One thought on failures, I could partition this based on whether the build succeeded, do you think that would be useful?

[syphar]

One thought on failures, I could partition this based on whether the build succeeded, do you think that would be useful?

It seems like the intent of this metric is more or less "build time capacity" or something like that, or how much time builds by crate consume. And for the time consumption in general it doesn't matter if it's a build failure or not.

The real interesting information will come when we start recording per-target build information in addition to the logs, like the duration, if the build did timeout or OOM, ...