improving and fixing failure detection

[shlomi-noach]

This PR will improve upon the failure detection process, by:

• prioritizing detection by severity
• more useful output

[nickvanw]

I believe the proper behavior here would be to pass analysisEntry into the goroutine:

go func(analysisEntry inst.ReplicationAnalysis) {
    // call things in here
}(analysisEntry)

[shlomi-noach]

@nickvanw thank you; I became accustomed to this paradigm following https://golang.org/doc/effective_go.html#channels (req := req).
I thought at first it looked funny, but then it is also very explicit and clear.

[shlomi-noach]

Time to elaborate. This PR fixes a subtle bug which does not manifest consistently. The subtle fix is on https://github.com/github/orchestrator/pull/196/files#diff-86933c5afedc1f1a02e011c53af7b039R1366

The executeCheckAndRecoverFunction() function was called within a goroutine even while analysisEntry was iterating replicationAnalysis, which means by the time the goroutine actually executed, analysisEntry would have switched to a/the next value.

This doesn't manifest in a consistent manner because of the nature of unpredictable concurrency. It did manifest more on prior array entries than on latter array entries.

When this PR is merged (being hammered right now) I'll release a version as this is a critical recovery fix. I'm surprised it hasn't surfaced sooner, to be honest.

[shlomi-noach]

This PR adds a distinction between an actionably-recoverable problem (e.g. DeadMaster) and one that hasn't got a recovering action (e.g. UnreachableMaster).

It allows orchestrator to register up to one actionable and one non-actionable entry in the backend topology_failure_detection table.

Detection is known to be of an escalating nature. A dying master may for example first be diagnosed as UnreachableMaster and then as DeadMaster. While recoveries work for such escalations (pending the subtle fix in this PR), detection only shows the first met analysis, and this causes for lacking visibility.

With the changes in this PR we will get more information via topology_failure_detection and, as result, via /api/audit-failure-detection and /web/audit-failure-detection.

[shlomi-noach]

an also more logging to be able to figure out what's up with detection at any step. And logging is Info rather than Debug because it's important™.

[shlomi-noach]

Recent commit also intentionally changes recovery iteration to random order.