Feature: Global configuration

[JoshKeegan]

Add a global configuration that would allow for defaults to be set for an entire test project.
This would be at the xRetry level (so would apply to specflow and xunit tests) and cover:

• maxRetries
• delay

Idea came from #71, but is implemeted there in a specflow-only manner.

[mtaghavi2005]

We already need to have this feature in our test project because there are many feature files (over 300 scenarios) and somehow tests are randomly failing. So, it would be great if we can set it globally for all scenarios. If it is ok for you, I will create PR for this feature.

[dylbarne]

@JoshKeegan in addition to the global configuration options you outlined, I think it would also be nice to provide some global condition to dynamically configure whether or not the test running executes a retry at all.

Usecase: be able to keep existing code flagged with [RetryFact] etc., but then conditionally disable the retry functionality temporarily during local development at a global level without making individual test method attribute changes.
Most often during initial development or later troubleshooting of flaky tests the retry logic just gets in the way since the test is likely failing the first time already.
Ideally would just disable retry functionality in a development/debugging context globally and the retries would only occur in a build pipeline context where this is more applicable.

[matthetherington]

This would be very useful for us. We have a test suite that is occasionally flakey when ran during CI due to limited resources, but fine locally.

I'd like a test to fail after just 1 attempt locally, so that error messages are surfaced quickly, but then retry 3 times when running the test suite in CI/CD - there doesn't appear to be a way to do this at present.

[matthetherington]

For anyone else in the same boat, I ended up making these:

public interface ICiAwareRetryAttribute
{
    private const int DefaultLocalRetries = 1;
    private const int DefaultCiRetries = 3;
    
    protected static bool IsCi()
    {
        return string.Equals(
            Environment.GetEnvironmentVariable("CI"),
            "true",
            StringComparison.OrdinalIgnoreCase
        ) || !string.IsNullOrWhiteSpace(Environment.GetEnvironmentVariable("GITHUB_ACTIONS"));
    }

    protected static int GetLocalRetryCount()
    {
        var environmentOverride = Environment.GetEnvironmentVariable("XRETRY_LOCAL_RETRIES");
        
        if(!string.IsNullOrEmpty(environmentOverride) && int.TryParse(environmentOverride, CultureInfo.InvariantCulture, out var localRetries))
        {
            return localRetries;
        }
        
        return DefaultLocalRetries;
    }
    
    protected static int GetCiRetryCount()
    {
        var environmentOverride = Environment.GetEnvironmentVariable("XRETRY_CI_RETRIES");
        
        if(!string.IsNullOrEmpty(environmentOverride) && int.TryParse(environmentOverride, CultureInfo.InvariantCulture, out var ciRetries))
        {
            return ciRetries;
        }
        
        return DefaultCiRetries;
    }
}

public sealed class CiAwareRetryFactAttribute : RetryFactAttribute, ICiAwareRetryAttribute
{
    public CiAwareRetryFactAttribute(
        int delayBetweenRetriesMs = 0
    )
        : base(ICiAwareRetryAttribute.IsCi() ?
            ICiAwareRetryAttribute.GetCiRetryCount() :
            ICiAwareRetryAttribute.GetLocalRetryCount(),
            delayBetweenRetriesMs) { }
}

public sealed class CiAwareRetryTheoryAttribute : RetryTheoryAttribute, ICiAwareRetryAttribute
{
    public CiAwareRetryTheoryAttribute(
        int delayBetweenRetriesMs = 0
    )
        : base(ICiAwareRetryAttribute.IsCi() ?
                ICiAwareRetryAttribute.GetCiRetryCount() :
                ICiAwareRetryAttribute.GetLocalRetryCount(),
            delayBetweenRetriesMs) { }
}

then using [CiAwareRetryFact] / [CiAwareRetryTheory] in place of [RetryFact] and [RetryTheory]