Repository navigation
[Testing Platform] Batch Filter #3528
Description
This would work alongside #3527.
A filter type that could specify a start index and an end index, and pass that filter through to test adapters.
Test adapters could implement this and skip/batch as appropriate. All the testing platform needs to do is pass through a new filter type with the start/end indexes on it.
This would enable users/build systems to run batches of tests on different processes/agents to speed up test runs using parallelism.
Logic could be as follows:
--dotnet run --count-tests
500
dotnet run --batch-filter 0 99
dotnet run --batch-filter 100 199
dotnet run --batch-filter 200 299
dotnet run --batch-filter 300 399
dotnet run --batch-filter 400 499
public sealed class BatchFilter : ITestExecutionFilter
{
public required int StartIndex { get; init; }
public required int EndIndex { get; init; }
}
Activity
Evangelink commentedon Aug 13, 2024
This is assuming that the tests are always returned in the same order right?
I don't see a direct big value in this shape (I would prefer to split based on list of UIDs) but I don't think that's a bad suggestion so I'll mark it as feature request and see if we have some more interest before we work on implementation.
thomhurst commentedon Aug 13, 2024
Yes you're right they'd need to be in the same order. But if framework providers register it like they do your treenode filter, they can choose whether to support it or not.
My tests are source generated so should be deterministic afaik
thomhurst commentedon Aug 13, 2024
Splitting on uids works too. Wouldn't require specific ordering then, but would be a very verbose dotnet run command
thomhurst commentedon Aug 13, 2024
Also are you planning to let frameworks build their own filters? That's all still internal code isn't it?
Evangelink commentedon Aug 13, 2024
Yes, my idea was to be able to provide a filter file with either simply the list of uids OR to have a structured file so we are not only expecting UIDs but we could potentially do mix up of filters.
Yes and yes. We could start opening with the
[Experimental]. While we do that, I think you should be able to implement some custom with "hacky" solution. Register an extension that will provide this command line and when executed simply stores the info in the services, from your test framework, retrieve this custom service and get the data (e.g. start/end index) and voila.cc Marco Rossignoli (@MarcoRossignoli)
thomhurst commentedon Aug 13, 2024
I was actually thinking about mixes of filters earlier and whether that was worth supporting. You'd need a new type like
AggregateFilter(likeAggregateException) that wraps around multiple. Might be worth splitting that into its own issue if that sounds like a good idea to you?Evangelink commentedon Aug 13, 2024
Yes definitely!
MarcoRossignoli commentedon Aug 13, 2024
I think that to have complete filtering we should offer
Users can OR them, treenode is already supporting the globbing part we should add something like
--filterByUid-iniline "uid uid uid"--filterByUid-fromfile filtersIds.txt(where the filtersIds.txt is a file where we have 1 id per line).thomhurst commentedon Aug 13, 2024
You already have
TestNodeUidListFilterbut I don't know how to invoke it from the cli. I assume it's to support test explorers mainly ATM.MarcoRossignoli commentedon Aug 13, 2024
Yep it's only used by the protocol atm. If we expose the api in the cli we can fill that object too.