Add precise_output argument to test optimize.

[Kmeakin]

Allow test optimize tests to take the precise-output argument, and to be updated by CRANELIFT_TEST_BLESS, similar to test compile

[jameysharp]

This is neat! It's clearly handy to be able to regenerate test expectations like this.

I don't know for sure that precise output tests are a good idea for optimizations though. Writing assertions about only the specific aspects we're trying to test makes the tests more resilient to unrelated changes in the optimizer.

I'm curious if @cfallin or @fitzgen have opinions on this.

[Kmeakin]

I don't know for sure that precise output tests are a good idea for optimizations though. Writing assertions about only the specific aspects we're trying to test makes the tests more resilient to unrelated changes in the optimizer.

Could we do a script to automatically insert the correct filecheck commands? LLVM has something similar: https://github.com/llvm/llvm-project/blob/main/llvm/utils/update_any_test_checks.py

[cfallin]

I don't know for sure that precise output tests are a good idea for optimizations though. Writing assertions about only the specific aspects we're trying to test makes the tests more resilient to unrelated changes in the optimizer.

I'm curious if @cfallin or @fitzgen have opinions on this.

Yes, I think we want to retain this property: many of the tests are checking just "this one value becomes X" rather than freezing the entire output. There is still some unnecessary capture of implementation details (e.g. the value number in the output exposes how many rewrites we went through) but less coupling is better here.

It would be nice to try to find ways to automate updates though!

[fitzgen]

Yeah for tests that are asserting one particular peephole applied, it is nice to not have the full precise output. There are certainly other tests that are written in a less targeted way, and could be migrated to precise output tests on a case-by-case basis. Because of this, it would be nice to support, even if we don't blanket enable it for every test optimize test.

[jameysharp]

Given that Nick thinks we'll find uses for this, let's go ahead and merge it. I've reviewed it and it's good, but I do have one thing I want changed before merge:

Let's share check_precise_output and update_test between all subtest modes that have a precise-output option. I think it makes sense to put these functions in cranelift/filetests/src/subtest.rs since they only rely on Context. Optionally, they could be methods on Context.

If I'm not mistaken, this copy of update_test is identical to the one in test_compile.rs, and this copy of check_precise_output is almost identical except for requiring the caller to provide a slice of strings. The latter is easily unified by moving the definition of actual to run in test_compile.rs.

Looking at this diff, there are several things I'd like to change in these functions, but they're all things that were pre-existing in test_compile.rs, so let's not change them in this PR.

[jameysharp]

Looks good to me, thank you!