Updated command line tool

[t00sa]

Create a new fab command line utility to replace the existing one. Feature list includes:

• Create separate build and system loggers
• Allow build and system levels to be set independently
• Add --quiet mode to suppress almost all output
• Add --detailed mode to show commands being executed
• Log everything to a file in the project directory
• Improved progress indicators with the optional rich library
  • Add spinners to tasks of unknown duration
  • Add progress bars to multiprocessing pools
  • Use rich log handlers
  • Use rich generalised exception handling
  • Support build targets, assuming they are defined in the build recipe
  • Support --fresh option to remove an existing workspace
  • Support -DFOO=BAR style command line options
  • Unrecognised options should be passed through to the build recipe
  • Support for getting help from from the build recipe argument parser, if any
• Basic recipe classes including:
  • A base class which sets up infrastructure for other recipes
  • A zero config recipe derived from the base class
  • LFRic-Baf support

[t00sa]

Note: the branch currently contains some sleep() calls to slow things down, to allow the progress monitoring to be tested. These will need to be removed before the changes go for review.

[t00sa]

Proof of concept with rich support and zero-config mode:

[hiker]

Hi Sam, thanks a lot for this. I would really suggest that we should have a chat first, because a lot of the things you have done in Fab are already in BAF. For example, a zero-config setup based on BAF can be done with (for time reasons I didn't bother adding command line options to specify the name, I hard-coded "my_prog"):

class FabCli(BafBase):

    def __init__(self):
        super().__init__("my_prog")

    def setup_site_specific_location(self):
        '''This will likely not be necessary once the base class is in Fab, since it will know where to find the compiler-specific setup files'''
        root_dir = Path(__file__).parent
        sys.path.insert(0, str(root_dir))
        sys.path.insert(0, str(root_dir / "site_specific"))

    def find_source_files_step(self):      # That's the only real code required
        find_source_files(self.config,
                          source_root=".")

if __name__ == "__main__":
    zero_config = FabCli()
    zero_config.build()

Yes, I am aware that your code does a lot more (different logger, spinning display, ...). But the above lines will already pickup compiler and flags from $FC, $CC $FFLAGS (and allow you to specify them as command line options if you prefer) allow you to specify if you want MPI or not (or openmp or not), compilation-profile (full-debug), site-specific setup, ...

You have mind of started re-implementing BAF to some degree :(

Let me discuss your todo list:

• Allow --file to override default script location

I don't really see a need for a wrapper script. Just start the python script. For zero-config, Fab will provide a fab command line tool (like the one above, with some additional command line options)

• Try to use FabFile in the current working directory if --file has not be set

• Use a built-in zero-config recipe if there is no FabFile

Not necessary at all :)

Improved logging

Create separate build and system loggers

I am not sure what exactly a build and what a system logger are supposed to be. I assume 'system' would be for the suite? But yes, that's great to have (well, using python logger we already have different loggers, and I have already done some work on logging for Jules to disable the Fab output by default).

Allow build and system levels to be set independently

Add --quiet mode to suppress almost all output

Add --detailed mode to show commands being executed

That's great!

Log everything to a file in the project directory

That is already the case with Fab.

Improved progress indicators with the optional rich library

Hmm - I don't know that, but I assume it will be disabled for batch build? Nothing as annoying as batch log files filled with rows after row of ===== 11% ==== :)

Add spinners to tasks of unknown duration

Again, please not for batch builds.

Add progress bars to multiprocessing pools

Use rich log handlers

That feels like a lot of eye candy, which imho could wait till 2027 :) It doesn't help us getting Fab used now.

Use rich generalised exception handling

Sure, you have a ticket for that. FWIW, Fab checks availability of a tool (and you can't add a tool to a ToolBox if it's doesn't run). So yes, there is just one kind of RunTime error raised, but it will always be caused by an exception raised by the tool. If a tool doesn't work/doesn't exist it will be caught way earlier. But agreed, some nice hierarchy of exceptions can't hurt :)

Extra command line options

Support build targets, assuming they are defined in the build recipe

What do you call build targets? Profile modes (that already exist), or the name of the program to execute?

Support --fresh option to remove an existing workspace

Yes, that's great to have.

Support -DFOO=BAR style command line options

For what exactly? But that's again easy to do in Baf. ATM we are using command line options (e.g. --mpi will automatically add USE_MPI in LFRic). Similarly, precision will be controlled via command line options (which will then be translated into preprocessor directives). The nice advantage of this is if the implementation should change (remember in the UM there was a lot of effort to replace preprocessor directives ;) ), we only need to change the build script, not all suites that might have long list of defines.

Unrecognised options should be passed through to the build recipe

This is related to the wrapper script, and I am not certain of the need to have one. BAF-based scripts will handle all command lines, and errors will then be reported.

Support for getting help from from the build recipe argument parser, if any

Done in BAF.

Basic recipe classes including:

Done in BAF.

A base class which sets up infrastructure for other recipes

That is BAF (or some part of it)

A zero config recipe derived from the base class

See my above example code.

My feeling is that we should sit together, and have a look at BAF. While I believe there is some resistance in merging BAF into Fab, I might try to create a PR for this. We can discuss this at the next meeting.

An additional general remark: You are doing a lot of once in one PR. Doing smaller steps would speed up the review process (and maybe us agreeing on priorities would also help a lot). IMHO, we need to be very strict with postponing non-critical items till some time next year :)