StatAnalysis takes a lot of time to run if the MPR is large (around 1 GB)

[pratiman-91]

Hi METplus team,

I am trying to use StatAnalysis for aggregating monthly stat outputs from using MPR (PointStats). The configuration works fine and takes around 4 minutes if the size of the MPR txt file is around 100 MB while it takes around more than 5 hours if the file size is more than 1 GB.

Please let me know if this can be speed-up.

Thanks
-Pratiman

[JohnHalleyGotway]

@pratiman-91, I see you're having issues running Stat-Analysis to aggregate matched pair (MPR) data. The process is taking much longer than you'd expect. I'd like to share a couple of details that may be involved...

Please take a look at MET issue #2690 which was the result of METplus Discussion #2364. The Stat-Analysis tool was writing a LOT of temp files across the NSF file system, which was very, very slow.

If you're running version 12.0 or later and it's acceptable on your system, try setting export MET_TMP_DIR="/tmp" to see what, if any, impact that has on runtime. That'll write to the local temporary directory which MIGHT be significantly faster than writing it across the file system.

An additional change to Stat-Analysis is slated for MET version 13.0, as described in MET issue #2698. It is included in the MET-13.0.0-beta1 development release. You shouldn't use development releases for anything other than testing, but we'd definitely appreciate any feedback you have.

Basically, Stat-Analysis has always performed a two-step process:

1. Parse the config file and command line to define a "default" job. Read all the input data specified with the -lookin command line options, apply the default job filtering criteria, and write the result to a temporary file.
2. For each job defined (either in the config file jobs entry or a single one on the command line), read data from that temporary file and perform the logic of that job.

The original design anticipated that people would be running multiple jobs with a config file, and applying common filtering criteria up front would save time in the long run. In practice, that may or may not actually be the case.

So in MET-13.0.0-beta1, if only a single job is defined (either through the config file or on the command line), it now SKIPS writing that temp file since doing so would NOT be more efficient.

You're welcome to try out that beta1 release and let us know if that improves your specific usage of Stat-Analysis.

Thanks, and please LMK how it goes.

[pratiman-91]

Hi @JohnHalleyGotway ,

Thank you for your quick reply. Currently, i am using version 6.1 of metplus, which I believe is MET version 12.1.1.

As suggested, I have tried it using Docker with the image dtcenter/metplus-analysis:7.0.0-beta1 and used export MET_TMP_DIR="/tmp". Unfortunately this is still taking similar time.

Here, I am using single monthly consolidated _mpr.txt file to run StatAnalysis. So, I believe it is not related with reading/writing multiple files.

-Pratiman

[pratiman-91]

Hi @JohnHalleyGotway ,

I am no expert on C++ code. However, I asked Claude about it and recieved this information that might be useful.

When running stat_analysis on large STAT files (e.g. ~1 GB), the tool generates millions of individual write() syscalls to its temporary file, one per retained line. This is caused by unbuffered (or minimally buffered) line-by-line I/O in process_stat_file() and has a measurable performance cost at scale.

In process_stat_file(), every retained STAT line is written individually:
while(f >> line) { if(line.is_header() || job.is_keeper(line)) { tmp_out << line; // one write() syscall per line } }

The default std::ofstream buffer is small (typically 8 KB), meaning at millions of lines, this syscall overhead accumulates significantly.

[JohnHalleyGotway]

@pratiman-91 thanks for the version details and even for testing out that beta1 version. I really appreciate the feedback.

Are you able to send me (or point me to, on NCAR HPC or NOAA HPC) a sample call to Stat-Analysis?

I'd need your single monthly consolidated _mpr.txt file, your Stat-Analysis config file (if you're using one), and the command line call.

That'd enable me to understand what's going on.

[pratiman-91]

@JohnHalleyGotway
I have copied the sample _mpr.txt and StatAnalysis configuration file to /glade/work/pate1061/StatAnalysis on NCAR HPC. Let me know if you are unable to access the files.

Thanks for your help!

[pratiman-91]

Hi @JohnHalleyGotway,

I would like to know if there are any updates on this.

Thanks!