Bugfix #3370 develop tc_rmw BEST tracks#3372
Merged
JohnHalleyGotway merged 17 commits intodevelopfrom May 7, 2026
Merged
Conversation
JohnHalleyGotway
moved this from 🩺 Needs Triage
to 🔎 In review
in METplus-13.0 Development
23 tasks
KathrynNewman
requested changes
Apr 29, 2026
Contributor
KathrynNewman
left a comment
KathrynNewman
left a comment
There was a problem hiding this comment.
Testing revealed that the bug fix enabled processing using best track data, but is missing desired functionality:
- In tc-pairs, the config option best_technique is used to list the input data following the Best Track format. This should be added to enable proper parsing of the Best Track files.
- Add warning to skip valid times not available in the model output.
… User's Guide accordingly.
…ique and update the unit tests.
…to the TrackPoint class. This enables me to keep report the track lines point by point in the output of TC-RMW rather than only having them for the entire track.
…ck points used.
Collaborator
Author
|
@KathrynNewman I've made all the changes we discussed on this branch, and they are ready for your review and testing in: |
…p_tc_rmw_best_tracks
|
KathrynNewman
approved these changes
May 7, 2026
Contributor
KathrynNewman
left a comment
KathrynNewman
left a comment
There was a problem hiding this comment.
Additional code updates tested, working as expected. Documentation updates look good.
github-project-automation
Bot
moved this from 🔎 In review
to 🏁 Done
in METplus-13.0 Development
JohnHalleyGotway
added a commit
that referenced
this pull request
May 7, 2026
* Hotfix to develop to fix the update_truth.yml workflow logic. This testing workflow run failed (https://github.com/dtcenter/MET/actions/runs/9209471209). Here we switch to a unique update truth branch name to avoid conflicts. * Avoid pushing directly to the develop or main_vX.Y branches since that is not necessary for the automation logic in MET. * #2904 Changed R path to R-4.4.0 (#2905) Co-authored-by: Howard Soh <hsoh@seneca.rap.ucar.edu> * Feature #2912 pb2nc error (#2914) * Feature 2717 convert unit.pl to unit.py (#2871) * created unit.py module in new internal/test_unit/python directory * added xml parsing to unit.py * added repl_env function * added reading of the remaining xml tags in build_tests function * progress on main function (putting together test commands) * a few more lines in the main function * minor updates * fixed how the test command was being run * added if name/main and command line parsing * fixed handling of no 'env' in cmd_only mode * handle params from xml that have \ after filename without space in between * added logging * added some more pieces to unit * more updates to unit.py, including running checks on output files * bug fixes, improved handling of output file names, improved handling of env vars, improved logging output * fixed how shell commands are run, and other minor fixes * added last bits from the perl script, fixed some bugs * created unit.py module in new internal/test_unit/python directory * added xml parsing to unit.py * added repl_env function * added reading of the remaining xml tags in build_tests function * progress on main function (putting together test commands) * a few more lines in the main function * minor updates * update scripts to call python unit test script instead of the old perl script * fix she-bang line to allow script to be run without python3 before it * add missing test_dir and exit_on_fail tags that are found in the rest of the unit test xml files * fix call to logger.warning * change tags named 'exists' to 'exist' to match the rest of the xml files * added logger to function * removed tab at end of line that was causing output file path to be excluded from the command * fix broken checks for output files * incorporated george's recommended changes * changed default to overwrite logs; allow for more than one xml file to be passed in command --------- Co-authored-by: Natalie babij <natalie.babij@natbabij-mac1.home> Co-authored-by: Natalie babij <natalie.babij@vpn68-237.vpnclient.ucar.edu> Co-authored-by: Natalie babij <natalie.babij@gsd-sslvpn-209.fsl.noaa.gov> Co-authored-by: Natalie Babij <nbabij@seneca.rap.ucar.edu> Co-authored-by: John Halley Gotway <johnhg@ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> Co-authored-by: j-opatz <jopatz@ucar.edu> * Bugfix 2867 point2grid qc unittest (#2913) * #2867 Added compute_adp_qc_flag and adjusted ADP QC flags * #2867 Added point2grid_GOES_16_ADP_Enterprise_high. Changed AOD QC flags to 0,1,2 (was 1,2,3) * #2867 Added get_nc_att_values_ * #2867 Added get_nc_att_values. Added the argument allow_conversion to get_nc_data(netCDF::NcVar *, uchar *data) * #2867 Read the ADP QC flag values and meanings attributes from DQF variable and set the QC high, meduium, low values to support Enterprise algorithm. Adjusted the ADP QC values by using AOD qc values * #2867 Cleanup * #2867 Corrected indent * #2867 Changed log message * #2867 Removed unused argument * #2867 Removed unused argument * Cleanup * #2867 Fix SonarQube findings * #2867 Deleted protected section with no members * #2867 Cleanup * #2867 FIxed SonarQube findings; unused local variables, decalare as const, etc * #2867 MOved include directives to top * #2867 Changed some argumenmt with references to avoid copying objects * #2867 Do not filter by QC flag if -qc is not given * #2867 Use enumj class for GOES QC: HIGH, MEDIUM, and LOW * #2867 Added log message back which were deleted accidently * #2867 Chaned statci const to constexpr * #2867 Initial release. Separated from nc_utils.h * @2867 Added nc_utils_core.h * #2867 Moved some blocks to nc_utils_core.h * #2867 Include nc_utils_core.h * #2867 Added const references * #2867 Some 'static const' were chnaged to constexpr * #2867 Changed -qc options (1,2,3 to 0,1 - high & medium) for AOD * #2867 Merged develop branch * #2867 Corrected the unit test name --------- Co-authored-by: Howard Soh <hsoh@seneca.rap.ucar.edu> * Feature #2911 tc_stat_set_hdr (#2916) * Per #2911, no real changes for Stat-Analysis. Just changing order of variables for consistency. * Per #2911, add StatHdrColumns::apply_set_hdr_opts(...) function to be used by TC-Stat. * Per #2911, move ByColumn to the TCStatJob base class and add HdrName and HdrValue to support the -set_hdr job command. * Per #2911, update GSI tools to call the newly added StatHdrColumns::apply_set_hdr_opts(...) function. * Per #2911, update logic of Stat-Analysis for consistency to make use of common apply_set_hdr_opts() function. * Per #2911, add DataLine::set_item() function to support -set_hdr options. * Per #2911, just update contents of error message * Per #2911, add TCStatLine member functions for has() and get_offset(). * Per #2911, update tc_stat to support applying -set_hdr to TC-Stat filter jobs. * Per #2911, revise TC-Stat config files to exercise the -set_hdr job command option * Per #2911, update TC-Stat documentation to mention the -set_hdr job command option * Per #2911, add note * Per #2911, as recommended by SonarQube, make some of these member functions const. * Bugfix #2856 develop ens_climo (#2918) * Per #2856, port over fixes from main_v11.1 to develop. * Per #2856, correct conditionals in set_job_controls.sh and tweak existing Ensemble-Stat configuration file to exercise the logic that's being impacted here. * Bugfix #2841 develop tang_rad_winds (#2921) * Per #2841, port over fixes from bugfix_2841_main_v11.1_tang_rad_winds for the develop branch * Per #2841, clarify in the docs that azimuths are defined in degrees counter-clockwise from due east. * Per #2841, just updating with output from enum_to_string. * Per #2841, tweak the documentation. * Per #2841, correct the location of using namespace lines. * Per #2841, update compute_tc_diag.py to no longer skip writing the radial and tangential wind diagnostics. * Per #2841, update compute_tc_diag.py to no longer skip writing radial and tangential wind diagnostics. * Revert "Per #2841, update compute_tc_diag.py to no longer skip writing radial and tangential wind diagnostics." This reverts commit f097345bedcfcca663e8fb4322eed5b5e00e19fd. * Revert "Per #2841, update compute_tc_diag.py to no longer skip writing the radial and tangential wind diagnostics." This reverts commit c0402151b038c59efab99c060cc5c390edf002f6. * Per #2841, update comp_dir.sh logic to include .dat in the files that are diffed * Replace tab with spaces * Per #2841, correct the units for the azimuth netcdf output variable * Per #2841, reverse the x dimension of the rotated latlon grid to effectively switch from counterclockwise rotation to clockwise. --------- Co-authored-by: MET Tools Test Account <met_test@seneca.rap.ucar.edu> * Feature #2601 seeps climo config (#2927) * #2601 Added seeps_grid_climo_name and seeps_point_climo_name * #2601 Added seeps_grid_climo_name * #2601 Removed SEEPS settings * #2601 Initial release * #2601 Changed to set the SEEPS climo by using the configuration * #2601 Removed SEESP settings at PointStatConfig_APCP and use PointStatConfig_SEEPS for SEEPSm testing * #2601 Updated descryption for seeps_grid_climo_name * #2601 Added a argument for the SEEPS clomo file * #2601 Added conf_key_seeps_grid_climo_name and conf_key_seeps_point_climo_name * #2601 Support the climo filename from the configuration * #2601 Corrected key for climo name * Removing duplicate word --------- Co-authored-by: Howard Soh <hsoh@seneca.rap.ucar.edu> Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Feature 2673 sonarqube beta5 redundant parentheses (#2930) * #2673 Removed redundant_parentheses * #2673 Removed redundant_parentheses * #2673 Removed redundant parentheses * #2673 Removed redundant parentheses --------- Co-authored-by: Howard Soh <hsoh@seneca.rap.ucar.edu> * Fix release checksum action (#2929) * Feature 2857 tripolar coordinates (#2928) * #2857 Added MetNcCFDataFile::build_grid_from_lat_lon_vars * #2857 Added NcCfFile::build_grid_from_lat_lon_vars * #2857 Check the coordinates attribute to find latitude, longitude, and time variables * #2857 Get the lat/lon variables from coordinates attribute if exists * #2857 Added two constants * #2857 Deleted debug messages * #2857 Added lat_vname and lon_vname for var_name_map * #2857 Added two unit tests: point2grid_sea_ice_tripolar and point2grid_sea_ice_tripolar_config * #2857 Initial release * #2857 Correct dictinary to get file_type * #2857 DO not check the time variable for point2grid * #2857 Added point2grid_tripolar_rtofs --------- Co-authored-by: Howard Soh <hsoh@seneca.rap.ucar.edu> * Feature 2932 v12.0.0-beta5 (#2933) * Per #2932, updating version and release notes * Per #2932, updating date on release notes * Per #2932, fixed formatting and links * Update release-notes.rst * Update release-notes.rst Removing inline backticks since they do not format the way I expected, especially when put inside bolded release notes. --------- Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * Feature fix release notes (#2934) * Fixing up release notes * Update release-notes.rst --------- Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * Per dtcenter/METplus#2643 discussion, add more detail about the budget interpolation method. * Feature #2924 fcst climo, PR 1 of 2 (#2939) * Per #2924, Update the MPR and ORANK output line types to just write duplicate existing climo values, update the header tables and MPR/ORANK documentation tables. * Per #2924, update get_n_orank_columns() logic * Per #2924, update the Stat-Analysis parsing logic to parse the new MPR and ORANK climatology columns. * Per #2924, making some changes to the vx_statistics library to store climo data... but more work to come. Committing this first set of changes that are incomplete but do compile. * Per #2924, this big set of changes does compile but make test produces a segfault for ensemble-stat * Per #2924, fix return value for is_keeper_obs() * Per #2924, move fcst_info/obs_info into the VxPairBase pointer. * Per #2924, update Ensemble-Stat to set the VxPairBase::fcst_info pointer * Per #2924 udpate handling of fcst_info and obs_info pointers in Ensemble-Stat * Per #2924, update the GSI tools to handle the new fcst climo columns. * Per #2924, add backward compatibility logic so that when old climo column names are requested, the new ones are used. * Per #2924, print a DEBUG(2) log message if old column names are used. * Per #2924, switch the unit tests to reference the updated MPR column names rather than the old ones. * Per #2924, working progress. Not fully compiling yet * Per #2924, another round of changes. Removing MPR:FCST_CLIMO_CDF output column. This compiles but not sure if it actually runs yet * Per #2924, work in progress * Per #2924, work in progress. Almost compiling again. * Per #2924, get it compiling * Per #2924, add back in support for SCP and CDP which are interpreted as SOCP and OCDP, resp * Per #2924, update docs about SCP and CDP threshold types * Per #2924, minor whitespace changes * Per #2924, fix an uninitialized pointer bug by defining/calling SeepsClimoGrid::init_from_scratch() member function. The constructor had been calling clear() to delete pointers that weren't properly initialized to nullptr. Also, simplify some map processing logic. * Per #2924, rename SeepsAggScore from seeps to seeps_agg for clarity and to avoid conflicts in member function implementations. * Per #2924, fix seeps compilation error in Point-Stat * Per #2924, fix bug in the boolean logic for handling the do_climo_cdp NetCDF output option. * Per #2924, add missing exit statement. * Per #2924, tweak threshold.h * Per #2924, define one perc_thresh_info entry for each enumerated PercThreshType value * Per #2924, simplify the logic for handling percentile threshold types and print a log message once when the old versions are still used. * Per #2924, update the string comparison return value logic * Per #2924, fix the perc thresh string parsing logic by calling ConcatString::startswith() * Per #2924, switch all instances of CDP to OCDP. Gen-Ens-Prod was writing NetCDF files with OCDP in the output variable names, but Grid-Stat was requesting that the wrong variable name be read. So the unit tests failed. * Per #2924, add more doc details * Per #2924, update default config file to indicate when climo_mean and climo_stdev can be set seperately in the fcst and obs dictionaries. * Per #2924, update the MET tools to parse climo_mean and climo_stdev separately from the fcst and obs dictionaries. * Per #2924, backing out new/modified columns to minimize reg test diffs * Per #2924, one more section to be commented out later. * Per #2924, replace several calls to strncmp() with ConcatString::startswith() to simplify the code * Per #2924, strip out some more references to OBS_CLIMO_... in the unit tests. * Per #2924, delete accidental file * Per #2924 fix broken XML comments * Per #2924, fix comments * Per #2924, address SonarQube findings * Per #2924, tweak a Point-Stat and Grid-Stat unit test config file to make the output more comparable to develop. * Per #2924, fix bug in the logic of PairDataPoint and PairDataEnsemble, when looping over the 3-dim array do not return when checking the climo and fcst values. Instead we need to continue to the next loop iteration. * Per #2924, address more SonarQube code smells to reduce the overall number in MET for this PR. * Per #2924, correct the logic for parsing climo data from MPR lines. * Per #2924, cleanup grid_stat.cc source code by making calls to DataPlane::is_empty() and Grid::nxy(). * Per #2924, remove unneeded ==0 * Hotfix to the develop branch for a copy/paste bug introduced by PR #2939 * Feature #2924 sal1l2_mae, PR 3 of 3 (#2943) * Per #2924, track SL1L2 and SAL1L2 MAE scores with separate variables since they are no longer the same value. I renamed the existing 'mae' as 'smae' and added a new 'samae' variable. Renaming the existing lets me use the compiler help find all references to it throughout the code. * Per #2924, update the User's Guide climatology details and equations. * Per #2924, some changes to aggr_stat_line.cc and series_analysis.cc to satisfy some SonarQube code smells. * Update develop to clarify masking poly options based on METplus Discussion dtcenter/METplus#2650 * Remove two semi-colons that are not actually necessary to avoid confusion. * Per dtcenter/METplus#2653 discussion, update the MTD usage statement to clarify that data specified in the fcst dictionary is read from the -single input files. * Feature #2924 fcst climo, PR 2 of 3 (#2942) * Per #2924, Update the MPR and ORANK output line types to just write duplicate existing climo values, update the header tables and MPR/ORANK documentation tables. * Per #2924, update get_n_orank_columns() logic * Per #2924, update the Stat-Analysis parsing logic to parse the new MPR and ORANK climatology columns. * Per #2924, making some changes to the vx_statistics library to store climo data... but more work to come. Committing this first set of changes that are incomplete but do compile. * Per #2924, this big set of changes does compile but make test produces a segfault for ensemble-stat * Per #2924, fix return value for is_keeper_obs() * Per #2924, move fcst_info/obs_info into the VxPairBase pointer. * Per #2924, update Ensemble-Stat to set the VxPairBase::fcst_info pointer * Per #2924 udpate handling of fcst_info and obs_info pointers in Ensemble-Stat * Per #2924, update the GSI tools to handle the new fcst climo columns. * Per #2924, add backward compatibility logic so that when old climo column names are requested, the new ones are used. * Per #2924, print a DEBUG(2) log message if old column names are used. * Per #2924, switch the unit tests to reference the updated MPR column names rather than the old ones. * Per #2924, working progress. Not fully compiling yet * Per #2924, another round of changes. Removing MPR:FCST_CLIMO_CDF output column. This compiles but not sure if it actually runs yet * Per #2924, work in progress * Per #2924, work in progress. Almost compiling again. * Per #2924, get it compiling * Per #2924, add back in support for SCP and CDP which are interpreted as SOCP and OCDP, resp * Per #2924, update docs about SCP and CDP threshold types * Per #2924, minor whitespace changes * Per #2924, fix an uninitialized pointer bug by defining/calling SeepsClimoGrid::init_from_scratch() member function. The constructor had been calling clear() to delete pointers that weren't properly initialized to nullptr. Also, simplify some map processing logic. * Per #2924, rename SeepsAggScore from seeps to seeps_agg for clarity and to avoid conflicts in member function implementations. * Per #2924, fix seeps compilation error in Point-Stat * Per #2924, fix bug in the boolean logic for handling the do_climo_cdp NetCDF output option. * Per #2924, add missing exit statement. * Per #2924, tweak threshold.h * Per #2924, define one perc_thresh_info entry for each enumerated PercThreshType value * Per #2924, simplify the logic for handling percentile threshold types and print a log message once when the old versions are still used. * Per #2924, update the string comparison return value logic * Per #2924, fix the perc thresh string parsing logic by calling ConcatString::startswith() * Per #2924, switch all instances of CDP to OCDP. Gen-Ens-Prod was writing NetCDF files with OCDP in the output variable names, but Grid-Stat was requesting that the wrong variable name be read. So the unit tests failed. * Per #2924, add more doc details * Per #2924, update default config file to indicate when climo_mean and climo_stdev can be set seperately in the fcst and obs dictionaries. * Per #2924, update the MET tools to parse climo_mean and climo_stdev separately from the fcst and obs dictionaries. * Per #2924, backing out new/modified columns to minimize reg test diffs * Per #2924, one more section to be commented out later. * Per #2924, replace several calls to strncmp() with ConcatString::startswith() to simplify the code * Per #2924, strip out some more references to OBS_CLIMO_... in the unit tests. * Per #2924, delete accidental file * Per #2924 fix broken XML comments * Per #2924, fix comments * Per #2924, address SonarQube findings * Per #2924, tweak a Point-Stat and Grid-Stat unit test config file to make the output more comparable to develop. * Per #2924, fix bug in the logic of PairDataPoint and PairDataEnsemble, when looping over the 3-dim array do not return when checking the climo and fcst values. Instead we need to continue to the next loop iteration. * Per #2924, address more SonarQube code smells to reduce the overall number in MET for this PR. * Per #2924, correct the logic for parsing climo data from MPR lines. * Per #2924, update MPR and ORANK line types to update/add FCST/OBS_CLIMO_MEAN/STDEV/CDF columns. * Per #2924, cleanup grid_stat.cc source code by making calls to DataPlane::is_empty() and Grid::nxy(). * Per #2924, remove unneeded ==0 * Per #2924, working on PR2. * Per #2924, update User's Guide with notional example of specifying climo_mean and climo_stdev separately in the fcst and obs dicts. * Per #2924, adding a new unit test. It does NOT yet run as expected. Will debug on seneca * Per #2924, pass the description string to the read_climo_data_plane*() function to provide better log messages * Per #2924, more work on consistent log messages * Per #2924, tweak the configuration to define both field, climo_mean, and climo_stdev in both the fcst and obs dictionaries * Per #2924, tweak the unit_climatology_mixed.xml test * Per #2924, only whitespace changes. * Per #2924, missed swapping MET #2924 changes in 3 test files * Per #2924, delete accidentally committed file * Per #2924, delete accidentally committed files * Per #2924, add support for GRIB1 time range indicator value of 123 used for the corresponding METplus Use Case. Note that there are 22 other TRI values not currently supported. * Adds caveat regarding longitudes appearing in DEBUG statements with a… (#2947) * Adds caveat regarding longitudes appearing in DEBUG statements with a different sign to the FAQ. * Update appendixA.rst Missing paren * Create install_met_env.cactus * Adding special script for installing beta5 on wcoss2 * Modifying script, including updates to eckit and atlas * Corrected version of bufr being used * Feature #2938 pb2nc_center_time (#2954) * Per #2938, define CRC_Array::add_uniq(...) member function which is now used in PB2NC * Per #2938, replace n_elements() with n() to make the code more concise. Refine log/warning message when multiple message center times are encountered. * Feature #1371 series_analysis (#2951) * Per #1371, add -input command line argument and add support for ALL for the CTC, MCTC, SL1L2, and PCT line types. * Per #1371, rename the -input command line option as -aggregate instead * Per #1371, work in progress * Per #1371, just comments * Per #1371, working on aggregating CTC counts * Per #1371, work in progress * Per #1371, update timing info using time stamps in the aggr file * Per #1371, close the aggregate data file * Per #1371, define set_event() and set_nonevent() member functions * Per #1371, add logic to aggregate MCTC and PCT counts * Merging changes from develop * Per #1371, work in progress aggregating all the line statistics types. Still have several issues to address * Per #1371, switch to using get_stat() functions * Per #1371, work in progress. More consolidation * Per #1371, correct expected output file name * Per #1371, consistent regridding log messages and fix the Series-Analysis PairDataPoint object handling logic. * Per #1371, check the return status when opening the aggregate file. * Per #1371, fix prc/pjc typo * Per #1371, fix the series_analysis PCT aggregation logic and add a test to unit_series_analysis.xml to demonstrate. * Per #1371, resolve a few SonarQube findings * Per #1371, make use of range-based for loop, as recommeded by SonarQube * Per #1371, update series-analysis to apply the valid data threshold properly using the old aggregate data and the new pair data. * Per #1371, update series_analysis to buffer data and write it all at once instead of storing data value by value for each point. * Per #1371, add useful error message when required aggregation variables are not present in the input -aggr file. * Per #1371, print a Debug(2) message listing the aggregation fields being read. * Per #1371, correct operator+= logic in met_stats.cc for SL1L2Info, VL1L2Info, and NBRCNTInfo. The metadata settings, like fthresh and othresh, were not being passed to the output. * Per #1371, the DataPlane for the computed statistics should be initialized to a field of bad data values rather than the default value of 0. Otherwise, 0's are reported for stats a grid points with no data when they should really be reported as bad data! * Per #1371, update logic of the compute_cntinfo() function so that CNT statistics can be derived from a single SL1L2Info object containing both scalar and scalar anomaly partial sums. These changes enable CNT:ANOM_CORR to be aggregated in the Series-Analysis tool. * Per #1371, fix logic of climo log message. * Per #1371, this is actually related to MET #2924. In compute_pctinfo() used obs climo data first, if provided. And if not, use fcst climo data. * Per #1371, fix indexing bug (+i instead of +1) when check the valid data count. Also update the logic of read_aggr_total() to return a count of 0 for bad data. * Per #1371, add logic to aggregate the PSTD BRIERCL and BSS statistics in the do_climo_brier() function. Tested manually to confirm that it works. * Per #1371, switch to using string literals to satisfy SonarQube * Per #1371, update series_analysis tests in unit_climatology_1.0deg.xml to demonstrate aggregating climo-based stats. * Per #1371, remove extra comment * Per #1371, skip writing the PCT THRESH_i columns to the Series-Analysis output since they are not used * Per #1371, fix the R string literals to remove \t and \n escape sequences. * Per #1371, update the read_aggr_data_plane() suggestion strings. * Per #1371, ignore unneeded PCT 'THRESH_' variables both when reading and writing ALL PCT columns. * Per #1371, update the test named series_analysis_AGGR_CMD_LINE to include data for the F42 lead time that had previously been included for the same run in the develop branch. Note however that the timestamps in the output file for the develop branch (2012040900_to_2012041100) were wrong and have been corrected here (2012040900_to_2012041018) to match the actual data. * Per #1371, update the -aggr note to warn users about slow runtimes * Feature 2948 cxx17 (#2953) * Per #2948, updating versions of ecbuild, eckit, and atlas * Per #2948, Adding MET_CXX_STANDARD * Per #2948, updated wording for MET_CXX_STANDARD description * Per #2948, updating script to work with two versions of ecbuild, eckit, and atlas * Per #2948, without this change, there are compilation problems if the user wants to compile wihtout python * Per #2948, fixing logic for MET_CXX_STANDARD * Per #2928, adding missing end bracket * Per #2948, fixed the logic for compiling versions of ecbuild, eckit, and atlas * Per 948, fixed syntax for setting CXXFLAGS * Per #2948, adding new Makefile.in files and configure and changing METbaseimage 3.2 to 3.3. * Per #2948, updating version of met base tag from 3.2 to 3.3 * Per #2948, adding --enable-all MET_CXX_STANDARD=11 job * Update compilation_options.yml * Per #2948, added a job10 for MET_CXX_STANDARD=14 * Per #2948, added brief documentation for the MET_CXX_STANDARD option --------- Co-authored-by: Julie Prestopnik <jpresto@seneca.rap.ucar.edu> Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * Feature 1729 set attr grid (#2955) * #1729 Allow to change to differnt grid size if the raw size is 0 * Added build_grid_by_grid_string and build_grid_by_grid_string * #1729 Calls build_grid_by_grid_string * #1729 Added set_attr_grid at the -field option * #1729 Set obs_type to TYPE_NCCF if the file_type is given at the config file * #1729 Support set_sttr_grid and changed Error messages to Warning * #1729 FIxed SonmarQube findings * #1729 Initial release for unit test * #1729 Added update_missing_values * #1729 Deleted a shadowed local variable * #2673 Added more is_eq * #2673 Added get_exe_duration * 2673 Reducded nested statements * 2673 Fixed SonarGube findings * 2673 Fixed SonarQube findings * 2673 Fixed SonarQube findings * #1729 Added aan unittest plot_data_plane_set_attr_grid * #1729 Added aan unittest point2grid_cice_set_attr_grid * #1729 Added changed back the verbose level * #1729 Corrected typo --------- Co-authored-by: Howard Soh <hsoh@seneca.rap.ucar.edu> * Bugfix #2958 develop BAGSS SEDI CI (#2959) * Bugfix 2936 point2grid gfs (#2964) * #2936 Support 1D lat/lon values * #2936 Initial release * #2936 Cast the data type to avoid a compile warning * #2936 Added an unittest point2grid_gfs_1D_lat_lon --------- Co-authored-by: Howard Soh <hsoh@seneca.rap.ucar.edu> * Bugfix 2968 point2grid set attr grid (#2969) * #2968 Corrected set_attr_grid for point2grid_cice_set_attr_grid * #2968 Compare the DataPlane size and the variable data size * #2968 nx and ny are not ignored with set_attr_grid * #2968 Compare the DataPlane size and the variable data size --------- Co-authored-by: Howard Soh <hsoh@seneca.rap.ucar.edu> * Feature 2937 update unit (#2944) * added single quotes around env var/val pairs in export statements in cmd only mode * updated logic in unit() to check exec return value against expected return value; created TEST xml file to test this feature * deleted TEST_ xml, added test with retval 1 to unit_ascii2nc --------- Co-authored-by: Natalie Babij <nbabij@seneca.rap.ucar.edu> * Feature #2887 categorical weights PR 1 of 2 (#2967) * Per #2887, update NumArray::vals() to return a reference to the vector rather a pointer to doubles. * Per #2887, switch over the whole ContingencyTable class heirarchy from storing integer counts to storing double-precision weights. * Add ContingencyTable::is_integer() member function to check whether the table contains all integers * Per #2887, update parse_stat_line.cc to get it to compile after changing PCT to store thresholds in a std::vector. * Per #2887, update PCTInfo::clear() logic. * Per #2887, update ctc_by_row() logic to create reproducible results with the develop branch. * Per #2887, update logic of define_prob_bins() to add a final >=1.0 threshold if needed. While ==0.1 works fine, I found that ==0.05 did not because the last >=1.0 threshold was missing likely do to floating point precision issues. This change should fix that problem. * Per #2887, update roc_auc() function to match the develop branch * Per #2887, fix bug if computation of far() * Per #2887, replaced all ==0 integer equality checks with calls to is_eq() instead and fix a couple of equations to snuff out diffs in some CTS statistics. * Per #2887, address some of the 34 SonarQube code smells flagged for this PR. Note that the compute_ci.h/.cc changes are necessary and good since we should be computing CI's using doubles instead of integer counts. * Per #2887, update run_sonarqube.sh to specify the target CXX standard as 11. The hope is that that will limit the findings to only those features available in the C++11 standard. * Per #2887, update to SonarQube version 6.1.0.4477 released on 6/27/2024. * Per #2887, updating build_met_sonarqube.sh to specify --std=c++11 since c++17 is used by default * Hotfix to develop to fix a bug introduced for MET #2887. Refine the define_prob_bins() utility function so that ==n probability thresholds result in the correct number of probability thresholds. We were adding an unncessary 10-th bin (from 1.07143 to 1.0) for the ==7 probability threshold type. * Fix typo in tc-pairs.rst * Update build_docker_and_trigger_metplus.yml The docs directory was moved up to the top-level of the repository but this workflow was not updated. Changing the ignore setting so that doc-only updates do not trigger the full METplus testing workflow. * Feature 2023 remove double quotes around keywords (#2974) * testing AREA and AUTO changes * Keywords B thru L * thru R * adding quotes back in for lower case items * S thru the end of the document * Removing double quotes around 3 key words * Per #2023, adding a label name for the Attributes section * Per #2023, adding an internal link for the MODE tool Attributes section. * Adding quotes around Valid basins entries * more double quote updates * more complex updates with Julie P help * removing double quotes * fixing typos * removing double quotes * unbolding SURFACE and putting it in double quotes * fixing grammar * grammar * fixing typo * fixing typo --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Feature #2924 parse_config (#2963) * Per #2924, remove GenEnsProd config file comment about parsing desc separately from each obs.field entry because the obs dictionary does not exist in the GenEnsProd config file. * Per #2924, update list of needed config entry names * Per #2924, remove const from the parent() member function so that we can perform lookups for the parent. * Per #2924, update the signature for and logic of the utility functions that retrieve the climatology data. Rather than requiring all the climo_mean and climo_stdev dictionary entries to be defined at the same config file context level, parse each one individually. This enables the METplus wrappers to only partially override this dictionary and still rely on the default values provided in MET's default configuration files. * Per #2924, update all calls to the climatology utility functions based on the new function signature. Also update the tools to check the number of climo fields separately for the forecast and observation climos. * Per #2924, update the parsing logic for the climatology regrid dictionary. Use config.fcst.climo_mean.regrid first, config.fcst.regrid second, and config.climo_mean.regrid third. Notably, DO NOT use config.regrid. This is definitely the problem with having regrid specified at mutliple config file context levels. It makes the logic for which to use when very messy. * Per #2924, forgot to add an else to print an error * Per #2924, remove extraneous semicolon * Per #2924, move 'fcst.regrid' into 'fcst.climo_mean.regrid'. Defining the climatology regridding logic inside fcst is problematic because it applies to the forecast data as well and you end up with the verification grid being undefined. So the climo regridding logic must be defined in 'climo_mean.regrid' either within the 'fcst' and 'obs' dictionaries or at the top-level config context. * Per #2924, based on PR feedback from @georgemccabe, add the Upper_Left, Upper_Right, Lower_Right, and Lower_Left interpolation methods to the list of valid options for regridding, as already indicated in the MET User's Guide. * Per #2924, update the logic of parse_conf_regrid() to (hopefully) make it work the way @georgemccabe expects it to. It now uses pointers to both the primary and default dictionaries and parses each entry individually. * Per #2924, need to check for non-null pointer before using it * Per #2924, revise the climo_name dictionary lookup logic when parsing the regrid dictionary. * Per #2924, update logic for handling RegridInfo * Per #2924, remove the default regridding information from the 'Searching' log message to avoid confusion. --------- Co-authored-by: MET Tools Test Account <met_test@seneca.rap.ucar.edu> * Feature #2924 parse_config PR 2 (#2975) * Per #2924, remove GenEnsProd config file comment about parsing desc separately from each obs.field entry because the obs dictionary does not exist in the GenEnsProd config file. * Per #2924, update list of needed config entry names * Per #2924, remove const from the parent() member function so that we can perform lookups for the parent. * Per #2924, update the signature for and logic of the utility functions that retrieve the climatology data. Rather than requiring all the climo_mean and climo_stdev dictionary entries to be defined at the same config file context level, parse each one individually. This enables the METplus wrappers to only partially override this dictionary and still rely on the default values provided in MET's default configuration files. * Per #2924, update all calls to the climatology utility functions based on the new function signature. Also update the tools to check the number of climo fields separately for the forecast and observation climos. * Per #2924, update the parsing logic for the climatology regrid dictionary. Use config.fcst.climo_mean.regrid first, config.fcst.regrid second, and config.climo_mean.regrid third. Notably, DO NOT use config.regrid. This is definitely the problem with having regrid specified at mutliple config file context levels. It makes the logic for which to use when very messy. * Per #2924, forgot to add an else to print an error * Per #2924, remove extraneous semicolon * Per #2924, move 'fcst.regrid' into 'fcst.climo_mean.regrid'. Defining the climatology regridding logic inside fcst is problematic because it applies to the forecast data as well and you end up with the verification grid being undefined. So the climo regridding logic must be defined in 'climo_mean.regrid' either within the 'fcst' and 'obs' dictionaries or at the top-level config context. * Per #2924, based on PR feedback from @georgemccabe, add the Upper_Left, Upper_Right, Lower_Right, and Lower_Left interpolation methods to the list of valid options for regridding, as already indicated in the MET User's Guide. * Per #2924, update the logic of parse_conf_regrid() to (hopefully) make it work the way @georgemccabe expects it to. It now uses pointers to both the primary and default dictionaries and parses each entry individually. * Per #2924, need to check for non-null pointer before using it * Per #2924, revise the climo_name dictionary lookup logic when parsing the regrid dictionary. * Per #2924, update logic for handling RegridInfo * Per #2924, remove the default regridding information from the 'Searching' log message to avoid confusion. * Per #2924, escape sequences, like \n, cannot be used inside R-string literals. * Per #2924, update the logic of check_climo_n_vx() * Per #2924, revise logic in read_climo_data_plane_array(). Check the number of climo fields provided. If there's 0, just return since no data has been requested. If there's 1, use it regardless of the number of input fields. If there's more than 1, just use the requested i_vx index value. * Per #2924, update Series-Analysis to set both i_fcst and i_obs when looping over the series entries. * Per #2924, no real change. Just whitespace. * Unrelated to #2924, superficial changes to formatting of method_name strings for consistency. * Per #2924, add a new series_analysis test that ERRORS OUT prior to this PR but works after the changes in this PR. --------- Co-authored-by: MET Tools Test Account <met_test@seneca.rap.ucar.edu> * Feature 2949 cxx11 doc (#2973) * Per #2949, updating installation instructions * Per #2949, adding missing colon from note directive * Per #2949, third attempt to get the new note to show up * Per #2949, modifying text and format * Per #2949, removing images in favor of code blocks for easier modification * Per #2949, modified wording for clarity * Per #2929, corrected typo * Update installation.rst No changes to content, only whitespace for consistency, mostly removing tabs. * Update docs/Users_Guide/installation.rst Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * Per #2949, testing variable replacement, expect failures * Per #2949, reverting to orignal state after testing --------- Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * Bugfix #2979 develop MTD Grid (#2981) * Per #2979, remove nc_grid.h/.cc and replace it with calls to the read_netcdf_grid(...) and write_netcdf_proj(...) library utility functions. Note that these changes do compile but I haven't tested whether they actually fix the underlying problem. Also note that nc_utils_local.h/.cc can also likely be replaced with calls to common library functions. * Per #2979, remove references to nc_grid.o from the MTD test code. * Per #2979, insert a newline in unit.py output between the env vars and the command. * Per #2979, insert a newline in unit.py output between the env vars and the command. * Per #2979, the write_netcdf_proj(...) utility function adds the lat and lon dimensions. Update mtd to NOT define those dimensions prior to calling write_netcdf_proj(...). * Per #2979, minor changes to is_eq() calls to fix compiler warning messages * Per #2979, for the develop branch, also replace nc_utils_local.h/.cc with calls to common library code. Also remove commented out code. * Per #2979, delete commented out code and make error/warning message formatting consistent. * Fixes for SonarQube --------- Co-authored-by: MET Tools Test Account <met_test@seneca.rap.ucar.edu> * Feature #2880 point2grid qc (#2984) * Per #2880, remove Point2Grid quality_mark_thresh config option and add obs_quality_inc and obs_quality_exc from the config file. * Per #2880, update point2grid docs and reformat whitespace throughout. * Per #2880, remove quality_mark_thresh and add obs_quality_inc and obs_quality_exc. * Unrelated to #2880, fix formatting of this R-string which cannot include any special formatting, such as \n. * Per #2880, not working quite right yet but this is progress * Per #2880, add a write_css(IntArray) utility function. * Per #2880, update NcPointObsData class to read the obs quality values from the input file. * Per #2880, update the log message about the quality control filter options applied. * Per #2880, tweak log messages. * Per #2880, tweak log messages. * Per #2880, add a Point2Grid unit test to demonstrate using the obs_quality_inc and obs_quality_exc options. * Per #2880, declare these get accessor functions as const to avoid SonarQube code smell. * Per #2880, many changes to the vx_nc_obs library and point2grid application to replace dynamically allocated memory with STL vectors to satisfy SonarQube code smells. * Per #2880, fix compilation error. * Per #2880, fix compilation error * Per #2880, revert skip_times back to vector<bool> since I wasn't postive the casting (int *) to (bool *) would actually work in the way I expect. This is safer. * Per #2880, rename the -qc command line option as -goes_qc, but still quietly support -qc * Per #2880, based on guidance from @hsoh, reset the var_cell_mapping vector for each loop iteration. * Spelling correction * Feature #2882 seeps qa (#2987) * Update seeps.h Change variable names to reduce ambiguity for interpretation and aid useability. * Update seeps.cc Pull through variable name changes and renaming of functions to aid legibility and clarity. Introduced some additional debug print statements. * Update grid-stat.rst Add documentation about the location of the gridded climatology files for SEEPS and which environment variable to use. * Replace read_seeps_scores() with get_seeps_climo_grid() * Manually merging Rachel's patch-1 changes. * Getting close to getting these seeps changes to compile. But it's failing in pair_data_point.cc * Per #2882, get branch feature_2882_seeps_qa compiling again. Recommend revisiting the volume of SEEPS-related Debug log messages and reducing them once its fully tested. * Per #2882, need to update the handling of the PPT24_seepsweights_grid.nc file name. Rename as _v12.0.nc for the updated version with the new names so that the existing regressions tests and nightly builds for main_v11.1 and develop continue to work. We can remove the _v12.0 once this feature branch is merged into develop but for the time being, we need both versions to exist. * Per #2882, rename the SEEPS columns from S12, S13, S21, S23, S31, S32 to the more descriptive ODFL, ODFH, OLFD, OLFH, OHFD, OHFL names. * Per #2882, update SEEPS details * Per #2882, store and report the weighted mean fcst and mean obs, just like the SEEPS score itself so that they're handled in a consistent manner. Note however that it's hard-coded to NOT write the weighted means/score, only the unweighted ones. * Per #2882, change SEEPS debug log levels and correct the storage of mean_fcst and mean_obs values. * Per #2882, correct SEEPS column name lookups * Per #2882, call is_bad_data() instead of is_eq(..., -9999.0) to get rid of compiler warning message. * Per #2882, add 2 more variations of the is_eq() function with mixed float and double inputs to satisfy compiler pb2nc compiler warnings. * Per #2882, switch from dynamically allocated arrays to std::vector * Per #2882, enhance Stat-Analysis to write the SEEPS line type to an output .stat file. * Per #2882, update the aggregated seeps computation to use better-initialized vectors. * Per #2882, resolve a few more SonarQube code smells. * Per #2882, now that this PR is ready to merge, remove the v12.0 version number from the gridded SEEPS climo file name ci-skip-all --------- Co-authored-by: mpm-meto <64001904+mpm-meto@users.noreply.github.com> * Hotfix to the develop branch for unit_grid_stat.xml to use the updated gridded seeps climo file name. * Feature #2887 categorical weights (#2988) * Per #2887, update NumArray::vals() to return a reference to the vector rather a pointer to doubles. * Per #2887, switch over the whole ContingencyTable class heirarchy from storing integer counts to storing double-precision weights. * Add ContingencyTable::is_integer() member function to check whether the table contains all integers * Per #2887, update parse_stat_line.cc to get it to compile after changing PCT to store thresholds in a std::vector. * Per #2887, update PCTInfo::clear() logic. * Per #2887, update ctc_by_row() logic to create reproducible results with the develop branch. * Per #2887, update logic of define_prob_bins() to add a final >=1.0 threshold if needed. While ==0.1 works fine, I found that ==0.05 did not because the last >=1.0 threshold was missing likely do to floating point precision issues. This change should fix that problem. * Per #2887, update roc_auc() function to match the develop branch * Per #2887, fix bug if computation of far() * Per #2887, replaced all ==0 integer equality checks with calls to is_eq() instead and fix a couple of equations to snuff out diffs in some CTS statistics. * Per #2887, address some of the 34 SonarQube code smells flagged for this PR. Note that the compute_ci.h/.cc changes are necessary and good since we should be computing CI's using doubles instead of integer counts. * Per #2887, update run_sonarqube.sh to specify the target CXX standard as 11. The hope is that that will limit the findings to only those features available in the C++11 standard. * Per #2887, update to SonarQube version 6.1.0.4477 released on 6/27/2024. * Per #2887, updating build_met_sonarqube.sh to specify --std=c++11 since c++17 is used by default * Per #2887, swap in a much simpler implementation of the ORSS statistic to match the equation listed in the MET User's Guide. * Per #2887, update grid_stat and library code to actually apply the grid_weight_flag settings to the computation of contingency table counts and statistics. * Per #2887, fix the handling of bad data in the ORSS equation. * Per #2887, add Npairs member to the ContingencyTable class, eliminate the n() accessor function, and carefully replace references to n() with n_pairs() for the integer number of matched pairs or total() with the double-precision sum of the weights. * Per #2887, reset Npairs = 0 for ContingencyTable::zero_out() * Per #2883, need to call set_n_pairs() in a few spots to set ECLV TOTAL column correctly ci-run-unit * Per #2887, call set_n_pairs() when aggregating PCT data in Series-Analysis ci-run-unit * Per #2887, update stat_analysis to parse the TOTAL column for the PCT and MCTC line types. * Pet #2882, call set_n_pairs() after set_size() ci-run-unit * Per #2887, reconfigure existing Ensemble-Stat unit test to request probabilistic output to see that it's impacted by the grid_weight_flag setting. * Per #2887, update Ensemble-Stat test to provide climo stdev data * Per #2887, add grid_weight_flag to the list of config options for Grid-Stat and Ensemble-Stat. * Per #2887, disable FHO output if grid_weight_flag != NONE. * Per #2887, revise the existing unit_grid_weight.xml unit tests for Grid-Stat to write CTC/CTS/MCTC/MCTS output and for the DESC column to be populated to indicate the type of grid weighting that was applied. * Per #2887, relatively small changes to drive down SonarQube code smells. Also, switch from total() to n_pairs() when computing confidence intervals. * Per #2887, more SonarQube tweaks * Per #2887, more SonarQube tweaks. * Per #2887, more SonarQube tweaks. * Per #2887, whitespace only changes. * Per #2287, fix path the seeps climo grid. * Per #2887, update the grid_weight_flag documentation. * Per #2887, tweak the wording. * Feature 2673 sonarqube beta6 vector (#2991) * 673 Replaceddynamic allocatin method (using vector instead of new) * 673 SonarQube findings: use vector for the dynamic memory allocation * 673 SonarQube findings: use vector for the dynamic memory allocation * #2673 Check None of met_data.shape * #2673 Check None of met_data.shape --------- Co-authored-by: Howard Soh <hsoh@seneca.rap.ucar.edu> * Feature #2279 point_weight_flag (#2993) * Per #2887, update NumArray::vals() to return a reference to the vector rather a pointer to doubles. * Per #2887, switch over the whole ContingencyTable class heirarchy from storing integer counts to storing double-precision weights. * Add ContingencyTable::is_integer() member function to check whether the table contains all integers * Per #2887, update parse_stat_line.cc to get it to compile after changing PCT to store thresholds in a std::vector. * Per #2887, update PCTInfo::clear() logic. * Per #2887, update ctc_by_row() logic to create reproducible results with the develop branch. * Per #2887, update logic of define_prob_bins() to add a final >=1.0 threshold if needed. While ==0.1 works fine, I found that ==0.05 did not because the last >=1.0 threshold was missing likely do to floating point precision issues. This change should fix that problem. * Per #2887, update roc_auc() function to match the develop branch * Per #2887, fix bug if computation of far() * Per #2887, replaced all ==0 integer equality checks with calls to is_eq() instead and fix a couple of equations to snuff out diffs in some CTS statistics. * Per #2887, address some of the 34 SonarQube code smells flagged for this PR. Note that the compute_ci.h/.cc changes are necessary and good since we should be computing CI's using doubles instead of integer counts. * Per #2887, update run_sonarqube.sh to specify the target CXX standard as 11. The hope is that that will limit the findings to only those features available in the C++11 standard. * Per #2887, update to SonarQube version 6.1.0.4477 released on 6/27/2024. * Per #2887, updating build_met_sonarqube.sh to specify --std=c++11 since c++17 is used by default * Per #2887, swap in a much simpler implementation of the ORSS statistic to match the equation listed in the MET User's Guide. * Per #2887, update grid_stat and library code to actually apply the grid_weight_flag settings to the computation of contingency table counts and statistics. * Per #2887, fix the handling of bad data in the ORSS equation. * Per #2887, add Npairs member to the ContingencyTable class, eliminate the n() accessor function, and carefully replace references to n() with n_pairs() for the integer number of matched pairs or total() with the double-precision sum of the weights. * Per #2887, reset Npairs = 0 for ContingencyTable::zero_out() * Per #2883, need to call set_n_pairs() in a few spots to set ECLV TOTAL column correctly ci-run-unit * Per #2887, call set_n_pairs() when aggregating PCT data in Series-Analysis ci-run-unit * Per #2887, update stat_analysis to parse the TOTAL column for the PCT and MCTC line types. * Pet #2882, call set_n_pairs() after set_size() ci-run-unit * Per #2887, reconfigure existing Ensemble-Stat unit test to request probabilistic output to see that it's impacted by the grid_weight_flag setting. * Per #2887, update Ensemble-Stat test to provide climo stdev data * Per #2887, add grid_weight_flag to the list of config options for Grid-Stat and Ensemble-Stat. * Per #2887, disable FHO output if grid_weight_flag != NONE. * Per #2887, revise the existing unit_grid_weight.xml unit tests for Grid-Stat to write CTC/CTS/MCTC/MCTS output and for the DESC column to be populated to indicate the type of grid weighting that was applied. * Per #2279, add the MaskSID struct to store information about station id names and corresponding weights. * Per #2279, add new PointWeightType enumeration along with code to parse it. * Per #2279, adding point_weight_flag option to all Point-Stat and Ensemble-Stat config file and tweaking whitespace. * Per #2279, add point_weight_flag to the Point-Stat and Ensemble-Stat config class. Also remove sue unneeded wgt_dp argument for the add_point_obs() functions. Plan to add logic to set the point weights only AFTER all the observations have been collected for each verification task. * Per #2279, use the default_weight contstant instead of the literal 1.0 value. * Per #2279, add stubs for actually applying the point_weight_flag settings. * Per #2279, fix PairBase to actually set point weight values parsed from station id masks. * Per #2279, trying to fix 2 sonarqurqube bugs * Per #2279, fix a couple bugs parsing the SID weights and add a new unit_point_weight.xml unit test to run Point-Stat on scalar and probability inputs weighting the stations by their elevation. Still need to add Ensemble-Stat calls. * Per #2279, fix small bug ci-run-unit * Per #2279, add ensemble_stat calls to unit_point_weight.xml * Per #2279, add documentation about the point_weight_flag configuration option. * Per #2279, working on debug and warning messages. * Per #2279, tweak the user's guide * Per #2279, switch MaskSID::sid_list from a vector of pairs to a simpler map named sid_map. * Per #2279, fix the madis2nc call to parse_sid_mask() * Per #2279, move MaskSID from vx_config over into dedicated vx_util/mask_sid.h and .cc to be consistent with mask_poly.h. I note that the members of the MaskSID struct were not being initialized properly. So making it a complete class was the right solution. * Per #2279, another change to make it compile. * Per #2279, more tweaks to get it to compile. * Feature #2893 output_obtype (#2995) * Per #2893, add obtype_as_obs_message_type_flag option for Point-Stat and Ensemble-Stat that applies when writing MPR, SEEPS_MPR, and ORANK line types. If true, write the input observation message type to the OBTYPE output column rather than the name of the group. * Per #2893, add option to default config files and fix compilation errors. * Per #2893, change the name from obtype_as_obs_message_type_flag to obtype_as_group_val_flag. * Per #2893, add docs ci-run-unit * Per #2893, add obtype_as_group_val_flag entry for all config files that already include the message_type_group_map entry. * Per #2893, modify existing PointStatConfig_WINDS config file to have it write MPR output with obtype_as_group_val_flag = TRUE. Note that I switched the message types from ADPSFC, SFCSHP, USERSF to just USERSF. The original reason for the 3 was to prove that the counts for ADPSFC + SFCSHP = USERSF. * Per #2893, update commit history for ensemble_stat and point_stat * Per #2893, revert back PointStatConfig_WINDS to minimize diffs * Per #2893, add a Point-Stat unit test to demonstrate setting the obtype_as_group_val_flag option. * Per #2893, need to reset the obtype after writing all MPR, SEEPS_MPR, and ORANK line types in Point-Stat * Per #2893, tweak Point-Stat configuration to also write HIRA ORANK and PCT outputs. * Feature 2673 sonarqube beta6 vector dictionary (#2994) * #2673 SonarQube findings: use vector for the dynamic memory allocation * #2673 Check if a vector is allocated * #2673 Check if a vector is allocated * #2673 SonarQube findings use vector * #2673 Resolved SonarQube findingds * #2673 SonarQube: Replaced f-strings to regular strings * #2673 Added staticmethod to read_mpr * #2673 SonarQube: removed unnecessary pass * #2673 SonarQube: use vector * #2673 Initialize vecvgtor variables * 673 Check if an user python command is empty * #2673 Changed 0 to nullptr * #2673 Rollback for s and av * #2673 Applied the static method call * #2673 Rollback using vector because of introducing a bug * 673 Cleanup commented out code --------- Co-authored-by: Howard Soh <hsoh@seneca.rap.ucar.edu> * Feature #2996 v12.0.0_beta6 (#2998) * Per #2996, add beta6 release notes. * Per #2996, slight change in order * Update docs/Users_Guide/release-notes.rst Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Feature #2673 wchar argv (#2999) * Per #2673, switch to STL vector to attempt to squash a SonarQube bug. * Per #2673, remove m_strlen() entirely. * Per #2673, try again ci-skip-unit * Per #2673, switching to 2 more vectors * Update the release date for MET-12.0.0-beta6 * Feature #3000 point2grid_warning (#3001) * Per #3000, change 9 warning messages to Debug(2) log messages to minimize the number of warning messages Point2Grid writes. However, do retain the warnings about problems writing the grid mapping output file. * Per #3000 very minor whitespace diffs * Feature #2631 static_file_updates (#3002) * Per #2631, regenerate ndbc_stations.xml for MET-12.0.0 on October 22, 2024. * Per #2631, add MET_BASE anchor * Per #2631, add a description of some of the static data files that need to be periodically updated. Note that this is NOT an exhaustive list. * Per #2631, consistent wording * Per #2631, subset by update frequency. * Per #2631, consistent wording. Fix typos. * Per #2631, fix typo and URL * Per #2631, fix masked array error in the get_precision(...) function I encountered when using the utility to inspect the diffs for this PR. * Updating to add MET_PYTHON_EXE * Update compiler * Update install_met_env.casper to use oneAPI compilers * Based on feedback from @Ho-ChunHuang-NOAA, replace references to GOES-16 and GOES-17 with GOES-East and GOES-West in the MET User's Guide. * Feature #2780 grib tables (#3005) * Per #2780, grib2_bom.txt only has minor change replacing 'Prelim:' with 'Prelim' * Per #2780, grib2_tigge.txt add new table after eliminating whitespace from units strings * Per #2780, grib2_all.txt increase the number of entries from 1019 to 1570. * Per #2780, grib2_kma.txt add new grib2 table file. * Per #2780, carefully merge the existing 175 MRMS GRIB2 table entries with updates from the 109 defined in the wgrib2 table. * Per #2780, grib2_ecmwf.txt add new ECMWF GRIB2 table. * Per #2780, grib2_dwd.txt add new DWD GRIB2 table. * Per #2870, correct units out of order * Per #2870, grib2_gpp_af.txt elimiate embedded whitespace from units string to avoid problems writing them to FCST_UNITS and OBS_UNITS output columns * Per #2780, grib2_ecmwf.txt eliminate whitespace from units string * Per #2780, add/update GRIB2 tables * Per #2870, remove duplicate entries I'd added. * Per #2870, grib2_all.txt update with more entries from this version of wgrib2: hera:/home/Wesley.Ebisuzaki/grib2/wgrib2/gribtables/ncep/gribtable.dat * Per #2780, grib2_mrms.txt add entries for FRANA as requested by @MarcelCaron-NOAA. * Per #2780, add new grib1_jra_55_200.txt file that did not previously exist for MET. * Per #2780, remove embedded whitespace from GRIB1 DWD tables. * Per #2780, fix all the units by removing embedded whitespace and making them more human-readable by replacing ** with ^ to indicate exponentiation. Also correct bad units for radiation flux in AF tables by listing them as W/m^2. * Per #2780 eliminate trailing whitespace * Per #2870, fix typo * Per #2870, fix typos * Per #2870, fix typo * Per #2780, grib1_ncep_129_7.txt fix units by replacing 'µg' with 'mcg' to represent micrograms. Also replace 'mm6' with 'mm^6' by double-checking against the GRIB1 docs. * Per #2780, switch to defining particulate matter units as '10^-6g/m^3' for consistent usage across NCEP GRIB tables. * Per #2780, update GRIB2 units to show exponentiation correctly by comparing against documentation at https://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc * Per #2780, more tweaks for consistent and corrected units * Per #2780, update Makefiles for new grib1_jra55_200.txt file * Per #2870, more tweaks for consistent formatting of units to eliminate the use of parantheses in the denominator. For example, replace 'kg/(m*s)' with 'kg/m/s'. * Feature #2729 tc_diag (#3011) * Per #2729, get rid of extra blank line in log message about manually deleting temp files. * Per #2729, fix log message describing the range/azimuth grid. * Per #2729, Update TC-Diag Python driver code to convert rotation angles from degree clockwise from due East to radians counter-clockwise from due East. * Per #2729, complete the TC-Diag documentation. * Per #2729, more tweaks * Per #2729, clarify that some diagnostics are directly extracted from the ATCF track file rather than being computed from the cylindrical coordinates grid. * Update docs/Users_Guide/tc-diag.rst Co-authored-by: KathrynNewman <knewman@ucar.edu> * Update docs/Users_Guide/tc-diag.rst Co-authored-by: KathrynNewman <knewman@ucar.edu> * Update docs/Users_Guide/tc-diag.rst Co-authored-by: KathrynNewman <knewman@ucar.edu> * Update docs/Users_Guide/tc-diag.rst Co-authored-by: KathrynNewman <knewman@ucar.edu> * Update docs/Users_Guide/tc-diag.rst Co-authored-by: KathrynNewman <knewman@ucar.edu> --------- Co-authored-by: KathrynNewman <knewman@ucar.edu> * Feature #2750 mtd_docs (#3014) * Per #2750, user-provided should be hyphenated. * Per #2750, add units to MTD output tables and specify order of forecast minus observation differences, where applicable. Also update some wording for consistency. * Unrelated to #2750, but prior to the v12.0.0 release, create a 12.0.0 flowchart, save as PNG, and use it to update the overview figure. * Per #2750, put X_DOT and Y_DOT back on separate lines since RTD interprets X_,Y_DOT as being a poorly defined link. * Feature 3009 net cdf docs (#3016) * Update data_io.rst Updated netCDF reading documentation * Update data_io.rst * Update data_io.rst * Update data_io.rst updated link * Update data_io.rst Fixed typo * Feature #3015 v12.0.0-rc1 (#3017) * Per #3015, update the release version number and add release notes. * Per #3015, add one upgrade instruction * update logic to be able to trigger METplus testing workflow for main_vX.Y and develop branches * fix typo in workflow * add quotes around value sent to trigger METplus workflow and remove unnecessary step * use custom GitHub Action to trigger METplus use case testing * Updating * Bugfix #3020 develop grid_stat_seeps (#3021) * Per #3020, add missing GridStatNcOutInfo::do_seeps flag and use it to determine if SEEPS information should be written to the Grid-Stat NetCDF matched pairs output file. * Unrelated to #3020, fix broken NetCDF cf-conventions links in the User's Guide. * Per #3020, no real changes. Just whitespace * Updated to reflect usage of oneAPI compilers * Updating file to reflect usage of oneAPI compilers * Hotfix to the develop branch after PR #3021 fixed a SEEPS bug. The GridStatConfig_SEEPS config file needs to be updated with nc_pairs_flag.seeps = TRUE in order for the same output to be produced by the unit tests. * Feature #2966 local_solar_time (#3008) * Per #2966, add new solar_time() function to the vx_solar library. * Per #2966, add support for new solar_time masking type. Also make log messages for consistent and eliminate the warning about -thresh not being specified becuase its fine to not specify a threshold. * Per #2966, add a units attribute to the output NetCDF mask variable. * Per #2966, modify solar azimuth and altitude strings to make the log messages align well. * Per #2966, add gen_vx_mask unit test to demonstrate the solar_time masking type. * Per #2966, add documentation about the -solar_time option * Per #2966, reduce SonarQube code smells in gen_vx_mask * Per #2966, reduce SonarQube findings * Per #2966, support multiple mask types with the same mask field being supported in a single run. Still need to update the user's guide. * Per #2966, add UTC * Per #2966, update gen_vx_mask docs about supporting multiple -type options in a single run * Per #2966, update logic to fix using data masking twice, add a unit test to demonstrate, and update the mask_type attribute to include the magic string for the gr…
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
Please review PR #3371 for the
main_v12.2branch at the same time as this one.This PR is similar to the changes for PR #3371 but for the develop branch and are much more involved.
These changes accomplish the following:
falseto atrueintc_rmw.cc, as in Bugfix #3370 main_v12.2 tc_rmw BEST tracks #3371.best_techniqueas a new entry in the TCRMWConfig file, just like in TC-Pairs and update the TC-RMW code to parse it and set it when parsing input ATCF lines.StringArray TrackLinesout of theTrackInfoclass and down into theTrackPointclass so that we can keep track of which input lines belong to which points.unit_tcrmw.xmlto use run TCRMW using a FULL input BEST track.Note that (3) is really an enhancement more than a bugfix, but it was convenient to bundle them together in this single PR.
Changes are proposed for 12 files:
libcode/vx_tc_utilsource file to moveTrackLinesfrom theTrackInfoclass intoTrackPoint.Expected Differences
Do these changes introduce new tools, command line arguments, or configuration file options? [Yes]
If yes, please describe:
Adds new
best_techniquetoTCRWMConfig_default.Do these changes modify the structure of existing or add new output data types (e.g. statistic line types or NetCDF variables)? [No]
If yes, please describe:
Fortunately, there are no impacts to the actual output from TCRMW.
Pull Request Testing
Describe testing already performed for these changes:
Defined and tested the new use case to ensure that it processes all 5 points of the filtered BEST track data as expected. Also double-checked that the existing version of tc_rmw in the develop branch ONLY processing a single track point... which is the bug.
Recommend testing for the reviewer(s) to perform, including the location of input datasets, and any additional instructions:
Review code changes and new unit test.
Could download/inspect the new output file as a sanity check.
Review doc updates in RTD for this PR.
Please find this code compiled and available for testing in
seneca:/d1/projects/MET/MET_pull_requests/met-13.0.0/beta2/MET-bugfix_3370_develop_tc_rmw_best_tracks/binDo these changes include sufficient documentation updates, ensuring that no errors or warnings exist in the build of the documentation? [Yes]
I'll note that the existing TC-RMW chapter should really be enhanced to include a section titled
Scientific and Statistical Aspectsthat goes into much great detail. I did NOT add that section, but I did update the intro to state that it can process both ADeck and BDeck data and point out that the track data and grid data no longer need to exactly line up.Do these changes include sufficient testing updates? [Yes]
Added one new TC-RMW unit test.
Will this PR result in changes to the MET test suite? [Yes]
If yes, describe the new output and/or changes to the existing output:
One new TC-RMW output file generated. No other changes.
Will this PR result in changes to existing METplus Use Cases? [No]
If yes, create a new Update Truth METplus issue to describe them.
Do these changes introduce new SonarQube findings? [No]
If yes, please describe:
While some existing issues are flagged as being in new code, the overall number of SQ issues are reduced by 49, from 13,804 to 13,755.
Please complete this pull request review by [Fri 5/1/26].
Pull Request Checklist
See the METplus Workflow for details.
Select: Reviewer(s) and Development issue
Select: Milestone as the version that will include these changes
Select: METplus-X.Y Support project for bugfix releases or MET-X.Y Development project for the next coordinated release