Implementation of MYNN-EDMF submodule (#2148)

This replaces the original (non submodule) MYNN with the public-facing
MYNN-EDMF submodule, which will the central point of development for the
MYNN-EDMF in WRF, MPAS, and CCPP.

TYPE: bug fix, enhancement, and new feature

KEYWORDS: refactored, submodule, MYNN-EDMF

SOURCE: Joseph Olson (NOAA/GSL)

DESCRIPTION OF CHANGES:

1. Submodule implementation pointing to
https://github.com/NCAR/MYNN-EDMF/
2. This is a refactored scheme (now k-only) which is called from a new
driver derived heavily from the MPAS driver.
3. The module names changed from *_mynn_* to *_mynnedmf_* to resolve a
version conflict in MPAS.
4. This version carries many modification that originated in FV3-related
development for RRFSv1, but it has since been tuned to better perform in
MPAS and WRF, unlike previous versions of the MYNN. The list of
modifications are fairly extensive but the most dycore-sensitive parts
of the MYNN are related to the subgrid clouds. From now on, all minor
commits are captured in the public-facing submodule repository and the
submodule will be updated in WRF at a lower frequency (probably every 6
months or so).

LIST OF MODIFIED FILES: 
M       .gitmodules
M       Makefile
M       Registry/Registry.EM_COMMON
M       clean
M       dyn_em/module_first_rk_step_part1.F
M       main/depend.common
A       phys/MYNN-EDMF
M       phys/Makefile
D       phys/module_bl_mynn.F
D       phys/module_bl_mynn_common.F
D       phys/module_bl_mynn_wrapper.F
M       phys/module_pbl_driver.F
M       phys/module_physics_init.F

TESTS CONDUCTED: 
1. The submodule itself has been tested in both WRF and MPAS for several
months now in NOAA-GSL realtime systems and case study tests, passing
all of our newly designed regression tests.
2. Has been compiled and run with intel and gnu compilers on Jet and
Hera.
3. Are the Jenkins tests all passing?

RELEASE NOTE: Submodule implementation of the MYNN-EDMF
(https://github.com/NCAR/MYNN-EDMF). The module names changed from
*_mynn_* to *_mynnedmf_* to resolve a version conflict in MPAS. This
version was originally developed within FV3/CCPP for RRFSv1, but has
been refactored (to a k-only scheme) resulting in a speed-up of about
10-15% and it has since been tuned to better perform in MPAS and WRF
compared to previous versions which were primarily developed for use in
FV3.

Author: joeolson42

.gitmodules

@@ -4,3 +4,6 @@
 [submodule ".ci/hpc-workflows"]
 	path = .ci/hpc-workflows
 	url = https://github.com/islas/hpc-workflows
+[submodule "phys/MYNN-EDMF"]
+	path = phys/MYNN-EDMF
+	url = https://github.com/NCAR/MYNN-EDMF

Makefile

@@ -124,6 +124,20 @@ wrf : framework_only
 	   echo "NoahMP submodule files populating WRF directories" ; \
 	   echo "------------------------------------------------------------------------------" ; \
 	fi
+	if [ \( ! -f phys/module_bl_mynnedmf.F \) -o \
+	    \( ! -f phys/module_bl_mynnedmf_common.F \) -o \
+	    \( ! -f phys/module_bl_mynnedmf_common.F \) ] ; then \
+	  echo " " ; \
+	  echo "------------------------------------------------------------------------------" ; \
+	  echo "Error Error Error MYNN-EDMF submodule files not populating WRF directories" ; \
+	  echo "------------------------------------------------------------------------------" ; \
+	  echo " " ; \
+	  exit 31 ; \
+	else \
+	  echo "------------------------------------------------------------------------------" ; \
+	  echo "MYNN-EDMF submodule files populating WRF directories" ; \
+	  echo "------------------------------------------------------------------------------" ; \
+	fi
 	if [ $(WRF_CHEM) -eq 1 ]    ; then $(MAKE) MODULE_DIRS="$(ALL_MODULES)" chemics ; fi
 	if [ $(WRF_EM_CORE) -eq 1 ]    ; then $(MAKE) MODULE_DIRS="$(ALL_MODULES)" em_core ; fi
 	if [ $(WRF_HYDRO) -eq 1 ]   ; then $(MAKE) MODULE_DIRS="$(ALL_MODULES)" wrf_hydro ; fi

Registry/Registry.EM_COMMON

@@ -1142,7 +1142,6 @@ state   real   sub_thl3D       ikj     misc        1         -      h         "s
 state   real   sub_sqv3D       ikj     misc        1         -      h         "sub_sqv3D"        "qv subsidence tendency from EDMF" "kg kg-1 s-1"
 state   real   det_thl3D       ikj     misc        1         -      h         "det_thl3D"        "thetaL detrainment tendency from EDMF" "K s-1"
 state   real   det_sqv3D       ikj     misc        1         -      h         "det_sqv3D"        "qv detrainment tendency from EDMF" "kg kg-1 s-1"
-state   integer  ktop_plume     ij     misc        1         -      h         "ktop_plume"       "k-level of highest pentrating plume"    ""
 state   real   maxMF            ij     misc        1         -      h         "maxMF"            "Maximum mass-flux (neg: all dry, pos: moist)"   "m/s * area"
 state   real   maxwidth         ij     misc        1         -      h         "maxwidth"         "Maximum plume width"   "m"
 state   real   ztop_plume       ij     misc        1         -      h         "ztop_plume"       "Height of tallest plume"   "m"
@@ -2476,7 +2475,7 @@ rconfig   integer     bl_mynn_mixlength   namelist,physics      1              1
 rconfig   integer     bl_mynn_edmf        namelist,physics      max_domains    1       irh      "bl_mynn_edmf"                  "0:off,1:activate mass-flux in mynn"                    ""
 rconfig   integer     bl_mynn_edmf_mom    namelist,physics      max_domains    1       irh      "bl_mynn_edmf_mom"              "0:off,1:activate mass-flux transport of momentum"      ""
 rconfig   integer     bl_mynn_edmf_tke    namelist,physics      max_domains    0       irh      "bl_mynn_edmf_tke"              "0:off,1:activate mass-flux transport of tke"           ""
-rconfig   integer     bl_mynn_mixscalars  namelist,physics      max_domains    0       irh      "bl_mynn_mixscalars"            "0:off,1:activate mixing of scalars (qnx, qnxfa) in MYNN"      ""
+rconfig   integer     bl_mynn_mixscalars  namelist,physics      max_domains    1       irh      "bl_mynn_mixscalars"            "0:off,1:activate mixing of scalars (qnx, qnxfa) in MYNN"      ""
 rconfig   integer     bl_mynn_output      namelist,physics      max_domains    0       irh      "bl_mynn_output"                "0:off,1:Allocate and output extra 3D arrays"      ""
 rconfig   integer     bl_mynn_cloudmix    namelist,physics      max_domains    1       irh      "bl_mynn_cloudmix"              "0:off,1:activate mixing of all cloud species"          ""
 rconfig   integer     bl_mynn_mixqt       namelist,physics      max_domains    0       irh      "bl_mynn_mixqt"                 "0:mix moisture species separate,1: mix total water"    ""
@@ -3179,7 +3178,7 @@ package   kepsscheme     bl_pbl_physics==17          -             scalar:tke_ad
 package   mrfscheme      bl_pbl_physics==99          -             -
 
 package   tkebudget      tke_budget==1               -             state:qSHEAR,qBUOY,qDISS,qWT,dqke
-package   mynn_dmp_edmf  bl_mynn_edmf==1             -             state:ktop_plume,ztop_plume,maxmf,maxwidth
+package   mynn_dmp_edmf  bl_mynn_edmf==1             -             state:ztop_plume,maxmf,maxwidth
 package   mynn_3Doutput  bl_mynn_output==1           -             state:edmf_a,edmf_w,edmf_thl,edmf_qt,edmf_ent,edmf_qc,sub_thl3D,sub_sqv3D,det_thl3D,det_sqv3D
 package   pbl_cloud      icloud_bl==1                -             state:cldfra_bl,qc_bl,qi_bl
 

clean

@@ -83,6 +83,8 @@ if ( "$arg" == '-a' || "$arg" == '-aa' ) then
     /bin/rm -f phys/module_sf_noahmpdrv.F phys/module_sf_noahmp_glacier.F \
                phys/module_sf_noahmp_groundwater.F phys/module_sf_noahmplsm.F \
                run/MPTABLE.TBL
+    /bin/rm -f phys/module_bl_mynnedmf.F phys/module_bl_mynnedmf_common.F \
+               phys/module_bl_mynnedmf_driver.F
   endif
 endif
 

dyn_em/module_first_rk_step_part1.F

@@ -1232,7 +1232,7 @@ SUBROUTINE first_rk_step_part1 (   grid , config_flags              &
      &        ,qcg=grid%qcg, grav_settling=config_flags%grav_settling     &
 !    &        ,K_m=grid%K_m, K_h=grid%K_h, K_q=grid%K_q                   &
      &        ,vdfg=grid%vdfg,maxwidth=grid%maxwidth,maxMF=grid%maxmf     &
-     &        ,ztop_plume=grid%ztop_plume,ktop_plume=grid%ktop_plume      &
+     &        ,ztop_plume=grid%ztop_plume                                 &
      &        ,spp_pbl=config_flags%spp_pbl                               &
      &        ,pattern_spp_pbl=grid%pattern_spp_pbl                       &
      &        ,restart=config_flags%restart,cycling=config_flags%cycling  &

main/depend.common

@@ -663,13 +663,13 @@ module_bl_gfsedmf.o: \
 	module_gfs_physcons.o 
 
 
-module_bl_mynn.o: \
-	module_bl_mynn_common.o 
+module_bl_mynnedmf.o: \
+	module_bl_mynnedmf_common.o
 
 
-module_bl_mynn_wrapper.o: \
-	module_bl_mynn.o \
-	module_bl_mynn_common.o 
+module_bl_mynnedmf_driver.o: \
+	module_bl_mynnedmf.o \
+	module_bl_mynnedmf_common.o
 
 
 module_bl_gwdo.o: \
@@ -735,7 +735,6 @@ module_bl_camuwpbl_driver.o: \
 
 
 module_sf_mynn.o: \
-	module_bl_mynn.o \
 	../share/module_model_constants.o \
 	../frame/module_wrf_error.o 
 
@@ -1308,8 +1307,8 @@ module_physics_init.o: \
 	module_bl_acm.o \
 	module_bl_myjpbl.o \
 	module_bl_qnsepbl.o \
-	module_bl_mynn.o \
-	module_bl_mynn_wrapper.o \
+	module_bl_mynnedmf.o \
+	module_bl_mynnedmf_driver.o \
 	module_bl_myjurb.o \
 	module_bl_boulac.o \
 	module_bl_camuwpbl_driver.o \
@@ -1486,8 +1485,8 @@ module_pbl_driver.o: \
 	module_bl_camuwpbl_driver.o \
 	module_bl_gfs.o \
 	module_bl_gfsedmf.o \
-	module_bl_mynn.o \
-	module_bl_mynn_wrapper.o \
+	module_bl_mynnedmf.o \
+	module_bl_mynnedmf_driver.o \
 	module_bl_fogdes.o \
 	module_bl_gwdo.o \
 	module_bl_gwdo_gsl.o \
@@ -2790,8 +2789,7 @@ module_bl_mfshconvpbl.o: \
 	../share/module_model_constants.o 
 
 
-module_bl_mynn_common.o: \
-	module_gfs_machine.o \
+module_bl_mynnedmf_common.o: \
 	../share/module_model_constants.o \
 	ccpp_kind_types.o
 

phys/CMakeLists.txt

@@ -42,9 +42,6 @@ target_sources(
                   module_bl_mrf.F
                   module_bl_myjpbl.F
                   module_bl_myjurb.F
-                  module_bl_mynn.F
-                  module_bl_mynn_common.F
-                  module_bl_mynn_wrapper.F
                   module_bl_qnsepbl.F
                   module_bl_shinhong.F
                   module_bl_temf.F
@@ -263,6 +260,11 @@ target_sources(
                   physics_mmm/mp_wsm6.F90
                   physics_mmm/mp_wsm6_effectRad.F90
                   physics_mmm/sf_sfclayrev.F90
+
+                  # MYNN-EDMF
+                  MYNN-EDMF/module_bl_mynnedmf.F90
+                  MYNN-EDMF/WRF/module_bl_mynnedmf_common.F90
+                  MYNN-EDMF/WRF/module_bl_mynnedmf_driver.F90
                   )
 
 

phys/MYNN-EDMF

@@ -43,9 +43,9 @@ MODULES = \
 	module_bl_myjpbl.o \
 	module_bl_qnsepbl.o \
 	module_bl_acm.o \
-        module_bl_mynn_common.o \
-        module_bl_mynn.o \
-        module_bl_mynn_wrapper.o \
+        module_bl_mynnedmf_common.o \
+        module_bl_mynnedmf.o \
+        module_bl_mynnedmf_driver.o \
 	module_bl_fogdes.o \
 	module_bl_gwdo.o \
 	module_bl_gwdo_gsl.o \
@@ -272,7 +272,17 @@ submodules :
 	else \
 		echo No action required for NoahMP submodule ; \
 	fi
-
+	@if [ \( ! -f module_bl_mynnedmf.F \) -o \( ! -f module_bl_mynedmf_common.F \) -o \
+	      \( ! -f module_bl_mynnedmf_driver.F \) ] ; then \
+	      	echo Pulling in MYNN-EDMF submodule ; \
+		( cd .. ; git submodule update --init --recursive ) ; \
+		ln -sf MYNN-EDMF/module_bl_mynnedmf.F90 module_bl_mynnedmf.F ; \
+		ln -sf MYNN-EDMF/WRF/module_bl_mynnedmf_common.F90 module_bl_mynnedmf_common.F ; \
+		ln -sf MYNN-EDMF/WRF/module_bl_mynnedmf_driver.F90 module_bl_mynnedmf_driver.F ; \
+	else \
+		echo No action required for MYNN-EDMF submodule ; \
+	fi
+	
 clean:
 	@ echo 'use the clean script'