Fix a MAgPIE run based on existing gdx files

[flohump]

For some applications it would be useful if MAgPIE could be fixed to the values of existing GDX files until a certain year.
Let's assume you have a run that is infeasible in 2050. Now you want to restart the run but keep the results until 2020 from the first run that was infeasible only in 2050 (based on magpie_y1995.gdx ... magpie_y2020.gdx). This makes sure that the results for 1995-2020 are identical to other model runs from the same scenario set. From 2025 onwards the model should use the gdx files from the first run for all years until 2045 as starting point (cfg$gms$s_use_gdx <- 2). This will push the solver to a slightly different trajectory from 2025 onwards and might lead to a feasible solution for 2050 and beyond.

Note 1: Re-running the model for the period 1995-2020 without gdx files (or cfg$gms$s_use_gdx <- 0) should yield the same result but of course needs much more time than just fixing all variables based on existing gdx files.
Note 2: Re-running the model for the period 1995-2020 with gdx files and cfg$gms$s_use_gdx <- 2 produces different results.

[orichters]

I discussed with @mishkos yesterday that this functionality would be also very useful for delayed transition scenarios in the NGFS project.

We have this functionality in REMIND, so it should be possible to copy that to MAgPIE.

Basically, the cozy function create_standard_fixings reads from the gdx file (always input_ref.gdx in the ./output subfolder), creating three files levs.gms (levels), fixings.gms (fixings), and margs.gms (marginals).

There is some zipping and unzipping involved for restarted runs to avoid doing this multiple times.

In loop.gms, we have a unique string cb20140305readinpositionforfinxingfiles (only valid with the typo!) which is changed into the gms files created, which will then be executed by GAMS.

Like that, we fix runs on older runs for all timesteps < cm_startyear.

But maybe there is a more elegant solution for that. I guess @christophbertram might provide more information.

[christophbertram]

Isn't this much easier to implement in MAgPIE, given its recursive solution approach (compared to the intertemporal optimization in REMIND)?
Without knowing what exactly the cfg$gms$s_use_gdx <- 2 is, in my view there should be settings 2020, 2025, ...20xx for this switch that result in a scenario in which the gdxs up to the chosen year are just used from before, and the first year to be solved anew (as the start of the recursive chain) is the next time step. I recall that you argued in the past that this would result in INFES, but I think that the underlying mechanisms are then to blame and need revision, as the real world functions like this (as of now (p0), everything prior to p0, and all decisions that have been made in the past were done with one singular set of assumptions about the then future (part of which has transpassed p < p0, different from expectations, and part of which is still to occur). And no matter what I expected what will happen tomorrow in the past, I'm free to completely readjust my expectation now based on all information I have now.

[tscheypidi]

quick update here: The situation is here quite different in MAgPIE compared to REMIND. The good thing is: In contrast to REMIND we do not need this feature to run delayed action scenarios due to the iterative nature of the model. Bad thing is that the fixing solution from REMIND is also not applicable to MAgPIE.

I did some more digging and revisited the GAMS manual. What GAMS provides is a Save and Restart which goes into the direction of what we would need but is not exactly the same. Hence, I wrote a mail to the GAMS developer team asking for a feature expansion in this direction. What we would need is the option to set restart points in the code, which write a save file every time they reach this point and would serve as an entry point when the model is started from such a savepoint.

[tscheypidi]

PR #512 partly adresses this issue by adding restart support to the model. It allows to interrupt and restart the model and still getting the exact same results like without interruption.
However, using this restart point in combination with a different scenario will most likely not yield the wanted behavior as the restart point contains all information about the run, including the whole parametrization of this run

[orichters]

Any progress on this issue (planned?), @tscheypidi / @pfuehrlich-pik?

[pfuehrlich-pik]

Not on my side, I'm still very busy with RESCUE/OptimESM.

[tscheypidi]

no, no plans

[orichters]

In the delayed transition scenarios of NGFS that are fixing in REMIND on a reference run until 2030, some of the deviations between the reference run and the delayed transition run are quite substantial:

   group                                   variables period            reldiff
 1 Agricultural Demand                             1 2025,2030         0.654 %
42 Fertilizer Use|Nitrogen                         1 2025,2030         2.03 %
54 Land Cover|Forest                               1 2025,2030         32 %
60 Price|Agriculture|Corn|Index                    1 2025,2030         13.4 %
73 Water Withdrawal|Irrigation                     1 2025,2030         20.1 %
74 Yield|Cereal                                    1 2025,2030         7.63 %
75 Yield|Oilcrops                                  1 2025,2030         10.8 %
76 Yield|Sugarcrops                                1 2025,2030         26.2 %

[orichters]

The problem:

• see here a summary of the problem for a delayed transition run: less +514g /p/tmp/oliverr/NGFS_v5/remind-2024-03-27/output/C_d_strain-rem-3/log.txt
• this is a 2°C run (rcp2p6) which follows the trajectory of a Current Policies run (rcp4p5) until 2030.

The REMIND way to achieve consistency (except for EDGE-Transport):

• specify path_gdx_ref to a different run or a fulldata.gdx in the config, and a cm_startyear which is the first free model year
• create_fixing_files, creating a gdxdump of the input_ref.gdx
• create_standard_fixings, basically three .gms file for .L, .M and .FX. There is a list of removals from this file, basically to be able to use older gdx files, avoiding specifying values for variables that are never declared. These files are then included in full.gms.
• see for example less /p/tmp/oliverr/debugging/fixings.gms, a file with a long list of such entries: vm_effGr.FX ('2030','CAZ','inco') = 1 ;
• in the REMIND code, one has to differentiate between ttot (all model timesteps) and t which is basically all ttot >= cm_startyear to avoid that fixings are overwritten. This is a bit of a delicate step because what to use when is not so easy to understand

[orichters]

@merfort added: Something else that might be relevant is the partial foresight that MAgPIE has regarding forestry (there is some price anticipation, right?). If I am not mistaken, here changes in the settings may also affect the historical period (or rather the period until the REMIND start year), even though all inputs from REMIND are the same.

[merfort]

Thanks Olli! For RESCUE there are, unfortunately, quite some differences between the reference NPi run (C_SSP2EU-NPi; red) and a delayed policy run (C_RESCUE-dir-v4p0-EocBudg500-OAE_off, blue), in which in REMIND everything is fixed to NPi for all years < 2035. Differences in Emissions|CO2|Land|+|Land-use Change are in particularly large.
The largest difference seems to be due to the NPi/NDC switch, since the NDC run (C_SSP2EU-NDC) is much closer to the delayed policy run. However, for the NDC run we have no fixing to the REMIND NPi in 2025 and 2030 (cm_startyear = 2025), so the remaining differences might also come from the differences between the respective REMIND runs.

title	magpie_scen	cm_startyear (REMIND)
C_SSP2EU-NPi	SSP2|NPI|nocc_hist|rcp4p5|ForestryExo	2025
C_SSP2EU-NDC	SSP2|NDC|nocc_hist|rcp4p5|ForestryExo	2025
C_RESCUE-dir-v4p0-EocBudg500-OAE_off	SSP2|NDC|nocc_hist|rcp1p9|ForestryExo	2035

Simply overwriting all MAgPIE reporting variables in 2025 and 2030 with the one from the NPi run (which is what I do so far for the emissions reporting), leads to quite a jump between 2030 and 2035. This is particularly problematic, since for RESCUE we are also reporting the land-use data (e.g. land-cover change) and so far we use all data from one the delayed policy run (no overwriting). I guess, that overwriting land-use data on a cluster level is anyways not viable, since the jumps on this lower resolution on the cluster level will be even larger. @tscheypidi @pfuehrlich-pik what are your thoughts on that?