Performance degradation in the RE lexemes on big JSONs

[ldn-softdev]

for RE lexemes (<>R, <>L, <>D) performance is decaying exponentially, proportionally to the JSON size (search size). The same effect (exponential decay in performance) might be observed for <..>j lexeme and generally for any template-interpolation operations.

[ldn-softdev]

Root-cause:

- the regression was introduced with the new way of generating auto-tokens ($a, $b, etc). Each time when RE lexeme is called, it's subjected now for a template-interpolation (as the new feature from v1.76) that in turn was calling generate_auto_tokens_(..) routine as part of the interpolation, which was making a couple of walks (with ><w and <>a lexemes across the entire search range).
Those walks, happening across entire search range, were rebuilding their caches each time (the cache from those walks wasn't preserved) they were called. Thus, the bigger a search space, the longer it takes to rebuild the walk-caches, the slower iteration step for each RE lexeme was - that is source of the exponential decay effect - the bigger JSON the slower it goes.

That regression does not impact functionality, only performance was affected - that's why it sneaked past UT, I'd need to add some performance testing to the suit.

Fix:

1. ensure those walks in generate_auto_tokens_(..) do preserve their caches (so that it does not cause rebuilding them each time, effectively removing performance exponential decay)
2. a bit of further performance improvements:
   • for RE lexemes: ensure that those REGEX that do not have interpolation tokens would not undergo interpolation every time
   • also ensure the same for <..>j lexeme (interpolation would be repeatedly attempted only for the templated lexemes)