provide a helper routine for building a static regex using std::lazy

[hbina]

Introduce a macro that lazily initialise a static regex expression, call it lazy_regex.
This is helpful whenever you want to avoid the compile-at-each-iteration pattern when using regexes across threads.
The code is simply,

macro_rules! lazy_regex_1 {
    ($re:literal $(,)?) => {{
        static RE: SyncOnceCell<regex::Regex> = SyncOnceCell::new();
        RE.get_or_init(|| Regex::new($re).unwrap())
    }};
}

or

macro_rules! lazy_regex_2 {
    ($re:literal $(,)?) => {{
        static RE: SyncLazy<regex::Regex> = SyncLazy::new(|| Regex::new($re).unwrap());
        &RE
    }};
}

Which is only a slight alteration from the one provided in the once_cell documentation. There are many possible variations of this --- there are many ways to lazily initialise a static variable. I think this is something that is common enough that it should be included.

Thank you.

[BurntSushi]

I don't understand your feature request and I don't understand your diff. Please include more details. Maybe show how the feature you're requesting is intended to be used?

[tesuji]

SyncOnceCell is nightly std feature. I don't think regex crate need to introduce a macro that depends on nightly or lazy_static or so.

[BurntSushi]

OK, so I didn't even know that SyncOnceCell had made its way into std already. @hbina Please include relevant context in feature requests.

In any case, we've gone years without this and I see no reason to get impatient now. Once std::lazy stabilizes, then I'll be happy to revisit this. But I have no plans to add another nightly-only API. In the mean time, I'd encourage folks to brainstorm on what the API should look like. One thing I'd like to know is whether we need a macro or not. I'd rather not.

And to head off the question that I keep getting from people: no const fn can't help us here, or at least, won't be able to help us until the vast majority of Rust is able to be constified and I don't see that happening any time soon. Therefore, I plan to behave as if const fn has no impact.

[matklad]

JFYI, this exists as a crate: https://crates.io/crates/regex-macro

[BurntSushi]

I'd rather wait to revisit this until std::lazy has been a thing, and unfortunately, it probably needs to be there for at least a year before I'm okay bumping the MSRV to depend on it.

With that said, it's not clear to me that this is something we want to do. It's pretty painless already to compose the Lazy API with Regex. For example:

static RE_GROUP: Lazy<Regex> =
    Lazy::new(|| Regex::new(r"^[-A-Za-z0-9]+$").unwrap());

That's already pretty good IMO. So I think if someone wants to propose a new regex-specific API here, whatever they come up with is going to need to have a meaningful advantage over the above.

[matklad]

That's already pretty good IMO

I'd maybe push back against this tiny bit. To give a concrete example from git-branchless,

Both this:

        lazy_static! {
            static ref RE: Regex = Regex::new(r"^([^ ]+) (.+)$").unwrap();
        };
        match RE.captures(line) {

And this:

        static RE: Lazy<Regex> = Lazy::new(|| Regex::new(r"^([^ ]+) (.+)$").unwrap());
        match RE.captures(line) {

do look to me meaningfully worse than a hypothetical:

        match regex!(r"^([^ ]+) (.+)$").captures(line) {

Fore these kind of code, the use of Lazy and the use of static both seem like irrelevant details (which of course is context dependent: in more "production" code you might very well want to be aware when exactly each regex is compiled, but in more "stripty" code you'd rather ignore this, and one often wants to use regex in a scripty code). Similarly, introducing an essentially useless name like RE adds to noise.

At the same time, yes, absolutely, at the moment this issue isn't really actionable on the regex side, because the underlying lazy evaluation primitive isn't yet available in std. And also the implementation is really tiny and is available on crates.io and even in once-cell docs (https://docs.rs/once_cell/latest/once_cell/#lazily-compiled-regex), so there's no practical reason for regex to provide something now.

[BurntSushi]

Yeah I'd love to re-open this and revisit it. I do personally like the regex!(...) code you've shown. It's nice.

The other angle to put on this here is that regex! actually used to be a thing, and it was for "compile time regex." There is a minute possibility of that coming back some day (years away, if ever). There is also the problem that some folks might interpret it as a compile time regex. (Not necessarily because it used to exist, but because it kind of looks like it. But maybe that's my own bias.)

[matklad]

std::sync::OnceLock is a thing as of Rust 1.70.0, so in about a year (1.79.0) we could in theory provide this.

[BurntSushi]

EDIT: I updated this comment on 2024-01-23. I would love to hear folks thoughts on this.

OK, so now that OnceLock is stable, we can revisit this. To summarize the above discussion:

• A regex! macro can make some uses of regexes much nicer, even when compared to using once_cell::sync::Lazy (which has a std counterpart called LazyLock, but isn't stable yet).
• The regex crate tries to be somewhat conservative with respect to MSRV. I've somewhat arbitrarily settled on "try to support the last year of Rust releases." Since OnceLock was just stabilized, this means it will probably be a year before we can add it. We could use conditional compilation to sniff the Rust version and only add it for newer versions of Rust, but I've done this sort of thing in the past and I'm kind of "over it." Plus, in this case, it's a public API and I imagine it could be confusing for folks on an older version reading the docs and not being able to use it for non-obvious reasons.
• Some folks might get the impression that regex!("...") is a compile time regex when in reality it's "just" a convenience. In the past, there was actually a procedural macro named regex! that was a compile time regex.

I believe the macro would be defined as this:

macro_rules! regex {
    ($re:literal $(,)?) => {{
        use {std::sync::OnceLock, regex::Regex};

        static RE: OnceLock<Regex> = OnceLock::new();
        RE.get_or_init(|| Regex::new($re).unwrap())
    }};
}

Its return type is a &'static Regex. It would appear in the crate root. So you'd use it like regex::regex!. I'd also add an analogous version of it to the bytes submodule that creates a regex::bytes::Regex instead.

I have some other concerns too that I've thought of:

• The macro hides an unwrap call. I don't think this is a huge deal personally since we can define the macro in a way that it only accepts a string literal. We could make the macro return a Result<Regex, Error> and force users to unwrap it, but that seems like a bit of a bummer. Overall I'm personally fine hiding the unwrap as long we clearly document it, but I do think there will be a contingent of users that will be unhappy with hiding the unwrap.
• The fact that you don't have to unwrap the result of regex!("...") is likely to reinforce the idea that it is a compile time regex. For example, "of course I wouldn't have to unwrap it, since it's a compile time regex my program would fail to compile if the regex were invalid."
• The macro as proposed unfortunately reinforces a pattern that leads to a pretty unfortunate performance bug in some cases. The underlying performance bug could just be fixed and that would mitigate this concern. (This has been mostly mitigated to the point that I no longer feel too concerned about this.)
• Using this macro seems to preclude setting any options on a RegexBuilder. I think for the most part that is fine since most options can be set inside the regex syntax itself (.e.g, (?i) instead of RegexBuilder::case_insensitive), but some can't. For example, size limits.
• The fully qualified name, regex::regex!, seems unfortunately redundant. @matklad suggested some alternatives below, but I think they're too verbose. We could just name it compile! so that its fully qualified use is regex::compile!. But its... unqualified use as just compile! is pretty awful.
• Last but not least, do we even need a macro for this? I suppose the macro helps us guarantee that the pattern is actually a literal and thus makes the unwrap more palatable, but this can also be written as a function that accepts a &'static str which is pretty close to requiring a literal.

Here's what this would look like as a function (or perhaps a Regex::lazy method or something):

fn regex(pattern: &'static str) -> &'static regex::Regex {
    use std::sync::OnceLock;
    static RE: OnceLock<regex::Regex> = OnceLock::new();
    RE.get_or_init(|| regex::Regex::new(pattern).unwrap())
}

Playground: https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=ab56dc43c81e9b16bfbf7fef5ab4805c

(A function doesn't work. This has to be a macro. See @matklad's rebuttal below.)

Note that if you're reading this and wondering "why not just make compile time regex instead," please see #1076 and #1012. This macro is more about codifying a very common pattern that is used to avoid the footgun of compiling a regex in a loop. (A compile time regex would also solve that problem, but as the issues linked explain, there is a lot more to it than the simple macro proposed here that a bunch of folks are already using in one form or another.)