FullText search does not work across HTML tags

[quicklywilliam]

Currently if you search an HTML document for text that includes an HTML tag, it will fail. For example, the following:

Looking at the code, I believe this is because the implementation of findAllText in cre.cpp searches are only conducted within nodes, so search text that spans multiple nodes will always fail.

[poire-z]

Yes, known issue and good analysis - but not easy to solve :/ Point B in koreader/koreader#10366 (comment)

[quicklywilliam]

Hah, I figured. I spent more time than I care to admit trying to work around it today! I may take a crack at it, if I am understanding it correctly it doesn't seem that insane. Ie, in your example of finding "foobar" in <P><B>The foo</B>bar sucks.</P>, after failing to find foobar in the first node we could do a second comparison that looks to see if the last bit of the node matches ("foo"), and if so we then keep track of the partial match and repeat the process with the remainder of the string ("bar"). That sound about right?

[poire-z]

Sounds like you paraphrased my:

Interestingly, the Firefox code is quite standalone, and we could nearly pick its idea as is: when a match has started, and we reach end of text node, reduce the pattern to what remains (If meeting "The foo" and "foobar" started matching up to "foo", go ahead and try to match "bar" in the next text node(s). If it fails rewind back to the previous text node and go on). We could do that for our plain text matching, but we can't do that with @zwim regexp matching, as we can't adjust the regexp as simply as plain text strings - so it would need to keep the existing single-text-node code for regexp search, and so 2 code paths, not nice :/

So, yes, sounds probably right :)
But I'm not sure it will be that easy: you'll need to somehow be recursive/nested as foobar may be in 3 or 4 nodes - and it shouldn't add too much cost (as you'll probably failing to find foobar in the first node in 99.99% of the text nodes :))
I would also have liked to address all the issues mentionned in that link (diacritics,   vs. regular space,...) at the same time, or keep that/hooks in mind when doing that. (+ the regexp issue, that I don't know if we should keep the existing deficient codepath for it, or just kill it).
You can of course have a try, maybe you'll find an elegant solution that will open for other elegant ideas for the other issues.