Copyright detection regression after implementing gibberish detection

[JonoYang]

Certain copyrights are not detected anymore as the gibberish detector identifies it as gibberish:

https://github.com/aboutcode-org/scancode-toolkit/blob/2402-detect-gibberish-copyright/tests/cluecode/data/copyrights/scilab-Scilab#L67

• an instance of Scilab (c) INRIA-ENPC. was not detected
• c) INRIA-ENPC. is identified as gibberish

https://github.com/aboutcode-org/scancode-toolkit/blob/2402-detect-gibberish-copyright/tests/cluecode/data/copyrights/misco4/linux-copyrights/Documentation/networking/arcnet-hardware.txt#L32

• this did not detect Copyright Waterloo Microsystems Inc. 1985
• @Copyright is identified as gibberish

https://github.com/aboutcode-org/scancode-toolkit/blob/2402-detect-gibberish-copyright/tests/cluecode/data/authors/trailing_date#L3C19-L3C59

• Alexander Kanavin <alex.kanavin@gmail.com> was not detected
• * : commit 3debe362faa62e5b381b880e3ba23aee07c85f6e Author: is detected as gibberish

Originally posted by @JonoYang in #4610 (comment)

[Jayant-kernel]

"Hi @JonoYang, I would like to work on this issue.
I can investigate why these copyright strings are being flagged as gibberish and try to fix the detection logic."

[VaaishnaviS]

Hi @JonoYang,
I have investigated the detection logic and found that the normalize() function is stripping legal symbols (like © and (c) ), which causes the Markov chain to flag these strings as gibberish.

I have developed a fix that introduces a whitelist of legal indicators to be checked before normalization. I am opening a Pull Request for this now.