crash when using --unknown-licenses

[armijnhemel]

Description

ERROR: failed to run scan plugin: licenses:
Traceback (most recent call last):
  File "/home/armijn/git/scancode-toolkit/src/scancode/cli.py", line 1078, in run_codebase_plugins
    plugin.process_codebase(codebase, **kwargs)
  File "/home/armijn/git/scancode-toolkit/src/licensedcode/plugin_license.py", line 192, in process_codebase
    license_detections = collect_license_detections(
  File "/home/armijn/git/scancode-toolkit/src/licensedcode/plugin_license.py", line 247, in collect_license_detections
    license_detection_objects = detections_from_license_detection_mappings(
  File "/home/armijn/git/scancode-toolkit/src/licensedcode/detection.py", line 516, in detections_from_license_detection_mappings
    LicenseDetectionFromResult.from_license_detection_mapping(
  File "/home/armijn/git/scancode-toolkit/src/licensedcode/detection.py", line 489, in from_license_detection_mapping
    matches = LicenseMatchFromResult.from_dicts(
  File "/home/armijn/git/scancode-toolkit/src/licensedcode/detection.py", line 614, in from_dicts
    return [LicenseMatchFromResult.from_dict(lmm) for lmm in license_match_mappings]
  File "/home/armijn/git/scancode-toolkit/src/licensedcode/detection.py", line 614, in <listcomp>
    return [LicenseMatchFromResult.from_dict(lmm) for lmm in license_match_mappings]
  File "/home/armijn/git/scancode-toolkit/src/licensedcode/detection.py", line 592, in from_dict
    rule = Rule.from_match_data(license_match_mapping)
  File "/home/armijn/git/scancode-toolkit/src/licensedcode/models.py", line 2315, in from_match_data
    return klass(
  File "<attrs generated init licensedcode.models.UnknownRule>", line 68, in __init__
    self.__attrs_post_init__()
  File "/home/armijn/git/scancode-toolkit/src/licensedcode/models.py", line 2552, in __attrs_post_init__
    self.tokens()
  File "/home/armijn/git/scancode-toolkit/src/licensedcode/models.py", line 2101, in tokens
    text.startswith(('http://', 'https://', 'ftp://'))
AttributeError: 'NoneType' object has no attribute 'startswith'

How To Reproduce

Unfortunately I cannot share any client data, so this will be a bit hard.

./scancode -l -n8 --json-pp=/tmp/out.json --unknown-licenses /path/to/source

System configuration

For bug reports, it really helps us to know:

• What OS are you running on? (Windows/MacOS/Linux) - Linux
• What version of scancode-toolkit was used to generate the scan file? - v32.0.0rc4
• What installation method was used to install/run scancode? (pip/source download/other) - Git checkout

[armijnhemel]

It finishes without crashing.

[AyanSinhaMahapatra]

Thanks for the report @armijnhemel , also want your feedback on how the --unknown-licenses is working, do you have any feedback/insights there? See #3300 for updates on this btw. Also feel free to close this if this is solved.