Extend process module

[maxbachmann]

Currently the process module has the following functions:

function	kind	explanation
extractOne	one x many	returns the best match as (choice, score, index/key)
extract	one x many	returns the best matches until limit as list[(choice, score, index/key)]
extract_iter	one x many	generator yielding (choice, score, index/key). Usage is not really recommended, since it is far slower than the others
cdist	many x many	returns all results as numpy matrix

It would be nice to have equivalents of extractOne / extract for many x many. They would need less memory than cdist, which can take a large amount of memory when len(queries) and len(choices) are large.

function	kind	explanation
-	many x many	returns the best matches as list[(choice, score, index)]
-	many x many	returns the best matches until limit as list[list[(choice, score, index)]]
-	one x many	returns all result without any sorting like cdist

A first thought might be to overload the existing extractOne / extract on the type passed as query / queries. However this is not possible, since the following is a valid usage of these methods:

extractOne(["hello", "world"], [["hello", "world"]])

which can not be distinguished from many x many. For this reason these functions need a new API.

Beside this in many cases users are not actually interested, but only care about finding elements with a score, which is better than the score_cutoff. These could potentially be implemented more efficiently, since the implementation could quit once it is known, that they are better than score_cutoff. These could be cases:

function	kind	explanation
-	many x many	returns matrix of bool
-	one x many	returns list of bool when there is a matching choice (e.g. https://stackoverflow.com/questions/70770842/matching-strings-within-two-lists/70780527#70780527)

This could be automatically done when the user passes dtype=bool.

Any suggestions on the naming of these new API's are welcome.

[maxbachmann]

The api will follow the following scheme:

class Compare:
    __init__(scorer, *):

   def set_seq1(a):      -> single
   def set_seq1_list(a): -> many

   def set_seq2(b):      -> single
   def set_seq2_list(b): -> many

   def all()
   def max(axis=None, initial=..., keepdims=False)
   def sorted(axis=None, keepdims=False, limit=None)

How about calling it extractMany? I would love to have this feature soon :)

Is there any chance it'll be in any of the upcoming releases or you're not yet interested in implementing this? Thank you :)

[maxbachmann]

I am still interested in adding this feature. However I do not expect that I will have the time to implement this until later this year.

[maxbachmann]

For the max/sorted functions it would make sense to store both the score and the corresponding index in the results. However I am unsure how this should be stored in numpy/pandas/xarray/... to make further usage simpler.

Options I am considering so far:

• two separate numpy arrays
• a numpy structured array (I think this would be a pain when further processing the results)

I do not have a lot of experience with the whole scientific Python stack. Is there a specific format which would make sense for the results to simplify further processing. @Unc3nZureD do you have any preference for the result format?

Sorry for the huge delay. Sadly I have no preference as I'm not familiar with them at all.

Currently, it could help me by moving away from the case where I've got let's say 100 patterns and a "database" of patterns, let's say with 1000 ones inside. Currently, I'm looping the 100 patterns on a single python thread and each time checking the DB. It'd be a lot more optimal if some multithreaded C/++ code could do it and just return an array of bool or float (depending if I need the actual value or not)

Honestly speaking I'm not even sure why scientific libs would be necessary, but it may be due to I'm a beginner.

Thank you!

[maxbachmann]

Some things about this could be simplified:

1. only make this many x many. When users pass in a list with a single element this is good enough of a representation for 1 x many. The default of keepdims=false takes care of this anyways.

2. allow dumping to a byte array and loading from it. This would allow generating optimized models ahead of time.

3. similar to the changes in the process module scorer kwargs should be passed via scorer_kwargs

class Compare:
    __init__(choices, *, scorer, processor=None, scorer_kwargs=None):

   def all(queries)
   def max(queries, axis=None, initial=..., keepdims=False)
   def sorted(queries, axis=None, keepdims=False, limit=None)