Asynchronous API calls

[prabhant]

Description

Get evaluations of size 10000 for each task when supplied with list of task ids.
Currently we get evaluations for multiple tasks like this:
openml.evaluations.list_evaluations(task = suite.tasks, function = 'area_under_roc_curve', output_format='dataframe', size=10000)

which returns a dataframe of 10000 length with mixture of tasks, to get 10000 evaluations for each task user has to make a for loop which can take a long time.

We can introduce a new parameter like size_per_task to get a set of dataframes. For example
openml.evaluations.list_evaluations(task = suite.tasks, function = 'area_under_roc_curve', output_format='dataframe', size_per_task=10000)
which returns N datasets of length 10000.

[mfeurer]

How would the API call you're proposing work? If I were to implement your suggestion, I would use a for-loop under the hood (which apparently would take a lot of time).

[joaquinvanschoren]

Discussion from Slack:

joaquin 9:16 PM
Hi all, we noticed that when we do any listing call that requires multiple requests (e.g. listing evaluations for multiple tasks or when we need more than 10.000 results) the calls to the server seem to be sequential rather than asynchronous. Would it be possible to make them asynchronous? I already asked Prabhant to look into it but also wanted to ask whether you think that this is a good idea or not. Thanks!

mfeurer 10:15 AM
Would it be possible to make them asynchronous?
I think this is both possible and straight-forward
but also wanted to ask whether you think that this is a good idea or not.
Some will like it, so I guess it is a good idea to be able to do such calls in parallel

Hi, what is the best place to make a list of extensions to the openml-python API? Should we do that in the openml-python docs (e.g. under ‘examples’ or under a new menu item? Or do we want to keep that separate?
I think the docs are a good place; either on the contributing page or a new page
to be clear about the parallelism, it's straight-forward if one is using joblib, I'm not sure about other libraries. However, wouldn't that be bad for the server
?

PGijsbers 11:11 AM
I agree on making async calls, async network IO should be basic and well supported.

joaquin 11:12 AM
I don’t think this would hurt the server

PGijsbers 11:12 AM
Python has built in async, but I'm not sure if that was Py3.6+ (at least, I believe their first versions weren't that good)
If more than one parallel call hurts the server we have to fix the server instead
in case the builtin async doesn't work, maybe have a look at trio (https://trio.readthedocs.io/en/stable/)

mfeurer 11:24 AM
I'd prefer using built-in tools or the joblib (I know it's not optimal in for this use case, but incredibly simple)
with joblib it'll be basically replacing the for-loop by a different expression

joaquin 11:56 AM
the server should be able to handle 100s of parallel requests
it would be wise to check for errors and relaunch requests if unsuccessful, just as a contingency

[joaquinvanschoren]

From Engineering call: joblib seems the best choice for doing asynchronous calls. Should be rather straightforward, maybe a few hours of work.

[amueller]

I might pick this up if it actually does help ;)
so we support python 3.5 if I want to try other solutions? Main downside of joblib is that it's an additional requirement and adds another dependency. But maybe adding a joblib dependency is good to solve #891 as well. I'll investigate.

[amueller]

How do you know how many calls you need? You don't know the length of the result ahead of time. You could try to run the query and check the server error message which contains the total length. That seems rather indirect, though...

[joaquinvanschoren]

Good question. If you do things synchronously, you can just repeat until you get fewer results than requested. If you work asynchronously, you need to know this beforehand.

Currently, the only way to get this from the REST API is to:

• Do an initial request without a size filter
• If there are more than 10000 results, it will return an error message with the total number of items.

I've opened a new issue here: openml/OpenML#1032

Let me know if this is blocking, or whether you can work around it for now by interpreting the error message.

[mfeurer]

Main downside of joblib is that it's an additional requirement and adds another dependency.

It's already kind of a dependency because scikit-learn is a dependency. Also, #880 requires joblib as a dependency.

so we support python 3.5

It's still the default in some Ubuntus.

[joaquinvanschoren]

Ok, so are we converging at solving this via joblib?
@amueller are you still interested in taking this up?
Do you need the new api call with the number of items for this?

[joaquinvanschoren]

We discussed this in the last engineering call and decided that we need a better view on the use cases for this. Hitting the server 100x as hard may not be the solution and a count API may also be quite slow. Paging (with offset) is also not ideal since it is wasteful (see https://use-the-index-luke.com/no-offset), so we may not want to support that indefinitely.

It may be better to simply raise the result limit from 10,000 to whatever number we can support, or to remove the limit entirely but impose additional constraints on the filters (e.g. only allow 1 task_id or 1 uploader_id) at a time.

The latter solution would still require async calls to get results from more than 1 task or uploader, but it would be more predictable, it's better on the server (no more multiple paging calls) and it would not require a count API, since you would be getting all results for a single task.

We need a decision on from the REST API team before moving on.

[janvanrijn]

This is not the conclusion that I drew from that conversation.

It may be better to simply raise the result limit from 10,000 to whatever number we can support, or to remove the limit entirely but impose additional constraints on the filters (e.g. only allow 1 task_id or 1 uploader_id) at a time.

Before rainsing the limits (they can be raised slightly), can you please once again try to explain the use-case that is currently not supported? I feel like "getting all runs of a certain task" is not something that you want to do in many cases.