Support retrieving partial responses from endpoints (refactor for pydantic `2.10`)

[devraj]

Endpoint such as searches allow providing a parameters called fields which is an enum of fields which the response ends up returning e.g href, id, name, type, division, serverDisplayName, notes

The string must not contain any spaces. Just alphanumerics, underscores, commas, and dots.

This would mean we would have to make all fields in our Response objects optional.

We should also validate that the input is in fact an acceptable response field name.

[devraj]

pydantic has official support for this as of v2.10.

Rather than make the values optional, we should consider moving up to v2.10 and use the partial validation feature

[devraj]

Using yield and async callbacks in Python serve different purposes, and the choice depends on the behavior you need:

• Callback Approach: Suitable for simple async tasks where you want to pass a result to a specific function once the task completes, as shown in the previous example.
• Using yield with Generators: Useful when you want to produce a series of results over time, especially when processing a large dataset or streaming data. In async scenarios, yield can be used in asynchronous generators with async for, which is ideal for processing multiple chunks of data as they become available.

[devraj]

I might have misunderstood this feature for our purposes, this partial validation allows validating partial streams (not objects)

from typing import List

from annotated_types import MinLen
from typing_extensions import Annotated

from pydantic import BaseModel, TypeAdapter


class MyModel(BaseModel):
    a: int
    b: Annotated[str, MinLen(5)]


ta = TypeAdapter(List[MyModel])
v = ta.validate_json(
    '[{"a": 1, "b": "12345"}, {"a": 1,',
    experimental_allow_partial=True,
)
print(v)
#> [MyModel(a=1, b='12345')]

see docs which means that in cases where a field or a type is relaxed we would still have to cater for that in our rules #68.

Gallagher CC will not send incomplete objects.

This may be irrelevant for our use case and might have to side with the solution we outline in #68.