[potix2]

This PR is about increasing typing coverage for the aws provider. Part of: #9708
I added type hints to the rest modules of amazon provider.

The typing coverage of amazon provider becomes 77.7.

This PR relates to #11434 and #11612.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.

[potix2]

We cannot pass None as the argument parameters. The API document is here.

I checked the behavior of submit_job with the following code.

#!/usr/bin/env python

import boto3
import json

cl = boto3.client('batch')
ap=None
params=None
overrides=None
resp = cl.submit_job(
        jobName='myjob-1',
        jobQueue='test-batch-job-queue',
        jobDefinition='test',
        arrayProperties=ap,
        parameters=params,
        containerOverrides=overrides,
        )
print(json.dumps(resp, indent=4, sort_keys=True))

Result:

Traceback (most recent call last):
  File "main.py", line 17, in <module>
    containerOverrides=overrides,
  File "/Users/potix2/.pyenv/versions/3.7.5/lib/python3.7/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Users/potix2/.pyenv/versions/3.7.5/lib/python3.7/site-packages/botocore/client.py", line 649, in _make_api_call
    api_params, operation_model, context=request_context)
  File "/Users/potix2/.pyenv/versions/3.7.5/lib/python3.7/site-packages/botocore/client.py", line 697, in _convert_to_request_dict
    api_params, operation_model)
  File "/Users/potix2/.pyenv/versions/3.7.5/lib/python3.7/site-packages/botocore/validate.py", line 297, in serialize_to_request
    raise ParamValidationError(report=report.generate_report())
botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid type for parameter arrayProperties, value: None, type: <class 'NoneType'>, valid types: <class 'dict'>
Invalid type for parameter parameters, value: None, type: <class 'NoneType'>, valid types: <class 'dict'>
Invalid type for parameter containerOverrides, value: None, type: <class 'NoneType'>, valid types: <class 'dict'>

[potix2]

script_arguments is passed to Glue.Client.start_job_run as Arguments whose type is dict.
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/glue.html#Glue.Client.start_job_run

[potix2]

job_description must not be None because its type is not Optional[T].

[potix2]

To avoid deeply nested if-statements I rearrange the order of code.

[potix2]

argmin's type is Optional[T] so we need to pass the default value.

[potix2]

It's too hard to add type hint to positions.

[potix2]

self.hook is initialized by None so its type is Optional[AWSDataSyncHook]. To eliminate the possibility that get_hook returns None, we need to rearrange the order of evaluation.

[potix2]

This returned value is not used anywhere and _create_datasync_task is private method.

[potix2]

This returned value is not used anywhere also.

[potix2]

This one is also.

[potix2]

This one is also.

[potix2]

fix typo

[potix2]

This attribute is directly referenced from child classes. To avoid None I defined it as cached_property.

[potix2]

I checked failed tests in local but it's successful. I try to trigger actions in CI.

[mik-laj]

@mlgruby Can you look at it?

[mlgruby]

Yup I can have a look

[mlgruby]

Nice work @potix2 , learned few things (that chached_property one) while reviewing

Thanks

[mlgruby]

This if is because of None default value of hook and hence mypy complaining for it?
I have seen several other places the same new if statement.

[potix2]

Yes. get_hook() returns an actual value, but in the original form mypy hence type error.

[mlgruby]

Well, I liked this one :)

[potix2]

Me,too. I would like to rewrite all get_hook() to this way but I think it's out of scope for this work, so I keep the original form.

[mik-laj]

Can you do a rebase? This is the only thing blocking me from merging this change.

[potix2]

I rebased but CI looks like hung up... After a few hours I'll try it again.

[mik-laj]

I rebased but CI looks like hung up... After a few hours I'll try it again.

Everything is green with me. Should I merge it or would you like to make any more changes?

[potix2]

@mik-laj It's ready for merging. I overlooked that static checking was successful because CI Build / Static checks: basic checks only was skipped.