AWS tasks must always tag the resources they create (ec2_vpc_nat_gateway)

[johnsimcall]

The ec2_vpc_nat_gateway module doesn't allow tags to be created. The workaround is to immediately run a second ec2_tag task to apply the OpenShiftCluster and OpenShiftClusterId tags. The aws_destroy.yml playbook needs those tags to do it's job.

This workaround is problematic though because we create multiple NAT Gateways (one per AZ) and we aren't guaranteed to have all of them be created successfully. If any NGW failed to get created --for any reason-- the ec2_tag task would not be run. This results in creating untagged NGWs that can't be deleted by the aws_destroy.yml playbook. When the NGWs aren't deleted, then the subnets they belong to and ultimately the VPC can't be deleted.

Here is an example of a partial failure caused by an insufficient EIP Allocation quota. Only one of the three NGWs was created successfully. The logs also show the results of the patch I've created to resolve the issue. The patch simply wraps the ec2_vpc_nat_gateway task in an Ansible Block with an always: section.

Thanks to Mike for reporting the issue originally. And thanks to Gerald for suggesting the solution!

TASK [infrastructure : Create NAT gateways] ******************************************************************************************************************
Wednesday 19 August 2020  05:32:34 +0000 (0:00:02.146)       0:00:19.863 ****** 
changed: [localhost] => (item=subnet-0bdde190730995f57) => {
    "ansible_loop_var": "item",
    "changed": true,
    "create_time": "2020-08-19T05:32:36+00:00",
    "item": "subnet-0bdde190730995f57",
    "nat_gateway_addresses": [
        {
            "allocation_id": "eipalloc-08f5b15b12fda58d1"
        }
    ],
    "nat_gateway_id": "nat-0e52ccf6f78cd7bfd",
    "state": "pending",
    "subnet_id": "subnet-0bdde190730995f57",
    "success": true,
    "vpc_id": "vpc-0ae3aed1910f59a4f"
}
failed: [localhost] (item=subnet-029f25413093ecfe2) => {
    "ansible_loop_var": "item",
    "changed": "False",
    "item": "subnet-029f25413093ecfe2",
    "success": false
}

MSG:

An error occurred (AddressLimitExceeded) when calling the AllocateAddress operation: The maximum number of addresses has been reached.
failed: [localhost] (item=subnet-06c05d404fea71bf2) => {
    "ansible_loop_var": "item",
    "changed": "False",
    "item": "subnet-06c05d404fea71bf2",
    "success": false
}

MSG:

An error occurred (AddressLimitExceeded) when calling the AllocateAddress operation: The maximum number of addresses has been reached.

TASK [infrastructure : Add NAT gateway tags] *****************************************************************************************************************
Wednesday 19 August 2020  05:32:38 +0000 (0:00:03.753)       0:00:23.617 ****** 
changed: [localhost] => (item=[{'msg': '', 'success': True, 'changed': True, 'create_time': '2020-08-19T05:32:36+00:00', 'nat_gateway_addresses': [{'allocation_id': 'eipalloc-08f5b15b12fda58d1'}], 'nat_gateway_id': 'nat-0e52ccf6f78cd7bfd', 'state': 'pending', 'subnet_id': 'subnet-0bdde190730995f57', 'vpc_id': 'vpc-0ae3aed1910f59a4f', 'invocation': {'module_args': {'subnet_id': 'subnet-0bdde190730995f57', 'if_exist_do_not_create': True, 'debug_botocore_endpoint_logs': False, 'validate_certs': True, 'state': 'present', 'wait': False, 'wait_timeout': 320, 'release_eip': False, 'ec2_url': None, 'aws_secret_key': None, 'aws_access_key': None, 'security_token': None, 'profile': None, 'region': None, 'eip_address': None, 'allocation_id': None, 'nat_gateway_id': None, 'client_token': None}}, 'failed': False, 'item': 'subnet-0bdde190730995f57', 'ansible_loop_var': 'item'}, 'us-gov-east-1a']) => {
    "added_tags": {
        "Name": "ocp4-jcall-redhatgov-io-us-gov-east-1a",
        "OpenShiftCluster": "ocp4.jcall.redhatgov.io",
        "OpenShiftClusterId": "ocp4-jcall-redhatgov-io"
    },
    "ansible_loop_var": "item",
    "changed": true,
    "item": [
        {
            "ansible_loop_var": "item",
            "changed": true,
            "create_time": "2020-08-19T05:32:36+00:00",
            "failed": false,
            "invocation": {
                "module_args": {
                    "allocation_id": null,
                    "aws_access_key": null,
                    "aws_secret_key": null,
                    "client_token": null,
                    "debug_botocore_endpoint_logs": false,
                    "ec2_url": null,
                    "eip_address": null,
                    "if_exist_do_not_create": true,
                    "nat_gateway_id": null,
                    "profile": null,
                    "region": null,
                    "release_eip": false,
                    "security_token": null,
                    "state": "present",
                    "subnet_id": "subnet-0bdde190730995f57",
                    "validate_certs": true,
                    "wait": false,
                    "wait_timeout": 320
                }
            },
            "item": "subnet-0bdde190730995f57",
            "msg": "",
            "nat_gateway_addresses": [
                {
                    "allocation_id": "eipalloc-08f5b15b12fda58d1"
                }
            ],
            "nat_gateway_id": "nat-0e52ccf6f78cd7bfd",
            "state": "pending",
            "subnet_id": "subnet-0bdde190730995f57",
            "success": true,
            "vpc_id": "vpc-0ae3aed1910f59a4f"
        },
        "us-gov-east-1a"
    ],
    "tags": {
        "Name": "ocp4-jcall-redhatgov-io-us-gov-east-1a",
        "OpenShiftCluster": "ocp4.jcall.redhatgov.io",
        "OpenShiftClusterId": "ocp4-jcall-redhatgov-io"
    }
}
skipping: [localhost] => (item=[{'msg': 'An error occurred (AddressLimitExceeded) when calling the AllocateAddress operation: The maximum number of addresses has been reached.', 'success': False, 'changed': 'False', 'failed': True, 'invocation': {'module_args': {'subnet_id': 'subnet-029f25413093ecfe2', 'if_exist_do_not_create': True, 'debug_botocore_endpoint_logs': False, 'validate_certs': True, 'state': 'present', 'wait': False, 'wait_timeout': 320, 'release_eip': False, 'ec2_url': None, 'aws_secret_key': None, 'aws_access_key': None, 'security_token': None, 'profile': None, 'region': None, 'eip_address': None, 'allocation_id': None, 'nat_gateway_id': None, 'client_token': None}}, 'item': 'subnet-029f25413093ecfe2', 'ansible_loop_var': 'item'}, 'us-gov-east-1b'])  => {
    "ansible_loop_var": "item",
    "changed": false,
    "item": [
        {
            "ansible_loop_var": "item",
            "changed": "False",
            "failed": true,
            "invocation": {
                "module_args": {
                    "allocation_id": null,
                    "aws_access_key": null,
                    "aws_secret_key": null,
                    "client_token": null,
                    "debug_botocore_endpoint_logs": false,
                    "ec2_url": null,
                    "eip_address": null,
                    "if_exist_do_not_create": true,
                    "nat_gateway_id": null,
                    "profile": null,
                    "region": null,
                    "release_eip": false,
                    "security_token": null,
                    "state": "present",
                    "subnet_id": "subnet-029f25413093ecfe2",
                    "validate_certs": true,
                    "wait": false,
                    "wait_timeout": 320
                }
            },
            "item": "subnet-029f25413093ecfe2",
            "msg": "An error occurred (AddressLimitExceeded) when calling the AllocateAddress operation: The maximum number of addresses has been reached.",
            "success": false
        },
        "us-gov-east-1b"
    ],
    "skip_reason": "Conditional result was False"
}
skipping: [localhost] => (item=[{'msg': 'An error occurred (AddressLimitExceeded) when calling the AllocateAddress operation: The maximum number of addresses has been reached.', 'success': False, 'changed': 'False', 'failed': True, 'invocation': {'module_args': {'subnet_id': 'subnet-06c05d404fea71bf2', 'if_exist_do_not_create': True, 'debug_botocore_endpoint_logs': False, 'validate_certs': True, 'state': 'present', 'wait': False, 'wait_timeout': 320, 'release_eip': False, 'ec2_url': None, 'aws_secret_key': None, 'aws_access_key': None, 'security_token': None, 'profile': None, 'region': None, 'eip_address': None, 'allocation_id': None, 'nat_gateway_id': None, 'client_token': None}}, 'item': 'subnet-06c05d404fea71bf2', 'ansible_loop_var': 'item'}, 'us-gov-east-1c'])  => {
    "ansible_loop_var": "item",
    "changed": false,
    "item": [
        {
            "ansible_loop_var": "item",
            "changed": "False",
            "failed": true,
            "invocation": {
                "module_args": {
                    "allocation_id": null,
                    "aws_access_key": null,
                    "aws_secret_key": null,
                    "client_token": null,
                    "debug_botocore_endpoint_logs": false,
                    "ec2_url": null,
                    "eip_address": null,
                    "if_exist_do_not_create": true,
                    "nat_gateway_id": null,
                    "profile": null,
                    "region": null,
                    "release_eip": false,
                    "security_token": null,
                    "state": "present",
                    "subnet_id": "subnet-06c05d404fea71bf2",
                    "validate_certs": true,
                    "wait": false,
                    "wait_timeout": 320
                }
            },
            "item": "subnet-06c05d404fea71bf2",
            "msg": "An error occurred (AddressLimitExceeded) when calling the AllocateAddress operation: The maximum number of addresses has been reached.",
            "success": false
        },
        "us-gov-east-1c"
    ],
    "skip_reason": "Conditional result was False"
}

PLAY RECAP ***************************************************************************************************************************************************
localhost                  : ok=16   changed=1    unreachable=0    failed=1    skipped=20   rescued=0    ignored=0   

Wednesday 19 August 2020  05:32:40 +0000 (0:00:01.561)       0:00:25.178 ****** 
=============================================================================== 
Playbook run took 0 days, 0 hours, 0 minutes, 25 seconds

root at openshift4-bundle in /app 
#