update watchtower version in amazon provider #25019

Activity

1. alexInhert

   added

   kind:featureFeature RequestsFeature Requests

   on Jul 13, 2022
2. potiuk

   added

   good first issue

   on Jul 13, 2022
3. potiuk commented on Jul 13, 2022

   

   potiuk

   on Jul 13, 2022

   Member

   More actions

   Would you like to attempt it @alexInhert ? I marked it as good first issue so maybe someone will pick it up, but if you need it badly then the most certain way is to implement it yourself,
4. potiuk commented on Jul 13, 2022

   

   potiuk

   on Jul 13, 2022

   Member

   More actions

   Or maybe @ferruzzi or @o-nikolas - this is already somehow planned by the MWAA team :) ?
5. eladkal

   added

   provider:amazonAWS/Amazon - related issuesAWS/Amazon - related issues

   area:providers

   kind:taskA task that needs to be completed as part of a larger issueA task that needs to be completed as part of a larger issue

   and removed

   kind:featureFeature RequestsFeature Requests

   on Jul 13, 2022
6. o-nikolas commented on Jul 15, 2022

   

   o-nikolas

   on Jul 15, 2022

   Contributor

   More actions

   @potiuk, We have no immediate plans, our needs are covered by at least 2.0.1, but I think upgrading is still a good idea! I can assist @alexInhert if they get stuck, since I've done version bumps for Watchtower in the past.

   But note: This change will need a careful check of Airflow logs, since v3 of Watcher can now include much more information in the logs than previous (due to lacking json deserialization support many things were previously just logged as null but which now are logged in full)

   👍React with 👍1Reacted by Jarek Potiuk
7. cBiscuitSurprise commented on Sep 29, 2023

   

   cBiscuitSurprise

   on Sep 29, 2023

   Contributor

   More actions

   I would like to give this issue a try.

   We have the option to override the default-json serializer to whatever we want. I would propose that we override this to preserve the 2.x behavior with a config option to allow opt-in to the new repr format using logging.REMOTE_LOG_KWARGS (a new config). This way we can upgrade to 3.x, but keep the old serializer.

   def json_serialize_legacy(o):
       """
       The legacy `watchtower@2.0.1` json serializer function that serialized
       datetime objects as ISO format andall other non-JSON-serializable to `null`
       """
       if isinstance(o, (date, datetime)):
           return o.isoformat()
   
   ...
   
   class CloudwatchTaskHandler(FileTaskHandler, LoggingMixin):
   ...
       def __init__(self, base_log_folder: str, log_group_arn: str, filename_template: str | None = None):
           ...
   
           remote_log_options: Dict[str, Any] = conf.get("logging", "REMOTE_LOG_KWARGS", {})
           self.opt_in_new_json_serialize = remote_log_options.get("opt_in_new_json_serialize", False)
   
   ...
   
       def set_context(self, ti):
           super().set_context(ti)
           self.handler = watchtower.CloudWatchLogHandler(
               ...
               json_serialize_default=None if self.opt_in_new_json_serialize else json_serialize_legacy,
           )

   This is my first interaction with this project, so I understand if it's not desirable to add config like this, just brainstorming. Let me know what you guys think.
8. cBiscuitSurprise commented on Oct 2, 2023

   

   cBiscuitSurprise

   on Oct 2, 2023

   Contributor

   More actions

   Is this class deprecated? Would that make this issue a non-issue (and I should find another issue)?

   airflow/airflow/utils/log/__init__.py

   Lines 23 to 28 in 6618c5f

   	__deprecated_classes = {
   	"cloudwatch_task_handler": {
   	"CloudwatchTaskHandler": (
   	"airflow.providers.amazon.aws.log.cloudwatch_task_handler.CloudwatchTaskHandler"
   	),
   	},
9. vincbeck

   closed this as completedin #34747on Oct 6, 2023
10. vincbeck

    added a commit that references this issue on Oct 6, 2023

    Upgrade watchtower to 3.0.1 (#25019) (#34747)

    c01abd1