Skip to content

The associated hook should be a subclass of 'DbApiHook'. Got VerticaHook #25488

Closed

Description

@dKosarevsky

Apache Airflow version

2.3.3 (latest released)

What happened

I trying to make custom VerticaToGoogleSheetsOperator to work with Vertica database & GoogleSheets, I've installed all the required provider and successfully tested the connection. When I run VerticaToGoogleSheetsOperator I got this error message

Error: The connection type is not supported by VerticaToGoogleSheetsOperator. The associated hook should be a subclass of 'DbApiHook'. Got VerticaHook

What you think should happen instead

I think it's a bug in BaseSQLOperator on line 94 and I know how to fix it with the issubclass check.

How to reproduce

import logging
import datetime
import numbers

from contextlib import closing
from typing import Any, Iterable, Mapping, Optional, Sequence, Union

from airflow.operators.sql import BaseSQLOperator
from airflow.providers.google.suite.hooks.sheets import GSheetsHook

logger = logging.getLogger(__name__)


class VerticaToGoogleSheetsOperator(BaseSQLOperator):
    """
    Copy data from SQL results to provided Google Spreadsheet;
    :param sql: The SQL to execute;
    :param spreadsheet_id: The Google Sheet ID to interact with;
    :param conn_id: the connection ID used to connect to the database;
    :param parameters: The parameters to render the SQL query with;
    :param spreadsheet_range: The A1 notation of the values to retrieve;
    :param gcp_conn_id: The connection ID to use when fetching connection info;
    """

    template_fields: Sequence[str] = (
        "sql",
        "spreadsheet_id",
        "spreadsheet_range",
    )

    template_fields_renderers = {"sql": "sql"}
    template_ext: Sequence[str] = (".sql",)

    ui_color = "#a0e08c"

    def __init__(
        self,
        *,
        sql: str,
        spreadsheet_id: str,
        spreadsheet_range: str,
        sql_conn_id: str = "dwh",
        append_values: bool = False,
        clear_sheet: bool = True,
        gcp_conn_id: str = "google_sheets",
        parameters: Optional[Union[Mapping, Iterable]] = None,
        **kwargs,
    ) -> None:
        super().__init__(**kwargs)

        self.sql = sql
        self.conn_id = sql_conn_id
        self.parameters = parameters
        self.gcp_conn_id = gcp_conn_id
        self.clear_sheet = clear_sheet
        self.append_values = append_values
        self.spreadsheet_id = spreadsheet_id
        self.spreadsheet_range = spreadsheet_range

    @staticmethod
    def _data_prep(data):
        for row in data:
            item_list = []
            for item in row:
                if isinstance(item, (datetime.date, datetime.datetime)):
                    item = item.isoformat()
                elif isinstance(item, int):  # To exclude int from the number check.
                    pass
                elif isinstance(item, numbers.Number):
                    item = float(item)
                item_list.append(item)
            yield item_list

    def _get_data(self):
        hook = self.get_db_hook()
        with closing(hook.get_conn()) as conn, closing(conn.cursor()) as cur:
            self.log.info("Executing query")
            cur.execute(self.sql, self.parameters or ())

            yield [field[0] for field in cur.description]
            yield from self._data_prep(cur.fetchall())

    def execute(self, context: Any) -> None:
        self.log.info("Getting data")
        values = list(self._get_data())

        self.log.info("Connecting to Google")
        sheet_hook = GSheetsHook(gcp_conn_id=self.gcp_conn_id)

        self.log.info(f"Uploading data to https://docs.google.com/spreadsheets/d/{self.spreadsheet_id}")

        if self.append_values:
            sheet_hook.append_values(
                spreadsheet_id=self.spreadsheet_id,
                range_=self.spreadsheet_range,
                values=values,
            )
        else:
            if self.clear_sheet:
                sheet_hook.clear(spreadsheet_id=self.spreadsheet_id, range_=self.spreadsheet_range)
            sheet_hook.update_values(
                spreadsheet_id=self.spreadsheet_id,
                range_=self.spreadsheet_range,
                values=values,
            )

Operating System

Debian GNU/Linux 11 (bullseye)

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==4.0.0
apache-airflow-providers-celery==3.0.0
apache-airflow-providers-cncf-kubernetes==4.1.0
apache-airflow-providers-common-sql==1.0.0
apache-airflow-providers-docker==3.0.0
apache-airflow-providers-elasticsearch==4.0.0
apache-airflow-providers-ftp==3.0.0
apache-airflow-providers-google==8.1.0
apache-airflow-providers-grpc==3.0.0
apache-airflow-providers-hashicorp==3.0.0
apache-airflow-providers-http==3.0.0
apache-airflow-providers-imap==3.0.0
apache-airflow-providers-microsoft-azure==4.0.0
apache-airflow-providers-microsoft-mssql==3.1.0
apache-airflow-providers-mysql==3.0.0
apache-airflow-providers-odbc==3.0.0
apache-airflow-providers-postgres==5.0.0
apache-airflow-providers-redis==3.0.0
apache-airflow-providers-sendgrid==3.0.0
apache-airflow-providers-sftp==3.0.0
apache-airflow-providers-slack==5.0.0
apache-airflow-providers-sqlite==3.0.0
apache-airflow-providers-ssh==3.0.0
apache-airflow-providers-telegram==3.0.0
apache-airflow-providers-vertica==3.1.0

Deployment

Official Apache Airflow Helm Chart

Deployment details

terraform, helm, k8s, cloud

Anything else

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Activity

  1. boring-cyborg commented on Aug 2, 2022

    @boring-cyborg

    Thanks for opening your first issue here! Be sure to follow the issue template!

  2. potiuk commented on Aug 2, 2022

    @potiuk
    Member

    This is a bug introduced in the common.sql provider and it will be fixed in the upcoming version - fixed in #25293

    For now the solution is to downgrade to previous version of Vertica 3.0.0 provider as a workaround (there are no functional differences in 3.1.0). You will be able to upgrade when we release a new common-sql provider in ~ 2 weeks.

  3. potiuk commented on Aug 2, 2022

    @potiuk
    Member

    Closing as duplicate of #25274

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions