Skip to content

add basic openlineage metrics to stats collector #31223

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
harels wants to merge 1 commit into from
Closed

add basic openlineage metrics to stats collector #31223

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion airflow/providers/openlineage/plugins/adapter.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@
from airflow.providers.openlineage.extractors import OperatorLineage
from airflow.providers.openlineage.utils.utils import OpenLineageRedactor
from airflow.utils.log.logging_mixin import LoggingMixin
from airflow.stats import Stats
from openlineage.client import OpenLineageClient, set_producer
from openlineage.client.facet import (
BaseFacet,
Expand Down Expand Up @@ -117,8 +118,10 @@ def emit(self, event: RunEvent):
self._client = self.get_or_create_openlineage_client()
event = self._redacter.redact(event, max_depth=20)
try:
return self._client.emit(event)
with Stats.timer("ol.emit.attempts"):

@tatiana tatiana May 12, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@harels, this is great. It will help with the telemetry open lineage events.
Would it make sense for us to add or modify an existing test to assert we're sending these to Statsd?

@pankajkoti pankajkoti Jun 12, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, yes a test would help here.

@sunank200 sunank200 Jun 13, 2023

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. Test should also include scenario for exception

return self._client.emit(event)
except requests.exceptions.RequestException:
Stats.incr("ol.emit.failed")
self.log.exception(f"Failed to emit OpenLineage event of id {event.run.runId}")

def start_task(
Expand Down