Repository navigation
EMR serverless, new operator #20215
Description
Description
A new EMR serverless has been announced and it is already available, see:
- https://aws.amazon.com/blogs/big-data/announcing-amazon-emr-serverless-preview-run-big-data-applications-without-managing-servers/
- https://aws.amazon.com/emr/serverless/
Having an operator for creating applications and submitting jobs to EMR serverless would be awesome.
Use case/motivation
New operator for working with EMR serverless.
Related issues
No response
Are you willing to submit a PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Reacted by Kevin Su, Tommy Wagner, Damon P. Cortesi, keshrisohit and Arianna
Activity
boring-cyborg commentedon Dec 11, 2021
Thanks for opening your first issue here! Be sure to follow the issue template!
yeshbash commentedon Dec 13, 2021
Looks like the feature is not Generally Available and I couldn't find the docs to the APIs. I'd be happy to pitch in with this operator if there are more details about it. Also, any guidance from the community is appreciated.
ferruzzi commentedon Dec 16, 2021
That's correct, this is in preview and there is no announced date for general release yet. I'll also be happy to help once the docs are available.
dacort commentedon Jan 14, 2022
@yeshbash @ferruzzi If you haven't already, please sign up for the preview here: https://pages.awscloud.com/EMR-Serverless-Preview.html
We've started to let folks in and you will receive info on how to access the preview when you're allowlisted.
Docs are here and I've also created an emr-serverless-samples repository that may be useful.
I'm on the EMR team and would be happy to help implement or review any PRs. 😁
treyyi commentedon Mar 7, 2022
Hi @dacort,
I am also interested in EMR Serverless features for airflow such as operators, sensors, or testings.
Once EMR serverless is released officially, I will try to provide some suggestions, and I am willing to help you even if it wouldn't be my PR!
(FYI, currently I am in data engineer position for operating EMR clusters and airflow server.)
dacort commentedon Mar 28, 2022
I just published some information about how to use (a custom, while in preview) boto3 to access the EMR Serverless API.
https://github.com/aws-samples/emr-serverless-samples/tree/main/examples/python-api
dacort commentedon Apr 29, 2022
Working on moto PR as well so we can get ahead of tests. :) getmoto/moto#5071
dacort commentedon May 16, 2022
Hi folks - we're working internally at AWS on this. If folks here are currently in the preview and interested in testing this, please reach out to me. I'm
dacorton Twitter/LinkedIn/and the Airflow Slack.treyyi commentedon May 17, 2022
@dacort I am in the preview now, so just sent you a message on Twtitter!