Repository navigation

Add transfer operator S3 to (generic) SQL - #28964

Closed

maggesssss wants to merge 71 commits into

apache:mainfrom

maggesssss:feat_s3_to_sql_transfer

Closed

Add transfer operator S3 to (generic) SQL#28964
maggesssss wants to merge 71 commits into
apache:mainfrom
maggesssss:feat_s3_to_sql_transfer

Commits on Dec 25, 2022
- added JSON linter to connection edit / add UI for field extra. On connection edit screen, existing extra data will be displayed idented
  maggesssss
  committed
  1fd240f
Commits on Jan 15, 2023
- Merge branch 'apache:main' into feat_conn_ui
  maggesssss
  authored
  b84cc56
- Released new S3 to SQL Transfer Operator Including Documentation and Tests
  
  root
  authored and
  root
  committed
  cd204a3
- added missing sql_conn_id parameter to example DAG to make it more clear and explicit
  
  root
  authored and
  root
  committed
  a1394dc
- fixed non-unique task_ids and operator variable names in S3 to Sql example DAG
  
  root
  authored and
  root
  committed
  02c3252
Commits on Jan 18, 2023
- Merge branch 'apache:main' into feat_s3_to_sql_transfer
  maggesssss
  authored
  172e6e9
- save WIP
  
  root
  authored and
  root
  committed
  f147113
- Merge branch 'backup' into feat_s3_to_sql_transfer
  
  root
  authored and
  root
  committed
  74a63ec
- Added parser
  maggesssss
  committed
  078ec76
- WIP: migrated example DAG to system tests
  maggesssss
  committed
  d13cd79
- removed example DAG (migrated to system tests)
  maggesssss
  committed
  e313bb7
- Update airflow/providers/amazon/aws/transfers/s3_to_sql.py
  
  maggesssss
  and
  vincbeck
  authored
  f6893a1
Commits on Jan 19, 2023
- Update airflow/providers/amazon/aws/transfers/s3_to_sql.py
  
  maggesssss
  and
  o-nikolas
  authored
  197dd54
- changed internal method _get_hook
  maggesssss
  committed
  49c921a
- removed unused imports
  maggesssss
  committed
  28f0cd5
- changed conn type from s3 to aws
  maggesssss
  committed
  feab443
- added sql_default conn_id
  maggesssss
  committed
  24cc4d5
- added seek(0) to tempfile after it was
  maggesssss
  committed
  07af1b7
- converted sample_data to raw
  maggesssss
  committed
  663903c
- added test for bad hook
  maggesssss
  committed
  dd5789f
Commits on Jan 21, 2023
- Skip DockerOperator task when it returns a provided exit code (#28996)
  
  hussein-awala
  authored and
  maggesssss
  committed
  8d928ba
- Clarify that we are using lowest supported version for static checks (#29024)
  
  potiuk
  authored and
  maggesssss
  committed
  0838e38
- Update owner list for release dockerhub image workflow (#29023)
  
  pierrejeambrun
  authored and
  maggesssss
  committed
  5eb58b2
- Update how PythonSensor returns values from python_callable (#28932)
  
  SoxMax
  authored and
  maggesssss
  committed
  5872798
- Update gantt chart UI to display queued state of tasks (#28686)
  
  authored and
  maggesssss
  committed
  232acf0
- Patch MockExecutor in tests instead of "prod" code (#29028)
  
  o-nikolas
  authored and
  maggesssss
  committed
  97ffd63
- Add log for AWS Glue Job Console URL (#28925)
  
  IAL32
  authored and
  maggesssss
  committed
  0f1dc02
- Add norm to triage group (#29029)
  
  jedcunningham
  authored and
  maggesssss
  committed
  36d913e
- Fix `project-name` generation for Providers `test-type` in Breeze (#29030)
  
  josh-fell
  authored and
  maggesssss
  committed
  2f02e06
- Fix params rendering in AzureSynapseHook Python API docs (#29041)
  
  josh-fell
  authored and
  maggesssss
  committed
  7d81008
- Pin ruff to specific version and prevent from "fixing" cli conftest (#29040)
  
  potiuk
  authored and
  maggesssss
  committed
  e1fa038
- Sanitize url_for arguments before they are passed (#29039)
  
  potiuk
  authored and
  maggesssss
  committed
  0a792ac
- Add support for write_on_empty in BaseSQLToGCSOperator (#28959)
  
  vchiapaikeo
  authored and
  maggesssss
  committed
  bd41cbd
- AIP-44 Initialize methods map for Internal API RPC endpoint in the method (#28841)
  
  mhenc
  authored and
  maggesssss
  committed
  8c1293f
- uniformize getting hook through cached property in aws sensors (#29001)
  
  vandonr-amz
  authored and
  maggesssss
  committed
  94f17dd
- Update provide_bucket_name() decorator to handle new conn_type (#28706)
  
  RachitSharma2001
  authored and
  maggesssss
  committed
  0e2acb3
- nice table at the end of system test execs (#29051)
  
  vandonr-amz
  authored and
  maggesssss
  committed
  6976dda
- Move project and license docs down in menu to start with developer-focused docs (#28956)
  
  BasPH
  authored and
  maggesssss
  committed
  7a1fdec
- Annotate and simplify code samples in DAGs doc (#29027)
  
  BasPH
  authored and
  maggesssss
  committed
  84cd030
- Fix kerberos authentication for the REST API. (#29054)
  
  authored and
  maggesssss
  committed
  8ce35a2
- Migrate Helm tests to `pytest` (#29063)
  
  Taragolis
  authored and
  maggesssss
  committed
  d683870
- logging poke info when external dag is not none and task_id and task_ids are none (#28097)
  
  surabathini
  authored and
  maggesssss
  committed
  b5f61a5
- AIP-44: Add CLI command for running standalone Internal API (#28425)
  
  mhenc
  authored and
  maggesssss
  committed
  094a540
- Fix pre-commit warning for exclude in inclusive-language check (#29057)
  
  potiuk
  authored and
  maggesssss
  committed
  b3c5629
- AIP-44 Migrate DagModel.get_paused_dag_ids to Internal API (#28693)
  
  vincbeck
  authored and
  maggesssss
  committed
  143eedf
- Renaming nose compatible methods in flavour of regular pytest naming (#29035)
  
  Taragolis
  authored and
  maggesssss
  committed
  e2a4727
- Capitalize dag to DAG (#29064)
  
  BasPH
  authored and
  maggesssss
  committed
  3c2dba5
- Fix SFTP operator's template fields processing (#29068)
  
  potiuk
  authored and
  maggesssss
  committed
  ce320d2
- fixing import error for dataset (#29007)
  
  bharat99k
  authored and
  maggesssss
  committed
  c1b2d83
- Properly test that audit log secrets are masked for connection objects (#29011)
  
  ephraimbuddy
  authored and
  maggesssss
  committed
  a2090a8
- Minor improvements to serde helpers (#28978)
  
  uranusjr
  authored and
  maggesssss
  committed
  3ef98e7
- Add deferrable mode for Big Query Transfer operator (#27833)
  
  MrGeorgeOwl
  authored and
  maggesssss
  committed
  bbf11d7
- Add deferrable mode to DataFusionStartPipelineOperator (#28690)
  
  VladaZakharova
  authored and
  maggesssss
  committed
  974039c
- Fix minor typo in breeze example command. (#29072)
  
  snjypl
  authored and
  maggesssss
  committed
  c7b2419
- Emit DataDog statsd metrics with metadata tags (#28961)
  
  hussein-awala
  authored and
  maggesssss
  committed
  e5009f9
- listener plugin example added (#27905)
  
  Bowrna
  authored and
  maggesssss
  committed
  8250a1c
- Refactor `TestLocalTaskJob.test_process_sigterm_works_with_retries` (#28990)
  
  Taragolis
  authored and
  maggesssss
  committed
  6a93ff6
- Mark `test_process_sigterm_works_with_retries` quarantined again (#29075)
  
  Taragolis
  authored and
  maggesssss
  committed
  b8fcd20
- FTP operator has logic in __init__ (#29073)
  
  snjypl
  authored and
  maggesssss
  committed
  cb8308c
- Upgrade bitnami/postgresql helm depenency from 10.5.3 to 12.1.9 (#29071)
  
  authored and
  maggesssss
  committed
  2269fd7
- Chart: add doc note about podtemplate images (#29032)
  
  Aakcht
  authored and
  maggesssss
  committed
  3cc6673
- bump (#28602)
  
  DrFaust92
  authored and
  maggesssss
  committed
  10427eb
- Check for run_id url param when linking to graph/gantt views (#29066)
  
  bbovenzi
  authored and
  maggesssss
  committed
  1b40759
- emit dagrun failed duration when timeout (#29076)
  
  hussein-awala
  authored and
  maggesssss
  committed
  ce46baa
- Migrate DagFileProcessorManager.clear_nonexistent_import_errors to Internal API (#28976)
  
  snjypl
  authored and
  maggesssss
  committed
  d870a9b
- save WIP
  
  root
  authored and
  maggesssss
  committed
  dcf7e87
- removed type-hint from property db_hook.
  maggesssss
  committed
  db11996
- fixed import sorting
  maggesssss
  committed
  db1cb02
- added flush() to
  maggesssss
  committed
  1d902ff
- removed unneccessary
  maggesssss
  committed
  42a9b76
- Merge branch 'apache:main' into feat_s3_to_sql_transfer
  maggesssss
  authored
  aa5e2da

Commits

Commits on Dec 25, 2022

Commits on Jan 15, 2023

Commits on Jan 18, 2023

Commits on Jan 19, 2023

Commits on Jan 21, 2023