Dump configurations in airflow scheduler logs based on the config it reads #22141

Activity

1. arvindeybram

   added

   kind:featureFeature RequestsFeature Requests

   on Mar 10, 2022
2. potiuk commented on Mar 10, 2022

   

   potiuk

   on Mar 10, 2022

   Member

   More actions

   I thhini it would be super helpful. I've seen many people who had problems with understanding what their configuraiton is.
   The airflow config list is not enough - the problem is that many people run airflow using systemd or other solutions and it would be great to know what ACTUALLY was used by the process that started.

   In many cases people do not realise that variables they have set in .bashrc are not set in systemd for example, and there might be plenty other reasons why running airflow config list provides different results than what the running airflow process sees.

   Big +1 on this one.

   ❤️React with ❤️1Reacted by Arvind Abraham

   🚀React with 🚀1Reacted by Ragil18
3. uranusjr commented on Mar 13, 2022

   

   uranusjr

   on Mar 13, 2022

   Member

   More actions

   Just dump everything to the log file when the scheduler launches, then?

   ❤️React with ❤️1Reacted by Arvind Abraham
4. uranusjr

   added

   good first issue

   on Mar 13, 2022
5. potiuk commented on Mar 13, 2022

   

   potiuk

   on Mar 13, 2022

   Last edited by potiuk

   Member

   More actions

   Yep. we even have some functions for this (used in CLI) - It also masks sensitive values, so this is important to use it.
6. tirkarthi commented on Mar 28, 2022

   

   tirkarthi

   on Mar 28, 2022

   Contributor

   More actions

   Hi, I would like to work on this feature and this is my first patch. I looked into airflow conf list that has show_config to print config. It doesn't seem to mask sensitive values. I wrote another function that initializes AirflowConfigParser and uses as_dict that masks sensitive values to read the masked config again using read_dict. I am not sure if the approach is correct. Patch with test as below

   airflow/airflow/cli/commands/config_command.py

   Lines 23 to 35 in ae6570f

   	from airflow.configuration import conf
   	from airflow.utils.cli import should_use_colors
   	from airflow.utils.code_utils import get_terminal_formatter
   	def show_config(args):
   	"""Show current application configuration"""
   	with io.StringIO() as output:
   	conf.write(output)
   	code = output.getvalue()
   	if should_use_colors(args):
   	code = pygments.highlight(code=code, formatter=get_terminal_formatter(), lexer=IniLexer())
   	print(code)

   Patch :

   diff --git a/airflow/cli/commands/scheduler_command.py b/airflow/cli/commands/scheduler_command.py
   index bc6e983ee..4fae84f43 100644
   --- a/airflow/cli/commands/scheduler_command.py
   +++ b/airflow/cli/commands/scheduler_command.py
   @@ -16,6 +16,7 @@
    # under the License.
    
    """Scheduler command"""
   +import io
    import signal
    from multiprocessing import Process
    from typing import Optional
   @@ -24,6 +25,7 @@ import daemon
    from daemon.pidfile import TimeoutPIDLockFile
    
    from airflow import settings
   +from airflow.configuration import AirflowConfigParser, conf
    from airflow.jobs.scheduler_job import SchedulerJob
    from airflow.utils import cli as cli_utils
    from airflow.utils.cli import process_subdir, setup_locations, setup_logging, sigint_handler, sigquit_handler
   @@ -38,6 +40,16 @@ def _create_scheduler_job(args):
        return job
    
    
   +def _get_masked_config():
   +    parser = AirflowConfigParser(strict=False, interpolation=None)
   +    config_dict = conf.as_dict()
   +    parser.read_dict(config_dict)
   +
   +    with io.StringIO() as output:
   +        parser.write(output)
   +        return output.getvalue()
   +
   +
    def _run_scheduler_job(args):
        skip_serve_logs = args.skip_serve_logs
        job = _create_scheduler_job(args)
   @@ -53,6 +65,7 @@ def _run_scheduler_job(args):
    def scheduler(args):
        """Starts Airflow Scheduler"""
        print(settings.HEADER)
   +    print(_get_masked_config())
    
        if args.daemon:
            pid, stdout, stderr, log_file = setup_locations(
   diff --git a/tests/cli/commands/test_scheduler_command.py b/tests/cli/commands/test_scheduler_command.py
   index 00b851f55..8946e6c52 100644
   --- a/tests/cli/commands/test_scheduler_command.py
   +++ b/tests/cli/commands/test_scheduler_command.py
   @@ -15,6 +15,9 @@
    # KIND, either express or implied.  See the License for the
    # specific language governing permissions and limitations
    # under the License.
   +import contextlib
   +import io
   +import os
    import unittest
    from unittest import mock
    
   @@ -87,3 +90,14 @@ class TestSchedulerCommand(unittest.TestCase):
                    scheduler_command.scheduler(args)
                finally:
                    mock_process().terminate.assert_called()
   +
   +    @mock.patch.dict(os.environ, {}, clear=True)
   +    @mock.patch("airflow.cli.commands.scheduler_command.SchedulerJob")
   +    @mock.patch("airflow.cli.commands.scheduler_command.Process")
   +    def test_scheduler_print_config(self, mock_process, mock_scheduler_job):
   +        args = self.parser.parse_args(['scheduler'])
   +        with conf_vars({("core", "sql_alchemy_conn_cmd"): 'echo hello'}):
   +            with contextlib.redirect_stdout(io.StringIO()) as temp_stdout:
   +                scheduler_command.scheduler(args)
   +                output = temp_stdout.getvalue()
   +                assert "sql_alchemy_conn = < hidden >" in output
   
7. potiuk commented on Mar 29, 2022

   

   potiuk

   on Mar 29, 2022

   Member

   More actions

   Just open a PR with it it (referring to this one - it will be easier to dicuss it).
8. tirkarthi commented on Mar 29, 2022

   

   tirkarthi

   on Mar 29, 2022

   Contributor

   More actions

   Thanks, created #22588
9. potiuk

   closed this as completedin #22588on Apr 4, 2022