Add --wait option to `databricks runs submit` CLI command

[smurching]

In addition to submitting a one-time job run through the databricks runs submit CLI, it would be great to additionally support an experimental --wait option to await its success (polling job status every ~30s). This would greatly simplify integration testing use cases, allowing us to generalize plugins like https://github.com/databricks/run-notebook across CI/CD providers.

In the future, we might consider:

• Extending this parameter to the databricks jobs run-now API to allow awaiting the result of a run of an existing job as well
• Making the polling interval configurable, with some fixed minimum to avoid DOSing the jobs /runs/get REST API endpoint used to poll for run status.

The --wait option suggestion is based off of a similar API change proposed in this PR. This functionality is also similar to that provided by dbx launch --trace, but has the added flexibility of being able to run jobs that are not declared in DBX's YAML format & without requiring the caller to install dbx and invoke dbx configure

[smurching]

I just tried implementing this via shell scripts around existing CLI commands - it was pretty tricky and also would be kind of annoying to test/maintain as a customer (mostly just testing the branching logic around job status & making sure to log information to facilitate debugging). See https://gist.github.com/smurching/aff4e7ed25f8ff2eb9d920cd837f48f8 for details. It could be an ok workaround in the near term though if we think adding a --wait may be tricky

[pietern]

Added in #487.

[edmondop]

How about adding it also to the databricks job run-now?