Databricks Job Runner

Trigger Databricks jobs to run.

Run Job

Trigger a Databricks job to run with the parameters specified.

Variables

Task Parameters

databricks_connection required

Credentials to use to access Databricks.

job_id required

Id of the Databricks job to run.

task_type required

Type of task that the Databricks job will run.

notebook_params

A map from keys to values. Accessible through the dbutils.widgets.get. Example {"name": "john doe", "age": "35"}

python_params

Array of strings. Passed to Python file as command-line parameters. Example ["john doe", "35"]

jar_params

Array of strings. Used to invoke the main function of the main class. Example: ["john doe", "35"]

spark_submit_params

Array of strings. Passed to spark-submit script as command-line parameters. Example ["--class", "org.apache.spark.examples.SparkPi"]

wait_till_done required

Enable to force the Task to wait until the DataBricks job has successfully completed or generated an error.

polling_interval

Number of seconds the Task will wait between each check for job completion.

You can't use this until you register with us!