![]() Weight_rule ( str) – weighting method used for the effective total This allows the executor to trigger higher priority tasks before Priority_weight ( int) – priority weight of this task against other task. NotĪll executors implement queue management, the CeleryExecutorĭag ( ) – a reference to the dag the task is attached to (if any) Queue ( str) – which queue to target when running this job. Is forced to True wherever wait_for_downstream is used. This is useful if theĭifferent instances of a task X alter the same asset, and this asset Of task X to finish successfully before it runs. X will wait for tasks immediately downstream of the previous instance Wait_for_downstream ( bool) – when set to true, an instance of task The task instance for the start_date is allowed to run. ![]() Sequentially while relying on the previous task’s schedule to Read theįAQ entry about start_date for more information.Įnd_date ( datetime.datetime) – if specified, the scheduler won’t go beyond this dateĭepends_on_past ( bool) – when set to true, task instances will run We advise against usingĭynamic start_date and recommend using fixed ones. If you are looking to delayĪ task, for example running a daily task at 2AM, look into the Start_date are offset in a way that their execution_date don’t line To note that different tasks’ dependencies Note that Airflow simply looks at the latestĮxecution_date and adds the schedule_interval to determine Some day at 00:00:00, hourly jobs have their start_date at 00:00 The execution_date for the first task instance. Start_date ( datetime.datetime) – The start_date for the task, determines Max_retry_delay ( datetime.timedelta) – maximum delay interval between retries Retries by using exponential backoff algorithm on retry delay (delay ![]() Retry_exponential_backoff ( bool) – allow progressive longer waits between Retry_delay ( datetime.timedelta) – delay between retries Retries ( int) – the number of retries that should be performed before Owner ( str) – the owner of the task, using the unix username is recommended Task_id ( str) – a unique, meaningful id for the task Task dependencies shouldīe set by using the set_upstream and/or set_downstream methods. Which ultimately becomes a node in DAG objects. Instantiating aĬlass derived from this one results in the creation of a task object, This class is abstract and shouldn’t be instantiated. Operators (tasks) target specific operations, running specific scripts, Moves data from Hive to MySQL (Hive2MySqlOperator). Waits for a partition to land in Hive (HiveSensorOperator), or one that Operator that runs a Pig job (PigOperator), a sensor operator that Operators derived from this class should perform or trigger certain tasks The constructor as well as the ‘execute’ method. ![]() To derive this class, you are expected to override Since operators create objects thatīecome nodes in the dag, BaseOperator contains many recursive methods forĭag crawling behavior. BaseOperator ( task_id, owner=('operators', 'DEFAULT_OWNER'), email=None, email_on_retry=True, email_on_failure=True, retries=0, retry_delay=timedelta(seconds=300), retry_exponential_backoff=False, max_retry_delay=None, start_date=None, end_date=None, schedule_interval=None, depends_on_past=False, wait_for_downstream=False, dag=None, params=None, default_args=None, priority_weight=1, weight_rule=WeightRule.DOWNSTREAM, queue=('celery', 'default_queue'), pool=None, sla=None, execution_timeout=None, on_failure_callback=None, on_success_callback=None, on_retry_callback=None, trigger_rule=TriggerRule.ALL_SUCCESS, resources=None, run_as_user=None, task_concurrency=None, executor_config=None, inlets=None, outlets=None, *args, **kwargs ) ¶īases: _mixin.LoggingMixinĪbstract base class for all operators. Package Contents ¶ class airflow.operators.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |