Airflow Executor

There are two types of executor - those that run tasks locally (inside the scheduler process), and those that run their tasks remotely (usually via a pool of workers)

Local Executor

• Sequence Executor: Default on with local install, running with SQLite which not accept multi process so only run in sequence not parallel.
• Local Executor: BaseExecutor has the option to receive a parallelism parameter to limit the number of process spawned, when this parameter is 0 the number of processes that LocalExecutor can spawn is unlimited. Can run parallel but only on 1 machine so not suitable for scale
•  SequentialExecutor could be thought of as a LocalExecutor with limited parallelism of just 1 worker, i.e. self.parallelism = 1

Remote Executor

• CeleryExecutor: CeleryExecutor is one of the ways you can scale out the number of workers. For this to work, you need to setup a Celery backend(Message Broker) (RabbitMQ, Redis, …). Celery là một hệ thống phân tán với nhiều worker nên có thể dễ dàng scale. Các task sẽ được gửi từ scheduler qua một message queue như rabbitmq, redis sau đó các message queue này sẽ phân chia task cho các worker.
• KubernetesExecutor: In contrast to CeleryExecutor, KubernetesExecutor does not require additional components such as Redis, but does require access to Kubernetes cluster.

Airflow SQL Sensor