# Airflow Dag Patterns

> Build production Apache Airflow DAGs with best practices for operators, sensors, testing, and deployment. Use when creating data pipelines, orchestrating workflows, or scheduling batch jobs.

Source: https://skillsagentes.com/skills/wshobson/agents/airflow-dag-patterns
Repository: https://github.com/wshobson/agents
Author: wshobson
License: MIT
Updated: hace 2 meses
Context cost: 48 tok installed, 784 tok once triggered, 3.6k tok with every bundled file
Bundle: 2 files, 14 KB
Permissions requested: none declared

## Install

```bash
npx -y skills add wshobson/agents --skill airflow-dag-patterns --agent claude-code
```

## What it does

- Proporciona patrones para diseñar DAGs de Apache Airflow con operadores, sensores y dependencias de tareas
- Da ejemplos de código listos como plantilla de DAG con default_args, retries y TaskFlow API
- Incluye buenas y malas prácticas para producción (idempotencia, timeouts, sensores en modo reschedule)

## Use it when

- Crear orquestación de pipelines de datos con Airflow
- Diseñar estructuras de DAG y dependencias
- Implementar operadores y sensores personalizados
- Testear DAGs de Airflow localmente o depurar ejecuciones fallidas

## What triggers it

- "Ayúdame a crear un DAG de Airflow para un pipeline ETL diario"
- "¿Cómo estructuro dependencias fan-out en Airflow?"
- "Necesito testear localmente mis DAGs de Airflow"

## Before you install

- Requiere un entorno de Apache Airflow configurado (por ejemplo con acceso a `dags/`).

## Files

- SKILL.md — 3 KB
- references/details.md — 11 KB

## SKILL.md

Reproduced verbatim from wshobson/agents under MIT. This section is the upstream document and is in English.

# Apache Airflow DAG Patterns

Production-ready patterns for Apache Airflow including DAG design, operators, sensors, testing, and deployment strategies.

## When to Use This Skill

- Creating data pipeline orchestration with Airflow
- Designing DAG structures and dependencies
- Implementing custom operators and sensors
- Testing Airflow DAGs locally
- Setting up Airflow in production
- Debugging failed DAG runs

## Core Concepts

### 1. DAG Design Principles

| Principle       | Description                         |
| --------------- | ----------------------------------- |
| **Idempotent**  | Running twice produces same result  |
| **Atomic**      | Tasks succeed or fail completely    |
| **Incremental** | Process only new/changed data       |
| **Observable**  | Logs, metrics, alerts at every step |

### 2. Task Dependencies

```python
# Linear
task1 >> task2 >> task3

# Fan-out
task1 >> [task2, task3, task4]

# Fan-in
[task1, task2, task3] >> task4

# Complex
task1 >> task2 >> task4
task1 >> task3 >> task4
```

## Quick Start

```python
# dags/example_dag.py
from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.operators.empty import EmptyOperator

default_args = {
    'owner': 'data-team',
    'depends_on_past': False,
    'email_on_failure': True,
    'email_on_retry': False,
    'retries': 3,
    'retry_delay': timedelta(minutes=5),
    'retry_exponential_backoff': True,
    'max_retry_delay': timedelta(hours=1),
}

with DAG(
    dag_id='example_etl',
    default_args=default_args,
    description='Example ETL pipeline',
    schedule='0 6 * * *',  # Daily at 6 AM
    start_date=datetime(2024, 1, 1),
    catchup=False,
    tags=['etl', 'example'],
    max_active_runs=1,
) as dag:

    start = EmptyOperator(task_id='start')

    def extract_data(**context):
        execution_date = context['ds']
        # Extract logic here
        return {'records': 1000}

    extract = PythonOperator(
        task_id='extract',
        python_callable=extract_data,
    )

    end = EmptyOperator(task_id='end')

    start >> extract >> end
```

## Detailed patterns and worked examples

Detailed pattern documentation lives in `references/details.md`. Read that file when the navigation tier above is insufficient.

## Best Practices

### Do's

- **Use TaskFlow API** - Cleaner code, automatic XCom
- **Set timeouts** - Prevent zombie tasks
- **Use `mode='reschedule'`** - For sensors, free up workers
- **Test DAGs** - Unit tests and integration tests
- **Idempotent tasks** - Safe to retry

### Don'ts

- **Don't use `depends_on_past=True`** - Creates bottlenecks
- **Don't hardcode dates** - Use `{{ ds }}` macros
- **Don't use global state** - Tasks should be stateless
- **Don't skip catchup blindly** - Understand implications
- **Don't put heavy logic in DAG file** - Import from modules

---

Skills Agentes — https://skillsagentes.com/skills/wshobson/agents/airflow-dag-patterns