Would you like a concrete example of replacing large XComs with an S3 + reference pattern?

What Are XComs? XComs (short for "cross-communications") are Airflow’s built-in mechanism for tasks to exchange small pieces of data. One task pushes a value (key + value + task_id + dag_id), and another task pulls it.

# Pushing (implicitly via return) def task_a(): return "user_id": 123 def task_b(**context): user_id = context["task_instance"].xcom_pull(task_ids="task_a", key="return_value") The Good: Where XComs Shine | Aspect | Rating | Notes | |--------|--------|-------| | Simplicity | ⭐⭐⭐⭐ | Zero extra setup – return values are automatically stored. | | DAG readability | ⭐⭐⭐⭐ | Data flow is explicit when using .xcom_pull() . | | Lightweight coordination | ⭐⭐⭐⭐⭐ | Perfect for passing IDs, flags, dates, small configs. | | Backend support | ⭐⭐⭐ | Works with any Airflow DB backend (Postgres, MySQL, SQLite, etc.). |