CONCEPTS

What is shuffle?

Why data moves

When computation needs to reorganize data by a new key, partition, or parallelism level, data may move across processes or machines. That redistribution can add serialization, network, disk, and downstream merge costs.

Understanding shuffle is a useful entry point for reading Spark stages, distributed joins, and many data-processing performance problems.