Blaze Support

Build Blaze

Blaze supports Celeborn as remote shuffle service. Below introduction is used to enable this feature.

First refer to Build From Source or Build With Docker to build Blaze.

Blaze Configuration

Currently, to use Blaze following configurations are required in spark-defaults.conf.

spark.shuffle.manager org.apache.spark.sql.execution.blaze.shuffle.celeborn.BlazeCelebornShuffleManager

# celeborn master
spark.celeborn.master.endpoints clb-master:9097

spark.celeborn.client.spark.shuffle.writer hash
spark.serializer org.apache.spark.serializer.KryoSerializer
spark.sql.adaptive.localShuffleReader.enabled false

Availability

Celeborn Version	Available in Blaze?
< 0.5.0	No
>= 0.5.0	Yes