PySpark#

The daft.pyspark module provides a way to create a PySpark session that can be run locally or backed by a ray cluster. This serves as a way to run the Daft query engine with a Spark compatible API.

For the full PySpark SQL API documentation, see the official PySpark documentation.

Installing Daft with Spark Connect support#

Daft supports Spark Connect through the optional spark dependency.

1

pip install -U "daft[spark]"

Example#

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

from daft.pyspark import SparkSession
from pyspark.sql.functions import col

# Create a local spark session
spark = SparkSession.builder.local().getOrCreate()

# Alternatively, connect to a Ray cluster
# You can use `ray get-head-ip <cluster_config.yaml>` to get the head ip!
spark = SparkSession.builder.remote("ray://<HEAD_IP>:6379").getOrCreate()

# Use spark as you would with the native spark library, but with a daft backend!
spark.createDataFrame([{"hello": "world"}]).select(col("hello")).show()

# Stop the Spark session
spark.stop()

Object Store Configuration (S3, Azure, GCS)#

For reading and writing remote files such as s3 buckets, you will need to pass in the credentials using daft's io config.

Here's a list of supported config values that you can set via spark.conf.set("<key>", "<value>")

S3 Configuration Options#

Azure Storage Configuration Options#

GCS Configuration Options#

HTTP Configuration Options#

Notable Differences#

A few methods do have some notable differences compared to PySpark.

explain#

The df.explain() method will output non-Spark compatible explain and instead will be the same as calling explain on a DataFrame.

show#

Similarly, df.show() will output Daft's dataframe output instead of native Spark's.