Skip to content

Support Filter pushdown in Spark Structured Streaming #55680

@jalpan-randeri

Description

@jalpan-randeri

Currently, Spark Structured streaming via the DSv2 api does not pushdown predicate. This results in more data being scan and filtered out at engine layer. This results in excessive I/O, driver bottlenecks and increased latency.

Relevant Iceberg issue - apache/iceberg#15692

PR on Spark side - #55679

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions