polars.LazyFrame.sink_parquet

LazyFrame.sink_parquet(
    path: str | Path | IO[bytes] | PartitionBy,
    *,
    compression: ParquetCompression = 'zstd',
    compression_level: int | None = None,
    statistics: bool | str | dict[str,
    bool] = True,
    row_group_size: int | None = None,
    data_page_size: int | None = None,
    maintain_order: bool = True,
    storage_options: StorageOptionsDict | None = None,
    credential_provider: CredentialProviderFunction | Literal['auto'] | None = 'auto',
    retries: int | None = None,
    sync_on_close: SyncOnCloseMethod | None = None,
    metadata: ParquetMetadata | None = None,
    arrow_schema: ArrowSchemaExportable | None = None,
    mkdir: bool = False,
    lazy: bool = False,
    engine: EngineType = 'auto',
    optimizations: QueryOptFlags = (,
), sinked_paths_callback: SinkedPathsCallback | None = None, ) → LazyFrame | None

Evaluate the query in streaming mode and write to a Parquet file.

This allows streaming results that are larger than RAM to be written to disk.

engine:StreamingDistributed

Parameters:

path

File path to which the file should be written.

compression{‘lz4’, ‘uncompressed’, ‘snappy’, ‘gzip’, ‘brotli’, ‘zstd’}

Choose “zstd” for good compression performance. Choose “lz4” for fast compression/decompression. Choose “snappy” for more backwards compatibility guarantees when you deal with older parquet readers.

compression_level

The level of compression to use. Higher compression means smaller files on disk.

• “gzip” : min-level: 0, max-level: 9, default: 6.
• “brotli” : min-level: 0, max-level: 11, default: 1.
• “zstd” : min-level: 1, max-level: 22, default: 3.

statistics

Write statistics to the parquet headers. This is the default behavior.

Possible values:

• True: enable default set of statistics (default). Some statistics may be disabled.
• False: disable all statistics
• “full”: calculate and write all available statistics.

• { "statistic-key": True / False, ... }. Available keys:

  • “min”: column minimum value (default: True)
  • “max”: column maximum value (default: True)
  • “distinct_count”: number of unique column values (default: False)
  • “null_count”: number of null values in column (default: True)

row_group_size

Size of the row groups in number of rows. If None (default), the chunks of the DataFrame are used. Writing in smaller chunks may reduce memory pressure and improve writing speeds.

data_page_size

Size limit of individual data pages. If not set defaults to 1024 * 1024 bytes

maintain_order

Maintain the order in which data is processed. Setting this to False will be slightly faster.

Warning

This functionality is considered unstable. It may be changed at any point without it being considered a breaking change.

storage_options

Options that indicate how to connect to a cloud provider.

The cloud providers currently supported are AWS, GCP, and Azure. See supported keys here:

• aws
• gcp
• azure
• Hugging Face (hf://): Accepts an API key under the token parameter: {'token': '...'}, or by setting the HF_TOKEN environment variable.

If storage_options is not provided, Polars will try to infer the information from environment variables.

credential_provider

Provide a function that can be called to provide cloud storage credentials. The function is expected to return a dictionary of credential keys along with an optional credential expiry time.

Warning

This functionality is considered unstable. It may be changed at any point without it being considered a breaking change.

retries

Number of retries if accessing a cloud instance fails.

Deprecated since version 1.37.1: Pass {“max_retries”: n} via storage_options instead.

sync_on_close: { None, ‘data’, ‘all’ }

Sync to disk when before closing a file.

• None does not sync.
• data syncs the file contents.
• all syncs the file contents and metadata.

Warning

This functionality is considered unstable. It may be changed at any point without it being considered a breaking change.

metadata

A dictionary or callback to add key-values to the file-level Parquet metadata.

Warning

This functionality is considered experimental. It may be removed or changed at any point without it being considered a breaking change.

arrow_schema

Provide a custom arrow schema to write to the file. This allows setting custom schema and field-level metadata. Names and dtypes must match.

Warning

This functionality is considered unstable. It may be changed at any point without it being considered a breaking change.

mkdir: bool

Recursively create all the directories in the path.

Warning

This functionality is considered unstable. It may be changed at any point without it being considered a breaking change.

lazy: bool

Wait to start execution until collect is called.

Warning

This functionality is considered unstable. It may be changed at any point without it being considered a breaking change.

engine

Select the engine used to process the query (default "auto"). A Engine instance may also be passed. Supported engine names are:

• "auto": use the engine set by Config.set_engine_affinity or the POLARS_ENGINE_AFFINITY environment variable, falling back to "streaming" if unset.
• "in-memory": use the in-memory engine before writing, this is the default engine.
• "streaming": use the streaming engine, which processes queries in batches, reducing memory pressure and often outperforming the in-memory engine. This will soon become the default engine of Polars.
• "gpu": use the CUDA GPU engine (requires an Nvidia GPU and cudf-polars). Pass a GPUEngine object for fine-grained control.

If the selected engine cannot run the query, Polars falls back to the streaming engine.

optimizations

The optimization passes done during query optimization.

This has no effect if lazy is set to True.

Warning

This functionality is considered unstable. It may be changed at any point without it being considered a breaking change.

sinked_paths_callback

Callable that will be called with information on sinked paths.

Warning

This functionality is considered unstable. It may be changed at any point without it being considered a breaking change.

Returns:

DataFrame

See also

PartitionBy

Examples

>>> lf = pl.scan_csv("/path/to/my_larger_than_ram_file.csv")  
>>> lf.sink_parquet("out.parquet")  

Sink to a BytesIO object.

>>> import io
>>> buf = io.BytesIO()  
>>> pl.LazyFrame({"x": [1, 2, 1]}).sink_parquet(buf)  

Split into a hive-partitioning style partition:

>>> pl.LazyFrame({"x": [1, 2, 1], "y": [3, 4, 5]}).sink_parquet(
...     pl.PartitionBy("./out/", key="x"),
...     mkdir=True
... )