polars.LazyFrame.sink_iceberg

LazyFrame.sink_iceberg(
    target: str | pyiceberg.table.Table,
    *,
    mode: Literal['append',
    'overwrite'],
    schema_mode: Literal['merge',
    'overwrite'] | None = None,
    snapshot_properties: dict[str,
    str] | None = None,
    catalog: pyiceberg.catalog.Catalog | polars.io.iceberg.IcebergCatalogConfig | None = None,
    storage_options: StorageOptionsDict | None = None,
    engine: EngineType = 'auto',
) → DataFrame

Sink a LazyFrame to an Iceberg table.

Warning

This functionality is currently considered unstable. It may be changed at any point without it being considered a breaking change.

Parameters:

target

A PyIceberg Table object, or a ‘namespace.table_name’ identifier string.

mode{‘append’, ‘overwrite’}

How to handle existing data.

• If ‘append’, will add new data.
• If ‘overwrite’, will replace table with new data.

schema_mode{‘merge’, ‘overwrite’}

How to handle differences between the incoming and table schemas.

Warning

This functionality is considered unstable. It may be changed at any point without it being considered a breaking change.

• If ‘merge’, evolve the table schema with incoming fields and compatible type promotions.
• If ‘overwrite’, replace the table schema with the incoming schema. This requires mode='overwrite'.
• If None, require the schemas to match.

snapshot_properties

Custom properties to add to the Iceberg snapshot summary.

catalog

PyIceberg catalog to load the table from if the provided target was a table identifier.

storage_options

Extra options for the storage backends supported by pyiceberg. For cloud storages, this may include configurations for authentication etc.

More info is available here.

engine

Engine used to produce rows for the local pyiceberg writer.

Returns:

DataFrame

Contains the new metadata path.