polars.DataFrame.write_ipc

DataFrame.write_ipc(
    file: str | Path | IO[bytes] | None,
    *,
    compression: IpcCompression = 'uncompressed',
    compat_level: CompatLevel | None = None,
    record_batch_size: int | None = None,
    storage_options: StorageOptionsDict | None = None,
    credential_provider: CredentialProviderFunction | Literal['auto'] | None = 'auto',
    retries: int | None = None,
) → BytesIO | None

Write to Arrow IPC binary stream or Feather file.

See “File or Random Access format” in https://arrow.apache.org/docs/python/ipc.html.

Changed in version 1.1: The future parameter was renamed compat_level.

Parameters:

file

Path or writable file-like object to which the IPC data will be written. If set to None, the output is returned as a BytesIO object.

compression{‘uncompressed’, ‘lz4’, ‘zstd’}

Compression method. Defaults to “uncompressed”.

compat_level

Use a specific compatibility level when exporting Polars’ internal data structures.

record_batch_size

Size of the record batches in number of rows.

Warning

This functionality is considered unstable. It may be changed at any point without it being considered a breaking change.

storage_options

Options that indicate how to connect to a cloud provider.

The cloud providers currently supported are AWS, GCP, and Azure. See supported keys here:

• aws
• gcp
• azure
• Hugging Face (hf://): Accepts an API key under the token parameter: {'token': '...'}, or by setting the HF_TOKEN environment variable.

If storage_options is not provided, Polars will try to infer the information from environment variables.

credential_provider

Provide a function that can be called to provide cloud storage credentials. The function is expected to return a dictionary of credential keys along with an optional credential expiry time.

Warning

This functionality is considered unstable. It may be changed at any point without it being considered a breaking change.

retries

Number of retries if accessing a cloud instance fails.

Deprecated since version 1.37.1: Pass {“max_retries”: n} via storage_options instead.

Examples

>>> import pathlib
>>>
>>> df = pl.DataFrame(
...     {
...         "foo": [1, 2, 3, 4, 5],
...         "bar": [6, 7, 8, 9, 10],
...         "ham": ["a", "b", "c", "d", "e"],
...     }
... )
>>> path: pathlib.Path = dirpath / "new_file.arrow"
>>> df.write_ipc(path)

Write to a BytesIO object by passing file=None. The returned buffer’s position is at the end of the written data, so call seek(0) before reading it back.

>>> buf = df.write_ipc(file=None)
>>> buf.seek(0)
0
>>> pl.read_ipc(buf).equals(df)
True