polars.LazyFrame.unique

LazyFrame.unique(
    subset: IntoExpr | Collection[IntoExpr] | None = None,
    *,
    keep: UniqueKeepStrategy = 'any',
    maintain_order: bool = False,
) → LazyFrame

Drop duplicate rows from this LazyFrame.

engine:In MemoryStreamingDistributed

Parameters:

subset

Column name(s), selector(s), or expressions to consider when identifying duplicate rows. If set to None (default), all columns are considered.

keep{‘first’, ‘last’, ‘any’, ‘none’}

Which of the duplicate rows to keep.

• ‘any’: Does not give any guarantee of which row is kept.

  This allows more optimizations.

• ‘none’: Don’t keep duplicate rows.
• ‘first’: Keep the first unique row.
• ‘last’: Keep the last unique row.

maintain_order

Keep the same order as the original DataFrame. This is more expensive to compute. Settings this to True blocks the possibility to run on the streaming engine.

Returns:

LazyFrame

LazyFrame with unique rows.

Notes

If you’re coming from Pandas, this is similar to pandas.DataFrame.drop_duplicates.

Examples

>>> lf = pl.LazyFrame(
...     {
...         "foo": [1, 2, 3, 1, 1],
...         "bar": ["a", "a", "a", "x", "x"],
...         "ham": ["b", "b", "b", "y", "y"],
...     }
... )

By default, all columns are considered when determining which rows are unique:

>>> lf.unique(maintain_order=True).collect()
shape: (4, 3)
┌─────┬─────┬─────┐
│ foo ┆ bar ┆ ham │
│ --- ┆ --- ┆ --- │
│ i64 ┆ str ┆ str │
╞═════╪═════╪═════╡
│ 1   ┆ a   ┆ b   │
│ 2   ┆ a   ┆ b   │
│ 3   ┆ a   ┆ b   │
│ 1   ┆ x   ┆ y   │
└─────┴─────┴─────┘

We can also consider only a subset of columns when determining uniqueness, controlling which row we keep when duplicates are found:

>>> lf.unique(subset="foo", keep="first", maintain_order=True).collect()
shape: (3, 3)
┌─────┬─────┬─────┐
│ foo ┆ bar ┆ ham │
│ --- ┆ --- ┆ --- │
│ i64 ┆ str ┆ str │
╞═════╪═════╪═════╡
│ 1   ┆ a   ┆ b   │
│ 2   ┆ a   ┆ b   │
│ 3   ┆ a   ┆ b   │
└─────┴─────┴─────┘
>>> lf.unique(subset="foo", keep="last", maintain_order=True).collect()
shape: (3, 3)
┌─────┬─────┬─────┐
│ foo ┆ bar ┆ ham │
│ --- ┆ --- ┆ --- │
│ i64 ┆ str ┆ str │
╞═════╪═════╪═════╡
│ 2   ┆ a   ┆ b   │
│ 3   ┆ a   ┆ b   │
│ 1   ┆ x   ┆ y   │
└─────┴─────┴─────┘
>>> lf.unique(subset="foo", keep="none", maintain_order=True).collect()
shape: (2, 3)
┌─────┬─────┬─────┐
│ foo ┆ bar ┆ ham │
│ --- ┆ --- ┆ --- │
│ i64 ┆ str ┆ str │
╞═════╪═════╪═════╡
│ 2   ┆ a   ┆ b   │
│ 3   ┆ a   ┆ b   │
└─────┴─────┴─────┘

Selectors can be used to define the “subset” parameter:

>>> import polars.selectors as cs
>>> lf.unique(subset=cs.string(), maintain_order=True).collect()
shape: (2, 3)
┌─────┬─────┬─────┐
│ foo ┆ bar ┆ ham │
│ --- ┆ --- ┆ --- │
│ i64 ┆ str ┆ str │
╞═════╪═════╪═════╡
│ 1   ┆ a   ┆ b   │
│ 1   ┆ x   ┆ y   │
└─────┴─────┴─────┘

We can also use an arbitrary expression in the “subset” parameter; in this example we use the part of the label in front of “:” to determine uniqueness:

>>> lf = pl.LazyFrame(
...     {
...         "label": ["xx:1", "xx:2", "yy:3", "yy:4"],
...         "value": [100, 200, 300, 400],
...     }
... )
>>> lf.unique(
...     subset=pl.col("label").str.extract(r"^(\w+):"),
...     maintain_order=True,
...     keep="first",
... ).collect()
shape: (2, 2)
┌───────┬───────┐
│ label ┆ value │
│ ---   ┆ ---   │
│ str   ┆ i64   │
╞═══════╪═══════╡
│ xx:1  ┆ 100   │
│ yy:3  ┆ 300   │
└───────┴───────┘