› › ›

Limit token count token filter

Limits the number of output tokens. The limit filter is commonly used to limit the size of document field values based on token count.

By default, the limit filter keeps only the first token in a stream. For example, the filter can change the token stream [ one, two, three ] to [ one ].

This filter uses Lucene’s LimitTokenCountFilter.

 If you want to limit the size of field values based on
_character length_, use the <<ignore-above,`ignore_above`>> mapping parameter.

Configurable parameters

max_token_count: (Optional, integer) Maximum number of tokens to keep. Once this limit is reached, any remaining tokens are excluded from the output. Defaults to 1.
consume_all_tokens: (Optional, Boolean) If true, the limit filter exhausts the token stream, even if the max_token_count has already been reached. Defaults to false.

Example

The following analyze API request uses the limit filter to keep only the first two tokens in quick fox jumps over lazy dog:

GET _analyze
{
  "tokenizer": "standard",
    "filter": [
    {
      "type": "limit",
      "max_token_count": 2
    }
  ],
  "text": "quick fox jumps over lazy dog"
}

The filter produces the following tokens:

[ quick, fox ]

Add to an analyzer

The following create index API request uses the limit filter to configure a new custom analyzer.

PUT limit_example
{
  "settings": {
    "analysis": {
      "analyzer": {
        "standard_one_token_limit": {
          "tokenizer": "standard",
          "filter": [ "limit" ]
        }
      }
    }
  }
}

Customize

To customize the limit filter, duplicate it to create the basis for a new custom token filter. You can modify the filter using its configurable parameters.

For example, the following request creates a custom limit filter that keeps only the first five tokens of a stream:

PUT custom_limit_example
{
  "settings": {
    "analysis": {
      "analyzer": {
        "whitespace_five_token_limit": {
          "tokenizer": "whitespace",
          "filter": [ "five_token_limit" ]
        }
      },
      "filter": {
        "five_token_limit": {
          "type": "limit",
          "max_token_count": 5
        }
      }
    }
  }
}

© 2023-2025 Elasticsearch
As of September 2024, Elasticsearch is available under a choice of three licenses: the Server Side Public License (SSPL), the Elastic License, or the AGPLv3 (OSI approved).
Elasticsearch and the Elasticsearch logo are trademarks of Elasticsearch B.V., registered in the U.S. and in other countries.
https://www.elastic.co/guide/en/elasticsearch/reference/7.17/analysis-limit-token-count-tokenfilter.html