› › ›

Character group tokenizer

The char_group tokenizer breaks text into terms whenever it encounters a character which is in a defined set. It is mostly useful for cases where a simple custom tokenization is desired, and the overhead of use of the pattern tokenizer is not acceptable.

Configuration

The char_group tokenizer accepts one parameter:

`tokenize_on_chars`	A list containing a list of characters to tokenize the string on. Whenever a character from this list is encountered, a new token is started. This accepts either single characters like e.g. `-`, or character groups: `whitespace`, `letter`, `digit`, `punctuation`, `symbol`.
`max_token_length`	The maximum token length. If a token is seen that exceeds this length then it is split at `max_token_length` intervals. Defaults to `255`.

Example output

resp = client.indices.analyze(
    tokenizer={
        "type": "char_group",
        "tokenize_on_chars": [
            "whitespace",
            "-",
            "\n"
        ]
    },
    text="The QUICK brown-fox",
)
print(resp)

response = client.indices.analyze(
  body: {
    tokenizer: {
      type: 'char_group',
      tokenize_on_chars: [
        'whitespace',
        '-',
        "\n"
      ]
    },
    text: 'The QUICK brown-fox'
  }
)
puts response

const response = await client.indices.analyze({
  tokenizer: {
    type: "char_group",
    tokenize_on_chars: ["whitespace", "-", "\n"],
  },
  text: "The QUICK brown-fox",
});
console.log(response);

POST _analyze
{
  "tokenizer": {
    "type": "char_group",
    "tokenize_on_chars": [
      "whitespace",
      "-",
      "\n"
    ]
  },
  "text": "The QUICK brown-fox"
}

returns

{
  "tokens": [
    {
      "token": "The",
      "start_offset": 0,
      "end_offset": 3,
      "type": "word",
      "position": 0
    },
    {
      "token": "QUICK",
      "start_offset": 4,
      "end_offset": 9,
      "type": "word",
      "position": 1
    },
    {
      "token": "brown",
      "start_offset": 10,
      "end_offset": 15,
      "type": "word",
      "position": 2
    },
    {
      "token": "fox",
      "start_offset": 16,
      "end_offset": 19,
      "type": "word",
      "position": 3
    }
  ]
}

© 2023-2025 Elasticsearch
As of September 2024, Elasticsearch is available under a choice of three licenses: the Server Side Public License (SSPL), the Elastic License, or the AGPLv3 (OSI approved).
Elasticsearch and the Elasticsearch logo are trademarks of Elasticsearch B.V., registered in the U.S. and in other countries.
https://www.elastic.co/guide/en/elasticsearch/reference/8.17/analysis-chargroup-tokenizer.html