› › ›

CJK width token filter

Normalizes width differences in CJK (Chinese, Japanese, and Korean) characters as follows:

Folds full-width ASCII character variants into the equivalent basic Latin characters
Folds half-width Katakana character variants into the equivalent Kana characters

This filter is included in Elasticsearch’s built-in CJK language analyzer. It uses Lucene’s CJKWidthFilter.

This token filter can be viewed as a subset of NFKC/NFKD Unicode normalization. See the analysis-icu plugin for full normalization support.

Example

resp = client.indices.analyze(
    tokenizer="standard",
    filter=[
        "cjk_width"
    ],
    text="ｼｰｻｲﾄﾞﾗｲﾅｰ",
)
print(resp)

response = client.indices.analyze(
  body: {
    tokenizer: 'standard',
    filter: [
      'cjk_width'
    ],
    text: 'ｼｰｻｲﾄﾞﾗｲﾅｰ'
  }
)
puts response

const response = await client.indices.analyze({
  tokenizer: "standard",
  filter: ["cjk_width"],
  text: "ｼｰｻｲﾄﾞﾗｲﾅｰ",
});
console.log(response);

GET /_analyze
{
  "tokenizer" : "standard",
  "filter" : ["cjk_width"],
  "text" : "ｼｰｻｲﾄﾞﾗｲﾅｰ"
}

The filter produces the following token:

シーサイドライナー

Add to an analyzer

The following create index API request uses the CJK width token filter to configure a new custom analyzer.

resp = client.indices.create(
    index="cjk_width_example",
    settings={
        "analysis": {
            "analyzer": {
                "standard_cjk_width": {
                    "tokenizer": "standard",
                    "filter": [
                        "cjk_width"
                    ]
                }
            }
        }
    },
)
print(resp)

response = client.indices.create(
  index: 'cjk_width_example',
  body: {
    settings: {
      analysis: {
        analyzer: {
          standard_cjk_width: {
            tokenizer: 'standard',
            filter: [
              'cjk_width'
            ]
          }
        }
      }
    }
  }
)
puts response

const response = await client.indices.create({
  index: "cjk_width_example",
  settings: {
    analysis: {
      analyzer: {
        standard_cjk_width: {
          tokenizer: "standard",
          filter: ["cjk_width"],
        },
      },
    },
  },
});
console.log(response);

PUT /cjk_width_example
{
  "settings": {
    "analysis": {
      "analyzer": {
        "standard_cjk_width": {
          "tokenizer": "standard",
          "filter": [ "cjk_width" ]
        }
      }
    }
  }
}

© 2023-2025 Elasticsearch
As of September 2024, Elasticsearch is available under a choice of three licenses: the Server Side Public License (SSPL), the Elastic License, or the AGPLv3 (OSI approved).
Elasticsearch and the Elasticsearch logo are trademarks of Elasticsearch B.V., registered in the U.S. and in other countries.
https://www.elastic.co/guide/en/elasticsearch/reference/8.17/analysis-cjk-width-tokenfilter.html