Home » Posts tagged 'Kibana'

Tag Archives: Kibana

Sort strings alphabetically rather than lexicographically in Elasticsearch?

Let’s say we have a text field “name” in an elasticsearch index with the following values: Siddhant, SIDTECHNOCRAT, and sid.

Now follow the conventions mentioned in String Sorting in Elasticsearch, which talks about using a text field that is not analyzed for sorting.

I am assuming that you’ve followed the conventions mentioned in the above link.

For the demo I am using Elasticsearch 6.4.1.

Let’s index the names:

PUT /my_index/_doc/1
{ "name": "Siddhant" }

PUT /my_index/_doc/2
{ "name": "SIDTECHNOCRAT" }

PUT /my_index/_doc/3
{ "name": "sid" }

Let’s sort the names:

GET /my_index/user/_search?sort=name.keyword

Output:

SIDTECHNOCRAT 
Siddhant 
sid

Wait!! weren’t you expecting the result to be sid, Siddhant and SIDTECHNOCRAT.

You’re getting the results in the above order because the bytes used to represent capital letters have a lower ASCII value than the bytes used to represent lowercase letters, and as an international accepted standard, Elasticsearch follows ASCII sort order which is why the names are sorted with lowest bytes first.

In other words we’re getting results in lexicographical order which is perfectly fine for a machine but does not make much sense to human beings (expecting results to be sorted in alphabetical order).

If you want the results to be sorted in alphabetical order you should index each name in a way that ES should ignore the case while indexing.

To achieve this create a custom analyzer combining keyword tokenizer and lowercase token filter.

Then configure the text field you want to sort with the custom analyzer:

PUT /my_index
{
  "settings" : {
    "analysis" : {
      "analyzer" : {
        "custom_keyword_analyzer" : {
          "tokenizer" : "keyword",
          "filter" : ["lowercase"]
        }
      }
    }
  },
  "mappings" : {
    "_doc" : {
      "properties" : {
        "name" : {
          "type" : "text",
          "fields" : {
            "raw" : {
              "type" : "text",
              "analyzer" : "custom_keyword_analyzer",
              "fielddata": true
            }
          }
        }
      }
    }
  }
}
  • keyword tokenizer is used to consider the string as a whole and not splitting up into tokens.
  • lowercase filter is used to convert the token into small letters.
  • custom_keyword_analyzer is used with the multifield raw to sort the results alphabetically.

Index your data:

POST my_index/_doc/1
{ "name" : "Siddhant" }

POST my_index/_doc/2
{ "name" : "SIDTECHNOCRAT" }

POST my_index/_doc/3
{ "name" : "sid" }

Perform sort:

GET my_index/_doc/_search?sort=name.raw

Output:

sid 
Siddhant 
SIDTECHNOCRAT

Bingo !! You’ve got what you were expecting.

String sorting in Elasticsearch

We should not sort on analyzed text field instead we should sort on not_analyzed text field.

Let’s understand this with an example:

Index some documents with a text field “name”.

POST my_index/_doc/1
{
  "name" : "technocrat sid"
}

POST my_index/_doc/2
{
  "name" : "siddhant01"
}

POST my_index/_doc/3
{
  "name" : "sid 01"
}

POST my_index/_doc/4
{
  "name" : "agnihotry siddhant"
}

Let’s sort the results in ascending order:

GET my_index/_search
{
  "sort": [
  {
    "name": {
      "order": "asc"
    }
  }
 ]
}

We get the results in the order:

sid 01

agnihotry siddhant

technocrat sid

siddhant 01

Wait !! Why did we not get the results in alphabetical order? We were expecting something like this:

agnihotry siddhant

sid 01

siddhant 01

technocrat sid

 

Reason that we did not get the results in the above order:

As we haven’t specified index mapping beforehand, we are relying on default mapping.  So in this case, the text field above will be analyzed with Standard Analyzer by default which mainly splits the text with spaces and removes stop words.

i.e. if we analyze “agnihotry siddhant”, it results in two terms “agnihotry” & “siddhant”.

which means when we index the text it is stored into tokens,

text --> tokens 
technocrat sid --> technocrat, sid 
siddhant01 --> siddhant01 
sid 01 --> sid, 01 
agnihotry siddhant --> agnihotry, siddhant

 

But we probably want to sort alphabetically on the first term, then on the second term, and so forth. In this case we should consider the text as whole instead of splitting it into tokens.

i.e. we should consider “technocrat sid”, “sid 01” and “agnihotry siddhant” as a whole which means we should not analyze the text field.

How do we not analyze a text field?

Before Elasticsearch 5.x

Before Elasticsearch 5.x text fields were stored as string. In order to consider a string field as a whole it should not be analyzed but we still need to perform a full text query on that same field.

So what we really want is to index the same field in two different ways, i.e. we want to sort and search on the same string field.

We can do this using multifield mapping:

"name": {
  "type": "string",
    "fields": {
      "raw": {
        "type":  "string",
        "index": "not_analyzed"
      }
   }
}  

The main name field is same as before: an analyzed full-text field. The new name.raw sub field is not_analyzed.

That means we can use the name field for search and name.raw field for sorting:

GET my_index/_search
{
  "sort": [
  {
    "name.raw": {
      "order": "asc"
    }
  }
 ]
}

After Elasticsearh 5.x

In Elasticsearch 5.x, the string type has been removed and there are now two new types: text, which should be used for full-text search, and keyword, which should be used for sort.

For instance, if you index the following document:

{
  "name": "sid"
}

Then the following dynamic mappings will be created:

{
  "name": {
    "type" "text",
    "fields": {
      "keyword": {
        "type": "keyword",
        "ignore_above": 256
      }
    }
  }
}

So you don’t have to specify not_analyzed explicitly for a text field after ES 5.x.

You can use name.keyword for sorting:

GET my_index/_search
{
  "sort": [
  {
    "name.keyword": {
      "order": "asc"
    }
  }
 ]
}

Elasticsearch plugin for Sentiment Analysis

I have created an Elasticsearch plugin for sentiment-analysis using Stanford CoreNLP libraries. The plugin is compatible with Elasticsearch 6.4.1.

Follow the below steps to use this plugin with your elasticsearch server:

1. Install the plugin

Windows: 

bin\elasticsearch-plugin install https://github.com/TechnocratSid/elastic-sentiment-analysis-plugin/releases/download/6.4.1/elastic-sentiment-analyis-plugin-6.4.1.zip

Unix:

sudo bin/elasticsearch-plugin install https://github.com/TechnocratSid/elastic-sentiment-analysis-plugin/releases/download/6.4.1/elastic-sentiment-analyis-plugin-6.4.1.zip

2. Starting Elasticsearch

How you start Elasticsearch depends on how you installed it. I’ve installed Elasticsearch on Windows with a .zip package, in my case I can start Elasticsearch from the command line using the following command:

.\bin\elasticsearch.bat

Note: To setup Elasticsearch follow the link Set up Elasticsearch.

3. Open Kibana

Perform the request mentioned below:

Example1:

POST _sentiment
{
"text" : "He is very happy"
}

Output: 

{
"sentiment_score": 3,
"sentiment_type": "Positive",
"very_positive": "38.0%",
"positive": "59.0%",
"neutral": "2.0%",
"negative": "0.0%",
"very_negative": "0.0%"
}

Example2:

POST _sentiment
{
"text" : "He is bad"
}

Output:

{
"sentiment_score": 1,
"sentiment_type": "Negative",
"very_positive": "1.0%",
"positive": "2.0%",
"neutral": "13.0%",
"negative": "66.0%",
"very_negative": "19.0%"
}

If you don’t want to use kibana use curl instead.

If you want to hack into the code check out the github link.

Elasticsearch 6.x Analyzers

Elasticsearch Analyzer is a wrapper which wraps three functions:
  • Character filter: Mainly used to strip off some unused characters or change some characters.
  • Tokenizer: Breaks a text into individual tokens(or words) based on certain factors like whitespace, ngram etc.
  • Token filter: It receives the individual tokens from tokenizer and then applies some filters on it (example changing uppercase terms to lowercase).

In a nutshell, an analyzer is used to tell elasticsearch how the text/phrase should be indexed and searched.

Why do we need analyzers?

Analyzers are generally used when you want to index a text or phrase. It is useful to break the text into words so that you can search on terms to get the document.

Example: Let’s say you have an index (my_index) with a field “intro” and you index a document:

{ “intro” : “Hi there I am sid”} 

The following requests are performed in Kibana:

  • Create an index my_index:
PUT my_index
  • Put index mapping:
PUT my_index/_mapping/doc
{
  "properties": {
   "intro" : {
    "type": "keyword",
    "index": true
   }
  }
}
  • Index data:
POST my_index/doc/1
{
  "intro": "Hi there I am sid"
}

keyword type is not analyzed so the above text “Hi there I am sid” is indexed as it is i.e. it is not split into tokens.

If you want to query the above document you will have to write the complete phrase

i.e. (find documents where intro = “Hi there I am sid”)

The query will return the indexed document:

GET my_index/_search
{
  "query": {
   "match": {
    "intro": "Hi there I am sid"
   }
  }
}

But this will not:

GET my_index/_search
{
  "query": {
   "match": {
    "intro": "Hi there"
   }
  }
}

But if the phrase is indexed as tokens then even if you query for a token (find documents where intro=”sid”) you’ll get the document.

POST my_index2/doc/1
{
  "intro": "Hi there I am sid"
}

Note: By default standard analyzer is used for all text fields and it provides grammar based tokenization.

GET my_index2/_search
{
  "query": {
   "match": {
    "intro": "sid"
   }
  }
}

The above query will return the document.

Hope this is helpful !

Reference: My stackoverflow answer.