We use Elasticsearch in NeetoCourse for our searching needs. Recently, we have made some changes to Elasticsearch config to improve the search experience. In this blog, we will share the changes we made and what we learned during the process.
Definitions
These are some of the terminologies in Elasticsearch that we use in this blog.
-
Document: A document in Elasticsearch is similiar to a row in a database table. It is a collection of key-value pairs.
-
Index: An index is a collection of documents. It is similar to a database table. Indexing is the process of creating the said index, and we can configure each step of this process.
-
Analyzer: An analyzer converts a string into a list of searchable tokens. Analyzer contains three functions: Character Filter, Tokenizer, and Token Filter.
-
Character Filter is a function that performs the process of filtering out certain characters from the input string. For example, to strip html tags and to get only the body.
-
Tokenizer: A tokenizer is a function that splits the input string into tokens. This can be based on whitespace, punctuation, or any other character.
-
Token Filter: A token filter is a function that performs the process of filtering out certain tokens from the input string. For example, it can be used to remove stop words (a, the, and, etc.) which serves no purpose in the search.
Analyzer
This is our analyzer setup:
1{ 2 default: { 3 tokenizer: "whitespace", 4 filter: ["lowercase", "autocomplete"] 5 }, 6 search: { 7 tokenizer: "whitespace", 8 filter: ["lowercase"] 9 }, 10 english_exact: { 11 tokenizer: "whitespace", 12 filter: [ 13 "lowercase" 14 ] 15 } 16}
default is the analyzer used for indexing and searching. The search terms from the user is passed through the search analyzer. english_exact is the analyzer used for exact matches.
Tokenizer
By default, Elasticsearch uses the standard tokenizer which splits the input string into tokens based on whitespace, punctuation, and any other character. Since our content is mostly based on technical concepts and programming, we cannot use the standard tokenizer. The whitespace tokenizer splits the input string into tokens based on whitespace which is suitable for our use case. Hence, we use the whitespace tokenizer for all the analyzers.
Filter
The lowercase filter is used to convert all the tokens to lowercase before storing it in the index. This is a common requirement as we want the search to be case-insensitive. We also use the custom autocomplete filter. Let's see its definition:
1{ 2 autocomplete: { 3 "type": "edge_ngram", 4 "min_gram": 3, 5 "max_gram": 20, 6 "preserve_original": true 7 } 8}
The custom autocomplete filter is an implementation of edge_ngram token filter. Let's see the result of this filter when applied on the phrase "Elephant in the room".
1{ 2 "tokens": [ 3 { 4 "token": "Ele", 5 "start_offset": 0, 6 "end_offset": 8, 7 "type": "<ALPHANUM>", 8 "position": 0 9 }, 10 { 11 "token": "Elep", 12 "start_offset": 0, 13 "end_offset": 8, 14 "type": "<ALPHANUM>", 15 "position": 0 16 }, 17 { 18 "token": "Eleph", 19 "start_offset": 0, 20 "end_offset": 8, 21 "type": "<ALPHANUM>", 22 "position": 0 23 }, 24 { 25 "token": "Elepha", 26 "start_offset": 0, 27 "end_offset": 8, 28 "type": "<ALPHANUM>", 29 "position": 0 30 }, 31 { 32 "token": "Elephan", 33 "start_offset": 0, 34 "end_offset": 8, 35 "type": "<ALPHANUM>", 36 "position": 0 37 }, 38 { 39 "token": "Elephant", 40 "start_offset": 0, 41 "end_offset": 8, 42 "type": "<ALPHANUM>", 43 "position": 0 44 }, 45 { 46 "token": "in", 47 "start_offset": 9, 48 "end_offset": 11, 49 "type": "<ALPHANUM>", 50 "position": 1 51 }, 52 { 53 "token": "the", 54 "start_offset": 12, 55 "end_offset": 15, 56 "type": "<ALPHANUM>", 57 "position": 2 58 }, 59 { 60 "token": "roo", 61 "start_offset": 16, 62 "end_offset": 20, 63 "type": "<ALPHANUM>", 64 "position": 3 65 }, 66 { 67 "token": "room", 68 "start_offset": 16, 69 "end_offset": 20, 70 "type": "<ALPHANUM>", 71 "position": 3 72 } 73 ] 74}
The edge_ngram filter creates n-grams starting from the first character. Since we specified min_gram as 3 and max_gram as 20, the filter will create n-grams from the first 3 characters to the last 20 characters. This means that a document will be created for every 3 to 20 letter sequence of a word. If we start typing "ELEP", then there will be a document in the index corresponding to "ELEP" and we will get the result "Elephant in the room".
The Query
The query object in the search request is equally important for getting relevant results. Elasticsearch sorts matching search results by relevance score, which measures how well each document matches a query. The query clause contains the criteria for calculating the relevance score. This is our query object:
1{ 2 bool: { 3 should: [ 4 { 5 simple_query_string: { 6 query: requestBody.searchTerm, 7 fields: ["content", "meta.pageTitle", "meta.chapterTitle"], 8 quote_field_suffix: ".exact", 9 }, 10 }, 11 { 12 match: { 13 content: { 14 query: requestBody.searchTerm, 15 ...baseFuzzyQueryConfig(), 16 }, 17 }, 18 }, 19 { 20 match: { 21 "meta.pageTitle": { 22 query: requestBody.searchTerm, 23 ...baseFuzzyQueryConfig(), 24 boost: 1.5, 25 }, 26 }, 27 }, 28 { 29 match: { 30 "meta.chapterTitle": { 31 query: requestBody.searchTerm, 32 ...baseFuzzyQueryConfig(), 33 boost: 1.5, 34 }, 35 }, 36 }, 37 ], 38 minimum_should_match: 1, 39 }, 40};
The bool and should clauses are used to create a compound query. The should clause means that atleast one of the queries should match. Here the match is the standard query used for full text search. Here content, meta.pageTitle and meta.chapterTitle specify the fields that we created while indexing the data.
We have provided a boost value of 1.5 for page title and chapter title. This is to make sure that a page or chapter title has more relevance score than any content in the middle of the page.
The simple_query_string query is used for exact matches, when the search term contains double quotes, the english_exact analyzer is used. The double quotes operator ( " ) is part of the several operators that can be used in the simple_query_string query.
Fuzzy searching
We also use fuzzy searching in the match query. Fuzzy searching helps in giving proper results even if there are typos in the search term. Elasticsearch uses Levenshtein distance to calculate the similarity between the search term and the indexed data. Previously, we used the fuzzy query to implement fuzzy searching.
1{ 2 content: { 3 value: requestBody.searchTerm, 4 ...extendedFuzzyQueryConfig(), 5 }, 6},
But this caused several issues like:
- Fuzzy results being prioritized over exact matches. For example, searching for "five" returned results for "dive", even when "five" was present in the content
- Fuzzy results not being returned when the search term contained multiple words.
Upon investigation, we found that the fuzzy query does not perform analysis on the search term. Instead we now use the match query with fuzziness parameter.
1const baseFuzzyQueryConfig = () => ({ 2 prefix_length: 0, 3 fuzziness: "AUTO", 4}); 5... 6{ 7 content: { 8 query: requestBody.searchTerm, 9 ...baseFuzzyQueryConfig(), 10 }, 11},
Conclusion
There is no silver bullet in case of Elasticsearch configuration. And there is no metric to determine if the search is giving quality results. We have tweaked our configuration based on trial and error and from user feedback. We hope these techniques are useful for anyone who is looking to improve their search experience.