Whenever you come across a problem you poke and tinker - and when that doesn’t work then you, at last, stop and think. One way out of a hole is explaining your problem by articulating it aloud. So, I was reading and then explaining how Elasticsearch used memory to the metal squirrel sitting on the desk when it occurred to me neither of us had a clue how it worked. So we will open Elasticsearch … just a smidge.
Below, I summarise how Elasticsearch uses memory. It will cover the operating system, Java and Lucene. You will need to understand virtual memory.
Elasticsearch and the operating system
Operating systems manage a partially filled and limited space - the physical addresses. While it lies to applications that they have an empty block of memory - the virtual address space.
Operating systems can take advantages of virtual memory in two ways:
First, by reading in files from disk and keeping them in memory - called memory-mapped files. We will see is important in making faster queries when Lucene uses segmented files.
Second, when an operating system is running out of physical memory it can move parts of that memory that aren’t being used and saving them temporarily on disk - we call this swapping memory out on Linux and paging memory on Windows. Operating systems cannot swap and service Elasticsearch’s queries leading to slower Elasticsearch responses. Elasticsearch wants to be configured to avoid any swapping.
Table comparing Memory-mapped Files and Swapping / Paging
|Operation||Memory-mapped Files||Swapping / Paging|
|Physical memory use||Decreases||Increases|
|When is it used?||Any time||Running out of physical memory|
|Why?||Queries run faster||Frees physical memory when the system has run out of options|
Elasticsearch and Java
Elasticsearch wants to work with as little interference from the other systems on the computer. For servers running on Java this means setting the minimum and maximum heap size the same to increase predictability of the sizing of the virtual machine.
Java has managed memory, it garbage collects heap allocations.. Java 8.0 frequently pauses while garbage collecting slowing Elasticsearch’s response by ten times which can cause the entire cluster to wait and even re-elect a paused master.
Elasticsearch and Lucene
Elasticsearch uses Apache Lucene, a Java library, for text search.
Lucene puts search structures into immutable files, called segments. Since the files won’t change an operating systems can cache them as memory-mapped files so the next index query can complete in memory without reading from the slower hard drive. Memory mapped caching is at the operating system level and outside the Java managed heap.
Cached segment structures are the Inverted Index and Doc Values. Inverted Indexes answer “what documents has this query text”. While Doc Values answer “what values are in this document” - since this is the opposite of Inverted - Doc Values are also called uninverted indexes.
Lucene’s Segmented Files
|Inverted Index||Memory-mapped||Words to documents|
|Doc Values / uninverted index / Column-store||Memory-mapped||Documents to values|
Field data / Field cache
Doc Values are not efficient for aggregating text of analyzed fields and you should use heap structures called field data or field cache in Elasticsearch and Lucene respectively.
Elasticsearch generates field data structures on the fly to be saved into the heap, the default usage is unbounded and monitoring is advisable - as it can make a substantial amount of Elasticsearch’s heap.
Norms are numbers that weight the importance of a match on that field. Norms are one byte each and are caculated per document, per field which is seen as an increase in heap use.