Elasticsearch use case

Portfolio

Click on an image for more details

System overview

Generic statistics about the application execution

Elasticsearch issued about 1 million storage system calls, most of which were write system calls (71%).
Besides the write operations, Elasticsearch also issued pread64, read, lseek, stat, fdatasync, pwrite64, fstat, openat, lstat, close and other operations.
Moreover, the Elasticsearch application spawned 42 processes and 118 threads, and accessed almost 4000 different file paths.

Percentage of data vs metadata

Shows the percentage of data and metadata-related system calls

The majority of system calls issued by Elasticsearch were data-related (88%).

Execution time per system call

Shows the average execution time for the top 5 system calls

The system call with the highest average execution time was fsync, which took 2.99 milliseconds on average.
fdatasync was the second most expensive system call, with an average execution time of 2.46 milliseconds.
The average execution time of the unlink system call was 0.36 milliseconds.

Top file types

Shows the type of files accessed by the application and the number of system calls issued for each file type

Elasticsearch accessed two types of files: regular files and directories.
The majority of the system calls were issued to regular files (about 1 million).

File path accesses

Shows the application file accesses over time

Some files exhibit a constant access pattern, even in the absence of client requests.
Every 30 seconds Elasticsearch submits at least 2 system calls to the node.lock file (blue line).
Every 2 minutes, Elasticsearch submits 9 system calls to .es_temp_file file (pink line).

File path accesses for .es_temp_file

Show the application file accesses over time for the .es_temp_file file (nanosecond time interval)

Elasticsearch always performs the same sequecence of system calls: lstat→openat→write→openat→fsync→close→close→lstat→unlink.
The system call with higher execution time is fsync (blue bar).
There is a duplication of openat and close system calls.