Fluent Bit use case

How to reproduce

This page describes how to reproduce the results presented in the paper for two setups:

• Vanilla: without tracing I/O events
• DIO: tracing and analyzing system calls with DIO

Fluent Bit configuration

• Source code: https://github.com/fluent/fluent-bit.git
• Commits:
  • for the erroneous access patterns: e6506b7b5364c77bec186d94e51c4b3b51e6fbac
  • for the correct access patterns: ba2c0d70ab5fb32a10e7b920e0ea821b7fb9b259
• Env Configurations (mounting a new file system):

  dd if=/dev/zero of=disk.img bs=1M count=128
  mkfs.ext4 disk.img
  mkdir -p /fluent-bit/tests/mnt
  mount -o loop disk.img /fluent-bit/tests/mnt
  

• Configurations (fluentbit.conf):

  [SERVICE]
      Flush        5
      Daemon       On
      Log_Level    trace
  
  [INPUT]
      Name tail
      Tag  examplelogs
      DB   exampledb
      Path /fluent-bit/tests/mnt/*.log
      Mem_Buf_Limit     5MB
      Skip_Long_Lines   Off
      Refresh_Interval  10
  
  [OUTPUT]
      Name  stdout
      Match *
  
  

Client application

• Description: A simple C program that:
  1. creates a file named app.log, writes 26 bytes to it and closes it.
  2. waits for 10 seconds.
  3. removes the file.
  4. waits for 10 seconds.
  5. creates a new file named app.log, writes 16 bytes to it and closes it.

DIO configuration

• Tracer Configurations:
  • Filter events by specific Process IDs (--pid):
  • Fluent Bit's PID
  • Client app's PID
  • Filter events by file paths (target_paths):
  • RocksDB folder: /rocksdb/test/kvstore

Docker images

• Image: taniaesteves/dio-fluentbit
  • TAG "v1.4.0" - Fluent Bit v1.4.0 with the erroneous access pattern
  • TAG "v2.0.5" - Fluent Bit v2.0.5 with the correct access pattern
• Run test:
  • Vanilla:
  docker run -it --rm --name fluentbit --pid=host --privileged --cap-add=ALL --net=host -v /lib/modules:/lib/modules -v /usr/src:/usr/src -v /sys/kernel/debug/:/sys/kernel/debug/ taniaesteves/dio-fluentbit:v1.4.0
  • DIO:
  docker run -it --rm --name fluentbit --pid=host --privileged --cap-add=ALL --net=host -v /lib/modules:/lib/modules -v /usr/src:/usr/src -v /sys/kernel/debug/:/sys/kernel/debug/ -v /tmp/dio_data:/dio_data -e CORRELATE_PATHS=true -e ES_SERVERS=http://<DIO_ES_URL>:<DIO_ES_PORT> taniaesteves/dio-fluentbit:v1.4.0 dio