Demonstrations of mdflush, the Linux bpftrace/eBPF version.


The mdflush tool traces flushes at the md driver level, and prints details
including the time of the flush:

# ./mdflush.bt
Tracing md flush requests... Hit Ctrl-C to end.
TIME     PID    COMM             DEVICE
03:13:49 16770  sync             md0
03:14:08 16864  sync             md0
03:14:49 496    kworker/1:0H     md0
03:14:49 488    xfsaild/md0      md0
03:14:54 488    xfsaild/md0      md0
03:15:00 488    xfsaild/md0      md0
03:15:02 85     kswapd0          md0
03:15:02 488    xfsaild/md0      md0
03:15:05 488    xfsaild/md0      md0
03:15:08 488    xfsaild/md0      md0
03:15:10 488    xfsaild/md0      md0
03:15:11 488    xfsaild/md0      md0
03:15:11 488    xfsaild/md0      md0
03:15:11 488    xfsaild/md0      md0
03:15:11 488    xfsaild/md0      md0
03:15:11 488    xfsaild/md0      md0
03:15:12 488    xfsaild/md0      md0
03:15:13 488    xfsaild/md0      md0
03:15:15 488    xfsaild/md0      md0
03:15:19 496    kworker/1:0H     md0
03:15:49 496    kworker/1:0H     md0
03:15:55 18840  sync             md0
03:16:49 496    kworker/1:0H     md0
03:17:19 496    kworker/1:0H     md0
03:20:19 496    kworker/1:0H     md0
03:21:19 496    kworker/1:0H     md0
03:21:49 496    kworker/1:0H     md0
03:25:19 496    kworker/1:0H     md0
[...]

This can be useful for correlation with latency outliers or spikes in disk
latency, as measured using another tool (eg, system monitoring). If spikes in
disk latency often coincide with md flush events, then it would make flushing
a target for tuning.

Note that the flush events are likely to originate from higher in the I/O
stack, such as from file systems. This traces md processing them, and the
timestamp corresponds with when md began to issue the flush to disks.

There is another version of this tool in bcc: https://github.com/iovisor/bcc