<!DOCTYPE html>
<html lang="en" data-content_root="../">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Eprobe - Event-based Probe Tracing — The Linux Kernel documentation</title>
<link rel="stylesheet" type="text/css" href="../_static/pygments.css?v=fa44fd50" />
<link rel="stylesheet" type="text/css" href="../_static/alabaster.css?v=3918102e" />
<script src="../_static/documentation_options.js?v=5929fcd5"></script>
<script src="../_static/doctools.js?v=9bcbadda"></script>
<script src="../_static/sphinx_highlight.js?v=dc90522c"></script>
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
<link rel="next" title="Fprobe - Function entry/exit probe" href="fprobe.html" />
<link rel="prev" title="Fprobe-based Event Tracing" href="fprobetrace.html" />
<link rel="stylesheet" href="../_static/custom.css" type="text/css" />
</head><body>
<div class="document">
<div class="sphinxsidebar" role="navigation" aria-label="Main">
<div class="sphinxsidebarwrapper">
<p class="logo"><a href="../index.html">
<img class="logo" src="../_static/logo.svg" alt="Logo of The Linux Kernel"/>
</a></p>
<h1 class="logo"><a href="../index.html">The Linux Kernel</a></h1>
<p class="blurb">6.18.50</p>
<search id="searchbox" style="display: none" role="search">
<h3 id="searchlabel">Quick search</h3>
<div class="searchformwrapper">
<form class="search" action="../search.html" method="get">
<input type="text" name="q" aria-labelledby="searchlabel" autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false"/>
<input type="submit" value="Go" />
</form>
</div>
</search>
<script>document.getElementById('searchbox').style.display = "block"</script>
<p>
<h3 class="kernel-toc-contents">Contents</h3>
<input type="checkbox" class="kernel-toc-toggle" id = "kernel-toc-toggle" checked>
<label class="kernel-toc-title" for="kernel-toc-toggle"></label>
<div class="kerneltoc" id="kerneltoc">
<ul>
<li class="toctree-l1"><a class="reference internal" href="../process/development-process.html">Development process</a></li>
<li class="toctree-l1"><a class="reference internal" href="../process/submitting-patches.html">Submitting patches</a></li>
<li class="toctree-l1"><a class="reference internal" href="../process/code-of-conduct.html">Code of conduct</a></li>
<li class="toctree-l1"><a class="reference internal" href="../maintainer/index.html">Maintainer handbook</a></li>
<li class="toctree-l1"><a class="reference internal" href="../process/index.html">All development-process docs</a></li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../core-api/index.html">Core API</a></li>
<li class="toctree-l1"><a class="reference internal" href="../driver-api/index.html">Driver APIs</a></li>
<li class="toctree-l1"><a class="reference internal" href="../subsystem-apis.html">Subsystems</a></li>
<li class="toctree-l1"><a class="reference internal" href="../locking/index.html">Locking</a></li>
</ul>
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../process/license-rules.html">Licensing rules</a></li>
<li class="toctree-l1"><a class="reference internal" href="../doc-guide/index.html">Writing documentation</a></li>
<li class="toctree-l1"><a class="reference internal" href="../dev-tools/index.html">Development tools</a></li>
<li class="toctree-l1"><a class="reference internal" href="../dev-tools/testing-overview.html">Testing guide</a></li>
<li class="toctree-l1"><a class="reference internal" href="../kernel-hacking/index.html">Hacking guide</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="index.html">Tracing</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="index.html#introduction-to-tracing">Introduction to Tracing</a></li>
<li class="toctree-l2 current"><a class="reference internal" href="index.html#core-tracing-frameworks">Core Tracing Frameworks</a><ul class="current">
<li class="toctree-l3"><a class="reference internal" href="ftrace.html">ftrace - Function Tracer</a></li>
<li class="toctree-l3"><a class="reference internal" href="ftrace-design.html">Function Tracer Design</a></li>
<li class="toctree-l3"><a class="reference internal" href="ftrace-uses.html">Using ftrace to hook to functions</a></li>
<li class="toctree-l3"><a class="reference internal" href="kprobes.html">Kernel Probes (Kprobes)</a></li>
<li class="toctree-l3"><a class="reference internal" href="kprobetrace.html">Kprobe-based Event Tracing</a></li>
<li class="toctree-l3"><a class="reference internal" href="fprobetrace.html">Fprobe-based Event Tracing</a></li>
<li class="toctree-l3 current"><a class="current reference internal" href="#">Eprobe - Event-based Probe Tracing</a></li>
<li class="toctree-l3"><a class="reference internal" href="fprobe.html">Fprobe - Function entry/exit probe</a></li>
<li class="toctree-l3"><a class="reference internal" href="ring-buffer-design.html">Lockless Ring Buffer Design</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="index.html#event-tracing-and-analysis">Event Tracing and Analysis</a></li>
<li class="toctree-l2"><a class="reference internal" href="index.html#hardware-and-performance-tracing">Hardware and Performance Tracing</a></li>
<li class="toctree-l2"><a class="reference internal" href="index.html#user-space-tracing">User-Space Tracing</a></li>
<li class="toctree-l2"><a class="reference internal" href="index.html#additional-resources">Additional Resources</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../fault-injection/index.html">Fault injection</a></li>
<li class="toctree-l1"><a class="reference internal" href="../livepatch/index.html">Livepatching</a></li>
<li class="toctree-l1"><a class="reference internal" href="../rust/index.html">Rust</a></li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../admin-guide/index.html">Administration</a></li>
<li class="toctree-l1"><a class="reference internal" href="../kbuild/index.html">Build system</a></li>
<li class="toctree-l1"><a class="reference internal" href="../admin-guide/reporting-issues.html">Reporting issues</a></li>
<li class="toctree-l1"><a class="reference internal" href="../tools/index.html">Userspace tools</a></li>
<li class="toctree-l1"><a class="reference internal" href="../userspace-api/index.html">Userspace API</a></li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../firmware-guide/index.html">Firmware</a></li>
<li class="toctree-l1"><a class="reference internal" href="../devicetree/index.html">Firmware and Devicetree</a></li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../arch/index.html">CPU architectures</a></li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../staging/index.html">Unsorted documentation</a></li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../translations/index.html">Translations</a></li>
</ul>
</div>
<script type="text/javascript"> <!--
var sbar = document.getElementsByClassName("sphinxsidebar")[0];
let currents = document.getElementsByClassName("current")
if (currents.length) {
sbar.scrollTop = currents[currents.length - 1].offsetTop;
}
--> </script>
<div role="note" aria-label="source link">
<h3>This Page</h3>
<ul class="this-page-menu">
<li><a href="../_sources/trace/eprobetrace.rst.txt"
rel="nofollow">Show Source</a></li>
</ul>
</div>
</div>
</div>
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body" role="main">
<section id="eprobe-event-based-probe-tracing">
<h1>Eprobe - Event-based Probe Tracing<a class="headerlink" href="#eprobe-event-based-probe-tracing" title="Link to this heading">¶</a></h1>
<dl class="field-list simple">
<dt class="field-odd">Author<span class="colon">:</span></dt>
<dd class="field-odd"><p>Steven Rostedt <<a class="reference external" href="mailto:rostedt%40goodmis.org">rostedt<span>@</span>goodmis<span>.</span>org</a>></p>
</dd>
</dl>
<ul class="simple">
<li><p>Written for v6.17</p></li>
</ul>
<section id="overview">
<h2>Overview<a class="headerlink" href="#overview" title="Link to this heading">¶</a></h2>
<p>Eprobes are dynamic events that are placed on existing events to either
dereference a field that is a pointer, or simply to limit what fields are
recorded in the trace event.</p>
<p>Eprobes depend on kprobe events so to enable this feature, build your kernel
with CONFIG_EPROBE_EVENTS=y.</p>
<p>Eprobes are created via the /sys/kernel/tracing/dynamic_events file.</p>
<section id="synopsis-of-eprobe-events">
<h3>Synopsis of eprobe_events<a class="headerlink" href="#synopsis-of-eprobe-events" title="Link to this heading">¶</a></h3>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span> e[:[EGRP/][EEVENT]] GRP.EVENT [FETCHARGS] : Set a probe
-:[EGRP/][EEVENT] : Clear a probe
EGRP : Group name of the new event. If omitted, use "eprobes" for it.
EEVENT : Event name. If omitted, the event name is generated and will
be the same event name as the event it attached to.
GRP : Group name of the event to attach to.
EVENT : Event name of the event to attach to.
FETCHARGS : Arguments. Each probe can have up to 128 args.
$FIELD : Fetch the value of the event field called FIELD.
@ADDR : Fetch memory at ADDR (ADDR should be in kernel)
@SYM[+|-offs] : Fetch memory at SYM +|- offs (SYM should be a data symbol)
$comm : Fetch current task comm.
+|-[u]OFFS(FETCHARG) : Fetch memory at FETCHARG +|- OFFS address.(\*3)(\*4)
\IMM : Store an immediate value to the argument.
NAME=FETCHARG : Set NAME as the argument name of FETCHARG.
FETCHARG:TYPE : Set TYPE as the type of FETCHARG. Currently, basic types
(u8/u16/u32/u64/s8/s16/s32/s64), hexadecimal types
(x8/x16/x32/x64), VFS layer common type(%pd/%pD), "char",
"string", "ustring", "symbol", "symstr" and "bitfield" are
supported.
(STRUCT)FIELD->MEMBER[->MEMBER] : If BTF is supported, typecast FIELD to
a pointer to STRUCT and then derference the pointer defined by
->MEMBER. Note that when this is used, the FIELD name does not
need to be prefixed with a '$'.
</pre></div>
</div>
</section>
<section id="types">
<h3>Types<a class="headerlink" href="#types" title="Link to this heading">¶</a></h3>
<p>The FETCHARGS above is very similar to the kprobe events as described in
<a class="reference internal" href="kprobetrace.html"><span class="doc">Kprobe-based Event Tracing</span></a>.</p>
<p>The difference between eprobes and kprobes FETCHARGS is that eprobes has a
$FIELD command that returns the content of the event field of the event
that is attached. Eprobes do not have access to registers, stacks and function
arguments that kprobes has.</p>
<p>If a field argument is a pointer, it may be dereferenced just like a memory
address using the FETCHARGS syntax.</p>
</section>
<section id="attaching-to-dynamic-events">
<h3>Attaching to dynamic events<a class="headerlink" href="#attaching-to-dynamic-events" title="Link to this heading">¶</a></h3>
<p>Eprobes may attach to dynamic events as well as to normal events. It may
attach to a kprobe event, a synthetic event or a fprobe event. This is useful
if the type of a field needs to be changed. See Example 2 below.</p>
</section>
</section>
<section id="usage-examples">
<h2>Usage examples<a class="headerlink" href="#usage-examples" title="Link to this heading">¶</a></h2>
<section id="example-1">
<h3>Example 1<a class="headerlink" href="#example-1" title="Link to this heading">¶</a></h3>
<p>The basic usage of eprobes is to limit the data that is being recorded into
the tracing buffer. For example, a common event to trace is the sched_switch
trace event. That has a format of:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>field:unsigned short common_type; offset:0; size:2; signed:0;
field:unsigned char common_flags; offset:2; size:1; signed:0;
field:unsigned char common_preempt_count; offset:3; size:1; signed:0;
field:int common_pid; offset:4; size:4; signed:1;
field:char prev_comm[16]; offset:8; size:16; signed:0;
field:pid_t prev_pid; offset:24; size:4; signed:1;
field:int prev_prio; offset:28; size:4; signed:1;
field:long prev_state; offset:32; size:8; signed:1;
field:char next_comm[16]; offset:40; size:16; signed:0;
field:pid_t next_pid; offset:56; size:4; signed:1;
field:int next_prio; offset:60; size:4; signed:1;
</pre></div>
</div>
<p>The first four fields are common to all events and can not be limited. But the
rest of the event has 60 bytes of information. It records the names of the
previous and next tasks being scheduled out and in, as well as their pids and
priorities. It also records the state of the previous task. If only the pids
of the tasks are of interest, why waste the ring buffer with all the other
fields?</p>
<p>An eprobe can limit what gets recorded. Note, it does not help in performance,
as all the fields are recorded in a temporary buffer to process the eprobe.</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># echo 'e:sched/switch sched.sched_switch prev=$prev_pid:u32 next=$next_pid:u32' >> /sys/kernel/tracing/dynamic_events
# echo 1 > /sys/kernel/tracing/events/sched/switch/enable
# cat /sys/kernel/tracing/trace
# tracer: nop
#
# entries-in-buffer/entries-written: 2721/2721 #P:8
#
# _-----=> irqs-off/BH-disabled
# / _----=> need-resched
# | / _---=> hardirq/softirq
# || / _--=> preempt-depth
# ||| / _-=> migrate-disable
# |||| / delay
# TASK-PID CPU# ||||| TIMESTAMP FUNCTION
# | | | ||||| | |
sshd-session-1082 [004] d..4. 5041.239906: switch: (sched.sched_switch) prev=1082 next=0
bash-1085 [001] d..4. 5041.240198: switch: (sched.sched_switch) prev=1085 next=141
kworker/u34:5-141 [001] d..4. 5041.240259: switch: (sched.sched_switch) prev=141 next=1085
<idle>-0 [004] d..4. 5041.240354: switch: (sched.sched_switch) prev=0 next=1082
bash-1085 [001] d..4. 5041.240385: switch: (sched.sched_switch) prev=1085 next=141
kworker/u34:5-141 [001] d..4. 5041.240410: switch: (sched.sched_switch) prev=141 next=1085
bash-1085 [001] d..4. 5041.240478: switch: (sched.sched_switch) prev=1085 next=0
sshd-session-1082 [004] d..4. 5041.240526: switch: (sched.sched_switch) prev=1082 next=0
<idle>-0 [001] d..4. 5041.247524: switch: (sched.sched_switch) prev=0 next=90
<idle>-0 [002] d..4. 5041.247545: switch: (sched.sched_switch) prev=0 next=16
kworker/1:1-90 [001] d..4. 5041.247580: switch: (sched.sched_switch) prev=90 next=0
rcu_sched-16 [002] d..4. 5041.247591: switch: (sched.sched_switch) prev=16 next=0
<idle>-0 [002] d..4. 5041.257536: switch: (sched.sched_switch) prev=0 next=16
rcu_sched-16 [002] d..4. 5041.257573: switch: (sched.sched_switch) prev=16 next=0
</pre></div>
</div>
<p>Note, without adding the “u32” after the prev_pid and next_pid, the values
would default showing in hexadecimal.</p>
</section>
<section id="example-2">
<h3>Example 2<a class="headerlink" href="#example-2" title="Link to this heading">¶</a></h3>
<p>If a specific system call is to be recorded but the syscalls events are not
enabled, the raw_syscalls can still be used (syscalls are system call
events are not normal events, but are created from the raw_syscalls events
within the kernel). In order to trace the openat system call, one can create
an event probe on top of the raw_syscalls event:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># cd /sys/kernel/tracing
# cat events/raw_syscalls/sys_enter/format
name: sys_enter
ID: 395
format:
field:unsigned short common_type; offset:0; size:2; signed:0;
field:unsigned char common_flags; offset:2; size:1; signed:0;
field:unsigned char common_preempt_count; offset:3; size:1; signed:0;
field:int common_pid; offset:4; size:4; signed:1;
field:long id; offset:8; size:8; signed:1;
field:unsigned long args[6]; offset:16; size:48; signed:0;
print fmt: "NR %ld (%lx, %lx, %lx, %lx, %lx, %lx)", REC->id, REC->args[0], REC->args[1], REC->args[2], REC->args[3], REC->args[4], REC->args[5]
</pre></div>
</div>
<p>From the source code, the <code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">sys_openat()</span></code> has:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>int sys_openat(int dirfd, const char *path, int flags, mode_t mode)
{
return my_syscall4(__NR_openat, dirfd, path, flags, mode);
}
</pre></div>
</div>
<p>The path is the second parameter, and that is what is wanted.</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># echo 'e:openat raw_syscalls.sys_enter nr=$id filename=+8($args):ustring' >> dynamic_events
</pre></div>
</div>
<p>This is being run on x86_64 where the word size is 8 bytes and the openat
system call __NR_openat is set at 257.</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># echo 'nr == 257' > events/eprobes/openat/filter
</pre></div>
</div>
<p>Now enable the event and look at the trace.</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># echo 1 > events/eprobes/openat/enable
# cat trace
# tracer: nop
#
# entries-in-buffer/entries-written: 4/4 #P:8
#
# _-----=> irqs-off/BH-disabled
# / _----=> need-resched
# | / _---=> hardirq/softirq
# || / _--=> preempt-depth
# ||| / _-=> migrate-disable
# |||| / delay
# TASK-PID CPU# ||||| TIMESTAMP FUNCTION
# | | | ||||| | |
cat-1298 [003] ...2. 2060.875970: openat: (raw_syscalls.sys_enter) nr=0x101 filename=(fault)
cat-1298 [003] ...2. 2060.876197: openat: (raw_syscalls.sys_enter) nr=0x101 filename=(fault)
cat-1298 [003] ...2. 2060.879126: openat: (raw_syscalls.sys_enter) nr=0x101 filename=(fault)
cat-1298 [003] ...2. 2060.879639: openat: (raw_syscalls.sys_enter) nr=0x101 filename=(fault)
</pre></div>
</div>
<p>The filename shows “(fault)”. This is likely because the filename has not been
pulled into memory yet and currently trace events cannot fault in memory that
is not present. When an eprobe tries to read memory that has not been faulted
in yet, it will show the “(fault)” text.</p>
<p>To get around this, as the kernel will likely pull in this filename and make
it present, attaching it to a synthetic event that can pass the address of the
filename from the entry of the event to the end of the event, this can be used
to show the filename when the system call returns.</p>
<p>Remove the old eprobe:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># echo 1 > events/eprobes/openat/enable
# echo '-:openat' >> dynamic_events
</pre></div>
</div>
<p>This time make an eprobe where the address of the filename is saved:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># echo 'e:openat_start raw_syscalls.sys_enter nr=$id filename=+8($args):x64' >> dynamic_events
</pre></div>
</div>
<p>Create a synthetic event that passes the address of the filename to the
end of the event:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># echo 's:filename u64 file' >> dynamic_events
# echo 'hist:keys=common_pid:f=filename if nr == 257' > events/eprobes/openat_start/trigger
# echo 'hist:keys=common_pid:file=$f:onmatch(eprobes.openat_start).trace(filename,$file) if id == 257' > events/raw_syscalls/sys_exit/trigger
</pre></div>
</div>
<p>Now that the address of the filename has been passed to the end of the
system call, create another eprobe to attach to the exit event to show the
string:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># echo 'e:openat synthetic.filename filename=+0($file):ustring' >> dynamic_events
# echo 1 > events/eprobes/openat/enable
# cat trace
# tracer: nop
#
# entries-in-buffer/entries-written: 4/4 #P:8
#
# _-----=> irqs-off/BH-disabled
# / _----=> need-resched
# | / _---=> hardirq/softirq
# || / _--=> preempt-depth
# ||| / _-=> migrate-disable
# |||| / delay
# TASK-PID CPU# ||||| TIMESTAMP FUNCTION
# | | | ||||| | |
cat-1331 [001] ...5. 2944.787977: openat: (synthetic.filename) filename="/etc/ld.so.cache"
cat-1331 [001] ...5. 2944.788480: openat: (synthetic.filename) filename="/lib/x86_64-linux-gnu/libc.so.6"
cat-1331 [001] ...5. 2944.793426: openat: (synthetic.filename) filename="/usr/lib/locale/locale-archive"
cat-1331 [001] ...5. 2944.831362: openat: (synthetic.filename) filename="trace"
</pre></div>
</div>
</section>
<section id="example-3">
<h3>Example 3<a class="headerlink" href="#example-3" title="Link to this heading">¶</a></h3>
<p>If syscall trace events are available, the above would not need the first
eprobe, but it would still need the last one:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># echo 's:filename u64 file' >> dynamic_events
# echo 'hist:keys=common_pid:f=filename' > events/syscalls/sys_enter_openat/trigger
# echo 'hist:keys=common_pid:file=$f:onmatch(syscalls.sys_enter_openat).trace(filename,$file)' > events/syscalls/sys_exit_openat/trigger
# echo 'e:openat synthetic.filename filename=+0($file):ustring' >> dynamic_events
# echo 1 > events/eprobes/openat/enable
</pre></div>
</div>
<p>And this would produce the same result as Example 2.</p>
</section>
</section>
</section>
</div>
</div>
</div>
<div class="clearer"></div>
</div>
<div class="footer">
©The kernel development community.
|
Powered by <a href="https://www.sphinx-doc.org/">Sphinx 8.1.3</a>
& <a href="https://alabaster.readthedocs.io">Alabaster 0.7.16</a>
|
<a href="../_sources/trace/eprobetrace.rst.txt"
rel="nofollow">Page source</a>
</div>
</body>
</html>