<!DOCTYPE html>
<html lang="en" data-content_root="../">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Event Tracing — The Linux Kernel documentation</title>
<link rel="stylesheet" type="text/css" href="../_static/pygments.css?v=fa44fd50" />
<link rel="stylesheet" type="text/css" href="../_static/alabaster.css?v=3918102e" />
<script src="../_static/documentation_options.js?v=5929fcd5"></script>
<script src="../_static/doctools.js?v=9bcbadda"></script>
<script src="../_static/sphinx_highlight.js?v=dc90522c"></script>
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
<link rel="next" title="Subsystem Trace Points: kmem" href="events-kmem.html" />
<link rel="prev" title="Lockless Ring Buffer Design" href="ring-buffer-design.html" />
<link rel="stylesheet" href="../_static/custom.css" type="text/css" />
</head><body>
<div class="document">
<div class="sphinxsidebar" role="navigation" aria-label="Main">
<div class="sphinxsidebarwrapper">
<p class="logo"><a href="../index.html">
<img class="logo" src="../_static/logo.svg" alt="Logo of The Linux Kernel"/>
</a></p>
<h1 class="logo"><a href="../index.html">The Linux Kernel</a></h1>
<p class="blurb">6.18.50</p>
<search id="searchbox" style="display: none" role="search">
<h3 id="searchlabel">Quick search</h3>
<div class="searchformwrapper">
<form class="search" action="../search.html" method="get">
<input type="text" name="q" aria-labelledby="searchlabel" autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false"/>
<input type="submit" value="Go" />
</form>
</div>
</search>
<script>document.getElementById('searchbox').style.display = "block"</script>
<p>
<h3 class="kernel-toc-contents">Contents</h3>
<input type="checkbox" class="kernel-toc-toggle" id = "kernel-toc-toggle" checked>
<label class="kernel-toc-title" for="kernel-toc-toggle"></label>
<div class="kerneltoc" id="kerneltoc">
<ul>
<li class="toctree-l1"><a class="reference internal" href="../process/development-process.html">Development process</a></li>
<li class="toctree-l1"><a class="reference internal" href="../process/submitting-patches.html">Submitting patches</a></li>
<li class="toctree-l1"><a class="reference internal" href="../process/code-of-conduct.html">Code of conduct</a></li>
<li class="toctree-l1"><a class="reference internal" href="../maintainer/index.html">Maintainer handbook</a></li>
<li class="toctree-l1"><a class="reference internal" href="../process/index.html">All development-process docs</a></li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../core-api/index.html">Core API</a></li>
<li class="toctree-l1"><a class="reference internal" href="../driver-api/index.html">Driver APIs</a></li>
<li class="toctree-l1"><a class="reference internal" href="../subsystem-apis.html">Subsystems</a></li>
<li class="toctree-l1"><a class="reference internal" href="../locking/index.html">Locking</a></li>
</ul>
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../process/license-rules.html">Licensing rules</a></li>
<li class="toctree-l1"><a class="reference internal" href="../doc-guide/index.html">Writing documentation</a></li>
<li class="toctree-l1"><a class="reference internal" href="../dev-tools/index.html">Development tools</a></li>
<li class="toctree-l1"><a class="reference internal" href="../dev-tools/testing-overview.html">Testing guide</a></li>
<li class="toctree-l1"><a class="reference internal" href="../kernel-hacking/index.html">Hacking guide</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="index.html">Tracing</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="index.html#introduction-to-tracing">Introduction to Tracing</a></li>
<li class="toctree-l2"><a class="reference internal" href="index.html#core-tracing-frameworks">Core Tracing Frameworks</a></li>
<li class="toctree-l2 current"><a class="reference internal" href="index.html#event-tracing-and-analysis">Event Tracing and Analysis</a><ul class="current">
<li class="toctree-l3 current"><a class="current reference internal" href="#">Event Tracing</a></li>
<li class="toctree-l3"><a class="reference internal" href="events-kmem.html">Subsystem Trace Points: kmem</a></li>
<li class="toctree-l3"><a class="reference internal" href="events-power.html">Subsystem Trace Points: power</a></li>
<li class="toctree-l3"><a class="reference internal" href="events-nmi.html">NMI Trace Events</a></li>
<li class="toctree-l3"><a class="reference internal" href="events-msr.html">MSR Trace Events</a></li>
<li class="toctree-l3"><a class="reference internal" href="events-pci.html">Subsystem Trace Points: PCI</a></li>
<li class="toctree-l3"><a class="reference internal" href="boottime-trace.html">Boot-time tracing</a></li>
<li class="toctree-l3"><a class="reference internal" href="histogram.html">Event Histograms</a></li>
<li class="toctree-l3"><a class="reference internal" href="histogram-design.html">Histogram Design Notes</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="index.html#hardware-and-performance-tracing">Hardware and Performance Tracing</a></li>
<li class="toctree-l2"><a class="reference internal" href="index.html#user-space-tracing">User-Space Tracing</a></li>
<li class="toctree-l2"><a class="reference internal" href="index.html#additional-resources">Additional Resources</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../fault-injection/index.html">Fault injection</a></li>
<li class="toctree-l1"><a class="reference internal" href="../livepatch/index.html">Livepatching</a></li>
<li class="toctree-l1"><a class="reference internal" href="../rust/index.html">Rust</a></li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../admin-guide/index.html">Administration</a></li>
<li class="toctree-l1"><a class="reference internal" href="../kbuild/index.html">Build system</a></li>
<li class="toctree-l1"><a class="reference internal" href="../admin-guide/reporting-issues.html">Reporting issues</a></li>
<li class="toctree-l1"><a class="reference internal" href="../tools/index.html">Userspace tools</a></li>
<li class="toctree-l1"><a class="reference internal" href="../userspace-api/index.html">Userspace API</a></li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../firmware-guide/index.html">Firmware</a></li>
<li class="toctree-l1"><a class="reference internal" href="../devicetree/index.html">Firmware and Devicetree</a></li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../arch/index.html">CPU architectures</a></li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../staging/index.html">Unsorted documentation</a></li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../translations/index.html">Translations</a></li>
</ul>
</div>
<script type="text/javascript"> <!--
var sbar = document.getElementsByClassName("sphinxsidebar")[0];
let currents = document.getElementsByClassName("current")
if (currents.length) {
sbar.scrollTop = currents[currents.length - 1].offsetTop;
}
--> </script>
<div role="note" aria-label="source link">
<h3>This Page</h3>
<ul class="this-page-menu">
<li><a href="../_sources/trace/events.rst.txt"
rel="nofollow">Show Source</a></li>
</ul>
</div>
</div>
</div>
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body" role="main">
<section id="event-tracing">
<h1>Event Tracing<a class="headerlink" href="#event-tracing" title="Link to this heading">¶</a></h1>
<dl class="field-list simple">
<dt class="field-odd">Author<span class="colon">:</span></dt>
<dd class="field-odd"><p>Theodore Ts’o</p>
</dd>
<dt class="field-even">Updated<span class="colon">:</span></dt>
<dd class="field-even"><p>Li Zefan and Tom Zanussi</p>
</dd>
</dl>
<section id="introduction">
<h2>1. Introduction<a class="headerlink" href="#introduction" title="Link to this heading">¶</a></h2>
<p>Tracepoints (see <a class="reference internal" href="tracepoints.html"><span class="doc">Using the Linux Kernel Tracepoints</span></a>) can be used
without creating custom kernel modules to register probe functions
using the event tracing infrastructure.</p>
<p>Not all tracepoints can be traced using the event tracing system;
the kernel developer must provide code snippets which define how the
tracing information is saved into the tracing buffer, and how the
tracing information should be printed.</p>
</section>
<section id="using-event-tracing">
<h2>2. Using Event Tracing<a class="headerlink" href="#using-event-tracing" title="Link to this heading">¶</a></h2>
<section id="via-the-set-event-interface">
<h3>2.1 Via the ‘set_event’ interface<a class="headerlink" href="#via-the-set-event-interface" title="Link to this heading">¶</a></h3>
<p>The events which are available for tracing can be found in the file
/sys/kernel/tracing/available_events.</p>
<p>To enable a particular event, such as ‘sched_wakeup’, simply echo it
to /sys/kernel/tracing/set_event. For example:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># echo sched_wakeup >> /sys/kernel/tracing/set_event
</pre></div>
</div>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>‘>>’ is necessary, otherwise it will firstly disable all the events.</p>
</div>
<p>To disable an event, echo the event name to the set_event file prefixed
with an exclamation point:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># echo '!sched_wakeup' >> /sys/kernel/tracing/set_event
</pre></div>
</div>
<p>To disable all events, echo an empty line to the set_event file:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># echo > /sys/kernel/tracing/set_event
</pre></div>
</div>
<p>To enable all events, echo <code class="docutils literal notranslate"><span class="pre">*:*</span></code> or <code class="docutils literal notranslate"><span class="pre">*:</span></code> to the set_event file:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># echo *:* > /sys/kernel/tracing/set_event
</pre></div>
</div>
<p>The events are organized into subsystems, such as ext4, irq, sched,
etc., and a full event name looks like this: <subsystem>:<event>. The
subsystem name is optional, but it is displayed in the available_events
file. All of the events in a subsystem can be specified via the syntax
<code class="docutils literal notranslate"><span class="pre"><subsystem>:*</span></code>; for example, to enable all irq events, you can use the
command:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># echo 'irq:*' > /sys/kernel/tracing/set_event
</pre></div>
</div>
<p>The set_event file may also be used to enable events associated to only
a specific module:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># echo ':mod:<module>' > /sys/kernel/tracing/set_event
</pre></div>
</div>
<p>Will enable all events in the module <code class="docutils literal notranslate"><span class="pre"><module></span></code>. If the module is not yet
loaded, the string will be saved and when a module is that matches <code class="docutils literal notranslate"><span class="pre"><module></span></code>
is loaded, then it will apply the enabling of events then.</p>
<p>The text before <code class="docutils literal notranslate"><span class="pre">:mod:</span></code> will be parsed to specify specific events that the
module creates:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># echo '<match>:mod:<module>' > /sys/kernel/tracing/set_event
</pre></div>
</div>
<p>The above will enable any system or event that <code class="docutils literal notranslate"><span class="pre"><match></span></code> matches. If
<code class="docutils literal notranslate"><span class="pre"><match></span></code> is <code class="docutils literal notranslate"><span class="pre">"*"</span></code> then it will match all events.</p>
<p>To enable only a specific event within a system:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># echo '<system>:<event>:mod:<module>' > /sys/kernel/tracing/set_event
</pre></div>
</div>
<p>If <code class="docutils literal notranslate"><span class="pre"><event></span></code> is <code class="docutils literal notranslate"><span class="pre">"*"</span></code> then it will match all events within the system
for a given module.</p>
</section>
<section id="via-the-enable-toggle">
<h3>2.2 Via the ‘enable’ toggle<a class="headerlink" href="#via-the-enable-toggle" title="Link to this heading">¶</a></h3>
<p>The events available are also listed in /sys/kernel/tracing/events/ hierarchy
of directories.</p>
<p>To enable event ‘sched_wakeup’:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># echo 1 > /sys/kernel/tracing/events/sched/sched_wakeup/enable
</pre></div>
</div>
<p>To disable it:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># echo 0 > /sys/kernel/tracing/events/sched/sched_wakeup/enable
</pre></div>
</div>
<p>To enable all events in sched subsystem:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># echo 1 > /sys/kernel/tracing/events/sched/enable
</pre></div>
</div>
<p>To enable all events:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># echo 1 > /sys/kernel/tracing/events/enable
</pre></div>
</div>
<p>When reading one of these enable files, there are four results:</p>
<blockquote>
<div><ul class="simple">
<li><p>0 - all events this file affects are disabled</p></li>
<li><p>1 - all events this file affects are enabled</p></li>
<li><p>X - there is a mixture of events enabled and disabled</p></li>
<li><p>? - this file does not affect any event</p></li>
</ul>
</div></blockquote>
</section>
<section id="boot-option">
<h3>2.3 Boot option<a class="headerlink" href="#boot-option" title="Link to this heading">¶</a></h3>
<p>In order to facilitate early boot debugging, use boot option:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>trace_event=[event-list]
</pre></div>
</div>
<p>event-list is a comma separated list of events. See section 2.1 for event
format.</p>
</section>
</section>
<section id="defining-an-event-enabled-tracepoint">
<h2>3. Defining an event-enabled tracepoint<a class="headerlink" href="#defining-an-event-enabled-tracepoint" title="Link to this heading">¶</a></h2>
<p>See The example provided in samples/trace_events</p>
</section>
<section id="event-formats">
<h2>4. Event formats<a class="headerlink" href="#event-formats" title="Link to this heading">¶</a></h2>
<p>Each trace event has a ‘format’ file associated with it that contains
a description of each field in a logged event. This information can
be used to parse the binary trace stream, and is also the place to
find the field names that can be used in event filters (see section 5).</p>
<p>It also displays the format string that will be used to print the
event in text mode, along with the event name and ID used for
profiling.</p>
<p>Every event has a set of <code class="docutils literal notranslate"><span class="pre">common</span></code> fields associated with it; these are
the fields prefixed with <code class="docutils literal notranslate"><span class="pre">common_</span></code>. The other fields vary between
events and correspond to the fields defined in the TRACE_EVENT
definition for that event.</p>
<p>Each field in the format has the form:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>field:field-type field-name; offset:N; size:N;
</pre></div>
</div>
<p>where offset is the offset of the field in the trace record and size
is the size of the data item, in bytes.</p>
<p>For example, here’s the information displayed for the ‘sched_wakeup’
event:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># cat /sys/kernel/tracing/events/sched/sched_wakeup/format
name: sched_wakeup
ID: 60
format:
field:unsigned short common_type; offset:0; size:2;
field:unsigned char common_flags; offset:2; size:1;
field:unsigned char common_preempt_count; offset:3; size:1;
field:int common_pid; offset:4; size:4;
field:int common_tgid; offset:8; size:4;
field:char comm[TASK_COMM_LEN]; offset:12; size:16;
field:pid_t pid; offset:28; size:4;
field:int prio; offset:32; size:4;
field:int success; offset:36; size:4;
field:int cpu; offset:40; size:4;
print fmt: "task %s:%d [%d] success=%d [%03d]", REC->comm, REC->pid,
REC->prio, REC->success, REC->cpu
</pre></div>
</div>
<p>This event contains 10 fields, the first 5 common and the remaining 5
event-specific. All the fields for this event are numeric, except for
‘comm’ which is a string, a distinction important for event filtering.</p>
</section>
<section id="event-filtering">
<h2>5. Event filtering<a class="headerlink" href="#event-filtering" title="Link to this heading">¶</a></h2>
<p>Trace events can be filtered in the kernel by associating boolean
‘filter expressions’ with them. As soon as an event is logged into
the trace buffer, its fields are checked against the filter expression
associated with that event type. An event with field values that
‘match’ the filter will appear in the trace output, and an event whose
values don’t match will be discarded. An event with no filter
associated with it matches everything, and is the default when no
filter has been set for an event.</p>
<section id="expression-syntax">
<h3>5.1 Expression syntax<a class="headerlink" href="#expression-syntax" title="Link to this heading">¶</a></h3>
<p>A filter expression consists of one or more ‘predicates’ that can be
combined using the logical operators ‘&&’ and ‘||’. A predicate is
simply a clause that compares the value of a field contained within a
logged event with a constant value and returns either 0 or 1 depending
on whether the field value matched (1) or didn’t match (0):</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>field-name relational-operator value
</pre></div>
</div>
<p>Parentheses can be used to provide arbitrary logical groupings and
double-quotes can be used to prevent the shell from interpreting
operators as shell metacharacters.</p>
<p>The field-names available for use in filters can be found in the
‘format’ files for trace events (see section 4).</p>
<p>The relational-operators depend on the type of the field being tested:</p>
<p>The operators available for numeric fields are:</p>
<p>==, !=, <, <=, >, >=, &</p>
<p>And for string fields they are:</p>
<p>==, !=, ~</p>
<p>The glob (~) accepts a wild card character (*,?) and character classes
([). For example:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>prev_comm ~ "*sh"
prev_comm ~ "sh*"
prev_comm ~ "*sh*"
prev_comm ~ "ba*sh"
</pre></div>
</div>
<p>If the field is a pointer that points into user space (for example
“filename” from sys_enter_openat), then you have to append “.ustring” to the
field name:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>filename.ustring ~ "password"
</pre></div>
</div>
<p>As the kernel will have to know how to retrieve the memory that the pointer
is at from user space.</p>
<p>You can convert any long type to a function address and search by function name:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>call_site.function == security_prepare_creds
</pre></div>
</div>
<p>The above will filter when the field “call_site” falls on the address within
“security_prepare_creds”. That is, it will compare the value of “call_site” and
the filter will return true if it is greater than or equal to the start of
the function “security_prepare_creds” and less than the end of that function.</p>
<p>The “.function” postfix can only be attached to values of size long, and can only
be compared with “==” or “!=”.</p>
<p>Cpumask fields or scalar fields that encode a CPU number can be filtered using
a user-provided cpumask in cpulist format. The format is as follows:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>CPUS{$cpulist}
</pre></div>
</div>
<p>Operators available to cpumask filtering are:</p>
<p>& (intersection), ==, !=</p>
<p>For example, this will filter events that have their .target_cpu field present
in the given cpumask:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>target_cpu & CPUS{17-42}
</pre></div>
</div>
</section>
<section id="setting-filters">
<h3>5.2 Setting filters<a class="headerlink" href="#setting-filters" title="Link to this heading">¶</a></h3>
<p>A filter for an individual event is set by writing a filter expression
to the ‘filter’ file for the given event.</p>
<p>For example:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># cd /sys/kernel/tracing/events/sched/sched_wakeup
# echo "common_preempt_count > 4" > filter
</pre></div>
</div>
<p>A slightly more involved example:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># cd /sys/kernel/tracing/events/signal/signal_generate
# echo "((sig >= 10 && sig < 15) || sig == 17) && comm != bash" > filter
</pre></div>
</div>
<p>If there is an error in the expression, you’ll get an ‘Invalid
argument’ error when setting it, and the erroneous string along with
an error message can be seen by looking at the filter e.g.:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># cd /sys/kernel/tracing/events/signal/signal_generate
# echo "((sig >= 10 && sig < 15) || dsig == 17) && comm != bash" > filter
-bash: echo: write error: Invalid argument
# cat filter
((sig >= 10 && sig < 15) || dsig == 17) && comm != bash
^
parse_error: Field not found
</pre></div>
</div>
<p>Currently the caret (‘^’) for an error always appears at the beginning of
the filter string; the error message should still be useful though
even without more accurate position info.</p>
</section>
<section id="filter-limitations">
<h3>5.2.1 Filter limitations<a class="headerlink" href="#filter-limitations" title="Link to this heading">¶</a></h3>
<p>If a filter is placed on a string pointer <code class="docutils literal notranslate"><span class="pre">(char</span> <span class="pre">*)</span></code> that does not point
to a string on the ring buffer, but instead points to kernel or user space
memory, then, for safety reasons, at most 1024 bytes of the content is
copied onto a temporary buffer to do the compare. If the copy of the memory
faults (the pointer points to memory that should not be accessed), then the
string compare will be treated as not matching.</p>
</section>
<section id="clearing-filters">
<h3>5.3 Clearing filters<a class="headerlink" href="#clearing-filters" title="Link to this heading">¶</a></h3>
<p>To clear the filter for an event, write a ‘0’ to the event’s filter
file.</p>
<p>To clear the filters for all events in a subsystem, write a ‘0’ to the
subsystem’s filter file.</p>
</section>
<section id="subsystem-filters">
<h3>5.4 Subsystem filters<a class="headerlink" href="#subsystem-filters" title="Link to this heading">¶</a></h3>
<p>For convenience, filters for every event in a subsystem can be set or
cleared as a group by writing a filter expression into the filter file
at the root of the subsystem. Note however, that if a filter for any
event within the subsystem lacks a field specified in the subsystem
filter, or if the filter can’t be applied for any other reason, the
filter for that event will retain its previous setting. This can
result in an unintended mixture of filters which could lead to
confusing (to the user who might think different filters are in
effect) trace output. Only filters that reference just the common
fields can be guaranteed to propagate successfully to all events.</p>
<p>Here are a few subsystem filter examples that also illustrate the
above points:</p>
<p>Clear the filters on all events in the sched subsystem:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># cd /sys/kernel/tracing/events/sched
# echo 0 > filter
# cat sched_switch/filter
none
# cat sched_wakeup/filter
none
</pre></div>
</div>
<p>Set a filter using only common fields for all events in the sched
subsystem (all events end up with the same filter):</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># cd /sys/kernel/tracing/events/sched
# echo common_pid == 0 > filter
# cat sched_switch/filter
common_pid == 0
# cat sched_wakeup/filter
common_pid == 0
</pre></div>
</div>
<p>Attempt to set a filter using a non-common field for all events in the
sched subsystem (all events but those that have a prev_pid field retain
their old filters):</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># cd /sys/kernel/tracing/events/sched
# echo prev_pid == 0 > filter
# cat sched_switch/filter
prev_pid == 0
# cat sched_wakeup/filter
common_pid == 0
</pre></div>
</div>
</section>
<section id="pid-filtering">
<h3>5.5 PID filtering<a class="headerlink" href="#pid-filtering" title="Link to this heading">¶</a></h3>
<p>The set_event_pid file in the same directory as the top events directory
exists, will filter all events from tracing any task that does not have the
PID listed in the set_event_pid file.</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># cd /sys/kernel/tracing
# echo $$ > set_event_pid
# echo 1 > events/enable
</pre></div>
</div>
<p>Will only trace events for the current task.</p>
<p>To add more PIDs without losing the PIDs already included, use ‘>>’.</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># echo 123 244 1 >> set_event_pid
</pre></div>
</div>
</section>
</section>
<section id="event-triggers">
<h2>6. Event triggers<a class="headerlink" href="#event-triggers" title="Link to this heading">¶</a></h2>
<p>Trace events can be made to conditionally invoke trigger ‘commands’
which can take various forms and are described in detail below;
examples would be enabling or disabling other trace events or invoking
a stack trace whenever the trace event is hit. Whenever a trace event
with attached triggers is invoked, the set of trigger commands
associated with that event is invoked. Any given trigger can
additionally have an event filter of the same form as described in
section 5 (Event filtering) associated with it - the command will only
be invoked if the event being invoked passes the associated filter.
If no filter is associated with the trigger, it always passes.</p>
<p>Triggers are added to and removed from a particular event by writing
trigger expressions to the ‘trigger’ file for the given event.</p>
<p>A given event can have any number of triggers associated with it,
subject to any restrictions that individual commands may have in that
regard.</p>
<p>Event triggers are implemented on top of “soft” mode, which means that
whenever a trace event has one or more triggers associated with it,
the event is activated even if it isn’t actually enabled, but is
disabled in a “soft” mode. That is, the tracepoint will be called,
but just will not be traced, unless of course it’s actually enabled.
This scheme allows triggers to be invoked even for events that aren’t
enabled, and also allows the current event filter implementation to be
used for conditionally invoking triggers.</p>
<p>The syntax for event triggers is roughly based on the syntax for
set_ftrace_filter ‘ftrace filter commands’ (see the ‘Filter commands’
section of <a class="reference internal" href="ftrace.html"><span class="doc">ftrace - Function Tracer</span></a>), but there are major
differences and the implementation isn’t currently tied to it in any
way, so beware about making generalizations between the two.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Writing into trace_marker (See <a class="reference internal" href="ftrace.html"><span class="doc">ftrace - Function Tracer</span></a>)
can also enable triggers that are written into
/sys/kernel/tracing/events/ftrace/print/trigger</p>
</div>
<section id="id1">
<h3>6.1 Expression syntax<a class="headerlink" href="#id1" title="Link to this heading">¶</a></h3>
<p>Triggers are added by echoing the command to the ‘trigger’ file:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># echo 'command[:count] [if filter]' > trigger
</pre></div>
</div>
<p>Triggers are removed by echoing the same command but starting with ‘!’
to the ‘trigger’ file:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># echo '!command[:count] [if filter]' > trigger
</pre></div>
</div>
<p>The [if filter] part isn’t used in matching commands when removing, so
leaving that off in a ‘!’ command will accomplish the same thing as
having it in.</p>
<p>The filter syntax is the same as that described in the ‘Event
filtering’ section above.</p>
<p>For ease of use, writing to the trigger file using ‘>’ currently just
adds or removes a single trigger and there’s no explicit ‘>>’ support
(‘>’ actually behaves like ‘>>’) or truncation support to remove all
triggers (you have to use ‘!’ for each one added.)</p>
</section>
<section id="supported-trigger-commands">
<h3>6.2 Supported trigger commands<a class="headerlink" href="#supported-trigger-commands" title="Link to this heading">¶</a></h3>
<p>The following commands are supported:</p>
<ul>
<li><p>enable_event/disable_event</p>
<p>These commands can enable or disable another trace event whenever
the triggering event is hit. When these commands are registered,
the other trace event is activated, but disabled in a “soft” mode.
That is, the tracepoint will be called, but just will not be traced.
The event tracepoint stays in this mode as long as there’s a trigger
in effect that can trigger it.</p>
<p>For example, the following trigger causes kmalloc events to be
traced when a read system call is entered, and the :1 at the end
specifies that this enablement happens only once:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># echo 'enable_event:kmem:kmalloc:1' > \
/sys/kernel/tracing/events/syscalls/sys_enter_read/trigger
</pre></div>
</div>
<p>The following trigger causes kmalloc events to stop being traced
when a read system call exits. This disablement happens on every
read system call exit:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># echo 'disable_event:kmem:kmalloc' > \
/sys/kernel/tracing/events/syscalls/sys_exit_read/trigger
</pre></div>
</div>
<p>The format is:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>enable_event:<system>:<event>[:count]
disable_event:<system>:<event>[:count]
</pre></div>
</div>
<p>To remove the above commands:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># echo '!enable_event:kmem:kmalloc:1' > \
/sys/kernel/tracing/events/syscalls/sys_enter_read/trigger
# echo '!disable_event:kmem:kmalloc' > \
/sys/kernel/tracing/events/syscalls/sys_exit_read/trigger
</pre></div>
</div>
<p>Note that there can be any number of enable/disable_event triggers
per triggering event, but there can only be one trigger per
triggered event. e.g. sys_enter_read can have triggers enabling both
kmem:kmalloc and sched:sched_switch, but can’t have two kmem:kmalloc
versions such as kmem:kmalloc and kmem:kmalloc:1 or ‘kmem:kmalloc if
bytes_req == 256’ and ‘kmem:kmalloc if bytes_alloc == 256’ (they
could be combined into a single filter on kmem:kmalloc though).</p>
</li>
<li><p>stacktrace</p>
<p>This command dumps a stacktrace in the trace buffer whenever the
triggering event occurs.</p>
<p>For example, the following trigger dumps a stacktrace every time the
kmalloc tracepoint is hit:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># echo 'stacktrace' > \
/sys/kernel/tracing/events/kmem/kmalloc/trigger
</pre></div>
</div>
<p>The following trigger dumps a stacktrace the first 5 times a kmalloc
request happens with a size >= 64K:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># echo 'stacktrace:5 if bytes_req >= 65536' > \
/sys/kernel/tracing/events/kmem/kmalloc/trigger
</pre></div>
</div>
<p>The format is:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>stacktrace[:count]
</pre></div>
</div>
<p>To remove the above commands:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># echo '!stacktrace' > \
/sys/kernel/tracing/events/kmem/kmalloc/trigger
# echo '!stacktrace:5 if bytes_req >= 65536' > \
/sys/kernel/tracing/events/kmem/kmalloc/trigger
</pre></div>
</div>
<p>The latter can also be removed more simply by the following (without
the filter):</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># echo '!stacktrace:5' > \
/sys/kernel/tracing/events/kmem/kmalloc/trigger
</pre></div>
</div>
<p>Note that there can be only one stacktrace trigger per triggering
event.</p>
</li>
<li><p>snapshot</p>
<p>This command causes a snapshot to be triggered whenever the
triggering event occurs.</p>
<p>The following command creates a snapshot every time a block request
queue is unplugged with a depth > 1. If you were tracing a set of
events or functions at the time, the snapshot trace buffer would
capture those events when the trigger event occurred:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># echo 'snapshot if nr_rq > 1' > \
/sys/kernel/tracing/events/block/block_unplug/trigger
</pre></div>
</div>
<p>To only snapshot once:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># echo 'snapshot:1 if nr_rq > 1' > \
/sys/kernel/tracing/events/block/block_unplug/trigger
</pre></div>
</div>
<p>To remove the above commands:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># echo '!snapshot if nr_rq > 1' > \
/sys/kernel/tracing/events/block/block_unplug/trigger
# echo '!snapshot:1 if nr_rq > 1' > \
/sys/kernel/tracing/events/block/block_unplug/trigger
</pre></div>
</div>
<p>Note that there can be only one snapshot trigger per triggering
event.</p>
</li>
<li><p>traceon/traceoff</p>
<p>These commands turn tracing on and off when the specified events are
hit. The parameter determines how many times the tracing system is
turned on and off. If unspecified, there is no limit.</p>
<p>The following command turns tracing off the first time a block
request queue is unplugged with a depth > 1. If you were tracing a
set of events or functions at the time, you could then examine the
trace buffer to see the sequence of events that led up to the
trigger event:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># echo 'traceoff:1 if nr_rq > 1' > \
/sys/kernel/tracing/events/block/block_unplug/trigger
</pre></div>
</div>
<p>To always disable tracing when nr_rq > 1:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># echo 'traceoff if nr_rq > 1' > \
/sys/kernel/tracing/events/block/block_unplug/trigger
</pre></div>
</div>
<p>To remove the above commands:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># echo '!traceoff:1 if nr_rq > 1' > \
/sys/kernel/tracing/events/block/block_unplug/trigger
# echo '!traceoff if nr_rq > 1' > \
/sys/kernel/tracing/events/block/block_unplug/trigger
</pre></div>
</div>
<p>Note that there can be only one traceon or traceoff trigger per
triggering event.</p>
</li>
<li><p>hist</p>
<p>This command aggregates event hits into a hash table keyed on one or
more trace event format fields (or stacktrace) and a set of running
totals derived from one or more trace event format fields and/or
event counts (hitcount).</p>
<p>See <a class="reference internal" href="histogram.html"><span class="doc">Event Histograms</span></a> for details and examples.</p>
</li>
</ul>
</section>
</section>
<section id="in-kernel-trace-event-api">
<h2>7. In-kernel trace event API<a class="headerlink" href="#in-kernel-trace-event-api" title="Link to this heading">¶</a></h2>
<p>In most cases, the command-line interface to trace events is more than
sufficient. Sometimes, however, applications might find the need for
more complex relationships than can be expressed through a simple
series of linked command-line expressions, or putting together sets of
commands may be simply too cumbersome. An example might be an
application that needs to ‘listen’ to the trace stream in order to
maintain an in-kernel state machine detecting, for instance, when an
illegal kernel state occurs in the scheduler.</p>
<p>The trace event subsystem provides an in-kernel API allowing modules
or other kernel code to generate user-defined ‘synthetic’ events at
will, which can be used to either augment the existing trace stream
and/or signal that a particular important state has occurred.</p>
<p>A similar in-kernel API is also available for creating kprobe and
kretprobe events.</p>
<p>Both the synthetic event and k/ret/probe event APIs are built on top
of a lower-level “dynevent_cmd” event command API, which is also
available for more specialized applications, or as the basis of other
higher-level trace event APIs.</p>
<p>The API provided for these purposes is describe below and allows the
following:</p>
<blockquote>
<div><ul class="simple">
<li><p>dynamically creating synthetic event definitions</p></li>
<li><p>dynamically creating kprobe and kretprobe event definitions</p></li>
<li><p>tracing synthetic events from in-kernel code</p></li>
<li><p>the low-level “dynevent_cmd” API</p></li>
</ul>
</div></blockquote>
<section id="dynamically-creating-synthetic-event-definitions">
<h3>7.1 Dynamically creating synthetic event definitions<a class="headerlink" href="#dynamically-creating-synthetic-event-definitions" title="Link to this heading">¶</a></h3>
<p>There are a couple ways to create a new synthetic event from a kernel
module or other kernel code.</p>
<p>The first creates the event in one step, using <code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">synth_event_create()</span></code>.
In this method, the name of the event to create and an array defining
the fields is supplied to <code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">synth_event_create()</span></code>. If successful, a
synthetic event with that name and fields will exist following that
call. For example, to create a new “schedtest” synthetic event:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>ret = synth_event_create("schedtest", sched_fields,
ARRAY_SIZE(sched_fields), THIS_MODULE);
</pre></div>
</div>
<p>The sched_fields param in this example points to an array of <code class="xref c c-struct broken_xref docutils literal notranslate"><span class="pre">struct</span>
<span class="pre">synth_field_desc</span></code>, each of which describes an event field by type and
name:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>static struct synth_field_desc sched_fields[] = {
{ .type = "pid_t", .name = "next_pid_field" },
{ .type = "char[16]", .name = "next_comm_field" },
{ .type = "u64", .name = "ts_ns" },
{ .type = "u64", .name = "ts_ms" },
{ .type = "unsigned int", .name = "cpu" },
{ .type = "char[64]", .name = "my_string_field" },
{ .type = "int", .name = "my_int_field" },
};
</pre></div>
</div>
<p>See <code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">synth_field_size()</span></code> for available types.</p>
<p>If field_name contains [n], the field is considered to be a static array.</p>
<p>If field_names contains[] (no subscript), the field is considered to
be a dynamic array, which will only take as much space in the event as
is required to hold the array.</p>
<p>Because space for an event is reserved before assigning field values
to the event, using dynamic arrays implies that the piecewise
in-kernel API described below can’t be used with dynamic arrays. The
other non-piecewise in-kernel APIs can, however, be used with dynamic
arrays.</p>
<p>If the event is created from within a module, a pointer to the module
must be passed to <code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">synth_event_create()</span></code>. This will ensure that the
trace buffer won’t contain unreadable events when the module is
removed.</p>
<p>At this point, the event object is ready to be used for generating new
events.</p>
<p>In the second method, the event is created in several steps. This
allows events to be created dynamically and without the need to create
and populate an array of fields beforehand.</p>
<p>To use this method, an empty or partially empty synthetic event should
first be created using <code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">synth_event_gen_cmd_start()</span></code> or
<code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">synth_event_gen_cmd_array_start()</span></code>. For <code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">synth_event_gen_cmd_start()</span></code>,
the name of the event along with one or more pairs of args each pair
representing a ‘type field_name;’ field specification should be
supplied. For <code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">synth_event_gen_cmd_array_start()</span></code>, the name of the
event along with an array of <code class="xref c c-struct broken_xref docutils literal notranslate"><span class="pre">struct</span> <span class="pre">synth_field_desc</span></code> should be
supplied. Before calling <code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">synth_event_gen_cmd_start()</span></code> or
<code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">synth_event_gen_cmd_array_start()</span></code>, the user should create and
initialize a dynevent_cmd object using <code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">synth_event_cmd_init()</span></code>.</p>
<p>For example, to create a new “schedtest” synthetic event with two
fields:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>struct dynevent_cmd cmd;
char *buf;
/* Create a buffer to hold the generated command */
buf = kzalloc(MAX_DYNEVENT_CMD_LEN, GFP_KERNEL);
/* Before generating the command, initialize the cmd object */
synth_event_cmd_init(&cmd, buf, MAX_DYNEVENT_CMD_LEN);
ret = synth_event_gen_cmd_start(&cmd, "schedtest", THIS_MODULE,
"pid_t", "next_pid_field",
"u64", "ts_ns");
</pre></div>
</div>
<p>Alternatively, using an array of <code class="xref c c-struct broken_xref docutils literal notranslate"><span class="pre">struct</span> <span class="pre">synth_field_desc</span></code> fields
containing the same information:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>ret = synth_event_gen_cmd_array_start(&cmd, "schedtest", THIS_MODULE,
fields, n_fields);
</pre></div>
</div>
<p>Once the synthetic event object has been created, it can then be
populated with more fields. Fields are added one by one using
<code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">synth_event_add_field()</span></code>, supplying the dynevent_cmd object, a field
type, and a field name. For example, to add a new int field named
“intfield”, the following call should be made:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>ret = synth_event_add_field(&cmd, "int", "intfield");
</pre></div>
</div>
<p>See <code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">synth_field_size()</span></code> for available types. If field_name contains [n]
the field is considered to be an array.</p>
<p>A group of fields can also be added all at once using an array of
synth_field_desc with <code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">add_synth_fields()</span></code>. For example, this would add
just the first four sched_fields:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>ret = synth_event_add_fields(&cmd, sched_fields, 4);
</pre></div>
</div>
<p>If you already have a string of the form ‘type field_name’,
<code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">synth_event_add_field_str()</span></code> can be used to add it as-is; it will
also automatically append a ‘;’ to the string.</p>
<p>Once all the fields have been added, the event should be finalized and
registered by calling the <code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">synth_event_gen_cmd_end()</span></code> function:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>ret = synth_event_gen_cmd_end(&cmd);
</pre></div>
</div>
<p>At this point, the event object is ready to be used for tracing new
events.</p>
</section>
<section id="tracing-synthetic-events-from-in-kernel-code">
<h3>7.2 Tracing synthetic events from in-kernel code<a class="headerlink" href="#tracing-synthetic-events-from-in-kernel-code" title="Link to this heading">¶</a></h3>
<p>To trace a synthetic event, there are several options. The first
option is to trace the event in one call, using <code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">synth_event_trace()</span></code>
with a variable number of values, or <code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">synth_event_trace_array()</span></code> with an
array of values to be set. A second option can be used to avoid the
need for a pre-formed array of values or list of arguments, via
<code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">synth_event_trace_start()</span></code> and <code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">synth_event_trace_end()</span></code> along with
<code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">synth_event_add_next_val()</span></code> or <code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">synth_event_add_val()</span></code> to add the values
piecewise.</p>
</section>
<section id="tracing-a-synthetic-event-all-at-once">
<h3>7.2.1 Tracing a synthetic event all at once<a class="headerlink" href="#tracing-a-synthetic-event-all-at-once" title="Link to this heading">¶</a></h3>
<p>To trace a synthetic event all at once, the <code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">synth_event_trace()</span></code> or
<code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">synth_event_trace_array()</span></code> functions can be used.</p>
<p>The <code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">synth_event_trace()</span></code> function is passed the trace_event_file
representing the synthetic event (which can be retrieved using
<code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">trace_get_event_file()</span></code> using the synthetic event name, “synthetic” as
the system name, and the trace instance name (NULL if using the global
trace array)), along with an variable number of u64 args, one for each
synthetic event field, and the number of values being passed.</p>
<p>So, to trace an event corresponding to the synthetic event definition
above, code like the following could be used:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>ret = synth_event_trace(create_synth_test, 7, /* number of values */
444, /* next_pid_field */
(u64)"clackers", /* next_comm_field */
1000000, /* ts_ns */
1000, /* ts_ms */
smp_processor_id(),/* cpu */
(u64)"Thneed", /* my_string_field */
999); /* my_int_field */
</pre></div>
</div>
<p>All vals should be cast to u64, and string vals are just pointers to
strings, cast to u64. Strings will be copied into space reserved in
the event for the string, using these pointers.</p>
<p>Alternatively, the <code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">synth_event_trace_array()</span></code> function can be used to
accomplish the same thing. It is passed the trace_event_file
representing the synthetic event (which can be retrieved using
<code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">trace_get_event_file()</span></code> using the synthetic event name, “synthetic” as
the system name, and the trace instance name (NULL if using the global
trace array)), along with an array of u64, one for each synthetic
event field.</p>
<p>To trace an event corresponding to the synthetic event definition
above, code like the following could be used:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>u64 vals[7];
vals[0] = 777; /* next_pid_field */
vals[1] = (u64)"tiddlywinks"; /* next_comm_field */
vals[2] = 1000000; /* ts_ns */
vals[3] = 1000; /* ts_ms */
vals[4] = smp_processor_id(); /* cpu */
vals[5] = (u64)"thneed"; /* my_string_field */
vals[6] = 398; /* my_int_field */
</pre></div>
</div>
<p>The ‘vals’ array is just an array of u64, the number of which must
match the number of field in the synthetic event, and which must be in
the same order as the synthetic event fields.</p>
<p>All vals should be cast to u64, and string vals are just pointers to
strings, cast to u64. Strings will be copied into space reserved in
the event for the string, using these pointers.</p>
<p>In order to trace a synthetic event, a pointer to the trace event file
is needed. The <code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">trace_get_event_file()</span></code> function can be used to get
it - it will find the file in the given trace instance (in this case
NULL since the top trace array is being used) while at the same time
preventing the instance containing it from going away:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>schedtest_event_file = trace_get_event_file(NULL, "synthetic",
"schedtest");
</pre></div>
</div>
<p>Before tracing the event, it should be enabled in some way, otherwise
the synthetic event won’t actually show up in the trace buffer.</p>
<p>To enable a synthetic event from the kernel, <code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">trace_array_set_clr_event()</span></code>
can be used (which is not specific to synthetic events, so does need
the “synthetic” system name to be specified explicitly).</p>
<p>To enable the event, pass ‘true’ to it:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>trace_array_set_clr_event(schedtest_event_file->tr,
"synthetic", "schedtest", true);
</pre></div>
</div>
<p>To disable it pass false:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>trace_array_set_clr_event(schedtest_event_file->tr,
"synthetic", "schedtest", false);
</pre></div>
</div>
<p>Finally, <code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">synth_event_trace_array()</span></code> can be used to actually trace the
event, which should be visible in the trace buffer afterwards:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>ret = synth_event_trace_array(schedtest_event_file, vals,
ARRAY_SIZE(vals));
</pre></div>
</div>
<p>To remove the synthetic event, the event should be disabled, and the
trace instance should be ‘put’ back using <code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">trace_put_event_file()</span></code>:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>trace_array_set_clr_event(schedtest_event_file->tr,
"synthetic", "schedtest", false);
trace_put_event_file(schedtest_event_file);
</pre></div>
</div>
<p>If those have been successful, <code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">synth_event_delete()</span></code> can be called to
remove the event:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>ret = synth_event_delete("schedtest");
</pre></div>
</div>
</section>
<section id="tracing-a-synthetic-event-piecewise">
<h3>7.2.2 Tracing a synthetic event piecewise<a class="headerlink" href="#tracing-a-synthetic-event-piecewise" title="Link to this heading">¶</a></h3>
<p>To trace a synthetic using the piecewise method described above, the
<code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">synth_event_trace_start()</span></code> function is used to ‘open’ the synthetic
event trace:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>struct synth_event_trace_state trace_state;
ret = synth_event_trace_start(schedtest_event_file, &trace_state);
</pre></div>
</div>
<p>It’s passed the trace_event_file representing the synthetic event
using the same methods as described above, along with a pointer to a
<code class="xref c c-struct broken_xref docutils literal notranslate"><span class="pre">struct</span> <span class="pre">synth_event_trace_state</span></code> object, which will be zeroed before use and
used to maintain state between this and following calls.</p>
<p>Once the event has been opened, which means space for it has been
reserved in the trace buffer, the individual fields can be set. There
are two ways to do that, either one after another for each field in
the event, which requires no lookups, or by name, which does. The
tradeoff is flexibility in doing the assignments vs the cost of a
lookup per field.</p>
<p>To assign the values one after the other without lookups,
<code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">synth_event_add_next_val()</span></code> should be used. Each call is passed the
same synth_event_trace_state object used in the <code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">synth_event_trace_start()</span></code>,
along with the value to set the next field in the event. After each
field is set, the ‘cursor’ points to the next field, which will be set
by the subsequent call, continuing until all the fields have been set
in order. The same sequence of calls as in the above examples using
this method would be (without error-handling code):</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>/* next_pid_field */
ret = synth_event_add_next_val(777, &trace_state);
/* next_comm_field */
ret = synth_event_add_next_val((u64)"slinky", &trace_state);
/* ts_ns */
ret = synth_event_add_next_val(1000000, &trace_state);
/* ts_ms */
ret = synth_event_add_next_val(1000, &trace_state);
/* cpu */
ret = synth_event_add_next_val(smp_processor_id(), &trace_state);
/* my_string_field */
ret = synth_event_add_next_val((u64)"thneed_2.01", &trace_state);
/* my_int_field */
ret = synth_event_add_next_val(395, &trace_state);
</pre></div>
</div>
<p>To assign the values in any order, <code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">synth_event_add_val()</span></code> should be
used. Each call is passed the same synth_event_trace_state object used in
the <code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">synth_event_trace_start()</span></code>, along with the field name of the field
to set and the value to set it to. The same sequence of calls as in
the above examples using this method would be (without error-handling
code):</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>ret = synth_event_add_val("next_pid_field", 777, &trace_state);
ret = synth_event_add_val("next_comm_field", (u64)"silly putty",
&trace_state);
ret = synth_event_add_val("ts_ns", 1000000, &trace_state);
ret = synth_event_add_val("ts_ms", 1000, &trace_state);
ret = synth_event_add_val("cpu", smp_processor_id(), &trace_state);
ret = synth_event_add_val("my_string_field", (u64)"thneed_9",
&trace_state);
ret = synth_event_add_val("my_int_field", 3999, &trace_state);
</pre></div>
</div>
<p>Note that <code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">synth_event_add_next_val()</span></code> and <code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">synth_event_add_val()</span></code> are
incompatible if used within the same trace of an event - either one
can be used but not both at the same time.</p>
<p>Finally, the event won’t be actually traced until it’s ‘closed’,
which is done using <code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">synth_event_trace_end()</span></code>, which takes only the
<code class="xref c c-struct broken_xref docutils literal notranslate"><span class="pre">struct</span> <span class="pre">synth_event_trace_state</span></code> object used in the previous calls:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>ret = synth_event_trace_end(&trace_state);
</pre></div>
</div>
<p>Note that <code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">synth_event_trace_end()</span></code> must be called at the end regardless
of whether any of the add calls failed (say due to a bad field name
being passed in).</p>
</section>
<section id="dynamically-creating-kprobe-and-kretprobe-event-definitions">
<h3>7.3 Dynamically creating kprobe and kretprobe event definitions<a class="headerlink" href="#dynamically-creating-kprobe-and-kretprobe-event-definitions" title="Link to this heading">¶</a></h3>
<p>To create a kprobe or kretprobe trace event from kernel code, the
<code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">kprobe_event_gen_cmd_start()</span></code> or <code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">kretprobe_event_gen_cmd_start()</span></code>
functions can be used.</p>
<p>To create a kprobe event, an empty or partially empty kprobe event
should first be created using <code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">kprobe_event_gen_cmd_start()</span></code>. The name
of the event and the probe location should be specified along with one
or args each representing a probe field should be supplied to this
function. Before calling <code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">kprobe_event_gen_cmd_start()</span></code>, the user
should create and initialize a dynevent_cmd object using
<code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">kprobe_event_cmd_init()</span></code>.</p>
<p>For example, to create a new “schedtest” kprobe event with two fields:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>struct dynevent_cmd cmd;
char *buf;
/* Create a buffer to hold the generated command */
buf = kzalloc(MAX_DYNEVENT_CMD_LEN, GFP_KERNEL);
/* Before generating the command, initialize the cmd object */
kprobe_event_cmd_init(&cmd, buf, MAX_DYNEVENT_CMD_LEN);
/*
* Define the gen_kprobe_test event with the first 2 kprobe
* fields.
*/
ret = kprobe_event_gen_cmd_start(&cmd, "gen_kprobe_test", "do_sys_open",
"dfd=%ax", "filename=%dx");
</pre></div>
</div>
<p>Once the kprobe event object has been created, it can then be
populated with more fields. Fields can be added using
<code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">kprobe_event_add_fields()</span></code>, supplying the dynevent_cmd object along
with a variable arg list of probe fields. For example, to add a
couple additional fields, the following call could be made:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>ret = kprobe_event_add_fields(&cmd, "flags=%cx", "mode=+4($stack)");
</pre></div>
</div>
<p>Once all the fields have been added, the event should be finalized and
registered by calling the <code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">kprobe_event_gen_cmd_end()</span></code> or
<code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">kretprobe_event_gen_cmd_end()</span></code> functions, depending on whether a kprobe
or kretprobe command was started:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>ret = kprobe_event_gen_cmd_end(&cmd);
</pre></div>
</div>
<p>or:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>ret = kretprobe_event_gen_cmd_end(&cmd);
</pre></div>
</div>
<p>At this point, the event object is ready to be used for tracing new
events.</p>
<p>Similarly, a kretprobe event can be created using
<code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">kretprobe_event_gen_cmd_start()</span></code> with a probe name and location and
additional params such as $retval:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>ret = kretprobe_event_gen_cmd_start(&cmd, "gen_kretprobe_test",
"do_sys_open", "$retval");
</pre></div>
</div>
<p>Similar to the synthetic event case, code like the following can be
used to enable the newly created kprobe event:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>gen_kprobe_test = trace_get_event_file(NULL, "kprobes", "gen_kprobe_test");
ret = trace_array_set_clr_event(gen_kprobe_test->tr,
"kprobes", "gen_kprobe_test", true);
</pre></div>
</div>
<p>Finally, also similar to synthetic events, the following code can be
used to give the kprobe event file back and delete the event:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>trace_put_event_file(gen_kprobe_test);
ret = kprobe_event_delete("gen_kprobe_test");
</pre></div>
</div>
</section>
<section id="the-dynevent-cmd-low-level-api">
<h3>7.4 The “dynevent_cmd” low-level API<a class="headerlink" href="#the-dynevent-cmd-low-level-api" title="Link to this heading">¶</a></h3>
<p>Both the in-kernel synthetic event and kprobe interfaces are built on
top of a lower-level “dynevent_cmd” interface. This interface is
meant to provide the basis for higher-level interfaces such as the
synthetic and kprobe interfaces, which can be used as examples.</p>
<p>The basic idea is simple and amounts to providing a general-purpose
layer that can be used to generate trace event commands. The
generated command strings can then be passed to the command-parsing
and event creation code that already exists in the trace event
subsystem for creating the corresponding trace events.</p>
<p>In a nutshell, the way it works is that the higher-level interface
code creates a <code class="xref c c-struct broken_xref docutils literal notranslate"><span class="pre">struct</span> <span class="pre">dynevent_cmd</span></code> object, then uses a couple
functions, <code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">dynevent_arg_add()</span></code> and <code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">dynevent_arg_pair_add()</span></code> to build up
a command string, which finally causes the command to be executed
using the <code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">dynevent_create()</span></code> function. The details of the interface
are described below.</p>
<p>The first step in building a new command string is to create and
initialize an instance of a dynevent_cmd. Here, for instance, we
create a dynevent_cmd on the stack and initialize it:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>struct dynevent_cmd cmd;
char *buf;
int ret;
buf = kzalloc(MAX_DYNEVENT_CMD_LEN, GFP_KERNEL);
dynevent_cmd_init(cmd, buf, maxlen, DYNEVENT_TYPE_FOO,
foo_event_run_command);
</pre></div>
</div>
<p>The dynevent_cmd initialization needs to be given a user-specified
buffer and the length of the buffer (MAX_DYNEVENT_CMD_LEN can be used
for this purpose - at 2k it’s generally too big to be comfortably put
on the stack, so is dynamically allocated), a dynevent type id, which
is meant to be used to check that further API calls are for the
correct command type, and a pointer to an event-specific <code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">run_command()</span></code>
callback that will be called to actually execute the event-specific
command function.</p>
<p>Once that’s done, the command string can by built up by successive
calls to argument-adding functions.</p>
<p>To add a single argument, define and initialize a <code class="xref c c-struct broken_xref docutils literal notranslate"><span class="pre">struct</span> <span class="pre">dynevent_arg</span></code>
or <code class="xref c c-struct broken_xref docutils literal notranslate"><span class="pre">struct</span> <span class="pre">dynevent_arg_pair</span></code> object. Here’s an example of the simplest
possible arg addition, which is simply to append the given string as
a whitespace-separated argument to the command:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>struct dynevent_arg arg;
dynevent_arg_init(&arg, NULL, 0);
arg.str = name;
ret = dynevent_arg_add(cmd, &arg);
</pre></div>
</div>
<p>The arg object is first initialized using <code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">dynevent_arg_init()</span></code> and in
this case the parameters are NULL or 0, which means there’s no
optional sanity-checking function or separator appended to the end of
the arg.</p>
<p>Here’s another more complicated example using an ‘arg pair’, which is
used to create an argument that consists of a couple components added
together as a unit, for example, a ‘type field_name;’ arg or a simple
expression arg e.g. ‘flags=%cx’:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>struct dynevent_arg_pair arg_pair;
dynevent_arg_pair_init(&arg_pair, dynevent_foo_check_arg_fn, 0, ';');
arg_pair.lhs = type;
arg_pair.rhs = name;
ret = dynevent_arg_pair_add(cmd, &arg_pair);
</pre></div>
</div>
<p>Again, the arg_pair is first initialized, in this case with a callback
function used to check the sanity of the args (for example, that
neither part of the pair is NULL), along with a character to be used
to add an operator between the pair (here none) and a separator to be
appended onto the end of the arg pair (here ‘;’).</p>
<p>There’s also a <code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">dynevent_str_add()</span></code> function that can be used to simply
add a string as-is, with no spaces, delimiters, or arg check.</p>
<p>Any number of dynevent_*<code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">_add()</span></code> calls can be made to build up the string
(until its length surpasses cmd->maxlen). When all the arguments have
been added and the command string is complete, the only thing left to
do is run the command, which happens by simply calling
<code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">dynevent_create()</span></code>:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>ret = dynevent_create(&cmd);
</pre></div>
</div>
<p>At that point, if the return value is 0, the dynamic event has been
created and is ready to use.</p>
<p>See the dynevent_cmd function definitions themselves for the details
of the API.</p>
</section>
</section>
</section>
</div>
</div>
</div>
<div class="clearer"></div>
</div>
<div class="footer">
©The kernel development community.
|
Powered by <a href="https://www.sphinx-doc.org/">Sphinx 8.1.3</a>
& <a href="https://alabaster.readthedocs.io">Alabaster 0.7.16</a>
|
<a href="../_sources/trace/events.rst.txt"
rel="nofollow">Page source</a>
</div>
</body>
</html>