<!DOCTYPE html>
<html lang="en" data-content_root="../">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Using the tracer for debugging — The Linux Kernel documentation</title>
<link rel="stylesheet" type="text/css" href="../_static/pygments.css?v=fa44fd50" />
<link rel="stylesheet" type="text/css" href="../_static/alabaster.css?v=3918102e" />
<script src="../_static/documentation_options.js?v=5929fcd5"></script>
<script src="../_static/doctools.js?v=9bcbadda"></script>
<script src="../_static/sphinx_highlight.js?v=dc90522c"></script>
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
<link rel="next" title="Using the Linux Kernel Tracepoints" href="tracepoints.html" />
<link rel="prev" title="Linux Tracing Technologies Guide" href="index.html" />
<link rel="stylesheet" href="../_static/custom.css" type="text/css" />
</head><body>
<div class="document">
<div class="sphinxsidebar" role="navigation" aria-label="Main">
<div class="sphinxsidebarwrapper">
<p class="logo"><a href="../index.html">
<img class="logo" src="../_static/logo.svg" alt="Logo of The Linux Kernel"/>
</a></p>
<h1 class="logo"><a href="../index.html">The Linux Kernel</a></h1>
<p class="blurb">6.18.50</p>
<search id="searchbox" style="display: none" role="search">
<h3 id="searchlabel">Quick search</h3>
<div class="searchformwrapper">
<form class="search" action="../search.html" method="get">
<input type="text" name="q" aria-labelledby="searchlabel" autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false"/>
<input type="submit" value="Go" />
</form>
</div>
</search>
<script>document.getElementById('searchbox').style.display = "block"</script>
<p>
<h3 class="kernel-toc-contents">Contents</h3>
<input type="checkbox" class="kernel-toc-toggle" id = "kernel-toc-toggle" checked>
<label class="kernel-toc-title" for="kernel-toc-toggle"></label>
<div class="kerneltoc" id="kerneltoc">
<ul>
<li class="toctree-l1"><a class="reference internal" href="../process/development-process.html">Development process</a></li>
<li class="toctree-l1"><a class="reference internal" href="../process/submitting-patches.html">Submitting patches</a></li>
<li class="toctree-l1"><a class="reference internal" href="../process/code-of-conduct.html">Code of conduct</a></li>
<li class="toctree-l1"><a class="reference internal" href="../maintainer/index.html">Maintainer handbook</a></li>
<li class="toctree-l1"><a class="reference internal" href="../process/index.html">All development-process docs</a></li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../core-api/index.html">Core API</a></li>
<li class="toctree-l1"><a class="reference internal" href="../driver-api/index.html">Driver APIs</a></li>
<li class="toctree-l1"><a class="reference internal" href="../subsystem-apis.html">Subsystems</a></li>
<li class="toctree-l1"><a class="reference internal" href="../locking/index.html">Locking</a></li>
</ul>
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../process/license-rules.html">Licensing rules</a></li>
<li class="toctree-l1"><a class="reference internal" href="../doc-guide/index.html">Writing documentation</a></li>
<li class="toctree-l1"><a class="reference internal" href="../dev-tools/index.html">Development tools</a></li>
<li class="toctree-l1"><a class="reference internal" href="../dev-tools/testing-overview.html">Testing guide</a></li>
<li class="toctree-l1"><a class="reference internal" href="../kernel-hacking/index.html">Hacking guide</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="index.html">Tracing</a><ul class="current">
<li class="toctree-l2 current"><a class="reference internal" href="index.html#introduction-to-tracing">Introduction to Tracing</a><ul class="current">
<li class="toctree-l3 current"><a class="current reference internal" href="#">Using the tracer for debugging</a></li>
<li class="toctree-l3"><a class="reference internal" href="tracepoints.html">Using the Linux Kernel Tracepoints</a></li>
<li class="toctree-l3"><a class="reference internal" href="tracepoint-analysis.html">Notes on Analysing Behaviour Using Events and Tracepoints</a></li>
<li class="toctree-l3"><a class="reference internal" href="ring-buffer-map.html">Tracefs ring-buffer memory mapping</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="index.html#core-tracing-frameworks">Core Tracing Frameworks</a></li>
<li class="toctree-l2"><a class="reference internal" href="index.html#event-tracing-and-analysis">Event Tracing and Analysis</a></li>
<li class="toctree-l2"><a class="reference internal" href="index.html#hardware-and-performance-tracing">Hardware and Performance Tracing</a></li>
<li class="toctree-l2"><a class="reference internal" href="index.html#user-space-tracing">User-Space Tracing</a></li>
<li class="toctree-l2"><a class="reference internal" href="index.html#additional-resources">Additional Resources</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../fault-injection/index.html">Fault injection</a></li>
<li class="toctree-l1"><a class="reference internal" href="../livepatch/index.html">Livepatching</a></li>
<li class="toctree-l1"><a class="reference internal" href="../rust/index.html">Rust</a></li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../admin-guide/index.html">Administration</a></li>
<li class="toctree-l1"><a class="reference internal" href="../kbuild/index.html">Build system</a></li>
<li class="toctree-l1"><a class="reference internal" href="../admin-guide/reporting-issues.html">Reporting issues</a></li>
<li class="toctree-l1"><a class="reference internal" href="../tools/index.html">Userspace tools</a></li>
<li class="toctree-l1"><a class="reference internal" href="../userspace-api/index.html">Userspace API</a></li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../firmware-guide/index.html">Firmware</a></li>
<li class="toctree-l1"><a class="reference internal" href="../devicetree/index.html">Firmware and Devicetree</a></li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../arch/index.html">CPU architectures</a></li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../staging/index.html">Unsorted documentation</a></li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../translations/index.html">Translations</a></li>
</ul>
</div>
<script type="text/javascript"> <!--
var sbar = document.getElementsByClassName("sphinxsidebar")[0];
let currents = document.getElementsByClassName("current")
if (currents.length) {
sbar.scrollTop = currents[currents.length - 1].offsetTop;
}
--> </script>
<div role="note" aria-label="source link">
<h3>This Page</h3>
<ul class="this-page-menu">
<li><a href="../_sources/trace/debugging.rst.txt"
rel="nofollow">Show Source</a></li>
</ul>
</div>
</div>
</div>
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body" role="main">
<section id="using-the-tracer-for-debugging">
<h1>Using the tracer for debugging<a class="headerlink" href="#using-the-tracer-for-debugging" title="Link to this heading">¶</a></h1>
<p>Copyright 2024 Google LLC.</p>
<dl class="field-list simple">
<dt class="field-odd">Author<span class="colon">:</span></dt>
<dd class="field-odd"><p>Steven Rostedt <<a class="reference external" href="mailto:rostedt%40goodmis.org">rostedt<span>@</span>goodmis<span>.</span>org</a>></p>
</dd>
<dt class="field-even">License<span class="colon">:</span></dt>
<dd class="field-even"><p>The GNU Free Documentation License, Version 1.2
(dual licensed under the GPL v2)</p>
</dd>
</dl>
<ul class="simple">
<li><p>Written for: 6.12</p></li>
</ul>
<section id="introduction">
<h2>Introduction<a class="headerlink" href="#introduction" title="Link to this heading">¶</a></h2>
<p>The tracing infrastructure can be very useful for debugging the Linux
kernel. This document is a place to add various methods of using the tracer
for debugging.</p>
<p>First, make sure that the tracefs file system is mounted:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>$ sudo mount -t tracefs tracefs /sys/kernel/tracing
</pre></div>
</div>
</section>
<section id="using-trace-printk">
<h2>Using trace_printk()<a class="headerlink" href="#using-trace-printk" title="Link to this heading">¶</a></h2>
<p><code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">trace_printk()</span></code> is a very lightweight utility that can be used in any context
inside the kernel, with the exception of “noinstr” sections. It can be used
in normal, softirq, interrupt and even NMI context. The trace data is
written to the tracing ring buffer in a lockless way. To make it even
lighter weight, when possible, it will only record the pointer to the format
string, and save the raw arguments into the buffer. The format and the
arguments will be post processed when the ring buffer is read. This way the
<code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">trace_printk()</span></code> format conversions are not done during the hot path, where
the trace is being recorded.</p>
<p><code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">trace_printk()</span></code> is meant only for debugging, and should never be added into
a subsystem of the kernel. If you need debugging traces, add trace events
instead. If a <code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">trace_printk()</span></code> is found in the kernel, the following will
appear in the dmesg:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>**********************************************************
** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE **
** **
** trace_printk() being used. Allocating extra memory. **
** **
** This means that this is a DEBUG kernel and it is **
** unsafe for production use. **
** **
** If you see this message and you are not debugging **
** the kernel, report this immediately to your vendor! **
** **
** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE **
**********************************************************
</pre></div>
</div>
</section>
<section id="debugging-kernel-crashes">
<h2>Debugging kernel crashes<a class="headerlink" href="#debugging-kernel-crashes" title="Link to this heading">¶</a></h2>
<p>There is various methods of acquiring the state of the system when a kernel
crash occurs. This could be from the oops message in printk, or one could
use kexec/kdump. But these just show what happened at the time of the crash.
It can be very useful in knowing what happened up to the point of the crash.
The tracing ring buffer, by default, is a circular buffer that will
overwrite older events with newer ones. When a crash happens, the content of
the ring buffer will be all the events that lead up to the crash.</p>
<p>There are several kernel command line parameters that can be used to help in
this. The first is “ftrace_dump_on_oops”. This will dump the tracing ring
buffer when a oops occurs to the console. This can be useful if the console
is being logged somewhere. If a serial console is used, it may be prudent to
make sure the ring buffer is relatively small, otherwise the dumping of the
ring buffer may take several minutes to hours to finish. Here’s an example
of the kernel command line:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>ftrace_dump_on_oops trace_buf_size=50K
</pre></div>
</div>
<p>Note, the tracing buffer is made up of per CPU buffers where each of these
buffers is broken up into sub-buffers that are by default PAGE_SIZE. The
above trace_buf_size option above sets each of the per CPU buffers to 50K,
so, on a machine with 8 CPUs, that’s actually 400K total.</p>
</section>
<section id="persistent-buffers-across-boots">
<h2>Persistent buffers across boots<a class="headerlink" href="#persistent-buffers-across-boots" title="Link to this heading">¶</a></h2>
<p>If the system memory allows it, the tracing ring buffer can be specified at
a specific location in memory. If the location is the same across boots and
the memory is not modified, the tracing buffer can be retrieved from the
following boot. There’s two ways to reserve memory for the use of the ring
buffer.</p>
<p>The more reliable way (on x86) is to reserve memory with the “memmap” kernel
command line option and then use that memory for the trace_instance. This
requires a bit of knowledge of the physical memory layout of the system. The
advantage of using this method, is that the memory for the ring buffer will
always be the same:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>memmap==12M$0x284500000 trace_instance=boot_map@0x284500000:12M
</pre></div>
</div>
<p>The memmap above reserves 12 megabytes of memory at the physical memory
location 0x284500000. Then the trace_instance option will create a trace
instance “boot_map” at that same location with the same amount of memory
reserved. As the ring buffer is broke up into per CPU buffers, the 12
megabytes will be broken up evenly between those CPUs. If you have 8 CPUs,
each per CPU ring buffer will be 1.5 megabytes in size. Note, that also
includes meta data, so the amount of memory actually used by the ring buffer
will be slightly smaller.</p>
<p>Another more generic but less robust way to allocate a ring buffer mapping
at boot is with the “reserve_mem” option:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>reserve_mem=12M:4096:trace trace_instance=boot_map@trace
</pre></div>
</div>
<p>The reserve_mem option above will find 12 megabytes that are available at
boot up, and align it by 4096 bytes. It will label this memory as “trace”
that can be used by later command line options.</p>
<p>The trace_instance option creates a “boot_map” instance and will use the
memory reserved by reserve_mem that was labeled as “trace”. This method is
more generic but may not be as reliable. Due to KASLR, the memory reserved
by reserve_mem may not be located at the same location. If this happens,
then the ring buffer will not be from the previous boot and will be reset.</p>
<p>Sometimes, by using a larger alignment, it can keep KASLR from moving things
around in such a way that it will move the location of the reserve_mem. By
using a larger alignment, you may find better that the buffer is more
consistent to where it is placed:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>reserve_mem=12M:0x2000000:trace trace_instance=boot_map@trace
</pre></div>
</div>
<p>On boot up, the memory reserved for the ring buffer is validated. It will go
through a series of tests to make sure that the ring buffer contains valid
data. If it is, it will then set it up to be available to read from the
instance. If it fails any of the tests, it will clear the entire ring buffer
and initialize it as new.</p>
<p>The layout of this mapped memory may not be consistent from kernel to
kernel, so only the same kernel is guaranteed to work if the mapping is
preserved. Switching to a different kernel version may find a different
layout and mark the buffer as invalid.</p>
<p>NB: Both the mapped address and size must be page aligned for the architecture.</p>
</section>
<section id="using-trace-printk-in-the-boot-instance">
<h2>Using trace_printk() in the boot instance<a class="headerlink" href="#using-trace-printk-in-the-boot-instance" title="Link to this heading">¶</a></h2>
<p>By default, the content of <code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">trace_printk()</span></code> goes into the top level tracing
instance. But this instance is never preserved across boots. To have the
<code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">trace_printk()</span></code> content, and some other internal tracing go to the preserved
buffer (like dump stacks), either set the instance to be the <code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">trace_printk()</span></code>
destination from the kernel command line, or set it after boot up via the
trace_printk_dest option.</p>
<p>After boot up:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>echo 1 > /sys/kernel/tracing/instances/boot_map/options/trace_printk_dest
</pre></div>
</div>
<p>From the kernel command line:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>reserve_mem=12M:4096:trace trace_instance=boot_map^traceprintk^traceoff@trace
</pre></div>
</div>
<p>If setting it from the kernel command line, it is recommended to also
disable tracing with the “traceoff” flag, and enable tracing after boot up.
Otherwise the trace from the most recent boot will be mixed with the trace
from the previous boot, and may make it confusing to read.</p>
</section>
</section>
</div>
</div>
</div>
<div class="clearer"></div>
</div>
<div class="footer">
©The kernel development community.
|
Powered by <a href="https://www.sphinx-doc.org/">Sphinx 8.1.3</a>
& <a href="https://alabaster.readthedocs.io">Alabaster 0.7.16</a>
|
<a href="../_sources/trace/debugging.rst.txt"
rel="nofollow">Page source</a>
</div>
</body>
</html>