<!DOCTYPE html>
<html lang="en" data-content_root="../">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<title>IP over InfiniBand — The Linux Kernel documentation</title>
<link rel="stylesheet" type="text/css" href="../_static/pygments.css?v=fa44fd50" />
<link rel="stylesheet" type="text/css" href="../_static/alabaster.css?v=3918102e" />
<script src="../_static/documentation_options.js?v=5929fcd5"></script>
<script src="../_static/doctools.js?v=9bcbadda"></script>
<script src="../_static/sphinx_highlight.js?v=dc90522c"></script>
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
<link rel="next" title="Intel Omni-Path (OPA) Virtual Network Interface Controller (VNIC)" href="opa_vnic.html" />
<link rel="prev" title="InfiniBand Midlayer Locking" href="core_locking.html" />
<link rel="stylesheet" href="../_static/custom.css" type="text/css" />
</head><body>
<div class="document">
<div class="sphinxsidebar" role="navigation" aria-label="Main">
<div class="sphinxsidebarwrapper">
<p class="logo"><a href="../index.html">
<img class="logo" src="../_static/logo.svg" alt="Logo of The Linux Kernel"/>
</a></p>
<h1 class="logo"><a href="../index.html">The Linux Kernel</a></h1>
<p class="blurb">6.18.50</p>
<search id="searchbox" style="display: none" role="search">
<h3 id="searchlabel">Quick search</h3>
<div class="searchformwrapper">
<form class="search" action="../search.html" method="get">
<input type="text" name="q" aria-labelledby="searchlabel" autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false"/>
<input type="submit" value="Go" />
</form>
</div>
</search>
<script>document.getElementById('searchbox').style.display = "block"</script>
<p>
<h3 class="kernel-toc-contents">Contents</h3>
<input type="checkbox" class="kernel-toc-toggle" id = "kernel-toc-toggle" checked>
<label class="kernel-toc-title" for="kernel-toc-toggle"></label>
<div class="kerneltoc" id="kerneltoc">
<ul>
<li class="toctree-l1"><a class="reference internal" href="../process/development-process.html">Development process</a></li>
<li class="toctree-l1"><a class="reference internal" href="../process/submitting-patches.html">Submitting patches</a></li>
<li class="toctree-l1"><a class="reference internal" href="../process/code-of-conduct.html">Code of conduct</a></li>
<li class="toctree-l1"><a class="reference internal" href="../maintainer/index.html">Maintainer handbook</a></li>
<li class="toctree-l1"><a class="reference internal" href="../process/index.html">All development-process docs</a></li>
</ul>
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../core-api/index.html">Core API</a></li>
<li class="toctree-l1"><a class="reference internal" href="../driver-api/index.html">Driver APIs</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="../subsystem-apis.html">Subsystems</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="../subsystem-apis.html#core-subsystems">Core subsystems</a></li>
<li class="toctree-l2"><a class="reference internal" href="../subsystem-apis.html#human-interfaces">Human interfaces</a></li>
<li class="toctree-l2 current"><a class="reference internal" href="../subsystem-apis.html#networking-interfaces">Networking interfaces</a><ul class="current">
<li class="toctree-l3"><a class="reference internal" href="../networking/index.html">Networking</a></li>
<li class="toctree-l3"><a class="reference internal" href="../netlabel/index.html">NetLabel</a></li>
<li class="toctree-l3 current"><a class="reference internal" href="index.html">InfiniBand</a></li>
<li class="toctree-l3"><a class="reference internal" href="../isdn/index.html">ISDN</a></li>
<li class="toctree-l3"><a class="reference internal" href="../mhi/index.html">MHI</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../subsystem-apis.html#storage-interfaces">Storage interfaces</a></li>
<li class="toctree-l2"><a class="reference internal" href="../subsystem-apis.html#other-subsystems">Other subsystems</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../locking/index.html">Locking</a></li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../process/license-rules.html">Licensing rules</a></li>
<li class="toctree-l1"><a class="reference internal" href="../doc-guide/index.html">Writing documentation</a></li>
<li class="toctree-l1"><a class="reference internal" href="../dev-tools/index.html">Development tools</a></li>
<li class="toctree-l1"><a class="reference internal" href="../dev-tools/testing-overview.html">Testing guide</a></li>
<li class="toctree-l1"><a class="reference internal" href="../kernel-hacking/index.html">Hacking guide</a></li>
<li class="toctree-l1"><a class="reference internal" href="../trace/index.html">Tracing</a></li>
<li class="toctree-l1"><a class="reference internal" href="../fault-injection/index.html">Fault injection</a></li>
<li class="toctree-l1"><a class="reference internal" href="../livepatch/index.html">Livepatching</a></li>
<li class="toctree-l1"><a class="reference internal" href="../rust/index.html">Rust</a></li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../admin-guide/index.html">Administration</a></li>
<li class="toctree-l1"><a class="reference internal" href="../kbuild/index.html">Build system</a></li>
<li class="toctree-l1"><a class="reference internal" href="../admin-guide/reporting-issues.html">Reporting issues</a></li>
<li class="toctree-l1"><a class="reference internal" href="../tools/index.html">Userspace tools</a></li>
<li class="toctree-l1"><a class="reference internal" href="../userspace-api/index.html">Userspace API</a></li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../firmware-guide/index.html">Firmware</a></li>
<li class="toctree-l1"><a class="reference internal" href="../devicetree/index.html">Firmware and Devicetree</a></li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../arch/index.html">CPU architectures</a></li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../staging/index.html">Unsorted documentation</a></li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../translations/index.html">Translations</a></li>
</ul>
</div>
<script type="text/javascript"> <!--
var sbar = document.getElementsByClassName("sphinxsidebar")[0];
let currents = document.getElementsByClassName("current")
if (currents.length) {
sbar.scrollTop = currents[currents.length - 1].offsetTop;
}
--> </script>
<div role="note" aria-label="source link">
<h3>This Page</h3>
<ul class="this-page-menu">
<li><a href="../_sources/infiniband/ipoib.rst.txt"
rel="nofollow">Show Source</a></li>
</ul>
</div>
</div>
</div>
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body" role="main">
<div class="language-selection">
English
<ul>
<li><a href="../translations/zh_CN/infiniband/ipoib.html">Chinese (Simplified)</a></li>
</ul>
</div>
<section id="ip-over-infiniband">
<h1>IP over InfiniBand<a class="headerlink" href="#ip-over-infiniband" title="Link to this heading">¶</a></h1>
<blockquote>
<div><p>The ib_ipoib driver is an implementation of the IP over InfiniBand
protocol as specified by RFC 4391 and 4392, issued by the IETF ipoib
working group. It is a “native” implementation in the sense of
setting the interface type to ARPHRD_INFINIBAND and the hardware
address length to 20 (earlier proprietary implementations
masqueraded to the kernel as ethernet interfaces).</p>
</div></blockquote>
<section id="partitions-and-p-keys">
<h2>Partitions and P_Keys<a class="headerlink" href="#partitions-and-p-keys" title="Link to this heading">¶</a></h2>
<blockquote>
<div><p>When the IPoIB driver is loaded, it creates one interface for each
port using the P_Key at index 0. To create an interface with a
different P_Key, write the desired P_Key into the main interface’s
/sys/class/net/<intf name>/create_child file. For example:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>echo 0x8001 > /sys/class/net/ib0/create_child
</pre></div>
</div>
<p>This will create an interface named ib0.8001 with P_Key 0x8001. To
remove a subinterface, use the “delete_child” file:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>echo 0x8001 > /sys/class/net/ib0/delete_child
</pre></div>
</div>
<p>The P_Key for any interface is given by the “pkey” file, and the
main interface for a subinterface is in “parent.”</p>
<p>Child interface create/delete can also be done using IPoIB’s
rtnl_link_ops, where children created using either way behave the same.</p>
</div></blockquote>
</section>
<section id="datagram-vs-connected-modes">
<h2>Datagram vs Connected modes<a class="headerlink" href="#datagram-vs-connected-modes" title="Link to this heading">¶</a></h2>
<blockquote>
<div><p>The IPoIB driver supports two modes of operation: datagram and
connected. The mode is set and read through an interface’s
/sys/class/net/<intf name>/mode file.</p>
<p>In datagram mode, the IB UD (Unreliable Datagram) transport is used
and so the interface MTU has is equal to the IB L2 MTU minus the
IPoIB encapsulation header (4 bytes). For example, in a typical IB
fabric with a 2K MTU, the IPoIB MTU will be 2048 - 4 = 2044 bytes.</p>
<p>In connected mode, the IB RC (Reliable Connected) transport is used.
Connected mode takes advantage of the connected nature of the IB
transport and allows an MTU up to the maximal IP packet size of 64K,
which reduces the number of IP packets needed for handling large UDP
datagrams, TCP segments, etc and increases the performance for large
messages.</p>
<p>In connected mode, the interface’s UD QP is still used for multicast
and communication with peers that don’t support connected mode. In
this case, RX emulation of ICMP PMTU packets is used to cause the
networking stack to use the smaller UD MTU for these neighbours.</p>
</div></blockquote>
</section>
<section id="stateless-offloads">
<h2>Stateless offloads<a class="headerlink" href="#stateless-offloads" title="Link to this heading">¶</a></h2>
<blockquote>
<div><p>If the IB HW supports IPoIB stateless offloads, IPoIB advertises
TCP/IP checksum and/or Large Send (LSO) offloading capability to the
network stack.</p>
<p>Large Receive (LRO) offloading is also implemented and may be turned
on/off using ethtool calls. Currently LRO is supported only for
checksum offload capable devices.</p>
<p>Stateless offloads are supported only in datagram mode.</p>
</div></blockquote>
</section>
<section id="interrupt-moderation">
<h2>Interrupt moderation<a class="headerlink" href="#interrupt-moderation" title="Link to this heading">¶</a></h2>
<blockquote>
<div><p>If the underlying IB device supports CQ event moderation, one can
use ethtool to set interrupt mitigation parameters and thus reduce
the overhead incurred by handling interrupts. The main code path of
IPoIB doesn’t use events for TX completion signaling so only RX
moderation is supported.</p>
</div></blockquote>
</section>
<section id="debugging-information">
<h2>Debugging Information<a class="headerlink" href="#debugging-information" title="Link to this heading">¶</a></h2>
<blockquote>
<div><p>By compiling the IPoIB driver with CONFIG_INFINIBAND_IPOIB_DEBUG set
to ‘y’, tracing messages are compiled into the driver. They are
turned on by setting the module parameters debug_level and
mcast_debug_level to 1. These parameters can be controlled at
runtime through files in /sys/module/ib_ipoib/.</p>
<p>CONFIG_INFINIBAND_IPOIB_DEBUG also enables files in the debugfs
virtual filesystem. By mounting this filesystem, for example with:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>mount -t debugfs none /sys/kernel/debug
</pre></div>
</div>
<p>it is possible to get statistics about multicast groups from the
files /sys/kernel/debug/ipoib/ib0_mcg and so on.</p>
<p>The performance impact of this option is negligible, so it
is safe to enable this option with debug_level set to 0 for normal
operation.</p>
<p>CONFIG_INFINIBAND_IPOIB_DEBUG_DATA enables even more debug output in
the data path when data_debug_level is set to 1. However, even with
the output disabled, enabling this configuration option will affect
performance, because it adds tests to the fast path.</p>
</div></blockquote>
</section>
<section id="references">
<h2>References<a class="headerlink" href="#references" title="Link to this heading">¶</a></h2>
<blockquote>
<div><dl class="simple">
<dt>Transmission of IP over InfiniBand (IPoIB) (RFC 4391)</dt><dd><p><a class="reference external" href="http://ietf.org/rfc/rfc4391.txt">http://ietf.org/rfc/rfc4391.txt</a></p>
</dd>
<dt>IP over InfiniBand (IPoIB) Architecture (RFC 4392)</dt><dd><p><a class="reference external" href="http://ietf.org/rfc/rfc4392.txt">http://ietf.org/rfc/rfc4392.txt</a></p>
</dd>
<dt>IP over InfiniBand: Connected Mode (RFC 4755)</dt><dd><p><a class="reference external" href="http://ietf.org/rfc/rfc4755.txt">http://ietf.org/rfc/rfc4755.txt</a></p>
</dd>
</dl>
</div></blockquote>
</section>
</section>
</div>
</div>
</div>
<div class="clearer"></div>
</div>
<div class="footer">
©The kernel development community.
|
Powered by <a href="https://www.sphinx-doc.org/">Sphinx 8.1.3</a>
& <a href="https://alabaster.readthedocs.io">Alabaster 0.7.16</a>
|
<a href="../_sources/infiniband/ipoib.rst.txt"
rel="nofollow">Page source</a>
</div>
</body>
</html>