<!DOCTYPE html>
<html lang="en" data-content_root="../../">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<title>SMB Direct - SMB3 over RDMA — The Linux Kernel documentation</title>
<link rel="stylesheet" type="text/css" href="../../_static/pygments.css?v=fa44fd50" />
<link rel="stylesheet" type="text/css" href="../../_static/alabaster.css?v=3918102e" />
<script src="../../_static/documentation_options.js?v=5929fcd5"></script>
<script src="../../_static/doctools.js?v=9bcbadda"></script>
<script src="../../_static/sphinx_highlight.js?v=dc90522c"></script>
<link rel="index" title="Index" href="../../genindex.html" />
<link rel="search" title="Search" href="../../search.html" />
<link rel="next" title="SPU Filesystem" href="../spufs/index.html" />
<link rel="prev" title="Mounting root file system via SMB (cifs.ko)" href="cifsroot.html" />
<link rel="stylesheet" href="../../_static/custom.css" type="text/css" />
</head><body>
<div class="document">
<div class="sphinxsidebar" role="navigation" aria-label="Main">
<div class="sphinxsidebarwrapper">
<p class="logo"><a href="../../index.html">
<img class="logo" src="../../_static/logo.svg" alt="Logo of The Linux Kernel"/>
</a></p>
<h1 class="logo"><a href="../../index.html">The Linux Kernel</a></h1>
<p class="blurb">6.18.50</p>
<search id="searchbox" style="display: none" role="search">
<h3 id="searchlabel">Quick search</h3>
<div class="searchformwrapper">
<form class="search" action="../../search.html" method="get">
<input type="text" name="q" aria-labelledby="searchlabel" autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false"/>
<input type="submit" value="Go" />
</form>
</div>
</search>
<script>document.getElementById('searchbox').style.display = "block"</script>
<p>
<h3 class="kernel-toc-contents">Contents</h3>
<input type="checkbox" class="kernel-toc-toggle" id = "kernel-toc-toggle" checked>
<label class="kernel-toc-title" for="kernel-toc-toggle"></label>
<div class="kerneltoc" id="kerneltoc">
<ul>
<li class="toctree-l1"><a class="reference internal" href="../../process/development-process.html">Development process</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../process/submitting-patches.html">Submitting patches</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../process/code-of-conduct.html">Code of conduct</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../maintainer/index.html">Maintainer handbook</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../process/index.html">All development-process docs</a></li>
</ul>
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../../core-api/index.html">Core API</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../driver-api/index.html">Driver APIs</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="../../subsystem-apis.html">Subsystems</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="../../subsystem-apis.html#core-subsystems">Core subsystems</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../subsystem-apis.html#human-interfaces">Human interfaces</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../subsystem-apis.html#networking-interfaces">Networking interfaces</a></li>
<li class="toctree-l2 current"><a class="reference internal" href="../../subsystem-apis.html#storage-interfaces">Storage interfaces</a><ul class="current">
<li class="toctree-l3 current"><a class="reference internal" href="../index.html">Filesystems in the Linux kernel</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../block/index.html">Block</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../cdrom/index.html">CD-ROM</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../scsi/index.html">SCSI Subsystem</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../target/index.html">TCM Virtual Device</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../nvme/index.html">NVMe Subsystem</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../../subsystem-apis.html#other-subsystems">Other subsystems</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../../locking/index.html">Locking</a></li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../../process/license-rules.html">Licensing rules</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../doc-guide/index.html">Writing documentation</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../dev-tools/index.html">Development tools</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../dev-tools/testing-overview.html">Testing guide</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../kernel-hacking/index.html">Hacking guide</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../trace/index.html">Tracing</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../fault-injection/index.html">Fault injection</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../livepatch/index.html">Livepatching</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../rust/index.html">Rust</a></li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../../admin-guide/index.html">Administration</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../kbuild/index.html">Build system</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../admin-guide/reporting-issues.html">Reporting issues</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../tools/index.html">Userspace tools</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../userspace-api/index.html">Userspace API</a></li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../../firmware-guide/index.html">Firmware</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../devicetree/index.html">Firmware and Devicetree</a></li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../../arch/index.html">CPU architectures</a></li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../../staging/index.html">Unsorted documentation</a></li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../../translations/index.html">Translations</a></li>
</ul>
</div>
<script type="text/javascript"> <!--
var sbar = document.getElementsByClassName("sphinxsidebar")[0];
let currents = document.getElementsByClassName("current")
if (currents.length) {
sbar.scrollTop = currents[currents.length - 1].offsetTop;
}
--> </script>
<div role="note" aria-label="source link">
<h3>This Page</h3>
<ul class="this-page-menu">
<li><a href="../../_sources/filesystems/smb/smbdirect.rst.txt"
rel="nofollow">Show Source</a></li>
</ul>
</div>
</div>
</div>
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body" role="main">
<section id="smb-direct-smb3-over-rdma">
<h1>SMB Direct - SMB3 over RDMA<a class="headerlink" href="#smb-direct-smb3-over-rdma" title="Link to this heading">¶</a></h1>
<p>This document describes how to set up the Linux SMB client and server to
use RDMA.</p>
<section id="overview">
<h2>Overview<a class="headerlink" href="#overview" title="Link to this heading">¶</a></h2>
<p>The Linux SMB kernel client supports SMB Direct, which is a transport
scheme for SMB3 that uses RDMA (Remote Direct Memory Access) to provide
high throughput and low latencies by bypassing the traditional TCP/IP
stack.
SMB Direct on the Linux SMB client can be tested against KSMBD - a
kernel-space SMB server.</p>
</section>
<section id="installation">
<h2>Installation<a class="headerlink" href="#installation" title="Link to this heading">¶</a></h2>
<ul>
<li><p>Install an RDMA device. As long as the RDMA device driver is supported
by the kernel, it should work. This includes both software emulators (soft
RoCE, soft iWARP) and hardware devices (InfiniBand, RoCE, iWARP).</p></li>
<li><p>Install a kernel with SMB Direct support. The first kernel release to
support SMB Direct on both the client and server side is 5.15. Therefore,
a distribution compatible with kernel 5.15 or later is required.</p></li>
<li><p>Install cifs-utils, which provides the <cite>mount.cifs</cite> command to mount SMB
shares.</p></li>
<li><p>Configure the RDMA stack</p>
<p>Make sure that your kernel configuration has RDMA support enabled. Under
Device Drivers -> Infiniband support, update the kernel configuration to
enable Infiniband support.</p>
<p>Enable the appropriate IB HCA support or iWARP adapter support,
depending on your hardware.</p>
<p>If you are using InfiniBand, enable IP-over-InfiniBand support.</p>
<p>For soft RDMA, enable either the soft iWARP (<cite>RDMA _SIW</cite>) or soft RoCE
(<cite>RDMA_RXE</cite>) module. Install the <cite>iproute2</cite> package and use the
<cite>rdma link add</cite> command to load the module and create an
RDMA interface.</p>
<p>e.g. if your local ethernet interface is <cite>eth0</cite>, you can use:</p>
<blockquote>
<div><div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>sudo<span class="w"> </span>rdma<span class="w"> </span>link<span class="w"> </span>add<span class="w"> </span>siw0<span class="w"> </span><span class="nb">type</span><span class="w"> </span>siw<span class="w"> </span>netdev<span class="w"> </span>eth0
</pre></div>
</div>
</div></blockquote>
</li>
<li><p>Enable SMB Direct support for both the server and the client in the kernel
configuration.</p>
<blockquote>
<div><p>Server Setup</p>
<div class="highlight-text notranslate"><div class="highlight"><pre><span></span>Network File Systems --->
<M> SMB3 server support
[*] Support for SMB Direct protocol
</pre></div>
</div>
<p>Client Setup</p>
<div class="highlight-text notranslate"><div class="highlight"><pre><span></span>Network File Systems --->
<M> SMB3 and CIFS support (advanced network filesystem)
[*] SMB Direct support
</pre></div>
</div>
</div></blockquote>
</li>
<li><p>Build and install the kernel. SMB Direct support will be enabled in the
cifs.ko and ksmbd.ko modules.</p></li>
</ul>
</section>
<section id="setup-and-usage">
<h2>Setup and Usage<a class="headerlink" href="#setup-and-usage" title="Link to this heading">¶</a></h2>
<ul>
<li><p>Set up and start a KSMBD server as described in the <a class="reference external" href="https://www.kernel.org/doc/Documentation/filesystems/smb/ksmbd.rst">KSMBD documentation</a>.
Also add the “server multi channel support = yes” parameter to ksmbd.conf.</p></li>
<li><p>On the client, mount the share with <cite>rdma</cite> mount option to use SMB Direct
(specify a SMB version 3.0 or higher using <cite>vers</cite>).</p>
<p>For example:</p>
<blockquote>
<div><div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>mount<span class="w"> </span>-t<span class="w"> </span>cifs<span class="w"> </span>//server/share<span class="w"> </span>/mnt/point<span class="w"> </span>-o<span class="w"> </span><span class="nv">vers</span><span class="o">=</span><span class="m">3</span>.1.1,rdma
</pre></div>
</div>
</div></blockquote>
</li>
<li><p>To verify that the mount is using SMB Direct, you can check dmesg for the
following log line after mounting:</p>
<blockquote>
<div><div class="highlight-text notranslate"><div class="highlight"><pre><span></span>CIFS: VFS: RDMA transport established
</pre></div>
</div>
</div></blockquote>
<p>Or, verify <cite>rdma</cite> mount option for the share in <cite>/proc/mounts</cite>:</p>
<blockquote>
<div><div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>cat<span class="w"> </span>/proc/mounts<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>cifs
</pre></div>
</div>
</div></blockquote>
</li>
</ul>
</section>
</section>
</div>
</div>
</div>
<div class="clearer"></div>
</div>
<div class="footer">
©The kernel development community.
|
Powered by <a href="https://www.sphinx-doc.org/">Sphinx 8.1.3</a>
& <a href="https://alabaster.readthedocs.io">Alabaster 0.7.16</a>
|
<a href="../../_sources/filesystems/smb/smbdirect.rst.txt"
rel="nofollow">Page source</a>
</div>
</body>
</html>