<!DOCTYPE html>
<html lang="en" data-content_root="../../../">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<title>POWER9 eXternal Interrupt Virtualization Engine (XIVE Gen1) — The Linux Kernel documentation</title>
<link rel="stylesheet" type="text/css" href="../../../_static/pygments.css?v=fa44fd50" />
<link rel="stylesheet" type="text/css" href="../../../_static/alabaster.css?v=3918102e" />
<script src="../../../_static/documentation_options.js?v=5929fcd5"></script>
<script src="../../../_static/doctools.js?v=9bcbadda"></script>
<script src="../../../_static/sphinx_highlight.js?v=dc90522c"></script>
<link rel="index" title="Index" href="../../../genindex.html" />
<link rel="search" title="Search" href="../../../search.html" />
<link rel="next" title="ARM" href="../arm/index.html" />
<link rel="prev" title="XICS interrupt controller" href="xics.html" />
<link rel="stylesheet" href="../../../_static/custom.css" type="text/css" />
</head><body>
<div class="document">
<div class="sphinxsidebar" role="navigation" aria-label="Main">
<div class="sphinxsidebarwrapper">
<p class="logo"><a href="../../../index.html">
<img class="logo" src="../../../_static/logo.svg" alt="Logo of The Linux Kernel"/>
</a></p>
<h1 class="logo"><a href="../../../index.html">The Linux Kernel</a></h1>
<p class="blurb">6.18.50</p>
<search id="searchbox" style="display: none" role="search">
<h3 id="searchlabel">Quick search</h3>
<div class="searchformwrapper">
<form class="search" action="../../../search.html" method="get">
<input type="text" name="q" aria-labelledby="searchlabel" autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false"/>
<input type="submit" value="Go" />
</form>
</div>
</search>
<script>document.getElementById('searchbox').style.display = "block"</script>
<p>
<h3 class="kernel-toc-contents">Contents</h3>
<input type="checkbox" class="kernel-toc-toggle" id = "kernel-toc-toggle" checked>
<label class="kernel-toc-title" for="kernel-toc-toggle"></label>
<div class="kerneltoc" id="kerneltoc">
<ul>
<li class="toctree-l1"><a class="reference internal" href="../../../process/development-process.html">Development process</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../process/submitting-patches.html">Submitting patches</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../process/code-of-conduct.html">Code of conduct</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../maintainer/index.html">Maintainer handbook</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../process/index.html">All development-process docs</a></li>
</ul>
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../../../core-api/index.html">Core API</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../driver-api/index.html">Driver APIs</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="../../../subsystem-apis.html">Subsystems</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="../../../subsystem-apis.html#core-subsystems">Core subsystems</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../../subsystem-apis.html#human-interfaces">Human interfaces</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../../subsystem-apis.html#networking-interfaces">Networking interfaces</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../../subsystem-apis.html#storage-interfaces">Storage interfaces</a></li>
<li class="toctree-l2 current"><a class="reference internal" href="../../../subsystem-apis.html#other-subsystems">Other subsystems</a><ul class="current">
<li class="toctree-l3"><a class="reference internal" href="../../../accounting/index.html">Accounting</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../cpu-freq/index.html">CPUFreq - CPU frequency and voltage scaling code in the Linux(TM) kernel</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../edac/index.html">EDAC Subsystem</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../fpga/index.html">FPGA</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../i2c/index.html">I2C/SMBus Subsystem</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../iio/index.html">Industrial I/O</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../pcmcia/index.html">PCMCIA</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../spi/index.html">Serial Peripheral Interface (SPI)</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../w1/index.html">1-Wire Subsystem</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../watchdog/index.html">Watchdog Support</a></li>
<li class="toctree-l3 current"><a class="reference internal" href="../../index.html">Virtualization Support</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../hwmon/index.html">Hardware Monitoring</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../accel/index.html">Compute Accelerators</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../security/index.html">Security Documentation</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../crypto/index.html">Crypto API</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../bpf/index.html">BPF Documentation</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../usb/index.html">USB support</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../PCI/index.html">PCI Bus Subsystem</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../misc-devices/index.html">Assorted Miscellaneous Devices Documentation</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../peci/index.html">PECI Subsystem</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../wmi/index.html">WMI Subsystem</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../tee/index.html">TEE Subsystem</a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../../../locking/index.html">Locking</a></li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../../../process/license-rules.html">Licensing rules</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../doc-guide/index.html">Writing documentation</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../dev-tools/index.html">Development tools</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../dev-tools/testing-overview.html">Testing guide</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../kernel-hacking/index.html">Hacking guide</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../trace/index.html">Tracing</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../fault-injection/index.html">Fault injection</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../livepatch/index.html">Livepatching</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../rust/index.html">Rust</a></li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../../../admin-guide/index.html">Administration</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../kbuild/index.html">Build system</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../admin-guide/reporting-issues.html">Reporting issues</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../tools/index.html">Userspace tools</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../userspace-api/index.html">Userspace API</a></li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../../../firmware-guide/index.html">Firmware</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../devicetree/index.html">Firmware and Devicetree</a></li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../../../arch/index.html">CPU architectures</a></li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../../../staging/index.html">Unsorted documentation</a></li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../../../translations/index.html">Translations</a></li>
</ul>
</div>
<script type="text/javascript"> <!--
var sbar = document.getElementsByClassName("sphinxsidebar")[0];
let currents = document.getElementsByClassName("current")
if (currents.length) {
sbar.scrollTop = currents[currents.length - 1].offsetTop;
}
--> </script>
<div role="note" aria-label="source link">
<h3>This Page</h3>
<ul class="this-page-menu">
<li><a href="../../../_sources/virt/kvm/devices/xive.rst.txt"
rel="nofollow">Show Source</a></li>
</ul>
</div>
</div>
</div>
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body" role="main">
<section id="power9-external-interrupt-virtualization-engine-xive-gen1">
<h1>POWER9 eXternal Interrupt Virtualization Engine (XIVE Gen1)<a class="headerlink" href="#power9-external-interrupt-virtualization-engine-xive-gen1" title="Link to this heading">ΒΆ</a></h1>
<dl class="simple">
<dt>Device types supported:</dt><dd><ul class="simple">
<li><p>KVM_DEV_TYPE_XIVE POWER9 XIVE Interrupt Controller generation 1</p></li>
</ul>
</dd>
</dl>
<p>This device acts as a VM interrupt controller. It provides the KVM
interface to configure the interrupt sources of a VM in the underlying
POWER9 XIVE interrupt controller.</p>
<p>Only one XIVE instance may be instantiated. A guest XIVE device
requires a POWER9 host and the guest OS should have support for the
XIVE native exploitation interrupt mode. If not, it should run using
the legacy interrupt mode, referred as XICS (POWER7/8).</p>
<ul>
<li><p>Device Mappings</p>
<p>The KVM device exposes different MMIO ranges of the XIVE HW which
are required for interrupt management. These are exposed to the
guest in VMAs populated with a custom VM fault handler.</p>
<ol class="arabic simple">
<li><p>Thread Interrupt Management Area (TIMA)</p></li>
</ol>
<p>Each thread has an associated Thread Interrupt Management context
composed of a set of registers. These registers let the thread
handle priority management and interrupt acknowledgment. The most
important are :</p>
<blockquote>
<div><ul class="simple">
<li><p>Interrupt Pending Buffer (IPB)</p></li>
<li><p>Current Processor Priority (CPPR)</p></li>
<li><p>Notification Source Register (NSR)</p></li>
</ul>
</div></blockquote>
<p>They are exposed to software in four different pages each proposing
a view with a different privilege. The first page is for the
physical thread context and the second for the hypervisor. Only the
third (operating system) and the fourth (user level) are exposed the
guest.</p>
<ol class="arabic simple" start="2">
<li><p>Event State Buffer (ESB)</p></li>
</ol>
<p>Each source is associated with an Event State Buffer (ESB) with
either a pair of even/odd pair of pages which provides commands to
manage the source: to trigger, to EOI, to turn off the source for
instance.</p>
<ol class="arabic simple" start="3">
<li><p>Device pass-through</p></li>
</ol>
<p>When a device is passed-through into the guest, the source
interrupts are from a different HW controller (PHB4) and the ESB
pages exposed to the guest should accommodate this change.</p>
<p>The passthru_irq helpers, <code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">kvmppc_xive_set_mapped()</span></code> and
<code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">kvmppc_xive_clr_mapped()</span></code> are called when the device HW irqs are
mapped into or unmapped from the guest IRQ number space. The KVM
device extends these helpers to clear the ESB pages of the guest IRQ
number being mapped and then lets the VM fault handler repopulate.
The handler will insert the ESB page corresponding to the HW
interrupt of the device being passed-through or the initial IPI ESB
page if the device has being removed.</p>
<p>The ESB remapping is fully transparent to the guest and the OS
device driver. All handling is done within VFIO and the above
helpers in KVM-PPC.</p>
</li>
<li><p>Groups:</p></li>
</ul>
<ol class="arabic simple">
<li><dl class="simple">
<dt>KVM_DEV_XIVE_GRP_CTRL</dt><dd><p>Provides global controls on the device</p>
</dd>
</dl>
</li>
</ol>
<blockquote>
<div><dl>
<dt>Attributes:</dt><dd><p>1.1 KVM_DEV_XIVE_RESET (write only)
Resets the interrupt controller configuration for sources and event
queues. To be used by kexec and kdump.</p>
<p>Errors: none</p>
<p>1.2 KVM_DEV_XIVE_EQ_SYNC (write only)
Sync all the sources and queues and mark the EQ pages dirty. This
to make sure that a consistent memory state is captured when
migrating the VM.</p>
<p>Errors: none</p>
<p>1.3 KVM_DEV_XIVE_NR_SERVERS (write only)
The kvm_device_attr.addr points to a __u32 value which is the number of
interrupt server numbers (ie, highest possible vcpu id plus one).</p>
<p>Errors:</p>
<blockquote>
<div><table class="docutils align-default">
<tbody>
<tr class="row-odd"><td><p>-EINVAL</p></td>
<td><p>Value greater than KVM_MAX_VCPU_IDS.</p></td>
</tr>
<tr class="row-even"><td><p>-EFAULT</p></td>
<td><p>Invalid user pointer for attr->addr.</p></td>
</tr>
<tr class="row-odd"><td><p>-EBUSY</p></td>
<td><p>A vCPU is already connected to the device.</p></td>
</tr>
</tbody>
</table>
</div></blockquote>
</dd>
</dl>
</div></blockquote>
<ol class="arabic simple" start="2">
<li><dl class="simple">
<dt>KVM_DEV_XIVE_GRP_SOURCE (write only)</dt><dd><p>Initializes a new source in the XIVE device and mask it.</p>
</dd>
</dl>
</li>
</ol>
<blockquote>
<div><dl class="simple">
<dt>Attributes:</dt><dd><p>Interrupt source number (64-bit)</p>
</dd>
</dl>
<p>The kvm_device_attr.addr points to a __u64 value:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>bits: | 63 .... 2 | 1 | 0
values: | unused | level | type
</pre></div>
</div>
<ul class="simple">
<li><p>type: 0:MSI 1:LSI</p></li>
<li><p>level: assertion level in case of an LSI.</p></li>
</ul>
<p>Errors:</p>
<blockquote>
<div><table class="docutils align-default">
<tbody>
<tr class="row-odd"><td><p>-E2BIG</p></td>
<td><p>Interrupt source number is out of range</p></td>
</tr>
<tr class="row-even"><td><p>-ENOMEM</p></td>
<td><p>Could not create a new source block</p></td>
</tr>
<tr class="row-odd"><td><p>-EFAULT</p></td>
<td><p>Invalid user pointer for attr->addr.</p></td>
</tr>
<tr class="row-even"><td><p>-ENXIO</p></td>
<td><p>Could not allocate underlying HW interrupt</p></td>
</tr>
</tbody>
</table>
</div></blockquote>
</div></blockquote>
<ol class="arabic simple" start="3">
<li><dl class="simple">
<dt>KVM_DEV_XIVE_GRP_SOURCE_CONFIG (write only)</dt><dd><p>Configures source targeting</p>
</dd>
</dl>
</li>
</ol>
<blockquote>
<div><dl class="simple">
<dt>Attributes:</dt><dd><p>Interrupt source number (64-bit)</p>
</dd>
</dl>
<p>The kvm_device_attr.addr points to a __u64 value:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>bits: | 63 .... 33 | 32 | 31 .. 3 | 2 .. 0
values: | eisn | mask | server | priority
</pre></div>
</div>
<ul class="simple">
<li><p>priority: 0-7 interrupt priority level</p></li>
<li><p>server: CPU number chosen to handle the interrupt</p></li>
<li><p>mask: mask flag (unused)</p></li>
<li><p>eisn: Effective Interrupt Source Number</p></li>
</ul>
<p>Errors:</p>
<blockquote>
<div><table class="docutils align-default">
<tbody>
<tr class="row-odd"><td><p>-ENOENT</p></td>
<td><p>Unknown source number</p></td>
</tr>
<tr class="row-even"><td><p>-EINVAL</p></td>
<td><p>Not initialized source number</p></td>
</tr>
<tr class="row-odd"><td><p>-EINVAL</p></td>
<td><p>Invalid priority</p></td>
</tr>
<tr class="row-even"><td><p>-EINVAL</p></td>
<td><p>Invalid CPU number.</p></td>
</tr>
<tr class="row-odd"><td><p>-EFAULT</p></td>
<td><p>Invalid user pointer for attr->addr.</p></td>
</tr>
<tr class="row-even"><td><p>-ENXIO</p></td>
<td><p>CPU event queues not configured or configuration of the
underlying HW interrupt failed</p></td>
</tr>
<tr class="row-odd"><td><p>-EBUSY</p></td>
<td><p>No CPU available to serve interrupt</p></td>
</tr>
</tbody>
</table>
</div></blockquote>
</div></blockquote>
<ol class="arabic simple" start="4">
<li><dl class="simple">
<dt>KVM_DEV_XIVE_GRP_EQ_CONFIG (read-write)</dt><dd><p>Configures an event queue of a CPU</p>
</dd>
</dl>
</li>
</ol>
<blockquote>
<div><dl class="simple">
<dt>Attributes:</dt><dd><p>EQ descriptor identifier (64-bit)</p>
</dd>
</dl>
<p>The EQ descriptor identifier is a tuple (server, priority):</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>bits: | 63 .... 32 | 31 .. 3 | 2 .. 0
values: | unused | server | priority
</pre></div>
</div>
<p>The kvm_device_attr.addr points to:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>struct kvm_ppc_xive_eq {
__u32 flags;
__u32 qshift;
__u64 qaddr;
__u32 qtoggle;
__u32 qindex;
__u8 pad[40];
};
</pre></div>
</div>
<ul class="simple">
<li><dl class="simple">
<dt>flags: queue flags</dt><dd><dl class="simple">
<dt>KVM_XIVE_EQ_ALWAYS_NOTIFY (required)</dt><dd><p>forces notification without using the coalescing mechanism
provided by the XIVE END ESBs.</p>
</dd>
</dl>
</dd>
</dl>
</li>
<li><p>qshift: queue size (power of 2)</p></li>
<li><p>qaddr: real address of queue</p></li>
<li><p>qtoggle: current queue toggle bit</p></li>
<li><p>qindex: current queue index</p></li>
<li><p>pad: reserved for future use</p></li>
</ul>
<p>Errors:</p>
<blockquote>
<div><table class="docutils align-default">
<tbody>
<tr class="row-odd"><td><p>-ENOENT</p></td>
<td><p>Invalid CPU number</p></td>
</tr>
<tr class="row-even"><td><p>-EINVAL</p></td>
<td><p>Invalid priority</p></td>
</tr>
<tr class="row-odd"><td><p>-EINVAL</p></td>
<td><p>Invalid flags</p></td>
</tr>
<tr class="row-even"><td><p>-EINVAL</p></td>
<td><p>Invalid queue size</p></td>
</tr>
<tr class="row-odd"><td><p>-EINVAL</p></td>
<td><p>Invalid queue address</p></td>
</tr>
<tr class="row-even"><td><p>-EFAULT</p></td>
<td><p>Invalid user pointer for attr->addr.</p></td>
</tr>
<tr class="row-odd"><td><p>-EIO</p></td>
<td><p>Configuration of the underlying HW failed</p></td>
</tr>
</tbody>
</table>
</div></blockquote>
</div></blockquote>
<ol class="arabic simple" start="5">
<li><dl class="simple">
<dt>KVM_DEV_XIVE_GRP_SOURCE_SYNC (write only)</dt><dd><p>Synchronize the source to flush event notifications</p>
</dd>
</dl>
</li>
</ol>
<blockquote>
<div><dl class="simple">
<dt>Attributes:</dt><dd><p>Interrupt source number (64-bit)</p>
</dd>
</dl>
<p>Errors:</p>
<blockquote>
<div><table class="docutils align-default">
<tbody>
<tr class="row-odd"><td><p>-ENOENT</p></td>
<td><p>Unknown source number</p></td>
</tr>
<tr class="row-even"><td><p>-EINVAL</p></td>
<td><p>Not initialized source number</p></td>
</tr>
</tbody>
</table>
</div></blockquote>
</div></blockquote>
<ul>
<li><p>VCPU state</p>
<p>The XIVE IC maintains VP interrupt state in an internal structure
called the NVT. When a VP is not dispatched on a HW processor
thread, this structure can be updated by HW if the VP is the target
of an event notification.</p>
<p>It is important for migration to capture the cached IPB from the NVT
as it synthesizes the priorities of the pending interrupts. We
capture a bit more to report debug information.</p>
<p>KVM_REG_PPC_VP_STATE (2 * 64bits):</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>bits: | 63 .... 32 | 31 .... 0 |
values: | TIMA word0 | TIMA word1 |
bits: | 127 .......... 64 |
values: | unused |
</pre></div>
</div>
</li>
<li><p>Migration:</p>
<p>Saving the state of a VM using the XIVE native exploitation mode
should follow a specific sequence. When the VM is stopped :</p>
<ol class="arabic simple">
<li><p>Mask all sources (PQ=01) to stop the flow of events.</p></li>
</ol>
<p>2. Sync the XIVE device with the KVM control KVM_DEV_XIVE_EQ_SYNC to
flush any in-flight event notification and to stabilize the EQs. At
this stage, the EQ pages are marked dirty to make sure they are
transferred in the migration sequence.</p>
<p>3. Capture the state of the source targeting, the EQs configuration
and the state of thread interrupt context registers.</p>
<p>Restore is similar:</p>
<ol class="arabic simple">
<li><p>Restore the EQ configuration. As targeting depends on it.</p></li>
<li><p>Restore targeting</p></li>
<li><p>Restore the thread interrupt contexts</p></li>
<li><p>Restore the source states</p></li>
<li><p>Let the vCPU run</p></li>
</ol>
</li>
</ul>
</section>
</div>
</div>
</div>
<div class="clearer"></div>
</div>
<div class="footer">
©The kernel development community.
|
Powered by <a href="https://www.sphinx-doc.org/">Sphinx 8.1.3</a>
& <a href="https://alabaster.readthedocs.io">Alabaster 0.7.16</a>
|
<a href="../../../_sources/virt/kvm/devices/xive.rst.txt"
rel="nofollow">Page source</a>
</div>
</body>
</html>