<!DOCTYPE html>
<html lang="en" data-content_root="../">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Squashfs 4.0 Filesystem — The Linux Kernel documentation</title>
<link rel="stylesheet" type="text/css" href="../_static/pygments.css?v=fa44fd50" />
<link rel="stylesheet" type="text/css" href="../_static/alabaster.css?v=3918102e" />
<script src="../_static/documentation_options.js?v=5929fcd5"></script>
<script src="../_static/doctools.js?v=9bcbadda"></script>
<script src="../_static/sphinx_highlight.js?v=dc90522c"></script>
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
<link rel="next" title="sysfs - _The_ filesystem for exporting kernel objects" href="sysfs.html" />
<link rel="prev" title="spu_run" href="spufs/spu_run.html" />
<link rel="stylesheet" href="../_static/custom.css" type="text/css" />
</head><body>
<div class="document">
<div class="sphinxsidebar" role="navigation" aria-label="Main">
<div class="sphinxsidebarwrapper">
<p class="logo"><a href="../index.html">
<img class="logo" src="../_static/logo.svg" alt="Logo of The Linux Kernel"/>
</a></p>
<h1 class="logo"><a href="../index.html">The Linux Kernel</a></h1>
<p class="blurb">6.18.50</p>
<search id="searchbox" style="display: none" role="search">
<h3 id="searchlabel">Quick search</h3>
<div class="searchformwrapper">
<form class="search" action="../search.html" method="get">
<input type="text" name="q" aria-labelledby="searchlabel" autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false"/>
<input type="submit" value="Go" />
</form>
</div>
</search>
<script>document.getElementById('searchbox').style.display = "block"</script>
<p>
<h3 class="kernel-toc-contents">Contents</h3>
<input type="checkbox" class="kernel-toc-toggle" id = "kernel-toc-toggle" checked>
<label class="kernel-toc-title" for="kernel-toc-toggle"></label>
<div class="kerneltoc" id="kerneltoc">
<ul>
<li class="toctree-l1"><a class="reference internal" href="../process/development-process.html">Development process</a></li>
<li class="toctree-l1"><a class="reference internal" href="../process/submitting-patches.html">Submitting patches</a></li>
<li class="toctree-l1"><a class="reference internal" href="../process/code-of-conduct.html">Code of conduct</a></li>
<li class="toctree-l1"><a class="reference internal" href="../maintainer/index.html">Maintainer handbook</a></li>
<li class="toctree-l1"><a class="reference internal" href="../process/index.html">All development-process docs</a></li>
</ul>
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../core-api/index.html">Core API</a></li>
<li class="toctree-l1"><a class="reference internal" href="../driver-api/index.html">Driver APIs</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="../subsystem-apis.html">Subsystems</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="../subsystem-apis.html#core-subsystems">Core subsystems</a></li>
<li class="toctree-l2"><a class="reference internal" href="../subsystem-apis.html#human-interfaces">Human interfaces</a></li>
<li class="toctree-l2"><a class="reference internal" href="../subsystem-apis.html#networking-interfaces">Networking interfaces</a></li>
<li class="toctree-l2 current"><a class="reference internal" href="../subsystem-apis.html#storage-interfaces">Storage interfaces</a><ul class="current">
<li class="toctree-l3 current"><a class="reference internal" href="index.html">Filesystems in the Linux kernel</a></li>
<li class="toctree-l3"><a class="reference internal" href="../block/index.html">Block</a></li>
<li class="toctree-l3"><a class="reference internal" href="../cdrom/index.html">CD-ROM</a></li>
<li class="toctree-l3"><a class="reference internal" href="../scsi/index.html">SCSI Subsystem</a></li>
<li class="toctree-l3"><a class="reference internal" href="../target/index.html">TCM Virtual Device</a></li>
<li class="toctree-l3"><a class="reference internal" href="../nvme/index.html">NVMe Subsystem</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../subsystem-apis.html#other-subsystems">Other subsystems</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../locking/index.html">Locking</a></li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../process/license-rules.html">Licensing rules</a></li>
<li class="toctree-l1"><a class="reference internal" href="../doc-guide/index.html">Writing documentation</a></li>
<li class="toctree-l1"><a class="reference internal" href="../dev-tools/index.html">Development tools</a></li>
<li class="toctree-l1"><a class="reference internal" href="../dev-tools/testing-overview.html">Testing guide</a></li>
<li class="toctree-l1"><a class="reference internal" href="../kernel-hacking/index.html">Hacking guide</a></li>
<li class="toctree-l1"><a class="reference internal" href="../trace/index.html">Tracing</a></li>
<li class="toctree-l1"><a class="reference internal" href="../fault-injection/index.html">Fault injection</a></li>
<li class="toctree-l1"><a class="reference internal" href="../livepatch/index.html">Livepatching</a></li>
<li class="toctree-l1"><a class="reference internal" href="../rust/index.html">Rust</a></li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../admin-guide/index.html">Administration</a></li>
<li class="toctree-l1"><a class="reference internal" href="../kbuild/index.html">Build system</a></li>
<li class="toctree-l1"><a class="reference internal" href="../admin-guide/reporting-issues.html">Reporting issues</a></li>
<li class="toctree-l1"><a class="reference internal" href="../tools/index.html">Userspace tools</a></li>
<li class="toctree-l1"><a class="reference internal" href="../userspace-api/index.html">Userspace API</a></li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../firmware-guide/index.html">Firmware</a></li>
<li class="toctree-l1"><a class="reference internal" href="../devicetree/index.html">Firmware and Devicetree</a></li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../arch/index.html">CPU architectures</a></li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../staging/index.html">Unsorted documentation</a></li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../translations/index.html">Translations</a></li>
</ul>
</div>
<script type="text/javascript"> <!--
var sbar = document.getElementsByClassName("sphinxsidebar")[0];
let currents = document.getElementsByClassName("current")
if (currents.length) {
sbar.scrollTop = currents[currents.length - 1].offsetTop;
}
--> </script>
<div role="note" aria-label="source link">
<h3>This Page</h3>
<ul class="this-page-menu">
<li><a href="../_sources/filesystems/squashfs.rst.txt"
rel="nofollow">Show Source</a></li>
</ul>
</div>
</div>
</div>
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body" role="main">
<section id="squashfs-4-0-filesystem">
<h1>Squashfs 4.0 Filesystem<a class="headerlink" href="#squashfs-4-0-filesystem" title="Link to this heading">¶</a></h1>
<p>Squashfs is a compressed read-only filesystem for Linux.</p>
<p>It uses zlib, lz4, lzo, xz or zstd compression to compress files, inodes and
directories. Inodes in the system are very small and all blocks are packed to
minimise data overhead. Block sizes greater than 4K are supported up to a
maximum of 1Mbytes (default block size 128K).</p>
<p>Squashfs is intended for general read-only filesystem use, for archival
use (i.e. in cases where a .tar.gz file may be used), and in constrained
block device/memory systems (e.g. embedded systems) where low overhead is
needed.</p>
<p>Mailing list (kernel code): <a class="reference external" href="mailto:linux-fsdevel%40vger.kernel.org">linux-fsdevel<span>@</span>vger<span>.</span>kernel<span>.</span>org</a>
Web site: github.com/plougher/squashfs-tools</p>
<section id="filesystem-features">
<h2>1. Filesystem Features<a class="headerlink" href="#filesystem-features" title="Link to this heading">¶</a></h2>
<p>Squashfs filesystem features versus Cramfs:</p>
<table class="docutils align-default">
<thead>
<tr class="row-odd"><th class="head"></th>
<th class="head"></th>
<th class="head"></th>
</tr>
</thead>
<tbody>
<tr class="row-even"><td><p>Max filesystem size</p></td>
<td><p>2^64</p></td>
<td><p>256 MiB</p></td>
</tr>
<tr class="row-odd"><td><p>Max file size</p></td>
<td><p>~ 2 TiB</p></td>
<td><p>16 MiB</p></td>
</tr>
<tr class="row-even"><td><p>Max files</p></td>
<td><p>unlimited</p></td>
<td><p>unlimited</p></td>
</tr>
<tr class="row-odd"><td><p>Max directories</p></td>
<td><p>unlimited</p></td>
<td><p>unlimited</p></td>
</tr>
<tr class="row-even"><td><p>Max entries per directory</p></td>
<td><p>unlimited</p></td>
<td><p>unlimited</p></td>
</tr>
<tr class="row-odd"><td><p>Max block size</p></td>
<td><p>1 MiB</p></td>
<td><p>4 KiB</p></td>
</tr>
<tr class="row-even"><td><p>Metadata compression</p></td>
<td><p>yes</p></td>
<td><p>no</p></td>
</tr>
<tr class="row-odd"><td><p>Directory indexes</p></td>
<td><p>yes</p></td>
<td><p>no</p></td>
</tr>
<tr class="row-even"><td><p>Sparse file support</p></td>
<td><p>yes</p></td>
<td><p>no</p></td>
</tr>
<tr class="row-odd"><td><p>Tail-end packing (fragments)</p></td>
<td><p>yes</p></td>
<td><p>no</p></td>
</tr>
<tr class="row-even"><td><p>Exportable (NFS etc.)</p></td>
<td><p>yes</p></td>
<td><p>no</p></td>
</tr>
<tr class="row-odd"><td><p>Hard link support</p></td>
<td><p>yes</p></td>
<td><p>no</p></td>
</tr>
<tr class="row-even"><td><p>“.” and “..” in readdir</p></td>
<td><p>yes</p></td>
<td><p>no</p></td>
</tr>
<tr class="row-odd"><td><p>Real inode numbers</p></td>
<td><p>yes</p></td>
<td><p>no</p></td>
</tr>
<tr class="row-even"><td><p>32-bit uids/gids</p></td>
<td><p>yes</p></td>
<td><p>no</p></td>
</tr>
<tr class="row-odd"><td><p>File creation time</p></td>
<td><p>yes</p></td>
<td><p>no</p></td>
</tr>
<tr class="row-even"><td><p>Xattr support</p></td>
<td><p>yes</p></td>
<td><p>no</p></td>
</tr>
<tr class="row-odd"><td><p>ACL support</p></td>
<td><p>no</p></td>
<td><p>no</p></td>
</tr>
</tbody>
</table>
<p>Squashfs compresses data, inodes and directories. In addition, inode and
directory data are highly compacted, and packed on byte boundaries. Each
compressed inode is on average 8 bytes in length (the exact length varies on
file type, i.e. regular file, directory, symbolic link, and block/char device
inodes have different sizes).</p>
</section>
<section id="using-squashfs">
<h2>2. Using Squashfs<a class="headerlink" href="#using-squashfs" title="Link to this heading">¶</a></h2>
<p>As squashfs is a read-only filesystem, the mksquashfs program must be used to
create populated squashfs filesystems. This and other squashfs utilities
are very likely packaged by your linux distribution (called squashfs-tools).
The source code can be obtained from github.com/plougher/squashfs-tools.
Usage instructions can also be obtained from this site.</p>
</section>
<section id="mount-options">
<h2>2.1 Mount options<a class="headerlink" href="#mount-options" title="Link to this heading">¶</a></h2>
<table class="docutils align-default">
<tbody>
<tr class="row-odd"><td><p>errors=%s</p></td>
<td><p>Specify whether squashfs errors trigger a kernel panic
or not</p>
<table class="docutils align-default">
<tbody>
<tr class="row-odd"><td><p>continue</p></td>
<td><p>errors don’t trigger a panic (default)</p></td>
</tr>
<tr class="row-even"><td><p>panic</p></td>
<td><p>trigger a panic when errors are encountered,
similar to several other filesystems (e.g.
btrfs, ext4, f2fs, GFS2, jfs, ntfs, ubifs)</p>
<p>This allows a kernel dump to be saved,
useful for analyzing and debugging the
corruption.</p>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
<tr class="row-even"><td><p>threads=%s</p></td>
<td><p>Select the decompression mode or the number of threads</p>
<p>If SQUASHFS_CHOICE_DECOMP_BY_MOUNT is set:</p>
<table class="docutils align-default">
<tbody>
<tr class="row-odd"><td><p>single</p></td>
<td><p>use single-threaded decompression (default)</p>
<p>Only one block (data or metadata) can be
decompressed at any one time. This limits
CPU and memory usage to a minimum, but it
also gives poor performance on parallel I/O
workloads when using multiple CPU machines
due to waiting on decompressor availability.</p>
</td>
</tr>
<tr class="row-even"><td><p>multi</p></td>
<td><p>use up to two parallel decompressors per core</p>
<p>If you have a parallel I/O workload and your
system has enough memory, using this option
may improve overall I/O performance. It
dynamically allocates decompressors on a
demand basis.</p>
</td>
</tr>
<tr class="row-odd"><td><p>percpu</p></td>
<td><p>use a maximum of one decompressor per core</p>
<p>It uses percpu variables to ensure
decompression is load-balanced across the
cores.</p>
</td>
</tr>
<tr class="row-even"><td><p>1|2|3|...</p></td>
<td><p>configure the number of threads used for
decompression</p>
<p>The upper limit is <code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">num_online_cpus()</span></code> * 2.</p>
</td>
</tr>
</tbody>
</table>
<p>If SQUASHFS_CHOICE_DECOMP_BY_MOUNT is <strong>not</strong> set and
SQUASHFS_DECOMP_MULTI, SQUASHFS_MOUNT_DECOMP_THREADS are
both set:</p>
<table class="docutils align-default">
<tbody>
<tr class="row-odd"><td><p>2|3|...</p></td>
<td><p>configure the number of threads used for
decompression</p>
<p>The upper limit is <code class="xref c c-func broken_xref docutils literal notranslate"><span class="pre">num_online_cpus()</span></code> * 2.</p>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</section>
<section id="squashfs-filesystem-design">
<h2>3. Squashfs Filesystem Design<a class="headerlink" href="#squashfs-filesystem-design" title="Link to this heading">¶</a></h2>
<p>A squashfs filesystem consists of a maximum of nine parts, packed together on a
byte alignment:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span> ---------------
| superblock |
|---------------|
| compression |
| options |
|---------------|
| datablocks |
| & fragments |
|---------------|
| inode table |
|---------------|
| directory |
| table |
|---------------|
| fragment |
| table |
|---------------|
| export |
| table |
|---------------|
| uid/gid |
| lookup table |
|---------------|
| xattr |
| table |
---------------
</pre></div>
</div>
<p>Compressed data blocks are written to the filesystem as files are read from
the source directory, and checked for duplicates. Once all file data has been
written the completed inode, directory, fragment, export, uid/gid lookup and
xattr tables are written.</p>
</section>
<section id="compression-options">
<h2>3.1 Compression options<a class="headerlink" href="#compression-options" title="Link to this heading">¶</a></h2>
<p>Compressors can optionally support compression specific options (e.g.
dictionary size). If non-default compression options have been used, then
these are stored here.</p>
</section>
<section id="inodes">
<h2>3.2 Inodes<a class="headerlink" href="#inodes" title="Link to this heading">¶</a></h2>
<p>Metadata (inodes and directories) are compressed in 8Kbyte blocks. Each
compressed block is prefixed by a two byte length, the top bit is set if the
block is uncompressed. A block will be uncompressed if the -noI option is set,
or if the compressed block was larger than the uncompressed block.</p>
<p>Inodes are packed into the metadata blocks, and are not aligned to block
boundaries, therefore inodes overlap compressed blocks. Inodes are identified
by a 48-bit number which encodes the location of the compressed metadata block
containing the inode, and the byte offset into that block where the inode is
placed (<block, offset>).</p>
<p>To maximise compression there are different inodes for each file type
(regular file, directory, device, etc.), the inode contents and length
varying with the type.</p>
<p>To further maximise compression, two types of regular file inode and
directory inode are defined: inodes optimised for frequently occurring
regular files and directories, and extended types where extra
information has to be stored.</p>
</section>
<section id="directories">
<h2>3.3 Directories<a class="headerlink" href="#directories" title="Link to this heading">¶</a></h2>
<p>Like inodes, directories are packed into compressed metadata blocks, stored
in a directory table. Directories are accessed using the start address of
the metablock containing the directory and the offset into the
decompressed block (<block, offset>).</p>
<p>Directories are organised in a slightly complex way, and are not simply
a list of file names. The organisation takes advantage of the
fact that (in most cases) the inodes of the files will be in the same
compressed metadata block, and therefore, can share the start block.
Directories are therefore organised in a two level list, a directory
header containing the shared start block value, and a sequence of directory
entries, each of which share the shared start block. A new directory header
is written once/if the inode start block changes. The directory
header/directory entry list is repeated as many times as necessary.</p>
<p>Directories are sorted, and can contain a directory index to speed up
file lookup. Directory indexes store one entry per metablock, each entry
storing the index/filename mapping to the first directory header
in each metadata block. Directories are sorted in alphabetical order,
and at lookup the index is scanned linearly looking for the first filename
alphabetically larger than the filename being looked up. At this point the
location of the metadata block the filename is in has been found.
The general idea of the index is to ensure only one metadata block needs to be
decompressed to do a lookup irrespective of the length of the directory.
This scheme has the advantage that it doesn’t require extra memory overhead
and doesn’t require much extra storage on disk.</p>
</section>
<section id="file-data">
<h2>3.4 File data<a class="headerlink" href="#file-data" title="Link to this heading">¶</a></h2>
<p>Regular files consist of a sequence of contiguous compressed blocks, and/or a
compressed fragment block (tail-end packed block). The compressed size
of each datablock is stored in a block list contained within the
file inode.</p>
<p>To speed up access to datablocks when reading ‘large’ files (256 Mbytes or
larger), the code implements an index cache that caches the mapping from
block index to datablock location on disk.</p>
<p>The index cache allows Squashfs to handle large files (up to 1.75 TiB) while
retaining a simple and space-efficient block list on disk. The cache
is split into slots, caching up to eight 224 GiB files (128 KiB blocks).
Larger files use multiple slots, with 1.75 TiB files using all 8 slots.
The index cache is designed to be memory efficient, and by default uses
16 KiB.</p>
</section>
<section id="fragment-lookup-table">
<h2>3.5 Fragment lookup table<a class="headerlink" href="#fragment-lookup-table" title="Link to this heading">¶</a></h2>
<p>Regular files can contain a fragment index which is mapped to a fragment
location on disk and compressed size using a fragment lookup table. This
fragment lookup table is itself stored compressed into metadata blocks.
A second index table is used to locate these. This second index table for
speed of access (and because it is small) is read at mount time and cached
in memory.</p>
</section>
<section id="uid-gid-lookup-table">
<h2>3.6 Uid/gid lookup table<a class="headerlink" href="#uid-gid-lookup-table" title="Link to this heading">¶</a></h2>
<p>For space efficiency regular files store uid and gid indexes, which are
converted to 32-bit uids/gids using an id look up table. This table is
stored compressed into metadata blocks. A second index table is used to
locate these. This second index table for speed of access (and because it
is small) is read at mount time and cached in memory.</p>
</section>
<section id="export-table">
<h2>3.7 Export table<a class="headerlink" href="#export-table" title="Link to this heading">¶</a></h2>
<p>To enable Squashfs filesystems to be exportable (via NFS etc.) filesystems
can optionally (disabled with the -no-exports Mksquashfs option) contain
an inode number to inode disk location lookup table. This is required to
enable Squashfs to map inode numbers passed in filehandles to the inode
location on disk, which is necessary when the export code reinstantiates
expired/flushed inodes.</p>
<p>This table is stored compressed into metadata blocks. A second index table is
used to locate these. This second index table for speed of access (and because
it is small) is read at mount time and cached in memory.</p>
</section>
<section id="xattr-table">
<h2>3.8 Xattr table<a class="headerlink" href="#xattr-table" title="Link to this heading">¶</a></h2>
<p>The xattr table contains extended attributes for each inode. The xattrs
for each inode are stored in a list, each list entry containing a type,
name and value field. The type field encodes the xattr prefix
(“user.”, “trusted.” etc) and it also encodes how the name/value fields
should be interpreted. Currently the type indicates whether the value
is stored inline (in which case the value field contains the xattr value),
or if it is stored out of line (in which case the value field stores a
reference to where the actual value is stored). This allows large values
to be stored out of line improving scanning and lookup performance and it
also allows values to be de-duplicated, the value being stored once, and
all other occurrences holding an out of line reference to that value.</p>
<p>The xattr lists are packed into compressed 8K metadata blocks.
To reduce overhead in inodes, rather than storing the on-disk
location of the xattr list inside each inode, a 32-bit xattr id
is stored. This xattr id is mapped into the location of the xattr
list using a second xattr id lookup table.</p>
</section>
<section id="todos-and-outstanding-issues">
<h2>4. TODOs and Outstanding Issues<a class="headerlink" href="#todos-and-outstanding-issues" title="Link to this heading">¶</a></h2>
</section>
<section id="todo-list">
<h2>4.1 TODO list<a class="headerlink" href="#todo-list" title="Link to this heading">¶</a></h2>
<p>Implement ACL support.</p>
</section>
<section id="squashfs-internal-cache">
<h2>4.2 Squashfs Internal Cache<a class="headerlink" href="#squashfs-internal-cache" title="Link to this heading">¶</a></h2>
<p>Blocks in Squashfs are compressed. To avoid repeatedly decompressing
recently accessed data Squashfs uses two small metadata and fragment caches.</p>
<p>The cache is not used for file datablocks, these are decompressed and cached in
the page-cache in the normal way. The cache is used to temporarily cache
fragment and metadata blocks which have been read as a result of a metadata
(i.e. inode or directory) or fragment access. Because metadata and fragments
are packed together into blocks (to gain greater compression) the read of a
particular piece of metadata or fragment will retrieve other metadata/fragments
which have been packed with it, these because of locality-of-reference may be
read in the near future. Temporarily caching them ensures they are available
for near future access without requiring an additional read and decompress.</p>
<p>In the future this internal cache may be replaced with an implementation which
uses the kernel page cache. Because the page cache operates on page sized
units this may introduce additional complexity in terms of locking and
associated race conditions.</p>
</section>
</section>
</div>
</div>
</div>
<div class="clearer"></div>
</div>
<div class="footer">
©The kernel development community.
|
Powered by <a href="https://www.sphinx-doc.org/">Sphinx 8.1.3</a>
& <a href="https://alabaster.readthedocs.io">Alabaster 0.7.16</a>
|
<a href="../_sources/filesystems/squashfs.rst.txt"
rel="nofollow">Page source</a>
</div>
</body>
</html>