Improve Swap Perfomance on your RPi

Move swap off your rootfs, and improve performance of your RPi under load!

Introduction

As shipped, the gentoo-on-rpi-64bit image utilizes, for convenience, a small swapfile on the system's root partition (at /var/cache/swap/swap1). However, if you use your RPi4/3 for memory-intensive tasks, such as large package compiles, you may find that when configured in this manner swap paging slows your system significantly (as RPi3s, and entry-level RPi4's, only have 1GiB of physical memory, and it is easy for the I/O to a single device to become congested).

To improve performance, you can set up a dedicated swap device on a different bus from your image's root filing system (rootfs). So, for example, if you have your image on a microSD card, you could set up a swap on USB stick (and vice versa). With this done, you can safely turn off your original rootfs swapfile (and delete it, to save space).

The process involved isn't difficult, and takes only a few minutes! In this short tutorial, I'll run through the necessary steps.

Formatting your New Swap Device

NB: in what follows, I'm going to assume you boot from (and have your rootfs on) microSD card, so you will want your swap device on USB. Obviously, adapt as necessary if you boot from (and have rootfs on) a USB-based device.

Select a high quality USB stick to use as your dedicated swap device (this doesn't need to be very high capacity - 4GB is more than adequate for most RPi workloads, particularly on the 2GiB and 4GiB RPi4 variants). Devices with a small physical form factor may be easier to manage, as you will not safely be able to remove them from your RPi when in use. Ensure that there is no important data currently on the device - it will be wiped by the following process.

Insert the stick into your RPi, and determine its allocated device path, by using lsblk, dmesg | tail, the Applications→Accessories→Disks app, or similar. The device path will be something like /dev/sda, /dev/sdb etc (depending on what other storage devices you have attached to the USB ports of your RPi). If the target device already contains partitions, you may see icons for these appear in 'translucent' form on your desktop shortly after you plug it in, but do not click on them, as we want all such partitions to remain unmounted.

If you are creating swap on a microSD card instead, its device path will almost certainly be /dev/mmcblk0 (but you should always check to make sure).

Next, create, on the target device, a fresh DOS partition table with a single swap partition spanning all available free space. Open a terminal window, and issue:

NB: substitute the actual allocated device path of your USB stick (e.g., /dev/sda, /dev/sdb etc.) for /dev/sdX in the following command. Advanced users can obviously modify the command if desired, for example to create a smaller size swap partition, or to change the partition index used.

demouser@pi64 ~ $ sed 's:\s*#.*$::g' <<EOF | sudo fdisk /dev/sdX
o  # create a new DOS partition table
n  # then create a new partition
p  # which is primary
1  # and is partition 1
   # use the default start sector
   # and default end sector, to fill the drive
t  # change the partition type
82 # to 'Linux swap'
w  # then commit changes and exit
EOF

In the above, sed is used simply to strip out all comments on each line, before passing to fdisk.

Once that completes, format the new partition for use as swap; issue:

demouser@pi64 ~ $ sudo mkswap --label="fast-swap" /dev/sdX1
Setting up swapspace version 1, size = 14.9 GiB (16004411392 bytes)
LABEL=fast-swap, UUID=4095c55e-f703-460e-bf7a-df83c6a32452

NB: substitute your actual target device's first partition path for /dev/sdX1 in the above command. So, if you passed say /dev/sda to fdisk in the previous step, use /dev/sda1 above; if you passed /dev/sdb, use /dev/sdb1; if you passed/dev/mmcblk0, use /dev/mmcblk0p1, and so on.

The actual UUID you see output will be different from the one above, obviously. Take a note of it, as we'll use it shortly.

Editing /etc/fstab to Reference the New Swap Partition

Next, we need to edit the /etc/fstab file, so it references the new swap partition. Before doing that, however, turn off the existing swapfile (the one at /var/cache/swaps/swap1):

demouser@pi64 ~ $ sudo swapoff --verbose --all

Then, open /etc/fstab for editing:

demouser@pi64 ~ $ sudo mousepad /etc/fstab &>/dev/null&

A mousepad editor window should open (running as the root account). Modify the line that currently reads:

/var/cache/swap/swap1 none swap sw 0 0

so that it now reads instead:

UUID=4095c55e-f703-460e-bf7a-df83c6a32452 none swap sw 0 0

NB: substitute the UUID value of 4095c55e-f703-460e-bf7a-df83c6a32452, with that printed (on your system) by mkswap earlier, in the above command. Incidentally, we use UUID-based naming here, as it is safer than using e.g. /dev/sda1 (since if the RPi3 is booted with multiple USB storage devices inserted, we cannot guarantee which will be /dev/sda, which /dev/sdb etc., whereas the UUID will always be unambiguous).

Leave the rest of the file as-is. Save, and exit mousepad.

Now, turn on the new swap to check it works:

demouser@pi64 ~ $ sudo swapon --all --verbose
swapon: /dev/sda1: found signature [pagesize=4096, signature=swap]
swapon: /dev/sda1: pagesize=4096, swapsize=16004415488, devsize=16004415488
swapon /dev/sda1

The output you see will differ from the above, obviously, but the important thing is to check that a swap signature is found, and that the mapped path (here, /dev/sda1) matches the one you passed to mkswap earlier.

Assuming that works (and that only your new swap partition was reported), delete the old swapfile to save space (and ensure it does not get recreated on next boot):

demouser@pi64 ~ $ sudo rm -v /var/cache/swap/swap1
demouser@pi64 ~ $ sudo rc-update del rpi3-expand-swap default

Testing and Use

With your swap device set up, you should now reboot your RPi, to ensure that it is properly activated (via /etc/fstab) at startup.

To do so, once your RPi4/3 comes back up, open a terminal window and issue:

demouser@pi64 ~ $ readlink --canonicalize /dev/disk/by-uuid/4095c55e-f703-460e-bf7a-df83c6a32452
/dev/sda1
demouser@pi64 ~ $ cat /proc/swaps
Filename    Type        Size       Used  Priority
/dev/sda1   partition   15629308   0     -2

NB: substitute the UUID value of 4095c55e-f703-460e-bf7a-df83c6a32452, with that printed (on your system) by mkswap earlier, in the above command. We use readlink to locate the 'standard form' name for this partition (/dev/sda1 or whatever) - if you have multiple devices on your USB bus, it is possible this will have changed on reboot (which is exactly why we used UUID-based naming in /etc/fstab, of course).

Again, your output will likely differ from the above, but the important thing to check is that the partition filename listed by /proc/swaps matches that output by the readlink command.

If so, then congratulations! You have successfully migrated your swapfile, and you should find that your system has much better performance under load.

Finally, one cautionary note: when your system is running, you must never unplug the swap device while it is active - doing so is likely to crash your system. If you do need to remove the device for some reason while the system is running, remember to turn off swapping first (via sudo swapoff --verbose --all).