Photo by Zach Inglis on Unsplash

Run Docker on your Raspberry Pi read-only file system (Raspbian)

Raspberry Pi is ideal for running NAS, trading bots, monitoring agents and many other things. With its low energy consumption and no noise it can be put somewhere, turned on, and just left there for months or even years. There is just one small issue — when the power goes out, SD card is often left in unusable (and even unrepairable) state.

My first Raspberry Pi was model 1B and while running it for a few months I have accumulated a small stack of broken SD cards. The reason was probably that I was actually writing data to the cards, so on almost every time power loss (which unfortunately happened from time to time) the system refused to boot. Needless to say, with a new RPi 4 I was determined not to do the same thing all over again.

There are a few hardware solutions to this problem that you should consider first. An USB flash drive didn’t fare much better than SD cards. Having an UPS (Uninterrupted Power Supply) however would help, and there seems to be a few available that are both cheap and good enough for this purpose. You can also build your own if you know what you are doing. If you go this route, be sure to trigger a controlled shutdown before the power runs out, otherwise you didn’t gain much! But since high uptime was not a requirement for my use case and I didn’t want to add new hardware to my setup, I decided to use a read-only filesystem.

I have followed an excellent blog post on making Raspberry Pi filesystem read-only and it worked like a charm. However for running Docker some additional changes are needed. This post outlines the steps needed.

Step 0: Make your filesystem readonly

Follow this guide on how to make your RPi readonly:

Make your Raspberry Pi file system read-only (Raspbian Buster)

Step 1: improvements

Before making the following changes, make sure you are in read-write mode (i.e., run rw).

First, let’s disable filesystem check for / and /boot in /etc/fstab — since both are read-only, we don’t expect many filesystem errors to occur. We do this by changing the last numbers in each line to 0:

PARTUUID=7f58cf5f-01  /boot  vfat  defaults,ro          0 0
PARTUUID=7f58cf5f-02  /      ext4  defaults,noatime,ro  0 0

Then we need to fix /tmp permissions. As in normal setup, it should be writeable by anyone, but only owner (root) can change its permissions. In /etc/fstab/, we change the /tmp line to read:

tmpfs    /tmp  tmpfs  mode=1777,nosuid,nodev  0 0

Still, if we reboot and check, we will notice that /tmp directory has invalid permissions after boot. The reason is that OS overrides /var/spool permissions, which in our case is linked to /tmp. To fix this we simply mount /var/spool on a separate tmpfs:

# rm /var/spool
# mkdir -m 755 /var/spool
# echo ‘tmpfs /var/spool tmpfs defaults,noatime,nosuid,nodev,noexec,mode=0755,size=64M 0 0’ >> /etc/fstab

After reboot, the /tmp permissions should be correct:

$ ls -ld /tmp
drwxrwxrwt 9 root root 260 Nov 29 10:41 /tmp

Step 2: Installing Docker

First things first:

# rw
# apt install docker.io

While Docker is running, it is changing many things within /var/lib/docker and will refuse to start if this directory is mounted read-only. So as a first step, we make sure that this directory is mounted on tmpfs:

# systemctl stop docker
# systemctl disable docker
# echo 'tmpfs  /var/lib/docker  tmpfs  defaults,noatime,nosuid,nodev,mode=0711  0  0' >> /etc/fstab
# rm -rf /var/lib/docker/*
# reboot

If we start docker now (systemctl start docker), it will work as it should, even in read-only mode. There are just two problems:

• any image that we pull from registry will not be persisted and will need to be re-downloaded after each reboot, and
• we are wasting memory, since all the Docker images are saved into RAM instead of on SD card now.

Step 3: Saving Docker images to persistent storage (SD card)

Docker puts both images and containers in the same directory structure. My first attempt was to try to figure out which is which and just persist the images. However this gets very complicated very soon, and it depends on the internals of /var/lib/docker/ directory for which I was unable to find good docs. The solution outlined below is simpler to implement because it saves the whole docker configuration (/var/lib/docker/) to a persistent storage. Whenever Docker service starts, all the data is copied to tmpfs automatically, with an exception of the images themselves which are only being linked to (to avoid wasting RAM).

Step 3.a: Script for saving existing Docker config to a persistent dir

This script should be saved to /usr/local/bin/dockersave.sh:

Don’t forget to make it executable:

# chmod +x /usr/local/bin/dockersave.sh

Step 3.b: Save initial config

# rw
# dockersave.sh
# ro

Step 3.c: Populating /var/lib/docker from persistent storage whenever Docker starts

Now, we just need to make sure that Docker will actually use the saved configuration when it starts.

This script should be saved to /usr/local/bin/systemctl.dockerload.sh:

Make it executable:

# chmod +x /usr/local/bin/systemctl.dockerload.sh

Then edit /lib/systemd/system/docker.service and add this line just before ExecStart=… line:

ExecStartPre=/usr/local/bin/systemctl.dockerload.sh

Now we reload systemd unit config (systemd doesn’t do that automatically) and start + enable Docker service:

# systemctl daemon-reload
# systemctl enable docker
# systemctl start docker

How to use this

You can use Docker as you would normally. After each service restart (on reboot or manually) all the Docker images will be reset and all the containers will vanish.

When you want to save Docker images to a persistent storage, you must:

• pull any Docker images you wish to persist (or remove those that are no longer needed)
• stop and remove any running containers (using docker stop and docker rm)
• stop Docker (systemctl stop docker)
• go into read-write mode (rw)
• run a script to save current docker configuration (dockersave.sh)
• go back to read-only mode (ro)
• start Docker (systemctl start docker)

Next time the Docker starts, it will have the images at its disposal.

One important thing to note: when using this solution, containers will _NOT_ be started automatically after reboot or service restart (because they are only in memory and will be lost). To start them, the easiest way is to use a systemd unit file.

This is it! I hope you found it useful. I️f you have any questions or suggestions, please start a conversation in the comments.