Linux has changed. Originally inspired by Unix, there were certain well understood but not well enforced rules that everyone understood. Programs did small things and used pipes to communicate. X Windows servers didn’t always run on your local machine. Nothing in /usr contributed to booting up the system.
These days, we have systemd controlling everything. If you run Chrome on one display, it is locked to that display and it really wants that to be the local video card. And moving /usr to another partition will easily prevent you from booting up, unless you take precautions. I moved /usr and I lived to tell about it. If you ever need to do it, you’ll want to hear my story.
A lot of people are critical of systemd — including me — but really it isn’t systemd’s fault. It is the loss of these principles as we get more programmers and many of them are influenced by other systems where things work differently. I’m not just ranting, though. I recently had an experience that brought all this to mind and, along the way, I learned a few things about the modern state of the boot process. The story starts with a friend giving me an Intel Compute Stick. But the problems I had were not specific to that hardware, but rather how modern Linux distributions manage their start-up process.
The Story
As I said, a friend of mine gave me an Intel Compute Stick. It was one of the early ones that was fairly anemic. In particular, it only had 1 GB of memory and 8 GB of flash storage. It booted an old version of Ubuntu and it was, as you’d expect, pretty slow. But I liked the form factor and I have a new workshop that could use a permanent PC, so I decided to upgrade it.
There were the usual problems. A BIOS upgrade broke the network. Upgrading to KDE Neon fixed the network, but the newer kernel had the dreaded C-State bug that caused hangs. Luckily, that’s easy to workaround. So after some effort, I had a reasonably working system. Sort of.
The Problem
The problem was the 8GB of flash. I put a 64GB SDCard in, but I didn’t want to boot from it. With Neon installed and a few other essential things, the flash was very close to 100% full. My plan was to move /opt, /home, and /usr over to the SDCard. I thought it would be easy.
Traditionally, this is a straightforward process. First, copy the files over using something that gets hidden files and links without changing them. Many people use rsync but usually use tar. Then you remove the old directory (I rename it first, just to be safe and delete it after everything is working). Finally, you make a new empty directory and change /etc/fstab to mount the disk at boot time.
For /home and /opt that was fine. The system will boot without difficulty and I had that working in no time. I knew /usr would be a bit harder but I figured I could be in a root shell without the GUI or just boot off a USB drive and do all the same work.
I actually anticipated four mounts. I mounted the entire SDCard at /sdcard. Then I did bind mounts from /sdcard/opt to /opt and /sdcard/home to /home. The /usr mount would also be a bind mount but it wasn’t going to be that easy.
The Bigger Problem
My attempts to move /usr caused the system to stop booting. Why? Turns out systemd handles mounting what’s in /etc/fstab and systemd requires things in /usr. I thought perhaps that systemd would be smart enough to boot the system if it didn’t have to read /etc/fstab so I decided to mount the SDCard using systemd’s native facilities.
Systemd can handle a mount just like a service. That means it will manage the mounting, and also weave it into the dependencies. So it is possible to require a service or other mount to be ready before mounting some disk and, of course, other services and mounts might depend on that disk, as well. In theory, that’s perfect because it doesn’t make sense to try to mount /sdcard/home, for example, before mounting /sdcard.
Here’s the mount definition in /etc/systemd/system/sdcard.mount
[Unit] Description=Main SDCard Mount DefaultDependencies=no Conflicts=umount.target Before=local-fs.target umount.target After=swap.target [Mount] What=/dev/disk/by-uuid/b3b6ac3b-2109-487c-af34-c49586412cea Where=/sdcard Type=ext4 Options=defaults,errors=remount-ro [Install] WantedBy=multi-user.target
Obviously, the UUID would change depending on your disk. Not so obviously, this file must be named sdcard.mount. If the mount point were, say, /usr/lib then the file would have to be usr-lib.mount.
Once that mount occurs, you can mount /home:
[Unit] Description=Home SDCard Mount DefaultDependenices=no Conflicts=umount.target Before=local-fs.target umount.target RequiresMountsFor=/sdcard [Mount] What=/sdcard/home Where=/home Type=none Options=bind,x-systemd.requires-mounts-for=/sdcard [Install] WantedBy=multi-user.target
Note that the mount requires /sdcard. Once you put these files in the right place and reload systems, you can start these units and the files will mount. You can enable them to cause them to start at boot. The /opt unit looks just the same, except for the file names.
Not So Fast!
This still leaves the problem with /usr. Oh sure, it is easy to write the unit, but the problem is that systemd needs some libraries out of /usr so the system will refuse to boot. I considered copying the libraries over to either /lib or somewhere in the initial RAM disk, but after it turned out to be quite a few, I decided against that.
I finally decided that having everything mounted early in the boot process would be the right answer. That way systemd can imagine it has a complete disk. I had actually considered using lvm to join the disks together, but decided that was bad for a lot of reasons and may have had the same problems anyway. I wanted control over what was on the SDcard vs the internal storage, so it was time to look at the initramfs scripts.
Booting (Some) Linux Systems
Most modern Linux distributions don’t boot your root file system directly. Instead, they have a compressed file system that loads into RAM and then boots. This system is responsible for getting the system ready for the real boot. Among other things, it mounts the root file system and then pivots to make it the real root.
For Debian-style distributions, this is the initramfs and you can find some user-definable scripts in /etc/initramfs-tools. The bulk of the predefined ones are in /usr/share/initramfs-tools. In the scripts directory you’ll see a number of subdirectories with suffixes of -top, -bottom, and -premount.
As you might imagine, init-* occurs at system initialization and local-* happens as local disks are mounted. Don’t confuse these with hook scripts. A hook script executes when you are building the initial file system. That helps if you need to modify the boot environment statically. The scripts we want are the ones that execute at boot time.
The top scripts run first, followed by the premount, and then the bottom. So init-top runs first and init-bottom is the last thing that runs. In between, the other scripts run and by local-bottom, the root file system should be ready to go.
Documentation and Gotchas
If you read the documentation, you’ll see that the scripts have a specific format. However, there are some examples that are misleading. For example, the template script shows sourcing /usr/share/initramfs-tools/hook-functions to load common functions. That’s great if /usr already exists, but for us it doesn’t. Some other scripts use a copy that is in the boot environment, located at /usr/share/initramfs-tools/scripts/functions. That’s what I used in my script:
#!/bin/sh
PREREQ=""
prereqs()
{
echo "$PREREQ"
}
case $1 in
prereqs)
prereqs
exit 0
;;
esac
. scripts/functions
# Note: our root is on /root right now, not /
mount /dev/mmcblk2p1 /root/sdcard
mount -o bind /root/sdcard/usr /root/usr
mount -o bind /root/sdcard/home /root/home
mount -o bind /root/sdcard/opt /root/opt
The only tricky part is that our eventual root file system isn’t at /, it is at /root, so the mounts reflect that.
The Result
Of course, I had to disable the systemd mounts for /opt and /home although I could have left them and not put them in this script. Now by the time systemd gets control it can find all the things in /usr it wants and the system boots. Moving those three directories left me with about 70% of the internal storage free and only took up a small fraction of the SDCard.
There are probably many other ways to do this. I mentioned lvm or you could revert back to the old init scripts. But this does work reliably and is very flexible once you get it all figured out.
The Intel stick is pretty small, but we’ve seen smaller. If you do try this at home, don’t forget that logging to eMMC devices isn’t always a good idea.
No comments:
Post a Comment