Ignoring Golden Rules
Sometimes you just start doing something with good intentions, thinking it will take you a few minutes… and finish a couple of days later, rethinking how many things went wrong and how much time of your life you spent doing a really unnecessary thing.
Well, that could be a sum of my last unnecessary IT adventure: swapping the boot device of my Raspberry from microSD to a USB-attached disk.
All Wrong From The Beginning
Last week, I got an email from Google: my 100 GB storage subscription was expiring. Since I’m trying to deGoogle for a few years, in 2025 I installed Immich on my homemade tiny server and started backing up my memories there. I’m quite happy with it: it works almost flawlessly and is stable. I finally felt ready to leave Google Photos, which undoubtedly works very well.
I pondered how to make the server even more robust, since it’s not a VM and it would be neither easy nor immediate to restore in case of failure.
First, I found raspiBackup to create quick and almost-incremental backups of the whole system (the /boot and / partitions, enough for me).
The most fragile aspect of the machine, which I had feared for almost nine months of running time, is the microSD on which the OS is running.
As a golden rule, microSDs tend to fail — they’re very human from this point of view.
For a long-running machine, it’s a good idea to replace it with an external disk.
I had a spare cheap SSD I bought from AliExpress a couple of years ago, and that should be more reliable than a microSD despite its quality. I put it in an even cheaper USB case from AUKEY and…
The USB case has no apparent issues with macOS or Windows, but on Linux (Raspbian) it kept disconnecting.
The dmesg kept updating every 30 seconds and partitions disappeared continuously.
I thought it was the cable, but even after swapping the 2.5" disk with another one I had the same results.
So the only solution was changing the enclosure.
I ordered a new one and waited for it.
With the new case, the SSD was seen with no issues.
Obstacle one cleared.
The Raspberry 5 Is Complicated
Last year I was lucky to buy a Pi5 before the crazy rise in prices, and I’m very happy with the choice of this version. I knew that it is the first one capable of booting from a USB drive out of the box, but I was already using a microSD with a quite complex configuration so I needed to migrate the data. I looked for ways to clone the content from the SD to a USB drive and found rpi-clone. In theory it is amazing; it would make everything with just one command! But of course it has some issues with this hardware.
I tried to use it, and the system didn’t boot from the USB drive.
Neither the help of Claude to edit the rpi-eeprom-config nor the cmdline.txt worked.
Plan failed. Oof. So I decided to make the smart move.1
Since the actual system is running RaspberryOS 12 Bookworm, I could flash a newer v13 –Trixie– on the SSD and move all the configuration manually. Luckily, 90% of the data are stored on a ZFS mirror and almost everything runs with Docker.
I stop for a moment, thinking about the golden rule…
…and I ignore it.
First, I make a backup with raspiBackup of the whole microSD on an external HDD (let’s call it Woody because of the brand — yeah, WD), then attach the USB drive and boot from it.
No issues. Whew, lucky me. Do consider that I do not have an external monitor to attach to the machine,2 so troubleshooting boot issues could be hard — or better, will be hard.
Then, rsync time!
After installing ZFS, I slowly move all the application files to the new Trixie installation, the systemd services, some customizations,…
Luckily I don’t have to reset any permissions since I use just a couple of users.
It took some time but it was quite straightforward and painless.
Only PiVPN required recreating configuration files; I wasn’t able to import the old ones.
And even after years, I am always amazed by the simplicity of Docker.
Everything is running. I release the tension — actually I was not worried at all because I had the working microSD and the backup on Woody the HDD. I even removed the microSD and archived it in a closet.
Never Let Your Computer Sense Your Urgency
The next morning, I had the even smarter idea to use a couple of zip ties to organize the USB cables of the hard disks – Woody and the boot SSD. Maybe they could be used to keep the two disks tight to the 3D-printed cover of the tiny server. It wouldn’t be elegant, but handy to move around in case of necessity.
I unplugged the disks, tidied the cables, reattached everything, and… the Raspberry did not boot. It powered on, but the network did not come up. I reinserted the microSD and it booted flawlessly.
What did I do instead? I booted from the microSD and mounted the SSD, then tried to dig into the logs to find some info. Well, guess what? Trixie does not store any logs by default. Thanks, Raspberry.
At a certain point, trying to figure out what I had done before powering off the computer, I plugged Woody in to restore some files, and… I got many errors.
The partition was not seen by the system.
I unmounted it, ran fsck… and got scared.
$ sudo e2fsck -f /dev/sdc1
e2fsck 1.47.0 (5-Feb-2023)
Error reading block 1232 (Input/output error). Ignore error<y>? yes
Force rewrite<y>? yes
Superblock has an invalid journal (inode 8).
Clear<y>? yes
*** journal has been deleted ***
Resize inode not valid. Recreate<y>? yes
Pass 1: Checking inodes, blocks, and sizes
Root inode is not a directory. Clear<y>?
I thought the whole disk was lost. And it was! All the backups were gone; the (running) microSD was my only plan to return to a functioning system.
I kept trying to boot from the SSD blindly but had no clue of the actual root problem. I wouldn’t have found anything in any case in logs because the cause was… random. And related to the power supplies.
Truth be told, as soon as I attached the monitor, the system booted from USB without any changes. That little bastard! Then I rebooted it, and the message appeared, clear as day. The tiny server has the official 27W USB-C power supply and a 12V/5A supply attached to the HAT dedicated to the two 3.5" HDDs of the ZFS pool. Despite this, when I finally got the external monitor attached to the Raspberry, the error said that the power was not enough for booting from a USB port(!).
At that point, the message itself already contained the solution: add the line usb_max_current_enable=1 at the end of the damn config.txt file.
After that, all the issues disappeared.
And when I reconnected Woody, all the backups were there!
As if the fsck did nothing!
That’s the actual mystery I haven’t found a solution for yet.
King of Regrets
I waited something like FOUR HOURS to get that damn monitor!
Four hours on this planet that no one will give me back.
And what bothers me is that I knew I was acting wrong.
I had suspected power issues, but the day before everything was fine; that’s why I called these errors random before.
I really would like to blame Claude, whom I asked for help again, who kept insisting on the cmdline.txt issue… But it was my fault.
I want to take full responsibility for this time lost.
I surely learned many things, especially about the boot procedure of the Raspberry! I try to look at the bright side of the story, but it’s true: when you want to make some simple things, sometimes you spend ten times the effort you thought — and maybe that they deserved.
And overall, the most important lesson in IT remains always the same: if it works and you are not forced to change it, let it be.
🎧 Impressed by Monolord’s last album
📖 I started No Country For Old Men and I’m loving it
Reactions
How to interact with this page
Via Webmention
Write about this page on your own website and send a
Webmention
to https://blog.scaglio.id/posts/ignoring-golden-rules/.
Your post will appear below.
By email
You can also send an evegreen
Sometimes you just start doing something with good intentions, thinking it will take you a few minutes… and finish a couple of days later, rethinking how many things went wrong and how much time of your life you spent doing a really unnecessary thing.
Well, that could be a sum of my last unnecessary IT adventure: swapping the boot device of my Raspberry from microSD to a USB-attached disk.
All Wrong From The Beginning
Last week, I got an email from Google: my 100 GB storage subscription was expiring. Since I’m trying to deGoogle for a few years, in 2025 I installed Immich on my homemade tiny server and started backing up my memories there. I’m quite happy with it: it works almost flawlessly and is stable. I finally felt ready to leave Google Photos, which undoubtedly works very well.
I pondered how to make the server even more robust, since it’s not a VM and it would be neither easy nor immediate to restore in case of failure.
First, I found raspiBackup to create quick and almost-incremental backups of the whole system (the /boot and / partitions, enough for me).
The most fragile aspect of the machine, which I had feared for almost nine months of running time, is the microSD on which the OS is running.
As a golden rule, microSDs tend to fail — they’re very human from this point of view.
For a long-running machine, it’s a good idea to replace it with an external disk.
I had a spare cheap SSD I bought from AliExpress a couple of years ago, and that should be more reliable than a microSD despite its quality. I put it in an even cheaper USB case from AUKEY and…
The USB case has no apparent issues with macOS or Windows, but on Linux (Raspbian) it kept disconnecting.
The dmesg kept updating every 30 seconds and partitions disappeared continuously.
I thought it was the cable, but even after swapping the 2.5" disk with another one I had the same results.
So the only solution was changing the enclosure.
I ordered a new one and waited for it.
With the new case, the SSD was seen with no issues.
Obstacle one cleared.
The Raspberry 5 Is Complicated
Last year I was lucky to buy a Pi5 before the crazy rise in prices, and I’m very happy with the choice of this version. I knew that it is the first one capable of booting from a USB drive out of the box, but I was already using a microSD with a quite complex configuration so I needed to migrate the data. I looked for ways to clone the content from the SD to a USB drive and found rpi-clone. In theory it is amazing; it would make everything with just one command! But of course it has some issues with this hardware.
I tried to use it, and the system didn’t boot from the USB drive.
Neither the help of Claude to edit the rpi-eeprom-config nor the cmdline.txt worked.
Plan failed. Oof. So I decided to make the smart move.1
Since the actual system is running RaspberryOS 12 Bookworm, I could flash a newer v13 –Trixie– on the SSD and move all the configuration manually. Luckily, 90% of the data are stored on a ZFS mirror and almost everything runs with Docker.
I stop for a moment, thinking about the golden rule…
…and I ignore it.
First, I make a backup with raspiBackup of the whole microSD on an external HDD (let’s call it Woody because of the brand — yeah, WD), then attach the USB drive and boot from it.
No issues. Whew, lucky me. Do consider that I do not have an external monitor to attach to the machine,2 so troubleshooting boot issues could be hard — or better, will be hard.
Then, rsync time!
After installing ZFS, I slowly move all the application files to the new Trixie installation, the systemd services, some customizations,…
Luckily I don’t have to reset any permissions since I use just a couple of users.
It took some time but it was quite straightforward and painless.
Only PiVPN required recreating configuration files; I wasn’t able to import the old ones.
And even after years, I am always amazed by the simplicity of Docker.
Everything is running. I release the tension — actually I was not worried at all because I had the working microSD and the backup on Woody the HDD. I even removed the microSD and archived it in a closet.
Never Let Your Computer Sense Your Urgency
The next morning, I had the even smarter idea to use a couple of zip ties to organize the USB cables of the hard disks – Woody and the boot SSD. Maybe they could be used to keep the two disks tight to the 3D-printed cover of the tiny server. It wouldn’t be elegant, but handy to move around in case of necessity.
I unplugged the disks, tidied the cables, reattached everything, and… the Raspberry did not boot. It powered on, but the network did not come up. I reinserted the microSD and it booted flawlessly.
What did I do instead? I booted from the microSD and mounted the SSD, then tried to dig into the logs to find some info. Well, guess what? Trixie does not store any logs by default. Thanks, Raspberry.
At a certain point, trying to figure out what I had done before powering off the computer, I plugged Woody in to restore some files, and… I got many errors.
The partition was not seen by the system.
I unmounted it, ran fsck… and got scared.
$ sudo e2fsck -f /dev/sdc1
e2fsck 1.47.0 (5-Feb-2023)
Error reading block 1232 (Input/output error). Ignore error<y>? yes
Force rewrite<y>? yes
Superblock has an invalid journal (inode 8).
Clear<y>? yes
*** journal has been deleted ***
Resize inode not valid. Recreate<y>? yes
Pass 1: Checking inodes, blocks, and sizes
Root inode is not a directory. Clear<y>?
I thought the whole disk was lost. And it was! All the backups were gone; the (running) microSD was my only plan to return to a functioning system.
I kept trying to boot from the SSD blindly but had no clue of the actual root problem. I wouldn’t have found anything in any case in logs because the cause was… random. And related to the power supplies.
Truth be told, as soon as I attached the monitor, the system booted from USB without any changes. That little bastard! Then I rebooted it, and the message appeared, clear as day. The tiny server has the official 27W USB-C power supply and a 12V/5A supply attached to the HAT dedicated to the two 3.5" HDDs of the ZFS pool. Despite this, when I finally got the external monitor attached to the Raspberry, the error said that the power was not enough for booting from a USB port(!).
At that point, the message itself already contained the solution: add the line usb_max_current_enable=1 at the end of the damn config.txt file.
After that, all the issues disappeared.
And when I reconnected Woody, all the backups were there!
As if the fsck did nothing!
That’s the actual mystery I haven’t found a solution for yet.
King of Regrets
I waited something like FOUR HOURS to get that damn monitor!
Four hours on this planet that no one will give me back.
And what bothers me is that I knew I was acting wrong.
I had suspected power issues, but the day before everything was fine; that’s why I called these errors random before.
I really would like to blame Claude, whom I asked for help again, who kept insisting on the cmdline.txt issue… But it was my fault.
I want to take full responsibility for this time lost.
I surely learned many things, especially about the boot procedure of the Raspberry! I try to look at the bright side of the story, but it’s true: when you want to make some simple things, sometimes you spend ten times the effort you thought — and maybe that they deserved.
And overall, the most important lesson in IT remains always the same: if it works and you are not forced to change it, let it be.
🎧 Impressed by Monolord’s last album
📖 I started No Country For Old Men and I’m loving it