BTRFS filesystem re-creation

Unmount faulty FS

The faulty filesystem is mounted readonly:

luc@kid2:~$ mount|grep sdd
/dev/sdd on /var/backups/urbackup type btrfs (ro,relatime,space_cache,subvolid=5,subvol=/)

sudo umount /var/backups/urbackup

Check disks

Checking the underlying disk hardware (readonly) using badblocks.  This takes time (about 3:30 minutes per percentage point on the 2TB drives so it'll keep the system busy for 700 minutes (!) :

sudo badblocks -s /dev/sdc
sudo badblocks -s /dev/sdd

Recreate BTRFS filesystem with first disk

After a long while, both commands ended without mentioning any error blocks so:

sudo mkfs.btrfs -f /dev/sdd

 

 btrfs-progs v5.16.2
See http://btrfs.wiki.kernel.org for more information.

NOTE: several default settings have changed in version 5.15, please make sure
      this does not affect your deployments:
      - DUP for metadata (-m dup)
      - enabled no-holes (-O no-holes)
      - enabled free-space-tree (-R free-space-tree)

Label:              (null)
UUID:               ddd8d501-3bd0-4afd-b501-66340f14f587
Node size:          16384
Sector size:        4096
Filesystem size:    1.82TiB
Block group profiles:
  Data:             single            8.00MiB
  Metadata:         DUP               1.00GiB
  System:           DUP               8.00MiB
SSD detected:       no
Zoned device:       no
Incompat features:  extref, skinny-metadata, no-holes
Runtime features:   free-space-tree
Checksum:           crc32c
Number of devices:  1
Devices:
   ID        SIZE  PATH
    1     1.82TiB  /dev/sdd

Copy the UUID (see above) and update the fstab entry so we can again mount it by default:

luc@kid2:~$ sudo vi /etc/fstab

luc@kid2:~$ sudo mount /var/backups/urbackup/
luc@kid2:~$ df -lm
Filesystem     1M-blocks  Used Available Use% Mounted on
tmpfs                198     2       197   1% /run
/dev/sda4          37910  5061     30892  15% /
tmpfs                988     0       988   0% /dev/shm
tmpfs                  5     0         5   0% /run/lock
/dev/sdb1         299365 51273    232815  19% /home
tmpfs                988     0       988   0% /run/qemu
/dev/sda2          80125 17764     58246  24% /var
tmpfs                198     1       198   1% /run/user/1000
/dev/sdd         1907730     4   1905665   1% /var/backups/urbackup

Re-install urbackup server:

luc@kid2:~$ sudo apt-get install urbackup-server
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
  urbackup-server
0 upgraded, 1 newly installed, 0 to remove and 12 not upgraded.
Need to get 0 B/4.553 kB of archives.
After this operation, 25,1 MB of additional disk space will be used.
Preconfiguring packages ...
Selecting previously unselected package urbackup-server.
(Reading database ... 150865 files and directories currently installed.)
Preparing to unpack .../urbackup-server_2.5.27.0-1ubuntu1~jammy_amd64.deb ...
Unpacking urbackup-server (2.5.27.0-1ubuntu1~jammy) ...
Setting up urbackup-server (2.5.27.0-1ubuntu1~jammy) ...
Processing triggers for man-db (2.10.2-1) ...

Add the second disk

sudo btrfs device add -f /dev/sdc /var/backups/urbackup/

sudo btrfs device usage /var/backups/urbackup/
/dev/sdd, ID: 1
   Device size:             1.82TiB
   Device slack:              0.00B
   Data,single:             8.00MiB
   Metadata,DUP:            2.00GiB
   System,DUP:             16.00MiB
   Unallocated:             1.82TiB

/dev/sdc, ID: 2
   Device size:             1.82TiB
   Device slack:              0.00B
   Unallocated:             1.82TiB

df -lm
Filesystem     1M-blocks  Used Available Use% Mounted on
tmpfs                198     2       197   1% /run
/dev/sda4          37910  5085     30868  15% /
tmpfs                988     0       988   0% /dev/shm
tmpfs                  5     0         5   0% /run/lock
/dev/sdb1         299365 51273    232815  19% /home
tmpfs                988     0       988   0% /run/qemu
/dev/sda2          80125 17764     58246  24% /var
tmpfs                198     1       198   1% /run/user/1000
/dev/sdd         3815459     4   3813393   1% /var/backups/urbackup

Change ownership then recreate tmp directory

I think this normally defaults to /tmp and thus the server will not try to create it.  In my case the root FS is too small and I created this directory elsewhere, but it had been removed by the re-creation of the BTRFS.  So:

 

sudo chown urbackup /var/backups/urbackup

sudo -u urbackup mkdir /var/backups/urbackup/tmp
sudo chmod 750 /var/backups/urbackup/tmp

Restart server:

sudo  systemctl restart urbackupsrv

 

Almost immediately I could connect.  The configuration seems still correct; but no clients have "recent backup" so immediately it started backing up the server itself and my laptop (both obviously active at the time).

BTRFS filesystem errors -- diagnostics and needs to recreate FS again

The BTRFS filesystem used for urbackup backups on /dev/sdd (/var/backups/urbackup) is remounted readonly even after a reboot.  Urbackup can't create new files when backing up so the new backups are failing.

The btrfs check command reports lots of errors.

The dmesg command doesn't report anything special for /dev/sdd nor /dev/sdc so hopefully the hardware is still OK.

Uninstall then reinstall urbackup... I thought that the urbackup DB was also on the failed FS but apparently I have a /var/urbackup directory which has the DB and configuration files.  However I had already done an uninstall so I had to reinstall. The ppa has disappeared probably during an Ubuntu upgrade so I had to add the repository back again:

• sudo add-apt-repository ppa:uroni/urbackup
• sudo apt-get install urbackup-server -y

This apt-get also pulled a lot of libraries and other packages, probably prerequisites.  During install, I chose to "keep" my configuration instead of replacing it with the one coming from the package.

Install went fine but the configure step failed. I still have /etc/urbackup/backupfolder containing the location of the (readonly) directory on BTRFS: /var/backups/urbackup

So the chown is failing due to that directory being readonly for the moment:

 

Setting up urbackup-server (2.5.27.0-1ubuntu1~jammy) ...
chown: changing ownership of '/var/backups/urbackup': Read-only file system
dpkg: error processing package urbackup-server (--configure):
 installed urbackup-server package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 urbackup-server
E: Sub-process /usr/bin/dpkg returned an error code (1)

Thus, remove urbackup-server package again and first work on the BTRFS issue:

sudo apt remove urbackup-server
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages were automatically installed and are no longer required:
  augeas-lenses cpu-checker db-util db5.3-util debootstrap dmeventd extlinux guestfish guestfs-tools guestmount hfsplus ibverbs-providers icoutils ipxe-qemu ipxe-qemu-256k-compat-efi-roms kpartx ldmtool libafflib0v5 libaugeas0 libbfio1 libboost-iostreams1.74.0 libboost-thread1.74.0 libcacard0 libconfig9
  libdate-manip-perl libdaxctl1 libdecor-0-0 libdecor-0-plugin-1-cairo libdevmapper-event1.02.1 libewf2 libfdt1 libgfapi0 libgfrpc0 libgfxdr0 libglusterfs0 libguestfs-hfsplus libguestfs-perl libguestfs-reiserfs libguestfs-tools libguestfs-xfs libguestfs0 libhfsp0 libhivex0 libibverbs1 libintl-perl libintl-xs-perl
  libiscsi7 libldm-1.0-0 liblvm2cmd2.03 libndctl6 libpmem1 libpmemobj1 librados2 librbd1 librdmacm1 librpm9 librpmio9 libsdl2-2.0-0 libslirp0 libspice-server1 libstring-shellquote-perl libsys-virt-perl libtsk19 libusbredirparser1 libvhdi1 libvirglrenderer1 libvirt0 libvmdk1 libwin-hivex-perl libxss1 libyajl2
  libyara8 lsscsi lvm2 msr-tools osinfo-db ovmf qemu-block-extra qemu-system-common qemu-system-data qemu-system-gui qemu-system-x86 qemu-utils rpm-common scrub seabios sleuthkit sqlite3 supermin thin-provisioning-tools virt-p2v zerofree
Use 'sudo apt autoremove' to remove them.
The following packages will be REMOVED:
  urbackup-server
0 upgraded, 0 newly installed, 1 to remove and 12 not upgraded.
1 not fully installed or removed.
After this operation, 25,1 MB disk space will be freed.
Do you want to continue? [Y/n]
(Reading database ... 151049 files and directories currently installed.)
Removing urbackup-server (2.5.27.0-1ubuntu1~jammy) ...
Processing triggers for man-db (2.10.2-1) ...