BTRFS filesystem re-creation

Unmount faulty FS

The faulty filesystem is mounted readonly:

luc@kid2:~$ mount|grep sdd
/dev/sdd on /var/backups/urbackup type btrfs (ro,relatime,space_cache,subvolid=5,subvol=/)

sudo umount /var/backups/urbackup

Check disks

Checking the underlying disk hardware (readonly) using badblocks.  This takes time (about 3:30 minutes per percentage point on the 2TB drives so it'll keep the system busy for 700 minutes (!) :

sudo badblocks -s /dev/sdc
sudo badblocks -s /dev/sdd

Recreate BTRFS filesystem with first disk

After a long while, both commands ended without mentioning any error blocks so:

sudo mkfs.btrfs -f /dev/sdd

 

 btrfs-progs v5.16.2
See http://btrfs.wiki.kernel.org for more information.

NOTE: several default settings have changed in version 5.15, please make sure
      this does not affect your deployments:
      - DUP for metadata (-m dup)
      - enabled no-holes (-O no-holes)
      - enabled free-space-tree (-R free-space-tree)

Label:              (null)
UUID:               ddd8d501-3bd0-4afd-b501-66340f14f587
Node size:          16384
Sector size:        4096
Filesystem size:    1.82TiB
Block group profiles:
  Data:             single            8.00MiB
  Metadata:         DUP               1.00GiB
  System:           DUP               8.00MiB
SSD detected:       no
Zoned device:       no
Incompat features:  extref, skinny-metadata, no-holes
Runtime features:   free-space-tree
Checksum:           crc32c
Number of devices:  1
Devices:
   ID        SIZE  PATH
    1     1.82TiB  /dev/sdd

Copy the UUID (see above) and update the fstab entry so we can again mount it by default:

luc@kid2:~$ sudo vi /etc/fstab

luc@kid2:~$ sudo mount /var/backups/urbackup/
luc@kid2:~$ df -lm
Filesystem     1M-blocks  Used Available Use% Mounted on
tmpfs                198     2       197   1% /run
/dev/sda4          37910  5061     30892  15% /
tmpfs                988     0       988   0% /dev/shm
tmpfs                  5     0         5   0% /run/lock
/dev/sdb1         299365 51273    232815  19% /home
tmpfs                988     0       988   0% /run/qemu
/dev/sda2          80125 17764     58246  24% /var
tmpfs                198     1       198   1% /run/user/1000
/dev/sdd         1907730     4   1905665   1% /var/backups/urbackup

Re-install urbackup server:

luc@kid2:~$ sudo apt-get install urbackup-server
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
  urbackup-server
0 upgraded, 1 newly installed, 0 to remove and 12 not upgraded.
Need to get 0 B/4.553 kB of archives.
After this operation, 25,1 MB of additional disk space will be used.
Preconfiguring packages ...
Selecting previously unselected package urbackup-server.
(Reading database ... 150865 files and directories currently installed.)
Preparing to unpack .../urbackup-server_2.5.27.0-1ubuntu1~jammy_amd64.deb ...
Unpacking urbackup-server (2.5.27.0-1ubuntu1~jammy) ...
Setting up urbackup-server (2.5.27.0-1ubuntu1~jammy) ...
Processing triggers for man-db (2.10.2-1) ...

Add the second disk

sudo btrfs device add -f /dev/sdc /var/backups/urbackup/

sudo btrfs device usage /var/backups/urbackup/
/dev/sdd, ID: 1
   Device size:             1.82TiB
   Device slack:              0.00B
   Data,single:             8.00MiB
   Metadata,DUP:            2.00GiB
   System,DUP:             16.00MiB
   Unallocated:             1.82TiB

/dev/sdc, ID: 2
   Device size:             1.82TiB
   Device slack:              0.00B
   Unallocated:             1.82TiB

df -lm
Filesystem     1M-blocks  Used Available Use% Mounted on
tmpfs                198     2       197   1% /run
/dev/sda4          37910  5085     30868  15% /
tmpfs                988     0       988   0% /dev/shm
tmpfs                  5     0         5   0% /run/lock
/dev/sdb1         299365 51273    232815  19% /home
tmpfs                988     0       988   0% /run/qemu
/dev/sda2          80125 17764     58246  24% /var
tmpfs                198     1       198   1% /run/user/1000
/dev/sdd         3815459     4   3813393   1% /var/backups/urbackup

Change ownership then recreate tmp directory

I think this normally defaults to /tmp and thus the server will not try to create it.  In my case the root FS is too small and I created this directory elsewhere, but it had been removed by the re-creation of the BTRFS.  So:

 

sudo chown urbackup /var/backups/urbackup

sudo -u urbackup mkdir /var/backups/urbackup/tmp
sudo chmod 750 /var/backups/urbackup/tmp

Restart server:

sudo  systemctl restart urbackupsrv

 

Almost immediately I could connect.  The configuration seems still correct; but no clients have "recent backup" so immediately it started backing up the server itself and my laptop (both obviously active at the time).