Renew LetsEncrypt certificate on kid2

sudo certbot renew --force-renewal

First attempt : failed

I had to poke a hole in the router for port 80, apparently certbot is designed to use this port and nothing else (for this renewal method) if I understand it right.  After that it worked fine:

luc@kid2:~$ sudo certbot renew --force-renewal

Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Processing /etc/letsencrypt/renewal/tuxic.dedyn.io.conf

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Renewing an existing certificate for tuxic.dedyn.io

Reloading apache server after certificate renewal

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Congratulations, all renewals succeeded:

  /etc/letsencrypt/live/tuxic.dedyn.io/fullchain.pem (success)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

BTRFS filesystem re-creation

Unmount faulty FS

The faulty filesystem is mounted readonly:

luc@kid2:~$ mount|grep sdd
/dev/sdd on /var/backups/urbackup type btrfs (ro,relatime,space_cache,subvolid=5,subvol=/)

sudo umount /var/backups/urbackup

Check disks

Checking the underlying disk hardware (readonly) using badblocks.  This takes time (about 3:30 minutes per percentage point on the 2TB drives so it'll keep the system busy for 700 minutes (!) :

sudo badblocks -s /dev/sdc
sudo badblocks -s /dev/sdd

Recreate BTRFS filesystem with first disk

After a long while, both commands ended without mentioning any error blocks so:

sudo mkfs.btrfs -f /dev/sdd

 

 btrfs-progs v5.16.2
See http://btrfs.wiki.kernel.org for more information.

NOTE: several default settings have changed in version 5.15, please make sure
      this does not affect your deployments:
      - DUP for metadata (-m dup)
      - enabled no-holes (-O no-holes)
      - enabled free-space-tree (-R free-space-tree)

Label:              (null)
UUID:               ddd8d501-3bd0-4afd-b501-66340f14f587
Node size:          16384
Sector size:        4096
Filesystem size:    1.82TiB
Block group profiles:
  Data:             single            8.00MiB
  Metadata:         DUP               1.00GiB
  System:           DUP               8.00MiB
SSD detected:       no
Zoned device:       no
Incompat features:  extref, skinny-metadata, no-holes
Runtime features:   free-space-tree
Checksum:           crc32c
Number of devices:  1
Devices:
   ID        SIZE  PATH
    1     1.82TiB  /dev/sdd

Copy the UUID (see above) and update the fstab entry so we can again mount it by default:

luc@kid2:~$ sudo vi /etc/fstab

luc@kid2:~$ sudo mount /var/backups/urbackup/
luc@kid2:~$ df -lm
Filesystem     1M-blocks  Used Available Use% Mounted on
tmpfs                198     2       197   1% /run
/dev/sda4          37910  5061     30892  15% /
tmpfs                988     0       988   0% /dev/shm
tmpfs                  5     0         5   0% /run/lock
/dev/sdb1         299365 51273    232815  19% /home
tmpfs                988     0       988   0% /run/qemu
/dev/sda2          80125 17764     58246  24% /var
tmpfs                198     1       198   1% /run/user/1000
/dev/sdd         1907730     4   1905665   1% /var/backups/urbackup

Re-install urbackup server:

luc@kid2:~$ sudo apt-get install urbackup-server
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
  urbackup-server
0 upgraded, 1 newly installed, 0 to remove and 12 not upgraded.
Need to get 0 B/4.553 kB of archives.
After this operation, 25,1 MB of additional disk space will be used.
Preconfiguring packages ...
Selecting previously unselected package urbackup-server.
(Reading database ... 150865 files and directories currently installed.)
Preparing to unpack .../urbackup-server_2.5.27.0-1ubuntu1~jammy_amd64.deb ...
Unpacking urbackup-server (2.5.27.0-1ubuntu1~jammy) ...
Setting up urbackup-server (2.5.27.0-1ubuntu1~jammy) ...
Processing triggers for man-db (2.10.2-1) ...

Add the second disk

sudo btrfs device add -f /dev/sdc /var/backups/urbackup/

sudo btrfs device usage /var/backups/urbackup/
/dev/sdd, ID: 1
   Device size:             1.82TiB
   Device slack:              0.00B
   Data,single:             8.00MiB
   Metadata,DUP:            2.00GiB
   System,DUP:             16.00MiB
   Unallocated:             1.82TiB

/dev/sdc, ID: 2
   Device size:             1.82TiB
   Device slack:              0.00B
   Unallocated:             1.82TiB

df -lm
Filesystem     1M-blocks  Used Available Use% Mounted on
tmpfs                198     2       197   1% /run
/dev/sda4          37910  5085     30868  15% /
tmpfs                988     0       988   0% /dev/shm
tmpfs                  5     0         5   0% /run/lock
/dev/sdb1         299365 51273    232815  19% /home
tmpfs                988     0       988   0% /run/qemu
/dev/sda2          80125 17764     58246  24% /var
tmpfs                198     1       198   1% /run/user/1000
/dev/sdd         3815459     4   3813393   1% /var/backups/urbackup

Change ownership then recreate tmp directory

I think this normally defaults to /tmp and thus the server will not try to create it.  In my case the root FS is too small and I created this directory elsewhere, but it had been removed by the re-creation of the BTRFS.  So:

 

sudo chown urbackup /var/backups/urbackup

sudo -u urbackup mkdir /var/backups/urbackup/tmp
sudo chmod 750 /var/backups/urbackup/tmp

Restart server:

sudo  systemctl restart urbackupsrv

 

Almost immediately I could connect.  The configuration seems still correct; but no clients have "recent backup" so immediately it started backing up the server itself and my laptop (both obviously active at the time).

BTRFS filesystem errors -- diagnostics and needs to recreate FS again

The BTRFS filesystem used for urbackup backups on /dev/sdd (/var/backups/urbackup) is remounted readonly even after a reboot.  Urbackup can't create new files when backing up so the new backups are failing.

The btrfs check command reports lots of errors.

The dmesg command doesn't report anything special for /dev/sdd nor /dev/sdc so hopefully the hardware is still OK.

Uninstall then reinstall urbackup... I thought that the urbackup DB was also on the failed FS but apparently I have a /var/urbackup directory which has the DB and configuration files.  However I had already done an uninstall so I had to reinstall. The ppa has disappeared probably during an Ubuntu upgrade so I had to add the repository back again:

• sudo add-apt-repository ppa:uroni/urbackup
• sudo apt-get install urbackup-server -y

This apt-get also pulled a lot of libraries and other packages, probably prerequisites.  During install, I chose to "keep" my configuration instead of replacing it with the one coming from the package.

Install went fine but the configure step failed. I still have /etc/urbackup/backupfolder containing the location of the (readonly) directory on BTRFS: /var/backups/urbackup

So the chown is failing due to that directory being readonly for the moment:

 

Setting up urbackup-server (2.5.27.0-1ubuntu1~jammy) ...
chown: changing ownership of '/var/backups/urbackup': Read-only file system
dpkg: error processing package urbackup-server (--configure):
 installed urbackup-server package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 urbackup-server
E: Sub-process /usr/bin/dpkg returned an error code (1)

Thus, remove urbackup-server package again and first work on the BTRFS issue:

sudo apt remove urbackup-server
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages were automatically installed and are no longer required:
  augeas-lenses cpu-checker db-util db5.3-util debootstrap dmeventd extlinux guestfish guestfs-tools guestmount hfsplus ibverbs-providers icoutils ipxe-qemu ipxe-qemu-256k-compat-efi-roms kpartx ldmtool libafflib0v5 libaugeas0 libbfio1 libboost-iostreams1.74.0 libboost-thread1.74.0 libcacard0 libconfig9
  libdate-manip-perl libdaxctl1 libdecor-0-0 libdecor-0-plugin-1-cairo libdevmapper-event1.02.1 libewf2 libfdt1 libgfapi0 libgfrpc0 libgfxdr0 libglusterfs0 libguestfs-hfsplus libguestfs-perl libguestfs-reiserfs libguestfs-tools libguestfs-xfs libguestfs0 libhfsp0 libhivex0 libibverbs1 libintl-perl libintl-xs-perl
  libiscsi7 libldm-1.0-0 liblvm2cmd2.03 libndctl6 libpmem1 libpmemobj1 librados2 librbd1 librdmacm1 librpm9 librpmio9 libsdl2-2.0-0 libslirp0 libspice-server1 libstring-shellquote-perl libsys-virt-perl libtsk19 libusbredirparser1 libvhdi1 libvirglrenderer1 libvirt0 libvmdk1 libwin-hivex-perl libxss1 libyajl2
  libyara8 lsscsi lvm2 msr-tools osinfo-db ovmf qemu-block-extra qemu-system-common qemu-system-data qemu-system-gui qemu-system-x86 qemu-utils rpm-common scrub seabios sleuthkit sqlite3 supermin thin-provisioning-tools virt-p2v zerofree
Use 'sudo apt autoremove' to remove them.
The following packages will be REMOVED:
  urbackup-server
0 upgraded, 0 newly installed, 1 to remove and 12 not upgraded.
1 not fully installed or removed.
After this operation, 25,1 MB disk space will be freed.
Do you want to continue? [Y/n]
(Reading database ... 151049 files and directories currently installed.)
Removing urbackup-server (2.5.27.0-1ubuntu1~jammy) ...
Processing triggers for man-db (2.10.2-1) ...

Linux Mint 21 Vanessa on main laptop

I used the GUI tool:

• sudo apt update
• sudo apt install mintupgrade
• sudo mintupgrade
  

This tool required/proposed to start a timeshift snapshot before upgrading.  I did create that configuration to store the snapshots on /media/luc/Sea1/timeshift/

Then it took quite a while to simulate/download/perform the upgrade.  It also removed some "non official" packages like the Belgian EID stuff and maybe some other Debian or Ubuntu packages (I don't remember which ones).  But finally ended OK and reboot was fine (no need to reinstall Grub or anything like that).

UrBackup agent still alive and kicking.  Darktable version unchanged (3.6.0).  Multi-head monitor configuration still OK.

Ubuntu 22.04 LTS on backup server

• do-release-upgrade ran on 25/10/2022
• After quite a while, upgrade finished with positive message
• Reboot: no need to reinstall grub this time (yeah)
• Piwigo still OK under DynDNS and HTTPS (but see remark about PHP below) so mysql still OK too.   
• UrBackup server still alive and kicking after the upgrade.
  
• however : PHP was upgraded from 7.4 to 8.1 and the apache2 configuration didn't get the message so apache failed to start.  Thus:

luc@kid2:~$ sudo a2dismod php7.4
Module php7.4 disabled.
To activate the new configuration, you need to run:
  systemctl restart apache2
luc@kid2:~$ sudo a2enmod php8.1
Considering dependency mpm_prefork for php8.1:
Considering conflict mpm_event for mpm_prefork:
Considering conflict mpm_worker for mpm_prefork:
Module mpm_prefork already enabled
Considering conflict php5 for php8.1:
Enabling module php8.1.
To activate the new configuration, you need to run:
  systemctl restart apache2

SSH keys

Continued to create ssh keys (one per environment) for my personal account on the 3 K8s servers.  Used ssh-keygen (with default parameters) and ssh-copy-id to copy the public key to all the workers.  So now I can login from the master to any of the workers on each cluster.  That's so many less password typing/pasting.